4th International Workshop on

Interactive Adaptive Learning (IAL2020)

Co-Located With virtual ECML PKDD 2020

14 September 2020 - Virtual Ghent (Belgium)

Get proceedings at ceur-ws.org

Image © by Michael Schmalenstroer (CC-BY-SA-3.0)

Please join the workshop with ZOOM:

The workshop will be fully virtual. We organized a zoom webinar which is accessible through Whova App or Whova Web.
See also the information at https://ecmlpkdd2020.net/attending/participate/


Science, technology, and commerce increasingly recognise the importance of machine learning approaches for data-intensive, evidence-based decision making.

Science, technology, and commerce increasingly recognise the importance of machine learning approaches for data-intensive, evidence-based decision making. This is accompanied by increasing numbers of machine learning applications and volumes of data. Nevertheless, the capacities of processing systems or human supervisors or domain experts remain limited in real-world applications. Furthermore, many applications require fast reaction to new situations, which means that first predictive models need to be available even if little data is yet available. Therefore approaches are needed that optimise the whole learning process, including the interaction with human supervisors, processing systems, and data of various kind and at different timings: techniques for estimating the impact of additional resources (e.g. data) on the learning progress; techniques for the active selection of the information processed or queried; techniques for reusing knowledge across time, domains, or tasks, by identifying similarities and adaptation to changes between them; techniques for making use of different types of information, such as labeled or unlabeled data, constraints or domain knowledge. Such techniques are studied for example in the fields of adaptive, active, semi-supervised, and transfer learning. However, this is mostly done in separate lines of research, while combinations thereof in interactive and adaptive machine learning systems that are capable of operating under various constraints, and thereby address the immanent real-world challenges of volume, velocity and variability of data and data mining systems, are rarely reported. Therefore, this workshop aims to bring together researchers and practitioners from these different areas, and to stimulate research in interactive and adaptive machine learning systems as a whole. It continues a successful series of events at ECML PKDD 2017 in Skopje (Workshop and Tutorial), IJCNN 2018 in Rio (Tutorial), ECML PKDD 2018 in Dublin (Workshop), and ECML PKDD 2019 in Würzburg (Workshop and Tutorial).

The workshop aims at discussing techniques and approaches for optimising the whole learning process, including the interaction with human supervisors, processing systems, and includes adaptive, active, semi-supervised, and transfer learning techniques, and combinations thereof in interactive and adaptive machine learning systems. Our objective is to bridge the communities researching and developing these techniques and systems in machine learning and data mining. Therefore, we welcome contributions that present a novel problem setting, propose a novel approach, or report experience with the practical deployment of such a system and raise unsolved questions to the research community.

In particular, we welcome contributions that address aspects including, but not limited to:

    Novel Techniques for Active, Semi-Supervised, Transfer Learning
  • methods for big, evolving, or streaming data,
  • methods for recent complex model structures such as deep learning neural networks or recurrent neural networks,
  • methods for interacting with imperfect or multiple oracles, e.g. learning from crowds,
  • methods for incorporating domain knowledge and constraints,
  • methods for timing the interaction and for combining different types of information,
  • online and ensemble methods for evolving models and systems, with specific switching and fusion techniques, and (inter-)active data integration techniques,
  • Innovative Use and Applications of Active, Semi-Supervised, Transfer Learning
  • for filtering, forgetting, resampling,
  • for active class or feature selection, e.g. from multi-modal data,
  • for detection of change, outliers, frauds, or attacks,
  • new interactive learning protocols and application scenarios, e.g., brain-computer interfaces, crowdsourcing, ...
  • in application in data-intensive science,
  • in applications with real-world deployment,
  • Techniques for Combined Interactive Adaptive Learning
  • methods combining adaptive, active, semi-supervised, or transfer learning techniques,
  • cost-aware methods and methods for estimating the impact of employing additional resources, such as data or processing capacities, on the learning progress,
  • methodologies for the evaluation of such techniques, and comparative studies,
  • methods for automating the control of an interactive adaptive learning process.

Important dates

The following timeline shows the most important dates for the workshop.

  • Submission open

    18 April 2020

    You can submit your contributions via EasyChair.

  • Submission deadline UPDATE

    9 June 2020 (EasyChair paper registration)
    16 June 2020 (PDF submission)

    Due to many requests, we decided to postpone the deadline for pdf upload for both tracks. Note, that it is still required to register the paper in EasyChair with authors, title, and abstract until 9 June.
  • Notification EXTENDED

    9 July 2020
    16 July 2020

  • Camera Ready EXTENDED

    28 July 2020
    4 August 2020

  • Workshop (Full Day)

    14 September 2020

    Co-Located With The The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery (ECML PKDD 2020).

Submit your contribution

Submission is now closed.

Full Paper Track

The full paper track covers new innovative contributions in the area of interactive adaptive learning. If you have a new method already evaluated briefly, a new tool to simplify interaction or some new insights the community might benefit from, please submit a regular paper. The page limit is 8-16 pages.

EasyChair Deadline: 9 June 2020
PDF Submission Deadline: 16 June 2020 (more details above)

Extended Abstract Track

The extended abstract track is ideal to discuss new ideas in the area of interactive adaptive learning. We encourage you to submit open challenges in research or industrial applications to initiate a discussion and find colleagues to collaborate with. The page limit is 2-4 pages.

EasyChair Deadline: 9 June 2020
PDF Submission Deadline: 16 June 2020 (more details above)

Indexed Publishing

All accepted papers will be published at ceur-ws.org (indexed by e.g. google scholar) or within Springer LNCS proceedings depending on the number of submissions. Reviews are single-blind.

LNCS Style

The paper must be be written in English and contain author names, affiliations, and email addresses. The paper must be in PDF using the LNCS format. See instructions here.


All accepted papers are presented in spotlight talks and/or poster sessions. At least one author of each accepted paper must be registered to the workshop.

Invited Talks

Kori Inkpen (Microsoft Research, USA)      
When Humans and AI Collide
As the use of AI in society grows and evolves, we see both opportunities and risks for these technologies. While AI has already shown strong performance in some areas, there are still many domains where the potential impact of AI will depend on the interaction between Humans and AI. So what happens when Humans and AI disagree? Who do you trust? And what happens when the Human, the AI, or both are biased? We need to continue to evolve our understanding of how humans and AI systems can work together, effectively harnessing the benefits of both systems, and mitigating their inherent biases. This talk will share results from our work on Human-AI complementarity, and the intersection of Human and AI bias.

Robert Munro (Machine Learning Consulting, USA)      
The rapid growth of Human-in-the-Loop Machine Learning
In the last few years Human-in-the-Loop Machine Learning has quickly become the dominant paradigm in many industries adopting AI for the first time. More than 90% of machine learning applications today are powered by supervised machine learning, including autonomous vehicles, in-home devices, and every item you purchase on-line. There are thousands of professional annotators and subject matter experts fine-tuning each underlying model by annotating new data. This talk will highlight different use cases in Human-in-the-Loop Machine Learning in industries including finance, healthcare, entertainment, retail, and disaster response.

Eyke Hüllermeier (University Paderborn, Germany)      
How to measure uncertainty in Uncertainty Sampling for Active Learning
Various strategies for active learning have been proposed in the machine learning literature. In uncertainty sampling, which is among the most popular approaches, the active learner sequentially queries the label of those instances for which its current prediction is maximally uncertain. The predictions as well as the measures used to quantify the degree of uncertainty, such as entropy, are traditionally of a probabilistic nature. Yet, alternative approaches to capturing uncertainty in machine learning, alongside with corresponding uncertainty measures, have been proposed in recent years. In particular, some of these measures seek to distinguish different sources and to separate different types of uncertainty, such as the reducible (epistemic) and the irreducible (aleatoric) part of the total uncertainty in a prediction. This talk elaborates on the usefulness of such measures for uncertainty sampling and compares their performance in active learning. To this end, uncertainty sampling is instantiated with different measures, the properties of the sampling strategies thus obtained are analyzed and compared in an experimental study.

Andreas Holzinger (Medical University Graz, Austria)      
From Explainable AI to Human-Centered AI
The problem of explainability is as old as AI itself and classic AI represented comprehensible retraceable approaches. Their weakness was in dealing with non-linearities and the intrinsic uncertainties of medical data. Advances in data-driven statistical machine learning have led to the current renaissance of AI, but the solutions are becoming increasingly complex and opaque. Due to increasing social, ethical, and legal aspects of AI in medicine, explainable AI (xAI) is attracting much interest within the international research community. While xAI deals with the implementation of transparency and traceability of statistical black‐box machine learning methods, there is a pressing need to go beyond xAI, e.g. to extent explainability with causability. The integrative backbone for this approach is in interactive machine learning with the human-in-the-loop because a human domain expert complements AI with implicit knowledge. Humans are robust, can generalize from few examples, understand relevant representations and concepts and are able to explain causal links between them. Consequently, more research is needed on how human experts explain their decisions by examining their strategies, as they are (but not always) able to describe the underlying explanatory factors. Formalized, these can be used to build structural causal models of human decision making and characteristics can be mapped back to train AI. Finally, such an AI-ecosystem needs advanced Human-AI interfaces, that allow to ask questions of why, but also to ask for counterfactuals, i.e. what-if. This interactivity between human and AI will contribute to enhance robustness, reliability, accountability, fairness and trust in AI and foster ethical responsible machine learning with the human-in-control.


The workshop will be fully virtual. More information will follow soon.

All times are in local Ghent time (MEST).
Time Program Presenter/Author
10:00 - 11:30 Session 1:
5m Introduction
45m Invited Talk: From Explainable AI to Human-Centered AI Andreas Holzinger (Medical University Graz, Austria)
20m Improving Unsupervised Domain Adaptation with Representative Selection Techniques I-Ting Chen and Hsuan-Tien Lin
20m On the Transferability of Deep Neural Networks for Recommender System Duc Nguyen, Hao Niu, Kei Yonekawa, Mori Kurokawa, Chihiro Ono, Daichi Amagata, Takuya Maekawa and Takahiro Hara
Break and Come Together
12:00 - 13:25 Session 2:
45m Invited Talk: How to measure uncertainty in Uncertainty Sampling for Active Learning Eyke Hüllermeier (University Paderborn, Germany)
20m Active Learning for LSTM-autoencoder-based Anomaly Detection in Electrocardiogram Readings Tomáš Šabata and Martin Holena
20m Towards Landscape Analysis in Adaptive Learning of Surrogate Models Zbyněk Pitra and Martin Holena
Break and Come Together
15:00 - 16:25 Session 3:
45m Invited Talk: The rapid growth of Human-in-the-Loop Machine Learning Robert Munro (Machine Learning Consulting, USA)
20m Learning active learning at the crossroads? Evaluation and discussion Louis Desreumaux and Vincent Lemaire
20m The Effects of Reluctant and Fallible Users in Interactive Online Machine Learning Agnes Tegen, Paul Davidsson and Jan A. Persson
Break and Come Together
17:00 - 18:30 Session 4:
45m Invited Talk: When Humans and AI Collide Kori Inkpen (Microsoft Research, USA)
20m VIAL-AD: Visual Interactive Labelling for Anomaly Detection - An approach and open research questions Andreas Theissler, Anna-Lena Kraft, Max Rudeck and Fabian Erlenbusch
20m Get a Human-In-The-Loop: Feature Engineering via Interactive Visualizations Dimitra Gkorou, Maialen Larranaga, Alexander Ypma, Faegheh Hasibi and Robert Jan van Wijk
5m Closing


Organizing Committee:
ial2020 (at) easychair.org

Georg Krempl

g.m.krempl (at) uu.nl
Utrecht University, Netherlands

Vincent Lemaire

vincent.lemaire (at) orange.com
Orange Labs, France

Daniel Kottke

daniel.kottke (at) uni-kassel.de
University of Kassel, Germany

Andreas Holzinger

a.holzinger (at) hci-kdd.org
Medical University Graz, Austria

Adrian Calma

adrian.calma (at) uni-kassel.de
vencortex, Germany

Steering Committee:

Robi Polikar

polikar (at) rowan.edu
Rowan University, USA

Bernhard Sick

bsick (at) uni-kassel.de
University of Kassel, Germany

Program Committee (tentative):

Albert Bifet (LTCI, Telecom ParisTech)
Alexis Bondu (Orange Labs)
Klemens Böhm (Karlsruhe Institute of Technology)
Martin Holena (Institute of Computer Science)
Dino Ienco (IRSTEA)
George Kachergis
Edwin Lughofer (Johannes Kepler University Linz)
Shreyasi Pathak (University of Twente)
Ingo Scholtes (University of Zurich)
Carlos Soares (LIAAD-INESCTEC, Porto)
Stefano Teso (Katholieke Universiteit Leuven)
Holger Trittenbach (Karlsruhe Institute of Technology)
Sebastian Tschiatschek