5th International Workshop on

Interactive Adaptive Learning (IAL2021)

Co-Located With ECML PKDD 2021

13 September 2021 - Virtual

Get full proceedings (soon at ceur-ws.org)


Science, technology, and commerce increasingly recognise the importance of machine learning approaches for data-intensive, evidence-based decision making.

Science, technology, and commerce increasingly recognise the importance of machine learning approaches for data-intensive, evidence-based decision making. Moreover, the number of machine learning applications and the volumes of data increase permanently. Nevertheless, the capacities of processing systems, human supervisors, or domain experts remain limited in real-world applications. Furthermore, many applications require a fast reaction to new situations, which means that first predictive models need to be available even if little data is yet available. Therefore, approaches that optimise the whole learning process are needed, including the interaction with human supervisors, processing systems, and data of various kind and at different timings: techniques for estimating the impact of additional resources (e.g. data) on the learning progress; methods for active selection of the information processed or queried; techniques for reusing knowledge across time, domains, or tasks, by identifying similarities and adaptation to changes between them; methods for making use of different types of information, such as labelled or unlabelled data, constraints, or domain knowledge. Such techniques are studied, for example, in the fields of adaptive, active, semi-supervised, and transfer learning -- mostly in separate lines of research. Combinations that are capable of operating under various constraints, and thereby address the inherent real-world challenges of volume, velocity, and variability of data and data mining systems, are rarely reported. Therefore, this workshop aims to bring together researchers and practitioners from these different areas, and to stimulate research in interactive and adaptive machine learning systems as a whole. It continues a successful series of events at ECML PKDD 2017 in Skopje (Workshop and Tutorial), IJCNN 2018 in Rio (Tutorial), ECML PKDD 2018 in Dublin (Workshop), ECML PKDD 2019 in Würzburg (Workshop and Tutorial), and ECML PKDD 2020 (hosted in Ghent, online Workshop).

The workshop aims at discussing techniques and approaches for optimising the whole learning process, including the interaction with human supervisors, processing systems, and includes adaptive, active, semi-supervised, and transfer learning techniques, and combinations thereof in interactive and adaptive machine learning systems make the topic of this workshop. Our objective is to bridge the communities researching and developing these techniques and systems in machine learning and data mining. Therefore, we welcome contributions that present a new problem setting, propose a novel approach, or report experience with the practical deployment of such a system and raise unsolved questions to the research community.

In particular, we welcome contributions that address aspects including, but not limited to:

    Novel Techniques for Active, Semi-Supervised, Transfer, or Weakly Supervised Learning
  • methods for big, evolving, or streaming data,
  • methods for recent complex model structures such as deep learning neural networks or recurrent neural networks,
  • methods for interacting with imperfect or multiple oracles, e.g. learning from crowds,
  • methods for incorporating domain knowledge and constraints,
  • methods for timing the interaction and for combining different types of information,
  • online and ensemble methods for evolving models and systems, with specific switching and fusion techniques, and (inter-)active data integration techniques,
  • Innovative Use and Applications of Active, Semi-Supervised, Transfer, or Weakly Supervised Learning
  • for filtering, forgetting, resampling,
  • for active class or feature selection, e.g. from multi-modal data,
  • for detection of change, outliers, frauds, or attacks,
  • new interactive learning protocols and application scenarios, e.g., brain-computer interfaces, crowdsourcing, ...
  • in application in data-intensive science,
  • in applications with real-world deployment,
  • Techniques for Combined Interactive Adaptive Learning
  • methods combining adaptive, active, semi-supervised, or transfer learning techniques,
  • cost-aware methods and methods for estimating the impact of employing additional resources, such as data or processing capacities, on the learning progress,
  • methodologies for the evaluation of such techniques, and comparative studies,
  • methods for automating the control of an interactive adaptive learning process.

Important dates

The following timeline shows the most important dates for the workshop.

  • Submission open

    23 April 2021

    You can submit your contributions via EasyChair.

  • Submission deadline EXTENDED

    23 June 2021
    7 July 2021

  • Notification POSTPONED

    26 July 2021
    9 August 2021

  • Camera Ready EXTENDED

    16 August 2021
    22 August 2021

  • Workshop (Full Day)

    13 September 2021

    Co-Located With The The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery (ECML PKDD 2021).

Submit your contribution

Submission is now closed.

Full Paper Track

The full paper track covers new innovative contributions in the area of interactive adaptive learning. If you have a new method already evaluated briefly, a new tool to simplify interaction or some new insights the community might benefit from, please submit a regular paper. The page limit is 8-16 pages.

Submission Deadline: 7 July 2021

Extended Abstract Track

The extended abstract track is ideal to discuss new ideas in the area of interactive adaptive learning. We encourage you to submit open challenges in research or industrial applications to initiate a discussion and find colleagues to collaborate with. The page limit is 2-4 pages.

Submission Deadline: 7 July 2021

Indexed Publishing

All accepted papers will be published at ceur-ws.org (indexed by e.g. google scholar) or within Springer LNCS proceedings depending on the number of submissions. Reviews are single-blind.

LNCS Style

The paper must be be written in English and contain author names, affiliations, and email addresses. The paper must be in PDF using the LNCS format. See instructions here.


All accepted papers are presented in spotlight talks and/or poster sessions. At least one author of each accepted paper must be registered to the workshop.

Invited Talks

Matthew Taylor (University of Alberta, Edmonton, Canada)      
Title: Reinforcement learning agents learning from the environment and from humans — what the heck do you want me to do?
Reinforcement learning (RL) agents can learn autonomously through interacting with an environment, but can learn much more quickly when also receiving advice from knowledgeable humans. Unfortunately, humans have biases, may not convey their ideas clearly, and are typically suboptimal. This talk will focus on how RL agents can benefit from human advice, while ultimately outperforming them. My hope is that the approaches discussed and the lessons learned will be applicable to many interactive learning settings beyond the human-RL framework.

Matthew E. Taylor (Matt) received his doctorate from the University of Texas at Austin in the summer of 2008, supervised by Peter Stone. He is a tenured associate professor in computer science at the University of Alberta, a Fellow-in-Residence at the Alberta Machine Intelligence Institute, and remains an adjunct professor at Washington State University. His current fundamental and applied research interests are in reinforcement learning, human-in-the-loop AI, multi-agent systems, and robotics.

Alison M Smith-Renner (Dataminr)      
Designing for the Human-in-the-Loop: Unpredictable Control in Interactive Machine Learning
Machine learning (ML) systems help people analyze, understand, and make decisions from data, but these systems also need help to personalize recommendations, maintain high classification accuracy, or adapt to new domains. Interactive ML techniques let users control ML models with iterative feedback. However, these models must balance user input and the underlying data, meaning they sometimes update slowly, poorly, or unpredictably—either by not incorporating user input as expected or by making other unexpected changes. Prior research in IML typically focuses on user control simply in terms of whether or how users provide model feedback, with little attention paid to users’ reactions when their feedback is not applied predictably. In this talk, I'll describe two studies examining user interaction in an IML case where controls can be unpredictable, yet are easy to validate: interactive topic modeling. This work advances our understanding of users' experience and expectations related to the control of IML systems.

Alison Smith-Renner is a Senior Research Scientist at Dataminr, where she designs, builds, and evaluates intelligent systems for augmenting human workflows. Her research interests lie at the intersection of AI and HCI, focusing on transparency and control for human-in-the-loop systems to engender appropriate trust and improve human performance. Alison received her Ph.D. in Computer Science from the University of Maryland, College Park. She is active in the explainable AI and human-centered AI research communities.


Time Program Presenter/Author
11:00 - 12:50 Session 1: Chair: Georg Krempl
10m Introduction
20m MetaREVEAL: RL-based Meta-learning from Learning Curves    Manh Hung Nguyen, Isabelle Guyon, Lisheng Sun-Hosoya and Nathan Grinsztajn
20m Uncertainty and Utility Sampling with Pre-Clustering    Zhixin Huang, Yujiang He, Stephan Vogt and Bernhard Sick
20m Evidential Nearest Neighbours in Active Learning    Daniel Zhu, Arnaud Martin, Yolande Le Gall, Jean-Christophe Dubois and Vincent Lemaire
20m SLAYER: A Semi-supervised Learning Approach for Drifting Data Streams under Extreme Verification Latency    Maria Arostegi, Jesus Lobo and Javier Del Ser
20m A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving    Maarten Bieshaar, Marek Herde, Denis Huselijc and Bernhard Sick
Lunch Break
14:30 - 16:30 Session 2: Chair: Vincent Lemaire
60m Invited Talk: Reinforcement learning agents learning from the environment and from humans — what the heck do you want me to do? Matthew Taylor (University of Alberta, Edmonton, Canada)
20m Active Class Selection with Uncertain Class Proportions    Mirko Bunse and Katharina Morik
20m Sample Noise Impact on Active Learning    Alexandre Abraham and Léo Dreyfus-Schmidt
20m Contrastive Representations for Label Noise Require Fine-Tuning    Pierre Nodet, Vincent Lemaire, Alexis Bondu and Antoine Cornuéjols
17:00 - 18:45 Session 3: Chair: Andreas Holzinger
60m Invited Talk: Designing for the Human-in-the-Loop: Unpredictable Control in Interactive Machine Learning Alison M Smith-Renner (Dataminr)
20m Combining Gaussian Processes with Neural Networks for Active Learning in Optimization    Jiří Růžička, Jan Koza, Jiří Tumpach, Zbyněk Pitra and Martin Holena
20m Stochastic Adversarial Gradient Embedding for Active Domain Adaptation    Victor Bouvier, Philippe Very, Clément Chastagnol, Myriam Tami and Hudelot Céline
5m Closing


Organizing Committee:
ial2021 (at) easychair.org

Georg Krempl

g.m.krempl (at) uu.nl
Utrecht University, Netherlands

Vincent Lemaire

vincent.lemaire (at) orange.com
Orange Labs, France

Daniel Kottke

daniel.kottke (at) uni-kassel.de
University of Kassel, Germany

Barbara Hammer

bhammer (at) techfak.uni-bielefeld.de
University of Bielefeld, Germany

Andreas Holzinger

a.holzinger (at) hci-kdd.org
Medical University Graz, Austria

Steering Committee:

Robi Polikar

polikar (at) rowan.edu
Rowan University, USA

Bernhard Sick

bsick (at) uni-kassel.de
University of Kassel, Germany

Adrian Calma

adrian.calma (at) uni-kassel.de
University of Kassel, Germany

Program Committee:

Michael Beigl (TECO, KIT)
Mirko Bunse (Dortmund University)
Klemens Böhm (Karlsruhe Institute of Technology)
Hudelot Céline (Ecole Centrale Paris)
Gregory Ditzler (University of Arizona)
Michael Granitzer (University of Passau)
Marek Herde (University of Kassel)
Martin Holena (Institute of Computer Science)
Denis Huseljic (University of Kassel)
Edwin Lughofer (Johannes Kepler University Linz)
Bernhard Pfahringer (University of Waikato)
Stefano Teso (Katholieke Universiteit Leuven)
Indre Zliobaite (University of Helsinki)