3rd International Tutorial & Workshop on

Interactive Adaptive Learning (IAL2019)

Co-Located With ECML PKDD 2019

16 September 2019 - Würzburg (Germany)

Get proceedings at ceur-ws.org

Get full proceedings

© by Zairon (CC-BY-SA-3.0)


Science, technology, and commerce increasingly recognise the importance of machine learning approaches for data-intensive, evidence-based decision making.

Science, technology, and commerce increasingly recognise the importance of machine learning approaches for data-intensive, evidence-based decision making. This is accompanied by increasing numbers of machine learning applications and volumes of data. Nevertheless, the capacities of processing systems or human supervisors or domain experts remain limited in real-world applications. Furthermore, many applications require fast reaction to new situations, which means that first predictive models need to be available even if little data is yet available. Therefore approaches are needed that optimise the whole learning process, including the interaction with human supervisors, processing systems, and data of various kind and at different timings: techniques for estimating the impact of additional resources (e.g. data) on the learning progress; techniques for the active selection of the information processed or queried; techniques for reusing knowledge across time, domains, or tasks, by identifying similarities and adaptation to changes between them; techniques for making use of different types of information, such as labeled or unlabeled data, constraints or domain knowledge. Such techniques are studied for example in the fields of adaptive, active, semi-supervised, and transfer learning. However, this is mostly done in separate lines of research, while combinations thereof in interactive and adaptive machine learning systems that are capable of operating under various constraints, and thereby address the immanent real-world challenges of volume, velocity and variability of data and data mining systems, are rarely reported. Therefore, this workshop and tutorial aims to bring together researchers and practitioners from these different areas, and to stimulate research in interactive and adaptive machine learning systems as a whole. It continues a successful series of events at ECML PKDD 2017 in Skopje (Workshop & Tutorial), IJCNN 2018 in Rio (Tutorial), and ECML PKDD 2018 in Dublin (Workshop). (Links to be completed)

The workshop aims at discussing techniques and approaches for optimising the whole learning process, including the interaction with human supervisors, processing systems, and includes adaptive, active, semi-supervised, and transfer learning techniques, and combinations thereof in interactive and adaptive machine learning systems. Our objective is to bridge the communities researching and developing these techniques and systems in machine learning and data mining. Therefore, we welcome contributions that present a novel problem setting, propose a novel approach, or report experience with the practical deployment of such a system and raise unsolved questions to the research community.

In particular, we welcome contributions that address aspects including, but not limited to:

    Novel Techniques for Active, Semi-Supervised, Transfer Learning
  • methods for big, evolving, or streaming data,
  • methods for recent complex model structures such as deep learning neural networks or recurrent neural networks,
  • methods for interacting with imperfect or multiple oracles, e.g. learning from crowds,
  • methods for incorporating domain knowledge and constraints,
  • methods for timing the interaction and for combining different types of information,
  • online and ensemble methods for evolving models and systems, with specific switching and fusion techniques, and (inter-)active data integration techniques,
  • Innovative Use and Applications of Active, Semi-Supervised, Transfer Learning
  • for filtering, forgetting, resampling,
  • for active class or feature selection, e.g. from multi-modal data,
  • for detection of change, outliers, frauds, or attacks,
  • new interactive learning protocols and application scenarios, e.g., brain-computer interfaces, crowdsourcing, ...
  • in application in data-intensive science,
  • in applications with real-world deployment,
  • Techniques for Combined Interactive Adaptive Learning
  • methods combining adaptive, active, semi-supervised, or transfer learning techniques,
  • cost-aware methods and methods for estimating the impact of employing additional resources, such as data or processing capacities, on the learning progress,
  • methodologies for the evaluation of such techniques, and comparative studies,
  • methods for automating the control of an interactive adaptive learning process.

This workshop is sponsored by:

Important dates

The following timeline shows the most important dates for the workshop.

  • Submission open

    17 May 2019

    You can submit your contributions via EasyChair.

  • Submission deadline EXTENDED

    14 June 2019
    28 June 2019

  • Notification EXTENDED

    12 July 2019
    24 July 2019

    ECML PKDD offers the early bird registration rate until 26 July 2019.

  • Camera Ready EXTENDED

    26 July 2019
    5 August 2019

  • Tutorial & Workshop (Full Day)

    16 September 2019

    Co-Located With The The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery (ECML PKDD 2019).

Submit your contribution

You can submit your contribution via EasyChair. Please use the link below.

Submission is now closed.

Full Paper Track

The full paper track covers new innovative contributions in the area of interactive adaptive learning. If you have a new method already evaluated briefly, a new tool to simplify interaction or some new insights the community might benefit from, please submit a regular paper. The page limit is 8-16 pages.

Submission Deadline: 28 June 2019

Extended Abstract Track

The extended abstract track is ideal to discuss new ideas in the area of interactive adaptive learning. We encourage you to submit open challenges in research or industrial applications to initiate a discussion and find colleagues to collaborate with. The page limit is 2-4 pages.

Submission Deadline: 28 June 2019

Indexed Publishing

All accepted papers will be published at ceur-ws.org (indexed by e.g. google scholar) or within Springer LNCS proceedings depending on the number of submissions. Reviews are single-blind.

LNCS Style

The paper must be be written in English and contain author names, affiliations, and email addresses. The paper must be in PDF using the LNCS format. See instructions here.


All accepted papers are presented in spotlight talks and/or poster sessions. At least one author of each accepted paper must be registered to the workshop.


Georg Krempl
Part 1 - Foundations of Interactive Adaptive Learning
This part starts with the classic stream mining paradigm. In its context, we discuss the challenges posed by non-stationarity and limitations in processing, storage, and supervision capacities. We briefly summarize related techniques, e.g. for incremental processing, forgetting, and change detection. Furthermore, we introduce techniques for optimising the interaction of a machine learning system with an oracle such as a human supervisor. We review active machine learning techniques, with focus on adaptive active learning for evolving and streaming data. We discuss recent advances and conclude with an overview on open research questions in adaptive active machine learning.

Andreas Holzinger
Part 2 - From Interactive ML to Explainable AI (ex-AI)
In this part, we focus on the role of humans in state-of-the-art decision systems. Thereby, we go beyond interactive machine learning to explainable artificial intelligence. How can this be realized? How can we include humans into the automated decision process and how can me measure their intelligence? To answer these questions, we will talk about different terms like interaction, reflection and discuss the underlying principles of intelligence and cognition. In the next part, we provide fundamentals to measure and evaluate human intelligence with biometric technologies, sensor arrays and affective computing to measure emotion and stress. The tutorial concludes with a discussion on ethical, legal and social issues of explainable AI systems.


The full proceedings of the workshop are available here: Get proceedings
Time Program Presenter/Author
9:00 - 10:30 Tutorial (Pt. 1):
Foundations of Interactive Adaptive Learning [slides] Georg Krempl
Morning Coffee and Tea Break with Poster Session
11:00 - 12:30 Tutorial (Pt. 2):
From Interactive ML to Explainable AI [slides] Andreas Holzinger
12:30 - 12:40 Spotlight Presentations of Posters see below
Lunch Break at conference venue with Poster Session
13:45 - 13:50 Workshop: Introduction Daniel Kottke
13:50 - 14:05 Session 1: Presentation of Full Papers
15m Toward Faithful Explanatory Active Learning with Self-explainable Neural Nets [slides] Stefano Teso
14:05 - 15:00 Invited Talk with Q&A on Evaluation of Interactive Machine Learning Systems [slides] Nadia Boukhelifa
Coffee & Tea Break with Poster Session
15:20 - 16:35 Session 2: Presentation of Full Papers
15m Validating One-Class Active Learning with User Studies - a Prototype and Open Challenges [poster] [slides] Holger Trittenbach, Adrian Englhardt, Klemens Böhm
15m RAL - Improving Stream-Based Active Learning by Reinforcement Learning Sarah Wassermann, Thibaut Cuvelier, Pedro Casas
15m Knowledge-based Selection of Gaussian Process Surrogates Zbyněk Pitra, Lukáš Bajer, Martin Holena
15m Explicit Control of Feature Relevance and Selection Stability Through Pareto Optimality [poster] [slides] Victor Hamer, Pierre Dupont
15m Deep Bayesian Semi-Supervised Active Learning for Sequence Labelling [poster] [slides] Tomáš Šabata, Juraj Eduard Páll and Martin Holeňa
16:35 - 16:45 Workshop: Closing Daniel Kottke
Special transport to opening ceremony for participants of this workshop.

Poster Session

Short Paper Presenter/Author
Combating Stagnation in Reinforcement Learning Through 'Guided Learning' With 'Taught-Response Memory' [poster] Keith Tunstead, Joeran Beel
Towards Active Simulation Data Mining [poster] Mirko Bunse, Amal Saadallah and Katharina Morik
Active Feature Acquisition For Opinion Stream Classification Under Drift [poster] Ranjith Shivakumaraswamy, Christian Beyer, Vishnu Unnikrishnan, Eirini Ntoutsi, Myra Spiliopoulou

Invited Talk

Nadia Boukhelifa
Evaluation of Interactive Machine Learning Systems
The evaluation of interactive machine learning systems remains a difficult task. These systems learn from and adapt to the human, but at the same time, the human receives feedback and adapts to the system. Getting a clear understanding of these subtle mechanisms of co-operation and co-adaptation is challenging. In this chapter, we report on our experience in designing and evaluating various interactive machine learning applications from different domains. We argue for coupling two types of validation: algorithm-centred analysis, to study the computational behaviour of the system; and human-centred evaluation, to observe the utility and effectiveness of the application for end-users. We use a visual analytics application for guided search, built using an interactive evolutionary approach, as an exemplar of our work. Our observation is that human-centred design and evaluation complement algorithmic analysis, and can play an important role in addressing the “black-box” effect of machine learning. Finally, we discuss research opportunities that require human-computer interaction methodologies, in order to support both the visible and hidden roles that humans play in interactive machine learning.
Get the slides here.


Organizing Committee:
ial2019 (at) easychair.org

Georg Krempl

g.m.krempl (at) uu.nl
Utrecht University, Netherlands

Vincent Lemaire

vincent.lemaire (at) orange.com
Orange Labs, France

Daniel Kottke

daniel.kottke (at) uni-kassel.de
University of Kassel, Germany

Andreas Holzinger

a.holzinger (at) hci-kdd.org
Medical University Graz, Austria

Adrian Calma

Darwin, USA


Steering Committee:

Robi Polikar

polikar (at) rowan.edu
Rowan University, USA

Bernhard Sick

bsick (at) uni-kassel.de
University of Kassel, Germany

Program Committee (to be completed):

Albert Bifet (LTCI, Telecom ParisTech)
Giacomo Boracchi (Politecnico di Milano)
Martin Holena (Institute of Computer Science)
Edwin Lughofer (Univ. Linz)
Tuan Pham Minh (Univ. Kassel)
Christin Seifert (Univ. of Twente)
Jurek Stefanowski (Poznan University)
Indre Zliobaite (Univ. of Helsinki)