Paper:
NEXT: A System for Real-World Development, Evaluation, and Application of Active Learning
Active learning methods automatically adapt data collection by selecting the most informative samples in order to accelerate machine learning. Because of this, real-world testing and comparing active learning algorithms requires collecting new datasets (adaptively), rather than simply applying algorithms to benchmark datasets, as is the norm in (passive) machine learning research. To facilitate the development, testing and deployment of active learning for real applications, we have built an open-source software system for large-scale active learning research and experimentation. The system, called NEXT, provides a unique platform for real-world, reproducible active learning research. This paper details the challenges of building the system and demonstrates its capabilities with several experiments. The results show how experimentation can help expose strengths and weaknesses of active learning algorithms, in sometimes unexpected and enlightening ways.

Review:
This paper describes an open-source platform, NEXT, for deploying general active learning algorithms in a real-world setting, where users or clients provide live, real-time feedback through a web interface to queries formed by active learning algorithms. Aiming at facilitating both research in active learning itself and research in other fields that use active learning, the NEXT system hides away engineering details such as load-balancing, logging, communication, locking, etc and lets researchers focus on algorithm or experiment design. Two use cases are presented to illustrate the capability of the system in comparing different algorithms in a real active learning setting, as opposed to many existing studies that conduct comparisons in a simulation mode.The system described in the paper is a great effort in facilitating active learning in the real world. Two suggestions on the presentation / doc side are as follows.(1) While the two example applications in the paper are nice, it would be great ifthere is also a template for a perhaps more typical active learning task, e.g., image or web page classification, that ML researchers/users can easily modify or extend.(2) The documentation of NEXT seems to focus on launching the system in the cloud, but most users would probably want to start by running the system locally to get a feeling of how it works. It is not clear from the documentation how to do that.