On the Limitations of Simulating Active Learning (2023)

Abstract: Active learning (AL) is a human-and-model-in-the-loop paradigm that iteratively selects informative unlabeled data for human annotation, aiming to improve data efficiency over random sampling. However, performing AL experiments with human annotations on-the-fly is a laborious and expensive process, thus unrealistic for academic research. An easy fix to this impediment is to simulate AL, by treating an already labeled and publicly available dataset as the pool of unlabeled data. In this position paper, we first survey recent literature and highlight the challenges across all different steps within the AL loop. We further unveil neglected caveats in the experimental setup that can significantly affect the quality of AL research. We continue with an exploration of how the simulation setting can govern empirical findings, arguing that it might be one of the answers behind the ever posed question “Why do Active Learning algorithms sometimes fail to outperform random sampling?”. We argue that evaluating AL algorithms on available labeled datasets might provide a lower bound as to their effectiveness in real data. We believe it is essential to collectively shape the best practices for AL research, especially now that the stellar engineering advances (e.g. ChatGPT) shift the research focus to data-driven approaches. To this end, we present guidelines for future work, hoping that by bringing these limitations to the community’s attention, we can explore ways to address them.

## introduction based on the assumption that "not all data is equal", active learning (al) @xcite aims to identify the most informative data for annotation from a pool (or a stream) of unlabeled data (i.e., data acquisition). with multiple rounds of model training, data acquisition and human annotation (figure 1 ), the goal is to achieve data efficiency. a data efficient al algorithm entails that a model achieves satisfactory performance on a held-out test set, by being trained with only a fraction of the acquired data. al has traditionally attracted wide attention in the natural language processing (nlp) community. it has been explored for machine translation @xcite @xcite ), text classification @xcite @xcite , part-of-speech tagging @xcite , coreference @xcite and entity resolution @xcite , named entity recognition @xcite @xcite , and natural language inference @xcite , inter alia. however, its potential value is still growing @xcite , driven by advancements in the state-of-the-art in language model pretraining @xcite . given the initial assumption that "not all data is equal", it is reasonable to expect researchers to seek out the "most valuable" data for pretraining or adapting their language models. the usual pool-based al setting is to acquire data from an unlabeled pool, label it, and use it to train a supervised model that, hopefully, obtains satisfactory performance on a test set for the task at hand. this is very similar to the general modelin-the-loop paradigm @xcite @xcite @xcite , with the main difference being the al-based data acquisition stage. the assumption is that, by iteratively selecting data for annotation according to an informativeness criterion, it will result into better model predictive performance compared to randomly sampling and annotate data of the same size. however, this does not always seem to be the case. a body of work has shown that al algorithms, that make use of uncertainty @xcite @xcite , diversity sampling @xcite @xcite or even more complex acquisition strategies @xcite @xcite , often fail to improve over a simple random sampling baseline @xcite @xcite @xcite . such findings pose a serious question on the practical usefulness of al, as they do not corroborate its initial core hypothesis that not all data is equally useful for training a model. in other words, if we cannot show that one subset of the data is "better" foot_0 than another, why do al in the first place? only a small body of work has attempted to explore the pain points of al. for instance, @xcite , leveraging visualisations from data maps @xcite , show that al algorithms tend to acquire collective outliers (i.e. groups of examples that deviate from the rest of the examples but cluster together), thus explaining the utter failure of eight al algorithms to outperform random sampling in visual question answering. building on this work, more recently snijders et al. (2023) corroborate these findings for the task of natural language inference and further show that uncertainty based al methods recover and even surpass random selection when hard-to-learn data points are removed from the pool. @xcite show that the benefits of al with cer-tain models and domains do not generalize reliably across models and tasks. this could be problematic since, in practice, one might not have the means to explore and compare alternative al strategies. they also show that an actively acquired dataset using a certain model-in-the-loop, may be disadvantageous for training models of a different family, raising the issue of whether the downsides inherent to al are worth the modest and inconsistent performance gains it tends to afford. in this paper, we aim to explore all possible limitations that researchers and practitioners currently face when doing research on al @xcite . we first describe the process of pool-based al (figure 1 ) and identify challenges in every step of the iterative process ( §2). next, we unearth obscure details that are often left unstated and under-explored ( §3). we then delve into a more philosophical discussion of the role of simulation and its connection to real practical applications ( §4). finally, we provide guidelines for future work ( §5) and conclusions ( §6), aspiring to promote neglected, but valuable, ideas to improve the direction of research in active learning. ## challenges in the active learning loop we first introduce the typical steps in the poolbased al setting @xcite and identify several challenges that an al practitioner has to deal with, across all steps (figure 2 ). foot_1 ## the fine print previously, we presented specific challenges across different steps in the al loop that researchers and practitioners need to address. still, these challenges have long been attracting the attention of the research community. interestingly, there are more caveats, that someone with no al experience might have never encountered or even imagined. hence, in this section we aim to unveil several such small details that still remain unexplored. ## active learning in simulated vs. real world settings is it truly logical to consider an already cleaned (preprocessed), typically published open-source labeled dataset as an unlabeled data pool for pool-based active learning simulation, with the expectation that any conclusions drawn will be applicable to real-world scenarios? the convenience and scalability of simulation make it an undoubtedly appealing approach for advancing machine learning research. in nlp, when tackling a specific task, for instance summarization, researchers often experiment with the limited availability of labeled summarization datasets, aiming to gain valuable insights and improve summarization models across various domains and languages. while this approach may not be ideal, it is a practical solution. what makes the sub-field of active learning different? admittedly, progress has, and will be made in al research by leveraging simulation environments, similar to other areas within machine learning. thus, there is no inherent requirement for a radically different approach in al. we believe that simulating al is indispensable for developing new methods and advancing the state-of-the-art. nonetheless, we argue that a slight distinction should be taken into account. al is an iterative process that aims to obtain the smallest possible amount of labeled data given a substantially larger pool of unlabeled data for maximizing predictive performance on a given task. the difference between developing models and constructing datasets lies in the fact that if a model is poorly trained, it can simply be retrained. conversely, in al, there exists a finite budget for acquiring annotations, and once it is expended, there is no going back. consequently, we must have confidence that the al stateof-the-art established through research simulations will perform equally well in practical applications. given these considerations, we advocate for a more critical approach to conducting simulation al experiments. we should be addressing all the challenges ( §2) and the experimental limitations ( §3) discussed previously, while acknowledging the disparities between the simulation environment and real-world applications ( §4.1). given that datasets tend to outlast models @xcite , we firmly believe that it is crucial to ensure the trustworthiness of al research findings and their generalizability to real-world active data collection. this will contribute to the generation of high-quality datasets that stand the test of time ( §3.4). ## guidelines for future work given the inherent limitations of simulated al settings, we propose guidelines to improve trustworthiness and robustness in al research. transparency our first recommendation is a call for transparency, which essentially means to report everything @xcite . every detail of the experimental setup, the implementation and the results, would be extremely helpful to properly evaluate the soundness of the experiments. we urge al researchers to make use of the appendix (or other means such as more detailed technical reports) to communicate interesting (or not) findings and problems, so that all details ( §3) are accessible. ## conclusion in this position paper, we examine the numerous challenges encountered throughout the various stages of the active learning pipeline. additionally, we provide a comprehensive overview of the often-overlooked limitations within the al research community, with the intention of illuminating obscure experimental design choices. furthermore, we delve into a thorough exploration of the limitations associated with simulation in al, engaging in a critical discussion regarding its potential as either a lower or upper bound on al performance. lastly, we put forth guidelines for future research directions, aimed at enhancing the robustness and credibility of al research for effective real-world applications. this perspective is particularly timely, particularly considering the notable advancements in modeling within the field nlp (e.g., chatgpt foot_7 , claude foot_8 ) . these advancements have resulted in a shift of emphasis towards a more data-centric approach in machine learning research, emphasizing the significance of carefully selecting relevant data to enhance models and ensure their alignment with human values.