Mind the User! Measures to More Accurately Evaluate the Practical Value of Active Learning Strategies (2023)

Abstract: One solution to limited annotation budgets is active learning (AL), a collaborative process of human and machine to strategically select a small but informative set of examples. While current measures optimize AL from a pure machine learning perspective, we argue that for a successful transfer into practice, additional criteria must target the second pillar of AL, the human annotator. In text classification, e.g., where practitioners regularly encounter datasets with an increased number of imbalanced classes, measures like F1 fall short when finding all classes or identifying rare cases is required. We therefore introduce four measures that reflect class-related demands that users place on data acquisition. In a comprehensive comparison of uncertainty-based, diversity-based, and hybrid query strategies on six different datasets, we find that strong F1 performance is not necessarily associated with full class coverage. Uncertainty sampling outperforms diversity sampling in selecting minority classes and covering classes more efficiently, while diversity sampling excels in selecting less monotonous batches. Our empirical findings emphasize that a holistic view is essential when evaluating AL approaches to ensure their usefulness in practice - the actual, but often overlooked, goal of development. To this end, standard measures for assessing the performance of text classification need to be complemented by such that more appropriately reflect user needs.

## introduction a well-known problem in supervised machine learning (ml) is scenarios where there are limited resources (e.g., budget or time) to annotate data. one approach to solving this problem is active learning (al; @xcite , a collaborative process between human and machine. through targeted query strategies, al aims to find a minimal subset of examples whose labels provide the most information for fitting a model. in text classification, many applications have been found to benefit from al, such as sentiment analysis, intent or topic detection (e.g., @xcite @xcite . in addition to these task-specific studies, increased efforts have been made to systematically evaluate the performance of al strategies across different use cases (e.g., @xcite @xcite . yet many academic studies ignore crucial realworld factors, leading to flawed assessments of practical utility. literature has pointed out several limitations, including: the difficulty of making a-priori forecasts about the practical value of strategies @xcite ; the fact that actively acquired datasets are often only effective coupled with the respective model @xcite ; the need for out-of-distribution generalization @xcite ; taking into account class imbalance that is regularly encountered in real-world text classification @xcite ; and the consideration of extreme multi-label scenarios @xcite . while these works seek to optimize al from a ml perspective, it has been largely neglected that users themselves can present significant challenges that may impact the success of al. for instance, it has been found that the effectiveness of al depends on the expertise of the annotators @xcite . furthermore, examples selected by acquisition functions tend to be more ambiguous in terms of class assignment, leading to an increase in annotation uncertainty @xcite and annotation time @xcite . such details can affect and even challenge the entire al process. we therefore argue that a successful transition from research to practice requires a more holistic evaluation that targets both pillars of al, the machine learner and the human annotator. in this work, we focus primarily on the requirements that the human annotator places on a successful al process. more precisely, we introduce evaluation measures that already take this perspective into account during the development phase of al approaches, further referred to as "user-centric" foot_0 . considering the frequent scenario of multi-class text classification with imbalanced classes @xcite , we contribute through four novel measures that capture classrelated demands in al. we compare different query strategies coupled with bert across six datasets and analyze the results from both a standard ml and a more user-centric perspective. our findings indicate that the proposed measures can provide important insights into strengths and weaknesses of al that complement existing approaches. ## related work in evaluating the performance of al, predictive accuracy has generally been the main focus @xcite . prior work has relied on taskspecific measures, such as accuracy and f 1 . less commonly, al-specific measures like deficiency @xcite were used. in addition, several measures have addressed desirable characteristics of query strategies, such as uncertainty of the acquired examples @xcite , diversity of the acquired examples @xcite , and representativeness w.r.t the full dataset @xcite . the majority of these measures focus on the input or feature space, but representativeness has also been measured in the output label space @xcite . another focus besides predictive accuracy has been on the computational effort @xcite . with a strong emphasis on ml performance, the current measures tend to overlook the human component in the real-world application of al. although user studies have proven helpful in uncovering user-centric pitfalls that can get in the way of practicality @xcite , they are expensive and time-consuming, which is why they are often avoided in research. to overcome this hurdle, @xcite suggested to simulate user factors from real-world applications when evaluating al in an experimental setup (i.e., benchmarking on an already labeled dataset). they addressed error-proneness in al and presented a theoretical framework for simulating annotation uncertainty of the user. our work follows this lead by incorporating user factors into the laboratory evaluation of al to provide a simple alternative to costly user studies. however, we focus on the requirements that users place on al applications in order for them to be considered beneficial in practice. in particular, we address the need for achieving high or full class coverage in a timely manner and covering minority classes. furthermore, as a solution approach to the annotation uncertainty problem modeled by @xcite , we hypothesize how examples should be acquired to reduce annotation errors and introduce a corresponding measure. ## methodology in this section, we first give a more formal introduction to al. then, we motivate and define the four user-centric measures that are central to this work. ## evaluation design we provide an overview of the study design next by going into detail about the dataset selection, the chosen classification model, the selection of query strategies, and the experimental setup. ## results in this section, we report the experimental results. we start by shedding light on the performance of the different query strategies as is common in the literature via a standard measure for classification tasks, in our case the f 1 score. using the newly introduced user-centric measures, we then shift our focus to analyzing additional indicators that can help select an appropriate query strategy for practical use. ## discussion we considered several measures that take into account aspects that may determine the practicality of active learning strategies with respect to specific application scenarios. for the datasets under consideration, it can be seen that the f 1 score, the rapidity of class coverage, and the minority-awareness in the batches advocate for the use of uncertainty-based acquisition functions, in particular bt, in practi-cal scenarios with multiple and imbalanced classes. however, core-sets offer the opportunity to add more variety to the monotonous task of annotation by filling batches with rather different classes and in a more balanced way. this may potentially help prevent annotation fatigue and thus human annotation errors that negatively impact al. in addition, such variation could be a plus in terms of usability. what is more, we found weaknesses in reaching full class coverage for all strategies. for random sampling and core-sets, we hypothesize that this is caused by extremely rare classes. however, for uncertainty sampling, the problem became even more apparent when excluding those classes. this is of particular interest since full supervision f 1 can be well achieved within the annotation budget (see figure 1 ). although the f 1 score and some user-centric measures recommend bt as a favorite, the lack of reliability in achieving full class coverage, which we have empirically determined, may become a decisive criterion for practical applicability. not only can it have a significant impact on human trust in al. this finding affects al in general, as the reliability of models strongly depends on the quality of the datasets. ## conclusion with our results, we were able to illustrate that different query strategies stand out in different aspects that might be desirable or even necessary from the user's perspective in the practical application of al. so what implications can be drawn for al research beyond this study? the main reason why research on al exists is its development and improvement for real-world use. in this, al is a collaberative interaction between human and machine. however, this particular feature of al seems to have gradually faded from the community's awareness, with the main focus being on optimizing the established performance measure for the particular machine learning task, e.g. classification. it is true that these established measures have important informational value about the methods. but there are additional requirements that arise specifically from the human factor inherent in the nature of al, which likewise impact the practical value of al. these should therefore be taken into account. therefore, we argue that future studies on al should report a wider range of measures in their experimental evaluation. with this broader foun-dation, practitioners will be able to make a more informed decision when selecting an al strategy based on academic findings in order to comply with their specific needs for a given application. for example, in applications where the annotation step is simultaneously used to analyze the dataset at hand, features such as a quick overview of all classes or, in particular, minority classes can be desired, as we have discussed in more detail in section 3.2. surely, the measures we have suggested are by no means exhaustive. therefore, this work should also serve as a motivation to cover other aspects of the human component of al in future research. ultimately, selecting an appropriate al strategy for some practical use case is a matter of balancing different needs. the suggested measures make an important contribution to this, as they enable more reflective decisions, especially in combination with common performance measures like the f 1 score. to sum up, al has the potential to support ml in scenarios where the annotation budget is limited. we have argued that in order to assist the transfer of such methods from research to practice, both the machine learner and the human annotator must be taken into account. considering the frequent use case of multi-class text classification with imbalanced classes, we introduced four measures that evaluate the acquired examples w.r.t. class-related requirements from the user's point of view. these measures are based on scientific literature and practical experience. our results show that as complete a picture as possible should be considered to avoid failures in practical application. the next step will be to conduct a user study to validate the usefulness of the metrics presented here. in future work, we will also investigate in more detail which influencing factors prevent a fast finding of all classes. this necessitates a study that investigates, among other aspects, the effect of data distribution on the class coverage of the different strategies in order to draw general conclusions.