ALANNO : An Active Learning Annotation System for Mortals (2023)

Abstract: Supervised machine learning has become the cornerstone of today’s data-driven society, increasing the need for labeled data. However, the process of acquiring labels is often expensive and tedious. One possible remedy is to use active learning (AL) – a special family of machine learning algorithms designed to reduce labeling costs. Although AL has been successful in practice, a number of practical challenges hinder its effectiveness and are often overlooked in existing AL annotation tools. To address these challenges, we developed ALANNO, an open-source annotation system for NLP tasks equipped with features to make AL effective in real-world annotation projects. ALANNO facilitates annotation management in a multi-annotator setup and supports a variety of AL methods and underlying models, which are easily configurable and extensible.

## introduction we are witnessing an ever-growing demand for data along with the rapid development of machine learning and deep learning algorithms. in particular, we need an abundance of labeled data to develop well-performing models, which is not easy to obtain. for many natural language processing (nlp) tasks, the labeling process, i.e., annotation, is often the most expensive and time-consuming part of developing machine learning models. the cognitive exertion of human annotators can affect their judgment, which further affects label validity. consequently, this manifests in poor agreement -a proxy for label reliability, which is a prerequisite for validity @xcite . poor label reliability and validity negatively affect the machine learning algorithm, as it is only as good as the data it consumes. designed to alleviate labeling issues and reduce annotation cost, active learning (al; @xcite ♠ equal contribution is a special family of machine learning algorithms. in contrast to the standard random selection of instances for labeling, a typical al method iteratively queries the most informative instances for the underlying model to achieve the best possible performance with the fewest possible labels. al has been shown to reduce annotation effort across machine learning applications, e.g., @xcite , especially in nlp, e.g., @xcite @xcite . despite the demonstrated successes of al, many challenges are involved in deploying al in realworld scenarios @xcite . unfortunately, these challenges are often overlooked in both research and practice. in particular, annotation tools that support al rarely address the problems of unbiased evaluation of al, imbalanced data, and stopping criteria for al. the lack of concrete solutions for these problems hinders the effectiveness of al. aside from the practical challenges in al, managing annotation campaigns is often very cumbersome, especially in multi-annotator setups (when multiple annotators are assigned to a single instance). specifically, assigning instances to multiple annotators can be painstaking, particularly if one aims to achieve balanced combinations of annotators across instances. while there are many serviceable frameworks for simulating al in idealized scenarios, e.g., @xcite @xcite , there are only a few tools for running real-world al annotation campaigns with multiple annotators, none of them explicitly addressing the practical al challenges. to facilitate the creation of high-quality nlp datasets at reduced annotation costs, we developed alanno (active learning annotation), an opensource annotation system with al strategies for data sampling. alanno's is specifically designed to address the practical challenges of al and fa-cilitate the management of multi-annotator annotation projects. in particular, alanno guides toward more quality labels with a novel method for the balanced assignment of unlabeled instances to annotators in a multi-annotator setup. we support building gold labels by monitoring the interannotator agreement with task-specific metrics and agreement-aware weighted aggregation of labels. equally important, alanno incorporates many features to address the major challenges of using al in practice. namely, we support guided learning @xcite for mitigating data imbalance, and we ensure trustworthy evaluation of the underlying model on an unbiased test set and a stopping criterion to maximize the effectiveness of al. as an essential practical solution, we enable a project-specific stopping criterion with a novel performance forecasting method based on bayesian regression. by estimating the performance of the underlying model with hypothetically enlarged labeled sets, we enable practitioners to determine on the spot whether further annotation will only have diminishing returns. lastly, alanno supports a wide range of state-of-the-art al methods from the literature, allowing seamless inclusion of new models or methods. in summary, our main contribution is alanno, an open-source al annotation system for nlp tasks, which features (1) practical strategies for applying al to real-world problems with a range of al methods and (2) annotation management facilitation in a multi-annotator setup with quality control. alanno enables non-experts in al to reap its benefits by accounting for key practical issues in annotation management and al. in two case studies, we demonstrate alanno's two key features -balanced data assignment and al performance forecasting. we also provide a short video 1 demonstration and release the code 2 under the apache 2.0 license. while alanno has been born out of several years of experience with nlp annotations for various tasks and has evolved with each new project, it remains highly configurable, allowing easy customization and extension. ## system overview we briefly describe the key aspects of alanno, which include projects, data assignment, label man- figure 1: organization of an alanno project. the lines between the icons indicate different lines of interaction, with the numbers denoting the temporal order. in brief, the project manager recruits annotators 1 , creates labels 2 , and imports the unlabeled data 3 , which are then appointed to annotators via an assignment algorithm 4 , 5 . the annotators use the created labels 6 to annotate the data 7 . projects. in alanno, the entire annotation process is encapsulated into a project, which handles the interactions between different parts of an annotation project, as depicted in figure 1 . a typical nlp annotation project is long-lasting and dynamic: annotators may be temporarily unavailable, new annotators may join an already-running project, and others may leave. alanno supports the managing of such a workforce dynamic. to separate the concerns and responsibilities, alanno defines two user roles: project managers, who are in charge of the annotation campaign, and annotators, whose task is to apply labels to the unlabeled data. at the moment, project managers can create three main types of projects: single-and multilabel classification, as well as sequence labeling tasks (e.g., named entity recognition). ## features motivated by our experience in annotation projects and the practical challenges that emerge when deploying al, we designed practical solutions that enable efficient labeling in real-world scenarios. we identified several key challenges, which, if not adequately addressed, may impair label quality and al efficiency. specifically, we focused on (1) label reliability, (2) unbiased evaluation of active learning models, (3) the stopping criterion for active learning, i.e., knowing when to terminate the active learning process, and (4) working with imbalanced data. ## case studies in the following case studies, we highlight the two essential features of alanno, namely balanced data assignment and al performance forecasting. case study 1: balanced assignment. to compare our quasi-monte carlo assignment method with uniform annotators combinations, we ran simulations of distributing unlabeled instances to annotators. as ## related work as the popularity of machine learning and deep learning grows, so does the need for annotated data. since high-quality data is imperative for highquality machine learning models, data annotation has become a lucrative industry, and new tools are constantly emerging. there are commercial tools such as prodigy, foot_0 v7, foot_1 and hasty. 5 however, these tools hide their full functionality behind a paywall. in contrast to the mentioned commercial system, several open-source annotation tools have appeared recently, such as label sleuth @xcite , label studio, foot_3 , inception @xcite , matilda @xcite , and paladin @xcite . label sleuth is an elegant annotation system designed to make nlp accessible for non-experts. the system enables al selection for labeling. however, it only supports simple binary classification with a single annotator per project. label studio is offered as an open-source system and a paid enterprise version. while the paid version supports active learning, the free version is limited to random selection. the system supports multiple annotators but with minimal functionalities in managing the annotations. inception is a highly configurable tool that supports al and multi-annotator setups. however, the system is hard to use, as it requires external libraries to integrate a model for al purposes. matilda is a platform for dialogue annotation in a multi-annotator setup with support for multiple languages. ## conclusion alanno is an open-source annotation system for natural language processing tasks powered by active learning. the system addresses the critical practical challenges of active learning in real-world annotation projects that have previously been overlooked. alanno enables non-experts in active learning to conduct effective annotation campaigns by supporting solutions for unbiased evaluation, stopping criterion for active learning, and class balancing. additionally, the system facilitates annotation management in a multi-annotator setup, emphasizing label quality through agreement monitoring, agreement-aware label aggregation, and a novel method for the balanced assignment of unlabeled instances to annotators.