ALT oolbox: A Set of Tools for Active Learning Annotation of Natural Language Texts (2022)

Abstract: We present ALToolbox – an open-source framework for active learning (AL) annotation in natural language processing. Currently, the framework supports text classification, sequence tagging, and seq2seq tasks. Besides state-of-the-art query strategies, ALToolbox provides a set of tools that help to reduce computational overhead and duration of AL iterations and increase annotated data reusability. The framework aims to support data scientists and researchers by providing an easy-to-deploy GUI annotation tool directly in the Jupyter IDE and an extensible benchmark for novel AL methods. We prepare a small demonstration of ALToolbox capabilities available online. The code of the framework is published under the MIT license.

## introduction the development of text processing applications based on machine learning (ml) usually requires a lot of labeled data. despite numerous annotated corpora designed for various tasks and available for resource-rich languages, in practice, the business logic of an application is often very specific and cannot be implemented using only public resources. manual annotation of natural language corpora is a tedious and time-consuming task, which can take up to 30-40% of the application development time. for rather simple tasks, the annotation of corpora can be organized using crowd-sourcing. however, crowd-sourcing is not suitable for specific domains like medicine, finance, information technology, or any other field that requires specific qualifications or knowledge of business logic. it is also problematic to apply crowd-sourcing when the annotation scheme is complex and requires some premature training of annotators. in each of the aforementioned cases, annotation of each instance becomes expensive because it requires hiring people with a high qualification or a specific skill set. active learning (al) is a well-known technique that speeds up data annotation by leveraging model output for selecting instances demonstrated to human experts @xcite . it focuses human effort on instances that are the most informative for model training, decreasing redundancy and filtering out noisy outliers. al helps to achieve a certain level of model performance using only a fraction of the labor required to exhaustively annotate a given dataset. in this work, we present altoolbox -an opensource framework that contains a comprehensive set of tools for practical al annotation in text classification, sequence tagging, and seq2seq tasks. the main goal of the framework is to support data scientists and researchers. they usually need to test new ideas very quickly, and the lack of annotation is a common obstacle to this. altoolbox aims to address several practical obstacles to deploying al: (1) data annotated with al should be reusable; (2) al should not consume excessive computational resources, while the annotation process should be interactive without delays for annotators; (3) annotation should be quick and fluent. there are many ui-centric academic and commercial annotation systems for end users that support al annotation: webanno @xcite , alpacatag @xcite , paladin @xcite , activeanno @xcite , famie @xcite , prodigy @xcite (a commercial system), and others. however, they lack many practical features that serve the goal of rapid annotation, compatibility with pipelines for data analysis and ides, and reusability of the annotated data. there are also several low-level al packages that focus on providing various query strategies and can be used as building blocks for more elaborated systems: libact @xcite , modal @xcite , baal (atighehchian et al., 2020), small-text @xcite . however, most of them also overlook the problem of reusability and computational efficiency. only small-text is specifically tailored to nlp tasks. the contributions of the proposed framework: method paladin activeanno alpacatag famie small-text altoolbox (ours) actune (yu et al., 2022) ✓ alps (yuan et al., 2020) ✓ ✓ badge (ash et al., 2020) ✓ ✓ ✓ bait @xcite ✓ bald @xcite ✓ batchbald @xcite ✓ bert-km (yuan et al., 2020) ✓ ✓ ✓ bleuvar @xcite ✓ breaking ties @xcite ✓ ✓ cal @xcite ✓ ✓ cluster-margin @xcite ✓ coreset @xcite ✓ ✓ discriminative al @xcite ✓ egl @xcite ✓ ✓ ensp @xcite ✓ entropy @xcite ✓ ✓ idds @xcite ✓ lc (lewis and gale, 1994) ✓ ✓ ✓ ✓ mnlp @xcite ✓ ✓ ✓ nsp @xcite ✓ seals @xcite ✓ ## framework description the altoolbox framework is a python library with several executable scripts, as well as a jupyter widget implemented in javascript. in this section, we describe the key features of the framework. ## related work the comparison of altoolbox with other frameworks from the related work on al in nlp is presented in first of all, altoolbox supports two most demanded nlp tasks: text classification and sequence tagging. it also works with abstractive text summarization, which is a seq2seq task. other frameworks support only one of the tasks: paladin, ac-tiveanno, and small-text work only with text classification, while alpacatag and famie support only sequence tagging. except for altoolbox and famie @xcite , the computational overhead and the al-caused time delays have been inexplicably dismissed in the prior art. famie entails training a bigger model in the background during the labeling of each batch while using a smaller one as a proxy for acquisition. such knowledge distillation makes the al annotation process more interactive but also carries an additional computational burden, requiring extra resources for training and running two models. on the contrary, the knowledge distillation within our framework reduces both the time needed to complete an al iteration and the overall amount of computation. we note that neither famie, nor other frameworks, address the asm problem that hinders the reusability of annotated data. the tools for model distillation and annotated data post-processing based on the plasm algorithm in our framework help to mitigate the asm, so a user, for example, can train xlnet using data acquired with distil-bert without significant performance penalties. most of the considered systems provide an elaborated gui for annotation by end-users. our framework aims to support data scientists and researchers and provides a fast-to-deploy minimalistic annotation system directly in the jupyter ide. none of the considered systems provides easyto-use scripts for conducting experiments with new al methods. altoolbox implements an extensible benchmarking tool that we hope will simplify research in al for nlp. one of the problems that are currently out of the scope of altoolbox is efficient task assignments to multiple annotators. proactive learning implemented in paladin addresses this problem. we consider this feature as future work.