Small-Text: Active Learning for Text Classification in Python (2023)

Abstract: We introduce small-text, an easy-to-use active learning library, which offers pool-based active learning for single- and multi-label text classification in Python. It features numerous pre-implemented state-of-the-art query strategies, including some that leverage the GPU. Standardized interfaces allow the combination of a variety of classifiers, query strategies, and stopping criteria, facilitating a quick mix and match, and enabling a rapid development of both active learning experiments and applications. With the objective of making various classifiers and query strategies accessible for active learning, small-text integrates several well-known machine learning libraries, namely scikit-learn, Pytorch, and Hugging Face transformers. The latter integrations are optionally installable extensions, so GPUs can be used but are not required. Using this new library, we investigate the performance of the recently published SetFit training paradigm, which we compare to vanilla transformer fine-tuning, finding that it matches the latter in classification accuracy while outperforming it in area under the curve. The library is available under the MIT License at https://github.com/webis-de/small-text , in version 1.3.0 at the time of writing.

## introduction text classification, like most modern machine learning applications, requires large amounts of training data to achieve state-of-the-art effectiveness. however, in many real-world use cases, labeled data does not exist and is expensive to obtain, especially when domain expertise is required. active learning @xcite solves this problem by repeatedly selecting unlabeled data instances that are deemed informative according to a so-called query strategy, and then having them labeled by an expert (see figure 1a ). a new model is then trained on all previously labeled data, and this process is repeated until a specified stopping criterion is met. active learning aims to minimize the amount of labeled data required while maximizing the effectiveness (increase per iteration) of the model, e.g., in terms of classification accuracy. an active learning setup, as shown in figure 1b , generally consists of up to three components on the system side: a classifier, a query strategy, and an optional stopping criterion. meanwhile, many approaches for each of these components have been proposed and studied. determining appropriate combinations of these approaches is only possible experimentally, and efficient implementations are often nontrivial. in addition, the components often depend on each other, for example, when a query strategy relies on parts specific to certain model classes, such as gradients @xcite or embeddings @xcite . the more such non-trivial combinations are used together, the more the reproduction effort increases, making a modular library essential. an obvious solution to the above problems is the use of open source libraries, which, among other benefits, accelerate research and facilitate technology transfer between researchers as well as into practice @xcite . while solu- core active learner pytorch integration transformers integration dataset abstraction classifiers query strategies query strategies dataset abstraction classifiers dataset abstraction classifiers required optional depends on installation packages package cpu gpu or cpu figure 2 : module architecture of small-text. the core module can optionally be extended with a pytorch and transformers integration, which enable to use gpu-based models and state-of-the-art transformer-based text classifiers of the hugging face transformers library, respectively. the dependencies between the module's packages have been omitted. the main goal of small-text is to offer state-ofthe-art active learning for text classification in a convenient and robust way for both researchers and practitioners. for this purpose, we implemented a modular pool-based active learning mechanism, illustrated in figure 2 , which exposes interfaces for classifiers, query strategies, and stopping criteria. the core of small-text integrates scikitlearn @xcite , enabling direct use of its classifiers. overall, the library provides thirteen query strategies, including some that are only usable on text data, five stopping criteria, and two integrations of well-known machine learning libraries, namely pytorch @xcite and transformers @xcite . the integrations ease the use of cuda-based gpu computing and transformer models, respectively. the modular architecture renders both integrations completely optional, resulting in a slim core that can also be used in a cpu-only scenario without unnecessary dependencies. given the ability to combine a considerable variety of classifiers and query strategies, we can easily build a vast number of combinations of active learning setups. the library provides relevant text classification baselines such as svm @xcite and @xcite , and many more can be used through scikit-learn. recent transformer mod-els such as bert @xcite are available through the transformers integration. this integration also includes a wrapper that enables the use of the recently published setfit training paradigm @xcite , which uses contrastive learning to fine-tune sbert embeddings @xcite in a sample efficient manner. as the query strategy, which selects the instances to be labeled, is the most salient component of an active learning setup, the range of alternative query strategies provided covers four paradigms at the time of writing: (i) confidence-based strategies: least confidence @xcite , prediction entropy @xcite , breaking ties @xcite , bald @xcite , cvirs @xcite , and contrastive active learning @xcite ; (ii) embedding-based strategies: badge @xcite , bert k-means @xcite , discriminative active learning @xcite , and seals @xcite ; (iii) gradient-based strategies: expected gradient length (egl; @xcite , egl-word @xcite , and eglsm @xcite ; and (iv) coreset strategies: greedy coreset @xcite and lightweight coreset @xcite . since there is an abundance of query strategies, this list will likely never be exhaustive-also because strategies from other domains, such as computer vision, are not always applicable to the text domain, e.g., when using the geometry of images @xcite , and thus will be disregarded here. furthermore, small-text includes a considerable amount of different stopping criteria: (i) stabilizing predictions @xcite , (iv) overall-uncertainty @xcite , (iii) classification-change @xcite , (ii) predicted change of f-measure @xcite , and (v) a criterion that stops after a fixed number of iterations. stopping criteria are often neglected in active learning although they exert a strong influence on labeling efficiency. the library is available via the python packaging index and can be installed with just a single command: pip install small-text. similarly, the integrations can be enabled using the extra requirements argument of python's setuptools, e.g., the transformers integration is installed using pip install small-text @xcite . the robustness of the implementation rests on extensive unit and integration tests. detailed examples, an api documentation, and common usage patterns are available in the online documentation. 1 ## library versus annotation tool we designed small-text for two types of settings: (i) experiments, which usually consist of either automated active learning evaluations or shortlived setups with one or more human annotators, and (ii) real-world applications, in which the final model is subsequently applied on unlabeled or unseen data. both cases benefit from a library which offers a wide range of well-tested functionality. to clarify on the distinction between a library and an annotation tool, small-text is a library, by which we mean a reusable set of functions and classes that can be used and combined within more complex programs. in contrast, annotation tools provide a graphical user interface and focus on the interaction between the user and the system. obviously, small-text is still intended to be used by annotation tools but remains a standalone library. in this way it can be used (i) in combination with an annotation tool, (ii) within an experiment setting, or (iii) as part of a backend application, e.g. a web api. as a library it remains compatible to all of these use cases. this flexibility is supported by the library's modular architecture which is also in concordance with software engineering best practices, where high cohesion and low coupling @xcite are known to contribute towards highly reusable software @xcite . as a result, small-text should be compatible with most annotations tools that are extensible and support text classification. ## code example in this section we show a code example to perform active learning with transformers models. dataset first, we create (for the sake of a simple example) a synthetic two-class spam dataset of 100 instances. the data is given by a list of texts and a list of integer labels. to define the tokenization strategy, we provide a transformers tokenizer. from these individual parts we construct a transformersdataset object which is a dataset abstraction that can be used by the interfaces in small-text. this yields a binary text classification dataset containing 50 examples of the positive class (spam) and the negative class (ham) each: since the active learner may need to instantiate a new classifier before the training step, a factory @xcite is responsible for creating new classifiers. finally, we set the query strategy to least confidence @xcite . initialization there is a chicken-and-egg problem for active learning because most query strategies rely on the model, and a model in turn is trained on labeled instances which are selected by the query strategy. this problem can be solved by either providing an initial model (e.g. through manual labeling), or by using cold start approaches @xcite . in this example we simulate a user-provided initialization by looking up the respective true labels and providing an initial model: from small_text import \ poolbasedactivelearner, \ random_initialization_balanced as init @xmath0, query_strategy, train) # provide initial data. @xmath1.y, @xmath2.initialize_data( indices_initial, train.y[indices_initial] ) 1 to provide an initial model in the experimental scenario (where true labels are accessible), smalltext provides sampling methods, from which we use the balanced sampling to obtain a subset whose class distribution is balanced (or close thereto). in a real-world application, initialization would be accomplished through a starting set of labels supplied by the user. alternatively, a cold start classifier or query strategy can be used instead. active learning loop after the previous code examples prepared the setting by loading a dataset, configuring the active learning setup, and providing an initial model, the following code block shows the actual active learning loop. in this example, we perform five queries during each of which ten instances are queried. during a query step the query strategy samples instances to be labeled. subsequently, new labels for each instance are provided and passed to the update method, and then a new model is trained. in this example, it is a simulated response relying on true labels, but in a real-world application this part is the user's response. 2022 ) is an active learning framework that provides an annotation interface and a benchmarking mechanism to develop new query strategies. while it has some overlap with small-text, it is not a library, but also focuses on text data, namely on text classification and sequence tagging. in the distinguishing characteristic of small-text is the focus on text classification, paired with a multitude of interchangeable components. it of- @xcite . the dataset type was abbreviated by n (news), s (sentiment), q (questions). ⋆: predefined test sets were available and adopted. ## experiment we perform an active learning experiment comparing an sbert model trained with the recent sentence transformers fine-tuning paradigm (set-fit; @xcite ) over a bert model trained with standard fine-tuning. setfit is a contrastive learning approach that trains on pairs of (dis)similar instances. given a fixed amount of differently labeled instances, the number of possible pairs is considerably higher than the size of the original set, making this approach highly sample efficient @xcite and therefore interesting for active learning. ## library adoption as recent publications have already adopted smalltext, we present four examples which have already successfully utilized it for their experiments. @xcite investigated the detection of abusive language using transformer-based active learning on six datasets of which two exhibited a balanced and four an imbalanced class distribution. they evalu-ated a pool-based binary active learning setup, and their main finding is that, when using active learning, a model for abusive language detection can be efficiently trained using only a fraction of the data. ## conclusion we introduced small-text, a modular python library, which offers state-of-the-art active learning for text classification. it integrates scikit-learn, pytorch, and transformers, and provides robust components that can be mixed and matched to quickly apply active learning in both experiments and applications, thereby making active learning easily accessible to the python ecosystem.