ALAMBIC : Active Learning Automation Methods to Battle Inefficient Curation (2023)

Abstract: In this paper, we present ALAMBIC, an open-source dockerized web-based platform for annotating text data through active learning for classification task. Active learning is known to reduce the need of labelling, a time-consuming task, by selecting the most informative instances among the unlabelled instances, reaching an optimal accuracy faster than by just randomly labelling data. ALAMBIC integrates all the steps from data import to customization of the (active) learning process and annotation of the data, with indications of the progress of the trained model that can be downloaded and used in downstream tasks. Its architecture also allows the easy integration of other types of model, features and active learning strategies. The code is available on https://github.com/Trusted-AI-Labs/ALAMBIC and a video demonstration is available on https://youtu.be/4oh8UADfEmY .

## introduction data annotation is crucial for any machine learning task. datasets of high-quality are obtained through manual labelling which requires both considerable time and sometimes also expertise. active learning aims to reduce the need for labelled data by starting from a partially labelled dataset and gradually selecting the most informative instances among the unlabelled instances @xcite . this incremental training of a machine learning model can thus actively select the instances it finds to be the most informative, asking the person involved to provide a true label for the selected data. several strategies exist to define informativeness, such as those based on uncertainty of the predictions obtained for unlabelled instances @xcite or based on the difference of their features with respect to the training set @xcite . the model is then trained with the newly obtained labelled set and can again select instances to be labeled. this process is repeated up until a specific criterion is reached, such as a desired accuracy or a maximum number of instances labelled. we present in this work alambic (active learning automation with methods to battle inefficient curation), an open-source dockerized web-based platform for active-learning-based text classification, allowing for a full customisation of the active learning process, from the choice of the model, its features and parameters, and the active learning strategy foot_0 . it allows the study of the usefulness of active learning on a given labelled dataset and model, as well as the annotation of text instances with active learning. ## related works several active learning libraries have been developed, such as alipy @xcite , modal (danka and horvath), scikit-activeml @xcite , for traditional machine learning methods and distil @xcite , smalltext @xcite , the low-resource text classification framework @xcite or the altoolbox @xcite which also contains active learning methods for deep learning. however, these libraries do not provide user interface or if they do, they require the user to be reasonably skilled in coding. tools such as prodigy foot_1 , aplenty @xcite , @xcite , paladin @xcite , 2021), or label sleuth @xcite tried to fill the gap by offering a user interface for the annotation process combined with an active learning setting. while these tools offer a friendly user-interface to annotate text classification or sequence labelling tasks, they either require a highto-expert knowledge in programming languages, are not open source or are limited in the choice of the parameters in the active learning process, such as the model or the selection strategy. at the time of writing, and to the best of our knowledge, alambic is the first free annotation tool for text classification with a user-friendly interface allowing a complete control of the user on the parameters of the features, model and active learning strategy, with no coding skills needed. moreover, in addition to support the annotation task, it also allows the study of different active learning strategies with chosen features and models to evaluate the best strategy within that specific context. its dockerized form also allows for an easy deployment and usage on diverse platforms. ## design and implementation alambic is a web-based platform built in python using django framework 3 , combined with celery 3 https://www.djangoproject.com/ foot_2 , redis foot_3 and postgresql foot_4 in a docker compose setting (fig. 1 ). the installation requires only few command lines and is quite easy, even for people not familiar with docker or github (see appendix a). the framework is divided in five main interfaces : (i) the import data interface, (ii) the setup and customisation of the (active) learning process, (iii) the progress of the active learning process, (iv) annotation interface and (v) the result interface. each of these interfaces will be described below in detail foot_5 . ## use cases the platform supposes that the user has a partially labelled dataset, a preferred model and features to use, and wants to label a portion of the remaining unlabelled dataset in a way that this portion will be the most informative to obtain a high-performance model. alambic can be used for two main use cases : alambic can be used to compare the performance of different active learning strategies, to determine which strategy to choose under given parameters/conditions. ## experiments and results we used the training set of the large movie review dataset @xcite , consisting in 25,000 highly polarized movie reviews, 12,500 positive and 12,500 negative. for the analysis, we selected randomly 500 instances from each class. using alambic, we tested several active learning methods, including an uncertainty-based strategy @xcite , i.e. strategy based on different measures of uncertainty computed with the prediction output of the model; core-set @xcite , a selection method which tries to select the most different from the current training set and representative subset of unlabelled instances; and of course the random sampling as a baseline. each active learning method was tested for five folds of cross-validation, with three repetitions of the active learning process for the same test set with different initial training seeds (of a size of 10% of the remaining dataset). this means that 15 experiments were conducted for each strategy. resulting performances are averaged for each iteration step. for each step, 50 instances are selected to be added. the experiment was conducted with a support vector machine (svm) and a random forest (rf), with the default parameters proposed in their implementation in scikit-learn. the text was first pre-processed by ignoring the stop words, then the term frequency -inverse document frequency (tf-idf) was computed for each term, with a minimum document frequency of 0.1 and a maximum of 0.9. only the top 3000 features were kept to build the vocabulary, ordered by term frequency across the corpus. figure 3 and 4 display the f-scores obtained for the subset of 1000 samples. we could observe that it would be preferable to use the uncertainty sampling method coupled with the svm, as we see that while it performs less efficiently in the first few iterations, it clearly performs better and reached an optimal performance earlier than the random sampling or core-set method. however, if one wanted to use a rf, then it would be better to use the coreset method, as while it does not perform well in the first iterations, it outperforms the outer two selection methods. moreover, for the rf, the uncertainty method has a similar performance as the random sampling. those observations highlight the importance of such analysis before using active learning in a real setting in order to choose the most optimal strategy. more advanced cases will be explored outside this paper. ## conclusion and future work we presented alambic, an open-source webbased platform for the study and use of active learning in text classification. it can help people with low-to-no coding skills to use in the most efficient way active learning methods to build high-quality nlp datasets at a faster pace. first, they can compare with their choice of parameters and models different active learning strategies and then annotate their data with the strategy obtaining the best performance. further developments of the tools include the addition of other nlp annotation tasks, the integration of deep learning models and their related active learning methods. while the later development would reduce the ease of deployment for non-experts in coding, this would offer more stateof-the-art models for further automatic annotation.