Combining Active Learning and Task Adaptation with BERT for Cost-Effective Annotation of Social Media Datasets (2023)

Abstract: Social media provide a rich source of data that can be mined and used for a wide variety of research purposes. However, annotating this data can be expensive, yet necessary for state-of-the-art pre-trained language models to achieve high prediction performance. Therefore, we combine pool-based active learning based on prediction uncertainty (an established method for reducing annotation costs) with unsupervised task adaptation through Masked Language Modeling (MLM). The results on three different datasets (two social media corpora, one benchmark dataset) show that task adaptation significantly improves results and that with only a fraction of the available training data, this approach reaches similar F1-scores as those achieved by an upper-bound baseline model fine-tuned on all training data. We hereby contribute to the scarce corpus of research on active learning with pre-trained language models and propose a cost-efficient annotation sampling and fine-tuning approach that can be applied to a wide variety of tasks and datasets.

## introduction approximately 59% of the population worldwide use social media @xcite . collectively, they post more than half a million comments on facebook each minute, and a grand total of 500 million tweets per day @xcite . these statistics indicate that social media are a virtually inexhaustible source of data, and a large part of this data can be accessed for research purposes. however, annotating this data, which is often necessary to achieve high prediction performance with pre-trained language models, can be an expensive and time-consuming process. one approach that has been proposed in previous research to reduce annotation costs is active learning (al), which aims at optimizing the annotation effort by selecting specific data points from an unlabeled data pool which are expected to contribute the most to a model's learning phase @xcite . although al has proven its usefulness throughout decades of research, it remains a data selection method, which makes it challenging to use as only tool for annotation cost reduction and to reach upper bound performance (achieved by supervised learning on all available data). in this work, we therefore exploit the capabilities of transformer-based pre-trained language models to learn from unsupervised data through their pretraining task. concretely, we combine al based on the prediction uncertainty of a model with unsupervised task adaptation through masked language modeling (mlm) to investigate whether this combination of techniques allows reaching the upper bound and with how much data. we test this approach by using different al protocols on three publicly available datasets (2 social media datasets and 1 benchmark) that are costly to annotate, e.g., because they contain many fine-grained labels or the task is difficult to learn with little data. the experiments presented in this work show that using task adaptation before al has a significant effect on model performance, and that substantially less data is needed to reach upper bound performance, therefore reducing annotation expenses. ## conclusion main findings the current work combines uncertainty-based al with task adaptation in order to learn from the data that could not be labeled due to limited annotation budget. it investigates the following research questions: the results of the experiments conducted on multiple datasets provide the following answers to these questions: future research directions as mentioned earlier, al with pre-trained language models remains understudied, in spite of it being an efficient annotation cost reduction method. future research directions for al with language models may include investigating the effect of calibration quality on uncertainty-based al. an additional direction worth investigating is combining al with other methods, such as data augmentation, weak supervision and domain adaptation, which are until now topics that are studied more extensively in other machine learning fields, such as computer vision @xcite @xcite @xcite .