PALS : Personalized Active Learning for Subjective Tasks in NLP (2023)

Abstract: For subjective NLP problems, such as classification of hate speech, aggression, or emotions, personalized solutions can be exploited. Then, the learned models infer about the perception of the content independently for each reader. To acquire training data, texts are commonly randomly assigned to users for annotation, which is expensive and highly inefficient. Therefore, for the first time, we suggest applying an active learning paradigm in a personalized context to better learn individual preferences. It aims to alleviate the labeling effort by selecting more relevant training samples. In this paper, we present novel Personalized Active Learning techniques for Subjective NLP tasks (PALS) to either reduce the cost of the annotation process or to boost the learning effect. Our five new measures allow us to determine the relevance of a text in the context of learning users personal preferences. We validated them on three datasets: Wiki discussion texts individually labeled with aggression and toxicity, and on Unhealthy Conversations dataset. Our PALS techniques outperform random selection even by more than 30%. They can also be used to reduce the number of necessary annotations while maintaining a given quality level. Personalized annotation assignments based on our controversy measure decrease the amount of data needed to just 25%-40% of the initial size.

## introduction the data acquisition process plays a crucial role in almost all natural language processing (nlp) studies. the relevant data can significantly improve the model performance. however, the low-quality data limits the model quality regardless of its advanced architecture. modern machine learning systems require large amounts of data to perform satisfactorily. although deep learning rapidly advances state-of-the-art results on a number of supervised learning tasks @xcite @xcite @xcite , we can observe that gains strongly depend on large annotated datasets @xcite @xcite . moreover, the data annotation process is very expensive @xcite . several lines of research, especially active learning, have developed mechanisms to reduce the amount of supervision required to achieve better predictive performance. they, however, focused on tasks themselves, i.e., such text selection that would increase agreement among annotators. this does not seem to be adequate for subjective nlp tasks, in which we allow the model to provide different decisions for different users and the same text @xcite . as a result, there are no active learning methods dedicated for subjective tasks and personalized reasoning. therefore, we introduce a new approach to active learning in nlp: pals -personalized active learning for subjective problems. its main contribution is modeling user preferences (perception) in the context of a given task. this allows us to exploit existing personalized models @xcite @xcite and complement them with more appropriate data acquisition. the main idea of pals is the selection of texts for annotation adapted to a given user that maximizes extraction of knowledge about the user preferences. it has been achieved by means of five measures described in sec. 4. our measures not only improve the overall performance of personalized models. they also significantly decrease the number of annotations needed. as a result, they reduce computing power, which saves money and cuts down the environmental toll. the final product of our pals methods is the optimization of the data collection process, which allows us to reduce the cost. less but more adequate data may provide the same training effect. alternatively, we acquire more information for the same cost. additionally, the annotation process can be stopped after achieving desirable quality. to develop efficient pals methods, we raised several questions. how many annotations should each participant evaluate? should we focus more on controversial or unambiguous texts? please note that currently there is no method for personalized selection of texts for annotation appropriate for subjective tasks -pals is the first. our main contribution in this work are: (1) posing a new problem regarding the effective acquisition of annotations from users in order to acquire knowledge about both the task and individual user perception supporting the learning of personalized reasoning models for subjective nlp tasks, (2) our novel pals solution -five measure-based methods, which capture the diversity in subjective text perception, (3) validation of our pals methods on three datasets and ten classification tasks related to the offensiveness of texts for different number of documents annotated by the user, (4) ablation study including the analysis of what fraction of the training set is sufficient to achieve performance comparable to a reference random selection. ## related work active learning (al) has been extensively studied and applied in numerous tasks over recent years. the costs and means of obtaining labeled datasets can be troublesome, and this fact became more prominent as the research in the domain of machine learning began to develop rapidly. the need of training classifiers using the least amount of data sprouted nearly 20 years ago, yet grew stronger in recent years. the more common al approaches use sampling based on model uncertainty using measures such as margin sampling @xcite , least model confidence @xcite and entropy @xcite . deep learning in al, also named deep active learning, can be a challenging task due to the tendency of rare uncertainty during inference and the requirement of a relatively large amount of training data. however, bayesian deep learning based approaches @xcite @xcite proved that using dropout, batch normalization, and ensembles were all useful ways of lowering uncertainty in model prediction. there have been numerous promising implementations of al in the field of natu-ral language processing. a significant amount of literature showed that the previously mentioned bayesian active learning outperformed classical uncertainty sampling. however, the newest approaches @xcite explored al with multiple strategies using bert based models for binary text classification tasks and proved that the performance of bert-based models managed to achieve even better scores. visual active learning @xcite is a generally known domain of al, which is explored in learning-based image classification. common implementations incorporate deep convolutional neural networks into al. the sample selection strategy works as a cost-effective improvement of the classifier through incorporating the analysis of high-confidence samples from an unlabeled set for feature learning. the selection of high-confidence samples leads to an automatic selection and interactively assigned pseudo labels. the vast majority of al work bases its mechanisms and score comparison on the generalized approach. the first attempts at developing personalization in al extended the bayesian al techniques @xcite through computing the probability of getting any rating, instead of the probability of getting a particular rating for a text. the algorithm is aimed at selecting items which have a very high probability of getting a rating from a certain user. this approach provides interesting scores, however, keeping the gold standard as a generalized measure. therefore, the area of personalization remains dependent on generalization to a certain degree @xcite @xcite . what we achieved in our work is the contrary, it is the person themselves who becomes the gold standard, through analyzing the annotations and ratings they submitted. this breakthrough not only may change if we analyze users' annotations in the al domain, but also in analyzing data universally. ## generalized vs. personalized active learning for text annotation active learning techniques should take into account the nature of the task to which they are applied. in the case of personalized classification, this task requires the model to combine the general knowledge about the task (nature of the text) with the individual preferences of the user (individual content perception). here, this knowledge balance is crucial, as a model too focused on the user will not take into account the linguistic aspects of the text. on the other hand, a classifier mainly exploiting linguistic knowledge will not be able to make personalized predictions given the user's point of view. to ensure this balance, we suggest the introduction of personalized active learning techniques. they should take into account the impression that the text evokes in the user and how much knowledge about the user's preferences should be gathered. to reveal individual user beliefs, a text selected for annotation must be controversial and the entire set of texts assigned to the user should contain contrasting examples. text subjectivity can be measured both in generalized and personalized ways. in the first case, we examine how diverse are the opinions that the text evokes among all annotators. the generalized ambiguity of a text can provide information about its potential to gain knowledge about the personal preferences of each annotator. however, this approach can mask detailed information about the preferences of individual people with unique points of view by treating all annotators holistically. to counteract this, knowledge of the general text controversy should be enriched with personalized information about the user's past decisions to assess how their individual subjectivity compares with the generalized subjectivity of other users @xcite @xcite @xcite . generalized active learning (gal) methods for text annotation focus on selection of documents that better reflect a given nlp problem averaged over the whole user community without considering any individual user preferences. as a result, every user may annotate the same texts, fig. 1 . usually their goal is to maximize the inter-annotator agreement level @xcite . for that reason, they do not appear to be suitable for subjective tasks like personalized offensiveness detection. we, however, introduce another idea -personalized active learning for subjective problems (pals), in which texts are selected for annotation individually for a particular user to better capture their specific content and task perception. in other words, we implement the human-centred perspective introduced in @xcite , which matches single users with subjective nlp tasks. this can further improve the performance of personalized architectures by reducing the number of redundant documents. such user-specific knowledge also facilitates distinguishing one user from another in a more efficient way. since we focus on subjective tasks, we experimentally confronted our methods with the baseline (random selection) rather than with generalized approaches, which were not designed to infer for subjective problems. ## active learning mechanisms we assume that, in the case of subjective tasks, active learning methods should respect user preferences. to achieve it, we prepared five measures: (1) controversy, (2) var ratio, (3) ratio distance, (4) stranger count, and (5) average minimal conformity. they take into account the diversity of the aggregated user beliefs or focus solely on the decisions of a single individual. texts with the greatest value of a given measure are selected to be annotated by the particular user. co-annotators (4) the stranger count measure (strangercount(d, u)) aims to compare the subjective preferences of the user with as large groups of unique users as possible, fig. 3 . its value for a specific document d in the context of user u is equal to the difference between the number of users who annotated d (|u d |) and the number of annotators (@xmath0: in our experiments, we used datasets obtained within the wikipedia detox project foot_0 and the unhealthy conversations dataset @xcite . figure 4 : the u-t dataset split. only the purple parts of the dataset are pruned in order to limit the amount of knowledge about the users in the dev and test folds the model can obtain ## experimental setup in order to validate our personalized active learning methods, we have decided to apply the following principles. the experiments that were performed on all three datasets using the binarized versions of the tasks. class imbalance was addressed by focusing on the positive, minority class {1} f1score (f1-binary) metric. ## results in the case of wikidetox aggression (the upper plot in fig. 5 ), the best results were achieved by the var ratio and controversy selection. average minimal conformity and stranger count demonstrated stable behavior, slightly different from the random annotations. however, the average minimal conformity was clearly better in the range of 3-5 annotations per user. the conformity distance measure achieved significantly worse results than all other measures. our experiments showed that choice of right measure allowed personalized reasoning models to provide better results for just 4 annotations per user compared to random selection with 14 annotations per user. the best results on the wikidetox toxicity data set (the middle in fig. 5 ) were observed for the controversy measure. however, the var ratio also performed better than the random assignment. the conformity distance and average minimal conformity turned out not to be the most stable but still slightly better for random baseline, especially in the range of 3-6/8 annotations per user. stranger count measure performed the worst. for the unhealthy conversations dataset (the bottom plot in fig. 5 ), the best results were achieved by the controversy measure. however, the average minimal conformity and the ratio distance still performed better than the random selection. surprisingly, the var ratio measure, which performed very well on the wikidetox datasets, provided worse results than the random baseline. it may be related to the high variance between the labels, which affects the final measure values. choosing the best pals measure for data annotation (controversy) allowed our model to outperform even by more than 30%, fig. 6 . it means that better selected training cases significantly boost quality of personalized reasoning since they deliver more information about user diversity. moreover, using the controversy or the var ratio measure allowed for achieving the similar results as for the random assignment, but using only about 25% of the data, fig. 7 . in other words, pals methods boost training so that it requires much less learning samples and computing resources. ## discussion by analyzing the results obtained on three datasets, we can come to the conclusion that the effectiveness of the model grows quickly with the addition of subsequent annotations for the users only up to a certain level, see fig. 5 . after it is exceeded, adding more samples to the set improves the performance only slightly, until it finally ceases to affect its performance at all. in the case of the wikidetox aggression dataset, the greatest increase in the efficiency of the model occurs in the range from 1 to 5 annotations per user, then the increase slows down until about 10 annotations is very insignificant. for wikidetox toxicity, the performance increases for up to 8 annotations per user, and then it fluctuates around a certain value. a similar effect is observable for unhealthy conversations but the inflection point occurs later -at 12 texts. we can also see that the method of selecting texts for annotation also affects the amount of data needed to reach saturation point. improving the quality of annotated texts still in the collection process reduces not only annotation costs but also training time, which positively impacts on the reduction of the carbon footprint emitted while training of deep learning models. most active learning approaches focus on the confidence of the model on unlabeled data. this works well for standard nlp methods where the text is the only source of knowledge. in this work, we show that, in personalized models, the subjectivity of people and their individual approach is also an essential factor that should be taken into account when selecting data samples for annotation. by selecting controversial texts and annotating them by different people, we gain more knowledge about their subjectivity, and therefore we improve the model quality of inference for these people. a common method of evaluating nlp models is counting metrics globally on all annotations of individuals or on majority votes of annotators. however, in this way we lose information about specific individuals or groups of people for whom our model gives significantly different results. for this reason, we can consider evaluating models for individual annotators and then averaging these results or comparing their distributions. ## conclusions and future work in this paper, for the first time, we raised a complex question: how to carry out the annotation process wisely and efficiently to reduce costs and/or to increase reasoning quality for subjective tasks solved in the personalized way? instead of commonly used random selection of texts to be annotated, we propose a novel paradigm: pals -personalized active learning methods suitable for subjective nlp tasks. they allow us to efficiently gather information about user preferences related to text perception. experimental studies on three datasets and ten problems revealed that the controversy measure appears to be the best out of all 20+ measures considered (only five best are described in the paper). the results show that pals methods increase quality of collected data and are better suited both to the task and to the user. as a result, the classifiers are able to improve their reasoning quality even by 30%. alternatively, pals methods can be used to reduce amount of data needed to reach a given performance; just 25%-40% is enough. thus, models using data provided by pals methods can learn in an extremely short time. this has a direct impact on reducing the costs of the entire annotation process. in the future work, we will focus on the problem of text diversity, collection representativeness for the task, and reinforcement learning methods that would adjust the selection process runtime. the code for all methods and experiments is publicly available foot_1 under the mit license.