“You are an expert annotator”: Automatic Best–Worst-Scaling Annotations for Emotion Intensity Modeling (2024)

Abstract: Labeling corpora constitutes a bottleneck to create models for new tasks or domains. Large language models mitigate the issue with automatic corpus labeling methods, particularly for categorical annotations. Some NLP tasks such as emotion intensity prediction, however, require text regression, but there is no work on automating annotations for continuous label assignments. Regression is considered more challenging than classification: The fact that humans perform worse when tasked to choose values from a rating scale lead to comparative annotation methods, including best–worst scaling. This raises the question if large language model-based annotation methods show similar patterns, namely that they perform worse on rating scale annotation tasks than on comparative annotation tasks. To study this, we automate emotion intensity predictions and compare direct rating scale predictions, pairwise comparisons and best–worst scaling. We find that the latter shows the highest reliability. A transformer regressor fine-tuned on these data performs nearly on par with a model trained on the original manual annotations.

## introduction labeling data with trained experts or via crowdsourcing is a resource-intensive and timeconsuming process @xcite @xcite . this motivates automated annotation methods, including weak supervison @xcite , zero-shot predictions @xcite , or, more recently, generative models @xcite @xcite . depending on the downstream task at hand, the labels to be assigned to a textual instance are categorical (e.g., in text classification), structured (for instance in parsing or named entity recognition), or continuous (e.g., emotion intensity, sentiment strength, or personality profiling predictions). annotating for continuous value labels comes with its own set of challenges. it can be difficult to obtain consistent labels from humans by asking them to assign a value from a rating scale @xcite . not only is it difficult for the annotator to rate texts consistently, but it is also difficult for researchers to design rating scales, as there are many design decisions which can bias the annotator, such as scale point descriptions and scale granularity. this lead to comparative annotation setups, in which annotators are tasked to compare multiple instances for the same task, which is easier to accomplish and has fewer design decisions to make. consider the two example sentences for sentiment strength: it is difficult to assign a value v(s i ) ∈ [-1; 1] to these sentences in isolation, or even in context, but it is straight-forward to decide that v(s 2 ) > v(s 1 ). best-worst scaling @xcite is such a comparison-based annotation method that has proven to be more reliable than assigning values on rating scales, when annotating continuous values. the idea is to task an annotator to decide which instance is the one with the highest and the lowest value. the tuple size can be varied but @xcite observe that quadruples provide a good trade-off between context and numbers of comparative judgements. in this paper, we question if bws is also an appropriate approach for large language model-based annotations. on the one hand, one might argue that comparative tasks are also more reliably conducted with language models. on the other hand, one might argue that a large language model (llm) has more access to other text instances implicitly from training data which it can compare a text to, than a human. that would be an argument that when using a llm for annotations, bws may not be necessary. we therefore set up prompts for continuous value assignments for two direct and two comparisonbased annotation approaches. we use rating scales @xcite , in which the model directly labels texts with a numerical value, in two variants: annotating single texts and tuples of four texts per prompt. our comparison-based approaches are paired comparisons @xcite in which every text is compared to every other text, and best-worst scaling (bws, @xcite , where "best" and "worst" instances are picked from a tuple. in our evaluation, we focus on comparisons against human annotations for the emotion intensity prediction task @xcite . we compare the llm-based annotations directly against human annotations and further train a transformer-based regressor both on the humanlabeled data and the llm-labeled data. the motivation for this regressor is to avoid the requirement to put together instances in tuples and query a potentially expensive api at inference time. we answer the following research questions: ## methods we now provide an overview of our automatic annotation method ( §3.1), describe our prompting strategies ( §3.2), and finally explain how the automatically created corpora are used to estimate a transformer-based regressor ( §3.3). ## experiments we perform experiments to compare the annotation methods on the emotion intensity prediction task, firstly to understand if bws, pc, rs, or rs-t perform differently and secondly to understand their performance in comparison to human annotators. finally, we perform additional experiments to understand the role of the tuple count, which can be flexibly adapted in automated annotations. ## conclusion & future work we proposed to automate annotations of text data with continuous labels with bws, which outperforms rating scales and paired comparisons in the case of emotion intensity predictions, when the original annotations were also annotated using bws. the predictions from a regression model, fine-tuned on automated annotations, perform nearly en par with the models fine-tuned on the original human annotations. we showed that we can improve the annotation quality by increasing the total number of tuples. in general, we conclude that bws is the better approach to annotate novel data sets for emotion intensity regression. the results of our experiments are encouraging for emotion intensity regression. we presume that these findings carry over to other regression tasks, but this still needs to be validated. candidates for other tasks would be the bws-labeled toxicity data set ruddit @xcite or the affective norms for english words dataset @xcite . word similarity assessment tasks could be an interesting case for evaluation as well @xcite . finally, it is important to study more open-source generative models, instead of relying on pay-locked and black-box models. the german research foundation, kl 2869/11-1). we thank the reviewers and the action editor at acl rolling review for their helpful feedback.