twinhter at L e W i D i-2025: Integrating Annotator Perspectives into BERT for Learning with Disagreements (2025)

Abstract: Annotator-provided information during labeling can reflect differences in how texts are understood and interpreted, though such variation may also arise from inconsistencies or errors. To make use of this information, we build a BERT-based model that integrates annotator perspectives and evaluate it on four datasets from the third edition of the Learning With Disagreements (LeWiDi) shared task. For each original data point, we create a new (text, annotator) pair, optionally modifying the text to reflect the annotator’s perspective when additional information is available. The text and annotator features are embedded separately and concatenated before classification, enabling the model to capture individual interpretations of the same input. Our model achieves first place on both tasks for the Par and VariErrNLI datasets. More broadly, it performs very well on datasets where annotators provide rich information and the number of annotators is relatively small, while still maintaining competitive results on datasets with limited annotator information and a larger annotator pool.

## introduction human language is often subjective and open to interpretation. in many nlp tasks, it's common for annotators to disagree sometimes for good reasons. but most traditional models ignore this variation and treat all labels as if there's only one correct answer. as a result, they may miss out on useful minority viewpoints and become less adaptable. the third edition of the learning with disagreements (lewidi) shared task at emnlp 2025 @xcite focuses on a critical challenge: building models that learn from disagreements rather than ignore them. the main objective of the task is to provide a unified evaluation framework for learning from disagreements. it introduces a benchmark including four datasets annotated with both soft labels and perspectivist annotations. here, soft labels represent probability distributions over possible classes, capturing the degree of annotator disagreement, while perspectivist predictions aim to recover the individual label choices of each annotator. participating teams are evaluated based on how accurately their models predict both types of outputs. in our approach, we built a simple but effective bert-based model @xcite that makes use of annotator perspectives during training. instead of collapsing multiple labels into one, we create a separate training instance for each annotator's view and combine it with their background information. this way, the model learns to understand how different kinds of annotators might interpret the same input differently. our approach performed well across all four shared task datasets. it was especially effective on tasks that involved a small set of annotators and provided natural language explanations alongside their labels. ## further analysis we conducted further analysis to understand how incorporating annotator information affects model performance. overall, models that leverage annotator information tend to outperform those that do not. for the par and varierrnli datasets, both of our annotator-aware models (tf-idf + rf and our proposed bert-based model) consistently surpassed the organizers' baselines and the bertbased models without annotator information. with a small and fixed set of annotators, the models can more easily capture individual behavior, helping them understand consistency in how samples are labeled. additionally, the inclusion of textual explanations allows the models to learn multiple perspectives from each annotator, resulting in richer, more multi-dimensional instance representations and reducing ambiguity compared to using raw labels alone. in contrast, for the mp and csc datasets, using annotator information did not lead to much improvement. these datasets only provide basic metadata (e.g., age, gender), and the number of annotators is much larger, making it harder for the model to learn how each annotator behaves. moreover, some annotator attributes were missing or not provided for certain examples, so we filled in missing values using the mode for each field. this imputation may have introduced noise and reduced the effectiveness of annotator-aware modeling. still, in mp, the model with annotator features performs slightly better. in the case of csc, our proposed model performed worse than the bert model that does not use annotator information. a likely explanation is that many annotators in csc lack associated metadata. as a result, we had to fill in missing values with default values, which may have introduced noise into the input representation and negatively affected the model's performance. these results highlight a general challenge: when the annotator pool is large and metadata is sparse or missing, modeling individual annotator behavior may become difficult. model rankings in task b largely reflect those in task a, indicating that understanding annotator behavior contributes to overall prediction quality. while annotator-aware modeling benefits datasets with small, information-rich annotator pools, generalizing to larger, sparse pools remains challenging. these results suggest that the approach is most effective when annotator numbers are limited and data is semantically rich, but its effectiveness may decrease as the pool grows and label distributions become sparse, highlighting an open question for future research. ## conclusion in this work, we presented a model that predicts labels for each (text, annotator) pair, aiming to capture individual annotator perspectives rather than just aggregated labels. we evaluated our method on four datasets covering sarcasm detection, irony detection, paraphrase detection, and natural language inference. our results show that including annotator information often leads to better performance, especially in datasets where annotator perspectives are clearly defined and consistent. however, for datasets with many annotators or missing metadata, the improvement is less clear, and in some cases, using annotator features may introduce noise. overall, our findings suggest that modelling individual perspectives is a promising direction for tasks involving subjective annotation. future work may explore more advanced architectures or evaluate on additional datasets to further understand the benefits and limitations of this approach.