D e M e V a at L e W i D i-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning (2025)

Abstract: This system paper presents the DeMeVa team’s approaches to the third edition of the Learning with Disagreements shared task (LeWiDi 2025; Leonardelli et al., 2025). We explore two directions: in-context learning (ICL) with large language models, where we compare example sampling strategies; and label distribution learning (LDL) methods with RoBERTa (Liu et al., 2019b), where we evaluate several fine-tuning methods. Our contributions are twofold: (1) we show that ICL can effectively predict annotator-specific annotations (perspectivist annotations), and that aggregating these predictions into soft labels yields competitive performance; and (2) we argue that LDL methods are promising for soft label predictions and merit further exploration by the perspectivist community.

## introduction in natural language processing (nlp), annotations are often treated as a gold standard, implying a single, unambiguous truth. however, for tasks that involve, among other things, cultural norms or subjectivity, human judgments can vary substantially, often reflecting diverse annotator backgrounds or personal perspectives @xcite . customary approaches that aggregate these diverging annotations with techniques like majority voting disregard the potential validity of pluralistic interpretations, which may lead to the loss of valuable information about both the data instances and the people who annotated them. the learning with disagreements (lewidi) shared task shifts the focus to learning from unaggregated crowd labels, whether through learning from soft labels or through aligning models with specific annotators' viewpoints (i.e., perspectivist training). the demeva team ranks 2nd overall on the leaderboard of the lewidi 3rd edition shared task (lewidi 2025; @xcite . in this system paper, we describe the contributions of the demeva team and discuss both our highest-scoring method and the other approaches that did not make it onto the leaderboard. we hope that our interpretation of these results will offer insights into learning with disagreement in nlp. we obtained our score on the leaderboard by employing in-context learning (icl) for perspectivist modeling. icl refers to the ability of pre-trained large language models (llms) to perform nlp tasks without task-specific training; in icl, these models are instead conditioned on input-output examples ("demonstrations") provided in the prompt @xcite . recent studies have demonstrated icl's success on a wide range of tasks (see e.g. @xcite . however, they have also shown that icl is sensitive to the choice, order, and format of demonstrations. we explore how and to what extent icl can be leveraged to steer llms toward the annotation patterns of individual annotators in natural language understanding. in parallel with perspectivist icl, our team also pursued alternative directions aimed at modeling label distributions. in this context, we drew on existing research from both nlp and other communities. specifically, we refer to studies in label distribution learning (ldl), a research vein that focuses on modeling probability distributions over full label spaces and which has its roots in the broader machine learning community. we note that some of the insights from ldl have not yet fully found their way into nlp-specific research. in our experiments, we build on such works by using two ldl-specific fine-tuning methods, neither of which has been widely applied in nlp: ordinal label distribution learning @xcite and predicting population-level label distributions via clustering @xcite . the structure of this paper is as follows. in section 2, we briefly reintroduce the datasets and subtasks of the lewidi shared task. next, we describe our icl approaches in section 3 and our ldlrelated fine-tuning strategies in section 4. finally, we make our concluding remarks in section 5. ## datasets, tasks, and evaluation metrics in this section, we discuss the datasets and evaluation metrics of the lewidi 2025 shared task. ## in-context learning recent work has explored in-context learning for steering language models toward diverse human label distributions, primarily focusing on personabased prediction for tasks like toxicity and hate speech detection @xcite @xcite . in that vein, many studies focus solely on the effect of steering models with persona descriptions @xcite @xcite ; in the meantime, prompts that also incorporate annotations have been shown to elicit better predictions @xcite . while these inquiries are mostly based on more widely used datasets, lewidi 2025 presents new challenges on tasks that have received limited attention so far in the domain of perspectivist nlp such as paraphrase evaluation and sarcasm detection. we explore different icl strategies on these novel datasets to advance perspective-aware modeling, leveraging state-of-the-art generative models: openai's gpt-4o (achiam et al., 2023), claude haiku 3.5 @xcite , and llama 3.1 70b-instruct @xcite . however, we do not explore persona-based steering as the lewidi 2025 datasets contain relatively few sociodemographic variables, making sociodemographic prompting infeasible. ## fine-tuning approaches in this section, we discuss the various fine-tuning approaches we have explored for task a. while the overall performance of these approaches was ranked lower on the leaderboard than the in-context learning methods from section 3-in part because we merely tackled a subset of task a-we believe that these fine-tuning methods are still a valuable contribution to understanding how we can learn from disagreements. all fine-tuning experiments were done using the base roberta @xcite model. ## conclusion in this paper, we introduced the two main approaches taken by the demeva team for the lewidi 2025 shared task. our comparison of icl approaches on perspectivist modeling, while not yielding fully conclusive results, suggested that sampling examples based on labels can help generative models calibrate their predictions-especially for numeric outputs like likert scale values. models calibrated in this way can trace and mimic annotators' behavior down to more specific, granular details. however, their reliance on common sense (possibly induced by rlhf) may hinder their ability to recognize plurality when it is not overtly expressed. the second contribution of this work is a call for the perspectivist nlp community to look outward. in particular, we can learn from how machine learning communities have addressed uncertainty and label distribution learning. while perspectivist nlp rightly centers the diversity of annotator perspectives, it can benefit from established techniques such as probabilistic modeling and smoothing methods that account for annotation noise and limited sample sizes. we have merely scratched the surface here by borrowing simple loss functions and a clustering method from ldl, but we believe that engaging with other fields can be beneficial to the perspectivist community as a whole.