LPI - RIT at L e W i D i-2025: Improving Distributional Predictions via Metadata and Loss Reweighting with D is C o (2025)

Abstract: The Learning With Disagreements (LeWiDi) 2025 shared task aims to model annotator disagreement through soft label distribution prediction and perspectivist evaluation, which focuses on modeling individual annotators. We adapt DisCo (Distribution from Context), a neural architecture that jointly models item-level and annotator-level label distributions, and present detailed analysis and improvements. In this paper, we extend DisCo by introducing annotator metadata embeddings, enhancing input representations, and multi-objective training losses to capture disagreement patterns better. Through extensive experiments, we demonstrate substantial improvements in both soft and perspectivist evaluation metrics across three datasets. We also conduct in-depth calibration and error analyses that reveal when and why disagreement-aware modeling improves. Our findings show that disagreement can be better captured by conditioning on annotator demographics and by optimizing directly for distributional metrics, yielding consistent improvements across datasets.

## introduction as machine learning systems increasingly mediate social, legal, and civic decision-making, their alignment with human values becomes paramount. however, as any participant in a democratic process knows well, human disagreement is always present. this includes many existing problems, such as hate speech detection, intent classification, or moral judgment. the lewidi 2025 shared task @xcite directly addresses this need by evaluating models on their ability to (1) predict soft label distributions derived from annotator disagreement and (2) approximate individual annotator behavior in a perspectivist setting. supervised learning typically resolves annotation disagreement by aggregating labels into a single ground truth, often via plurality vote. however, doing so can obscure valuable minority perspectives, especially on subjective or contentious content @xcite @xcite @xcite @xcite @xcite . however, preserving and modeling this disagreement can improve system robustness, fairness, and social accountability. tasks such as multipico @xcite , paraphrase, vari-errnli, and csc @xcite exemplify domains where capturing nuanced human perspectives, rather than just the majority opinion, is essential for ethical and practical deployment. lewidi-2025 challenges systems to go beyond single-label classification and instead model the full distribution of possible human responses. the core challenge lies in modeling disagreement when annotation is both sparse and noisy. annotators may vary in reliability, background, and interpretation, and most datasets provide only a few annotations per item. moreover, models must predict not only soft aggregate distributions but also simulate individual annotator responses, requiring them to generalize from partial supervision over complex, entangled signal sources. compounding this difficulty is the need for robust evaluation across both soft (e.g., manhattan, wasserstein) and perspectivist (e.g., error rate, normalized absolute distance) metrics, which test a model's fidelity to human-like prediction under both collective and individual frames. the four datasets utilized in the shared task are conversational sarcasm corpus (csc), multipico (mp), paraphrase (par), and varierr nli (ven). we adapt the disco @xcite ) model to the lewidi 3rd edition datasets. disco consumes item-annotator pairs as input and jointly predicts three interconnected distributions: the specific label an individual annotator would assign, the soft label distribution over all annotators for that item, and the annotator's own distribution over all items. while disco demonstrated the value of jointly modeling item-and annotator-level distributions, it treated annotators as one-hot ids and optimized losses misaligned with evaluation. we address both limitations by embedding annotator metadata and by designing loss functions directly tied to disagreement-aware metrics, enabling more interpretable and robust models. for the post-evaluation phase, we made the following contributions: with these updates, we observed a substantial improvement in the scores for three datasets: csc, mp, and par. additionally, this placed us as rank 4 instead of 7 for par and rank 5 instead of 9 for mp in the post-evaluation phase. ## background the lewidi shared task has emerged as a focal point for advancing methods that embrace, rather than suppress, annotator variation, since its inception @xcite . the third edition, lewidi-2025 @xcite , further extends these efforts by evaluating both distributional and perspectivist modeling across diverse datasets. lewidi-2025 focuses on four core benchmark datasets, each designed to probe different facets of human interpretative variation. please refer to appendix a.1 for further information on the datasets. the lewidi evaluation draws on two complementary research traditions. first, item-annotator modeling, the goal is to explicitly account for individual annotator behaviors when aggregating labels. @xcite 's foundational model represents each annotator's reliability via a latent confusion matrix, enabling joint estimation of true item labels and per-annotator error rates. subsequent work extended this framework with fully bayesian treatments @xcite and introduced clustering techniques to group annotators by shared labeling patterns @xcite . in the second paradigm, label distribution learning (ldl) reframes "ground truth" not as a single class but as a probability distribution over all possible labels. under this view, models are trained to match the full annotator-derived distribution rather than just the majority vote. early ldl work demonstrated strong performance in tasks like facial age estimation @xcite and has since been applied to diverse applications, from short text parsing @xcite to climate forecasting @xcite , showing that distributional targets can yield richer, more nuanced predictions. by learning shared embeddings for both items and annotators, disco effectively regularizes sparse annotation settings and pools context across related examples. in experiments on six publicly available datasets, disco matched or exceeded stateof-the-art ldl approaches, such as multinomial mixture models combined with cnns, and outperformed annotator-modeling baselines like crowd-layer across both single-label and distributional evaluation metrics. since semeval-2023, researchers have continued to push toward richer annotator-aware modeling. irel @xcite conditions toxicity predictions on anonymized user metadata-integrating each annotator's identity embedding directly into both the model input and the loss function to improve alignment with individual judgments. cicl_dms @xcite , by contrast, builds on large pre-trained language models and explores ensemble learning, multi-task fine-tuning, and gaussian process calibration to better match the full distribution of annotator labels. together, these contributions underscore a growing emphasis on leveraging demographic, behavioral, and contextual signals to capture the nuances of human disagreement. ## system overview our system builds upon the disco (distribution from context) architecture originally proposed by @xcite . to adapt it for the lewidi-2025 task, we introduced several targeted enhancements, including the use of task-specific sentence encoders, integration of annotator metadata via pretrained embeddings, and modified loss functions to reflect task evaluation metrics. these adaptations enable the model to generalize more effectively from sparse supervision and better capture the complexity of annotator behavior and disagreement. disco is designed to jointly model individual annotator responses, aggregate item-level label distributions, and annotator-level behavior distributions in a unified probabilistic framework. each data item x @xmath0, and its associated annotations from n annotators are collected in the matrix @xmath1. we denote the vector of responses for item m as y •,m and the histogram of these responses as #y •,m . similarly, each annotator n's behavior across all items is summarized by y n,• and its histogram #y n,• . this setup is illustrated in figure 1 . in the encoder (figure 2 ), item and annotator inputs are mapped into separate subspaces. the item vector x m is projected via a learnable matrix w @xmath2, while the one-hot annotator identifier a n is projected through w @xmath3. these embeddings are concatenated and passed through a two-layer mlp with softsign activations and a residual connection: the decoder takes the joint code z e and outputs three softmax-normalized vectors: z @xmath4,m | x m , a n , z @xmath5, and z @xmath6. training minimizes a composite loss that combines the negative log-likelihood of observed annotator responses with kl divergence terms that align predicted and empirical label distributions at both the item and annotator levels. at inference time, for an unseen item x m without a specific annotator id, we embed x m to obtain z i and tile it across all annotator embeddings in w a to form n joint codes. each code is decoded to yield per-annotator distributions, which are then aggregated by expectation or majority vote to pro-metadata "ann1":{ "gender":"male", "age":"26", "nationality":"chinese", "education":"master student" }, ## results we report the official results of our submitted system (under the name "lpi-rit") on the final leaderboard of the lewidi 2025 shared task. compared to the two official baselines, our system outperformed the random baseline across all submitted tasks except for paraphrase, but performed worse than the most frequent label baseline. for soft labels, our results were 1.45 (csc), 0.54 (mp), and 3.71 (par) while in the perspectivist task, they were 0.33 (csc), 0.32 (mp), and 0.44 (par). despite not achieving top rankings, our system provided a consistent output across tasks and served as a solid implementation of the disco modeling framework. these results highlight several areas for improvement-particularly in soft-label prediction on csc and in modeling individual annotator behavior under the perspectivist setup-while affirming the feasibility of generalizing disco to the lewidi setting without extensive task-specific modifications. in the post-evaluation phase, we introduced several improvements to the disco model, including the use of annotator metadata, expanded preprocessing support, stronger sentence encoders, and loss functions better aligned with soft-label and perspectivist objectives. these changes led to consistent gains across all datasets. ## discussion having established that disco_new consistently outperforms both og and baselines, we now analyze how and why these improvements occur. subsequent comparisons and analyses, the original and updated models are referred to as disco_og and disco_new, respectively. across all datasets and both tasks, the postevaluation model (disco_new) consistently outperforms both our original submission (disco_og) and the strongest lewidi baselines. on csc and par, disco_new reduces error substantially in both soft-label and perspectivist metrics, while on mp the gains are smaller but still clear. these results demonstrate that the proposed extensions-metadata embeddings and task-aligned loss functions-yield tangible improvements over the baseline disco architecture and most frequent baselines.