Opt- ICL at L e W i D i-2025: Maximizing In-Context Signal from Rater Examples via Meta-Learning (2025)

Abstract: Many natural language processing (NLP) tasks involve subjectivity, ambiguity, or legitimate disagreement between annotators. In this paper, we outline our system for modeling human variation. Our system leverages language models’ (LLMs) in-context learning abilities, along with a two-step meta-learning training procedure for 1) post-training on many datasets requiring in-context learning and 2) specializing the model via in-context meta-learning to the particular data distribution of interest. We also evaluate the performance of our system submission to the Learning With Disagreements (LeWiDi) competition, where it was the overall winner on both tasks. Additionally, we perform an ablation study to measure the importance of each system component. We find that including rater examples in-context is crucial for our system’s performance, dataset-specific fine-tuning is helpful on the larger datasets, post-training on other in-context datasets is helpful on one of the competition datasets, and that performance improves with model scale.

## introduction natural language processing (nlp) evaluations typically assume that there is a single correct answer (a.k.a., "ground truth") and view annotator disagreement as a source of noise to be eliminated, generally attributing rating variation to poor instructions, incomplete task specification, or noisy data. however, oftentimes annotator disagreement can be a useful signal of subjectivity, ambiguity, or multiple reasonable interpretations @xcite . properly integrating this disagreement can be important for robustness, uncertainty calibration, and representing multiple viewpoints. to address this, more and more have argued for focusing on methods for integrating human variation into evaluation and modeling @xcite , including annotations from people from diverse backgrounds @xcite , and aligning ai systems with pluralistic values @xcite . in order to inspire work towards these goals, the learning with disagreements (lewidi) competition @xcite consists of four datasets across two tasks for modeling disagreement: one task for predicting how a particular annotator's ratings ("perspectivist" task) and one for predicting the distribution of labels that a pool of annotators gave ("soft label" task). in this system paper, we outline our system submission. our system (opt-icl, for optimizing in-context learning) takes a fully perspectivist approach, trying to predict how an individual annotator rated each instance and then aggregating individual predictions into a distribution for the soft task. it primarily leverages llms' in-context learning ability @xcite , including an annotator's train ratings directly in-context at inference time. on top of a pre-trained autoregressive language model, we additionally perform two steps of training: post-training in order to enhance the models' in-context learning abilities and teach a unified prompt format (or, spectrum tuning, see @xcite , and dataset-specific finetuning. both training steps can be seen as forms of meta-learning @xcite , where the model is tasked with learning how best to fit to the in-context rater examples. our main contributions include: our proposed system for modeling disagreement ( §3), which was the overall winner on both competition tasks, and an ablation study outlining the effect of each system component ( §4). in particular, we find that: mp csc par ven average rank (error rate ↓) (abs. dist. ↓) (abs. dist. ↓) (error rate ↓) ours .289 (1) .156 (1) .119 (2) .270 (2) -best other team .300 (2) .172 (2) .080 (1) .124 (1) -most frequent label baseline .316 .239 .362 .345 -random label baseline .499 .352 .367 .497 -ours (rank) 1 1 2 (2-way tie) 2 (4-way tie) 1.5 (1) best other team (name) demeva (2) demeva (2) twinther (1) twinther (1) demeva 2 (2) soft task mp csc par ven average rank (manh. dist. ↓) (wass. dist. ↓) (wass. dist. ↓) (manh. dist. ↓) ours .422 (1) .746 (1) 1.20 (1) .449 (3) -best other team .428 (1) .792 (1) .983 (1) .233 (1) -most frequent label baseline .518 1.17 3.23 .595 -random label baseline .687 1.54 3.35 .676 -ours (rank) 1 (2-way tie) 1 (2-way tie) 1 (3-way tie) 3 (3-way tie) 1.5 (1) best other team (name) promotiongo (1) demeva (1) twinther (1) twinther (1) demeva 2.75 (2) the learning with disagreements competition (lewidi, @xcite aims to evaluate machine learning systems' ability to engage with and model human variation. the competition spans four datasets which contain subjective judgments where raters may disagree: in addition, some basic demographic information is provided about annotators for all datasets. for dataset statistics, see using these datasets, the competition constitutes two tasks: a "soft labeling" task, where the goal is to predict a probability distribution over possible labels that best match the human annotator label distribution and a "perspectivist" task, where the goal is to take on the perspective on an individual annotator and predict that particular annotator's label given prior demonstrations from that rater and (optionally) some demographic information. for additional information on the competition setup, please refer to @xcite . ## system overview our system consists of three components: specifically, our system uses the google/gemma-3-12b-pt (gemma team et al., 2025) language model. ## results we now outline how our system performed compared to others in the competition. then, we ablate mp csc par ven (error rate ↓) (abs. dist. ↓) (abs. dist. ↓) (error rate ↓) opt-icl (spect + sft + demographics + icl) .289 (1) .156 (1) .119 (1) .270 (1) prompt ablations no demographics .295 (2) .156 (1) .122 (1) .268 (1) no many-shot icl (one example) .305 (3) .185 (3) .216 (3) .321 (2) training ablations no sft .316 (4) .191 (3) .123 (1) .257 (1) no spect .303 (3) .157 (1) .120 (1) .247 (1) no sft, no spect (12b-pt) .336 (5) .192 (3) .129 (1) .243 (1) model size ablations (no train) 1b-pt (no sft, no spect) .341 (6) .219 (5) .308 (4) .429 (3) 4b-pt (no sft, no spect) .351 (7) .201 (4) .174 (2) .314 (2) 12b-pt (no sft, no spect) .336 (5) .192 (3) .129 (1) .243 (1) 27b-pt (no sft, no spect) .312 (4) .176 (2) .120 (1) .246 (1) soft task mp csc par ven (manh. dist. ↓) (wass. dist. ↓) (wass. dist. ↓) (manh. dist. ↓) ours (spect + sft + demographics + icl) .422 (1) .746 (1) 1.20 (1) .449 (1) prompt ablations no demographics .430 (2) .751 (1) 1.17 (1) .458 (1) no many-shot icl (one-example) .448 (3) .851 (2) 2.27 (3) .484 (1) training ablations no sft .486 (5) .963 (3) 1.15 (1) .446 (1) no spect .450 (3) .749 (1) 1.21 (1) .418 (1) no sft, no spect (12b-pt) .507 (6) .959 (3) 1.21 (1) .427 (1) model size ablations (no train) 1b-pt (no sft, no spect) .511 (7) 1.13 (5) 3.24 (4) .703 (3) 4b-pt (no sft, no spect) .526 (8) 1.03 (4) 1.75 (2) .519 (2) 12b-pt (no sft, no spect) .507 (6) .959 (3) 1.21 (1) .427 (1) 27b-pt (no sft, no spect) .462 (4) .875 (2) 1.11 (1) .413 (1) ## discussion and conclusion in summary, our system was able to perform strongly across the board and was the overall winner on both tasks. however, it did perform particularly well (1st) on mp and csc, which had many unique annotators and larger training datasets, and performed less well on par/ven (perspectivist: 2nd on par, 2nd on ven; soft: 1st on par, 3rd on ven), which had only four annotators each and much smaller training sets. our approach has many advantages, including: 1) a single model for each dataset, 2) potential adaptation at test time to new raters; 3) strong performance even in the limited data regime; 4) no dataset-specific assumptions; 5) same system for perspectivist and soft tasks. however, some limitations include expensive inference (see app a.3), foot_2 as prompt lengths are quite long in order to contain in-context rater examples and that the method is unable to effectively leverage additional rater demonstrations that do not fit in the context window. in our ablation study, we found that incontext demonstrations are crucial for performance, dataset-specific tuning helps given enough data, spectrum tuning helped on mp, and performance improves with model size (but scale alone does not make up for dataset-specific training). some interesting directions for future work include: 1) how performance scales with the number of in-context rater examples (including going beyond 3,000-token prompts), 2) whether selecting particular in-context examples at inference can outperform random selection, 3) the effect of including rater explanations on performance, 4) how sang michael xie, aditi raghunathan, percy liang, and tengyu ma. 2022. an explanation of in-context learning as implicit bayesian inference. preprint, arxiv:2111.02080.