Aligning NLP Models with Target Population Perspectives using PAIR : Population-Aligned Instance Replication (2025)

Abstract: Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR: Population-Aligned Instance Replication, a post-processing method that adjusts training data to better reflect target population characteristics without collecting additional annotations. Using simulation studies on offensive language and hate speech detection with varying annotator compositions, we show that non-representative pools degrade model calibration while leaving accuracy largely unchanged. PAIR corrects these calibration problems by replicating annotations from underrepresented annotator groups to match population proportions. We conclude with recommendations for improving the representativity of training data and model performance.

## introduction and inspiration when a hate speech detection model flags harmless expressions as toxic, or a content moderation system fails to identify genuinely harmful content, the root cause often lies not in the model architecture, but in who annotated the training data. while natural language processing (nlp) models aim to serve broad populations, the human judgments used to train these systems often come from crowdworkers and convenience samples. and the demographics, cultural contexts, and worldviews of these annotators often differ from those of the communities the models ultimately impact @xcite . these non-representative annotator pools can have real consequences, because annotator characteristics like age, education level, and cultural background impact how content is annotated (sap et al., 2022; fleisig et al., 2023; kirk 1 the code for experiments is available at https:// github.com/soda-lmu/pair . collecting surveys weighting towards target population representative opinions analysis on adjusted data collecting annotations weighting towards target population model training on adjusted data: better performance? ## related work several strands of related work inform our approach to identifying and mitigating bias due to the use of non-representative annotators: annotator impact on data and models. annotator characteristics and attitudes significantly influence annotation quality, particularly for subjective tasks like toxicity detection @xcite @xcite . for example, annotators' political views and racial attitudes affect their toxicity judgments @xcite . models trained on non-representative annotator pools inherit these biases and generalize poorly @xcite @xcite @xcite @xcite . annotator demographics. several researchers advocate collecting annotator demographics to assess representation and identify biases @xcite @xcite @xcite . 2 however, collecting and releasing these data can raise privacy concerns @xcite . recent works have also used demographics to prompt the large language models (argyle et al., 2023), and some find that these are less effective in subjective contexts @xcite . ## pair algorithm: adjustment via pseudo-population to adjust an annotator pool to better reflect a target population, we propose the pair algorithm, which constructs a pseudo-population through poststratification, weight normalization, and deterministic replication. this adjustment strategy is inspired by established methods in survey sampling @xcite . post-stratification aligns a sample more closely with population-level distributions @xcite . annotators are grouped into strata based on demographic or behavioral characteristics. for each unit i in stratum s, a post-stratification weight is computed as: these weights have only relative meaning and are invariant to multiplication by a constant (k): normalization useful if research teams have a target number of annotations per observation in mind, for either computational or design reasons, or if some weights given by eq. 1 are very small and round to one. to generate a pseudo-population, we apply deterministic replication: each unit is replicated n i times where while we focus in this initial study on deterministic replication, alternative implementations are possible, including resampling-based replication or direct incorporation of weights into model training. ## annotation simulation and model training to address our research questions, we conduct a simulation study on offensive language and hate speech detection. we imagine a population made up of equal shares of two types of people: those more likely to perceive offensive language and hate speech and those less likely. we create three datasets of simulated annotations which differ in the mix of the annotator types. we then create a fourth dataset, using the pair algorithm, to fix the imbalance in the annotators. we fine-tune roberta models on the four datasets and evaluate the effect of annotator composition on model performance (rq1) and the ability of the pair algorithm to improve performance (rq2). ## results we show results separately for the ol and hs models. ## limitations our study explores bias-aware data simulation and evaluation in a controlled setting, which necessarily involves simplifying assumptions and methodological constraints. we outline key areas where future work could broaden the applicability and robustness of our findings. stylized biases and simulated data. our simulation makes strong assumptions about annotator behavior: there are only two types of annotators, and, within each type, annotators behave similarly. real-world annotator biases may be more nuanced or context-dependent. the simulated annotators might not be representative of a stable opinion group @xcite . future work could incorporate more realistic biases and refine the proposed simulations and statistical techniques. ## ethical considerations in this simulation study, we experiment on a publicly available dataset collected in our previous study @xcite , which contains offensive and hateful tweets. we do not support the views expressed in these tweets. the simulation study itself does not collect any new data or raise any ethical considerations.