Modelling Variability in Human Annotator Simulation (2024)

Abstract: Human annotator simulation (HAS) serves as a cost-effective substitute for human evaluation tasks such as data annotation and system assessment. It is important to incorporate the variability present in human evaluation into HAS, since it helps capture diverse subjective interpretations and mitigate potential biases and over-representation. This work introduces a novel framework for modelling variability in HAS. Conditional softmax flow (S-CNF) is proposed to model the distribution of subjective human annotations, which leverages diverse human annotations via meta-learning. This enables efficient generation of annotations that exhibit human variability for unlabelled input. In addition, a wide range of evaluation metrics are adopted to assess the capability and efficiency of HAS systems in predicting the aggregated behaviours of human annotators, matching the distribution of human annotations, and simulating the inter-annotator disagreements. Results demonstrate that the proposed method achieves state-of-the-art performance on two real-world human evaluation tasks: emotion recognition and toxic speech detection.

## introduction human evaluation is fundamental to machine learning research. it guides processes such as data annotation and model assessment, including for instance perceptual quality evaluation @xcite @xcite , annotation generation for weak supervision @xcite , and model optimization based on human preference @xcite @xcite @xcite . collecting human annotations or evaluations often requires substantial resources and may expose human annotators to distressing and harmful content in sensitive tasks (e.g., toxic speech detection, suicidal risk prediction, and depression detection). this inspires the exploration of human annotator simulation (has) as a scalable and cost-effective alternative, which facilitates large-scale dataset evaluation, benchmarking, and system comparisons. variability is a unique aspect of real-world human evaluation. individual variations in cognitive biases, cultural backgrounds, and personal experiences @xcite @xcite can lead to variability in human interpretation @xcite . it has been argued that achieving a single deterministic "ground truth" in subjective tasks like human evaluation is not feasible, nor essential @xcite . therefore, has should incorporate the variability present in human evaluation rather than solely relying on majority opinions. this mitigates potential biases and over-representation in scenarios where dominant opinions could potentially overshadow minority viewpoints, thus promoting fairness and inclusivity. this work investigates modelling subjective human annotation distributions and simulating human-like annotations. we propose a novel framework which formulates has as a zero-shot density estimation problem. a new model, conditional softmax flows (s-cnfs), is proposed which leverages diverse human annotations via meta-learning. this enables efficient generation of annotations that exhibit human variability for unlabelled input. to the best of our knowledge, this is the first work that incorporates human variability into has without requiring human annotators to be dynamically involved in the process, while remaining scalable for large crowd-sourced datasets. moreover, a range of evaluation metrics are adopted to assess the capability and efficiency of has systems regarding prediction of the majority opinion, estimation of the human annotation distribution, and simulation of the inter-annotator disagreements. the proposed approach is evaluated on two real-world applications: emotion recognition and toxic speech detection. empirical results demonstrate that the proposed method achieves state-of-the-art performance on modelling variability in has. ## a meta-learning framework for has this paper proposes a novel framework for has that meta-learns a conditional softmax flow (s-cnf) to estimate the human annotation distribution p(η|x) across all training events in d. the proposed model learns to learn (i.e., meta-learns) how to estimate the underlying distribution of human annotations d i for any given descriptor x i by leveraging the diverse human annotations, rather than designing a proxy variable to summarize d i as in the first approach described in sec. 2.3. unlike the second approach in sec. 2.3 which separately models each individual human annotator with a different model, our method is compatible with large crowd-sourced datasets since it amortizes across annotators with a single s-cnf model. moreover, our model is a zero-shot human annotation simulator which can estimate the human annotation distribution p(η * |x * ) for any unseen test descriptor x * without access to any human annotations d * for x * , in contrast to the third method in sec. 2.3 which requires human annotators to be dynamically involved in the process of labelling x * . ## experimental setup the proposed framework for variability-aware has is evaluated on two real-world applications for speech and natural language processing: emotion class labelling and toxic speech detection. a wide range of evaluation metrics are adopted to assess the performance of has systems. ## results and analysis the proposed method is evaluated according to the setup in sec. 4. the evaluation results along with case study of representative examples demonstrates the superior capability of the proposed method in capturing the aggregated behaviours of human annotators, matching the distribution of human annotations, and simulating the variability of human interpretation. for each metric, mean value with standard error over three independent runs are reported for all methods. ## conclusions this paper studied human annotator simulation (has), a cost-effective alternative to generating human-like annotations for automatic data labelling and model evaluation. a novel framework is proposed to incorporate the variability of human evaluations into has. this framework leverages diverse annotations to estimate the distribution of human annotations by meta-learning a conditional softmax flow (s-cnf) on large crowd-sourced datasets. this overcomes the drawbacks of prior work and enables efficient generation of annotations that exhibit human variability for unlabelled test inputs. the proposed method clearly and consistently outperformed a wide range of methods on emotion class labelling and toxic speech detection, achieving the best performance for human annotation distribution matching and inter-annotator disagreement simulation. it is hoped that the proposed method could help mitigate unfair biases and overrepresentation in has and reduce the exposure of human annotators to potentially harmful content, thus promoting ethical ai practices.