Collective Human Opinions in Semantic Textual Similarity (2023)

Abstract: Despite the subjective nature of semantic textual similarity (STS) and pervasive disagreements in STS annotation, existing benchmarks have used averaged human ratings as gold standard. Averaging masks the true distribution of human opinions on examples of low agreement, and prevents models from capturing the semantic vagueness that the individual ratings represent. In this work, we introduce USTS, the first Uncertainty-aware STS dataset with ∼15,000 Chinese sentence pairs and 150,000 labels, to study collective human opinions in STS. Analysis reveals that neither a scalar nor a single Gaussian fits a set of observed judgments adequately. We further show that current STS models cannot capture the variance caused by human disagreement on individual instances, but rather reflect the predictive confidence over the aggregate dataset.

## introduction semantic textual similarity (sts) is a fundamental natural language understanding (nlu) task, involving the prediction of the degree of semantic equivalence between two pieces of text (s1,s2). sts has been approached in various ways, ranging from early efforts using string-or knowledge-based measures and count-based co-occurrence models @xcite @xcite , to modern neural networks. broadly speaking, the goal of the sts task is to train models to make a similarity assessment that matches what a human would make. gold-standard scores are typically assigned by asking multiple raters to label a pair of sentences and then taking the average @xcite @xcite @xcite @xcite . the underlying assumption here is that there is a single ''true'' similarity score between s1 and s2, and that this label can be approximated by averaging multiple-possibly noisy-human ratings. while this assumption might be reasonable in settings such as educational testing with well-defined knowledge or norms @xcite , it is not the case for more subjective nlu tasks. @xcite show that in natural language inference (nli), disagreements often persist even if more ratings are collected or when the amount of context provided to raters is increased. high disagreement has been observed in a number of existing nli datasets @xcite . in sts, concerns about inconsistent judgments have been raised, particularly for difficult boundary cases in complex domains, where even expert annotators can disagree about the ''true'' label @xcite . identifying and discarding ''noisy'' labels during training can reduce generalization error @xcite . we reexamine whether the disagreement observed among raters should be attributed to ''noise'' and resolved via dismissing, or should rather treated as an inherent quality of the sts labels. specifically, our primary contributions are: we study the human assessments and investigate how best to integrate them into a gold label across varying degrees of observed human disagreement. in terms of annotation, for general text (e.g., news, glosses, or image descriptions), it has mostly been performed using crowdsourcing via platforms such as amazon mechanical turk with five crowd workers @xcite . for knowledgerich domains such as clinical and biomedical text, on the other hand, a smaller number of expert annotators has been used, such as two clinical experts for medsts @xcite . raters are asked to assess similarity independently on the basis of semantic equivalence using a continuous value in range @xcite . then a gold label is computed by averaging these human ratings. is averaging appropriate? averaging has been the standard approach to generating gold labels since @xcite . however, this approach relies on the assumption that there is a well-defined gold-standard interpretation + score, and that any variance in independent ratings is arbitrary rather than due to systematic differences in interpretation. an example of this no. 1 low human disagreement s1 kenya supreme court upholds election result. s2 kenya sc upholds election result. old label 5.0 new label n (μ = 4.9, σ = 0.1) ## aggregation of human judgments for the high-agreement cases of usts-u, gold labels can be approximated by aggregating multiple annotations into either a scalar or a single gaussian distribution. however, for low-agreement examples, how to aggregate the human ratings remains an open question. are all distributions unimodal gaussian? though most distributions of human assessments can be assumed to be sampled from an underlying (generative) distribution defined by a single gaussian, we observed judgments that a unimodal gaussian struggles to fit. the annotations of examples no. 2 and 3 in figure 4 exhibit clear bior tri-modal distributions. how often, then, and to what extent do multimodal distributions fit better? we answer this question by fitting human judgments using a gaussian mixture model (gmm), where the number of components is selected during training. this means the model can still choose to fit the distribution with only one gaussian component where appropriate. if additional components yield a better fit to the judgments, i.e., larger log likelihood is observed than using a unimodal distribution, we consider the human judgments to exhibit a multimodal distribution. ## analysis of model predictions most sts models predict a pointwise similarity score rather than of a distribution over values. @xcite estimated the uncertainty for continuous labels by mc-dropout and gaussian process regression (gpr). however, due to the lack of gold distributions, they only evaluate outputs using expected calibration error (ece) and negative log-probability density (nlpd), assessing the predictive reliability. it's unknown whether these uncertainty-aware models mimic human disagreements, i.e., the predicted deviation reflects the variance of human judgments. to explore this, we experiment over usts and incorporate distributional divergence (i.e., kullback-leibler divergence [kl]) into the evaluation, to observe the fit between the distribution of collective human judgments and the model predictive probability. we also examine the ability of different models to capture the averaged score for low-agreement cases, and whether a well-calibrated model fits the distribution of annotations better. ## multilingual usts before extending usts into a multilingual benchmark, we question the validity of previous approaches involving direct transfer of annotations collected for one language to other languages @xcite . this strategy assumes that the nuanced semantics of the component sentences is not changed under translation, and hence the label will be identical. to test whether this assumption is reasonable, we analyze the impact of language on the annotations, and discuss whether such ratings are transferable across languages. specifically, we establish whether the label distribution varies based on language, and how annotator proficiency affects the distribution given the same text. collecting labels taking english as a pivot language, we employ native english speakers (''nt'') and bilingual raters whose mother language is mandarin chinese, including 5 professional translators (''pt''), 5 overseas students (''os''), and 5 general users (''gu''). each annotator assigns labels to 100 examples sampled from each of usts-u and usts-c (the same data set used in section 3.4), which have been manually post-edited by professional translators to ensure content alignment. ## discussion we focus on the sts task in this work. however, the methods we propose can be transferred to other subjective textual regression tasks, such as sentiment analysis (sa) rating and machine translation quality estimation in the format of direct assessment (da). similar findings stemming from task subjectivity may be relevant to other types of nlp tasks relying on human annotation. high disagreement among annotators may occur due to ambiguous labeling, where it is challenging to compile guidelines that are widely accepted and consistently interpreted by all individual annotators. lan usts-u usts-c usts zh 0.42 0.45 0.44 en 0.40 0.43 0.41 in practice, it may be difficult to estimate the distribution of human annotations in instances where multiple annotators are difficult to source, such as occurs in clinical and biomedical sts due to the need for highly specialized knowledge. transfer learning, which relies on patterns learned from general-purpose usts, provides a means to predict such a distribution, if noisily. we propose to explore the direct estimation of individual ratings by in-context learning based on large language models (llms), e.g., @xcite and chatgpt. 9 llms are able to perform in-context learn-perform a new task via inference alone, by conditioning on a few labeled pairs as part of the input @xcite . chatgpt appears to be highly effective at style transfer and tailoring of content to specific audiences such as five-year old children or domain experts, through learning about language style and tone from interactional data and individual preferences. this allows it to generate more personalized responses @xcite . @xcite show that assigning chatgpt a persona 9 https://openai.com/blog/chatgpt . additionally, @xcite show that generative llms can be used to automatically generate labeled sts datasets using targeted instructions. this data can be utilized to improve the quality of sentence embeddings. together, these imply that llms may have utility in generating personalised semantic similarity assessments, based on annotator meta data (e.g., age, educational background, or domain expertise). simulating variation in judgments between individual annotators using synthetic personalized ratings could mitigate ineffective training and evaluation caused by aggregation, given that neither a scalar nor a single gaussian fits the set of observed judgments adequately for instances of high disagreement. ## conclusion we presented the first uncertainty-aware sts corpus, consisting of 15k chinese examples with more than 150k annotations. the dataset is intended to promote the development of sts systems from the perspective of capturing inherent disagreements in sts labeling, and establish less biased and more nuanced gold labels when large variances exist among individual ratings. we additionally examine the models' ability to capture the averaged opinion and the distribution of collective human judgments. results show that the uncertainty captured by current models is not explained by the semantic uncertainty that results in disagreements among humans. rather, it tends to reflect the predictive confidence over the whole data set. we also found that the text language and language proficiency of annotators affect labeling consistency.