Can Large Language Models Capture Dissenting Human Voices? (2023)

Abstract: Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely align with the human disagreement distribution has not been well-studied, especially within the scope of natural language inference (NLI). In this paper, we evaluate the performance and alignment of LLM distribution with humans using two different techniques to estimate the multinomial distribution: Monte Carlo Estimation (MCE) and Log Probability Estimation (LPE). As a result, we show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution. The inference and human alignment performances plunge even further on data samples with high human disagreement levels, raising concerns about their natural language understanding (NLU) ability and their representativeness to a larger human population.

## introduction natural language inference (nli) has long served as a fundamental testbed to evaluate the ability of a model to recognize entailment and capture plausible inference relations between pairs of sentences @xcite @xcite . when constructing datasets, conventional processes result in a single label per instance even if multiple annotators contribute, which limits the full representation of diverse opinions that might arise in a larger human population. thus, recent datasets have become more attentive to incorporating multiple interpretations @xcite @xcite to capture dissenting human opinions. meanwhile, instruction fine-tuning large language models (llms) has elicited remarkable generalizability to diverse unseen tasks @xcite . not only can they generate free-form texts, but they can also select one answer from multiple options given in the input prompt. however, while many works study user interaction and conversational usage @xcite , limited works evaluate these instruction-following llms on a foundational nli task. therefore, we aim to answer the following questions: can llms capture dissenting voices that naturally arise in the dataset? are llms representative of the voices of the annotators in inference tasks? with this in mind, we jointly assess on a number of instruction-following llms, flan-t5 @xcite , flan-ul2 @xcite , opt-iml-max @xcite , and gpt-3 @xcite , on their performance on human opinion distribution datasets -chaosnli @xcite and pk2019 @xcite . for the process of using the model output distribution as an estimate of human disagreement distribution, we offer novel estimation methods: monte carlo estimation (mce) and log probability estimation (lpe) (figure 1 ). we find that the state-of-the-art gpt-3 model does not outperform smaller models such as finetuned bert @xcite and partially finetuned flan-t5-xxl in solving inference problems. furthermore, it yields higher jensen-shannon distance (jsd) @xcite and distribution calibration error (dce) @xcite than bert for the chaosnli datasets. each model is optimized using different estimation methods and prompt types, where gpt/flan-t5-xxl attains the best performances in nli capability and human alignment when using lpe/mce. our paper's contributions are as follows: ## methods we estimate and quantify dissenting human voices using the multinomial soft-label distribution of llms with two proposed methods: ## discussion llms do not perform well in nli. despite minimal, unknown, or absence of exposure to the nli task, we anticipated that state-of-the-art llms such as gpt-3 and stable vicuna could reason with this relatively basic inference problem. the models are trained with billions of parameters and are known to be effective in helping real-world users solve diverse, complex tasks @xcite . however, the unforeseen poor performance of these models casts doubt as to whether they possess true general language understanding abilities. the problem is exacerbated for distilled models (e.g. stable vicuna) that are fine-tuned using proprietary llms, a performance discrepancy issue similarly raised by @xcite . since smaller llms fully or partially trained with nli tasks could perform much better than the mue and ne models, this hints at a task-specific latent factor in nli tasks where supervised training is beneficial and required for a wider definition of natural language understanding. in fact, as these llms can simply be fine-tuned to perform better for nli tasks, a stricter evaluation criterion is needed to assess the genuine understanding capability of llms. characterizing disagreement with respect to ambiguity and uncertainty previous studies relate multiple annotations not only to disagreement @xcite , but also to ambiguity @xcite @xcite , and mostly to uncertainty @xcite @xcite @xcite . the definitions of ambiguity, uncertainty, and disagreement have the potential to be conflated and disambiguated. in our paper, we use the multinomial soft label estimate of a model as a representation of "disagreement". when estimating this distribution with mce, our modeling assumption treats each query to the model is analogous to asking an individual annotator to provide a label. in contrast, lpe is analogous to asking an individual to assign the scores to each option. whereas most works exploit disagreement or uncertainty to improve various nlp task performances @xcite @xcite our study focuses on evaluating the models. we find that using both methods for estimating the multinomial label distribution by querying the language model are not calibrated well with the human annotations. other domain tasks are transferable to nli. our work can be expanded to test llms on other nlp applications @xcite such as question answering @xcite , fact verification @xcite , and toxic language detection @xcite . further, our method can be applied for tasks that contain disagreements since they are easily transferable to nli tasks @xcite like the qnli dataset from however, we should consider the target tasks when tracing "human disagreement" only when it is a significant signal that needs to be captured. for example, since it is important to include diverse opinions, we can easily apply our methods to detect disagreements in hate speech @xcite . in contrast, spotting disagreement in the arithmetic reasoning task @xcite might be less important since it often requires a logical step-by-step reasoning procedure to obtain an accurate answer. how can we better align llms to represent dissenting voices? we point out the current limitation of utilizing llms to represent a larger human population, especially when disagreements are present. the causes of this phenomenon are indiscernible due to the entanglement of miscalibration of out-of-distribution (ood) inference, additional noise due to disagreement and ambiguity, prompt sensitivity, and more aspects that are yet to be identified. even though simple remedies of temperature scaling @xcite , incorporating logit bias, constrained decoding @xcite , or direct supervision to multiple annotations @xcite might mitigate the misalignment, these methods are unrealistic and not scalable due to the exhaustive hyperparameter tuning and additional data collection required to represent the population of interest. however, as llm applications are becoming more ubiquitous, it is important for them to faith-fully represent a larger population, preferably including the voices of minorities. thus, we suggest that future llms could be improved to reflect human disagreements in diverse means, for example, by fine-tuning with ambiguous instances @xcite . as llms are shown to be aware of their ignorance @xcite and have the ability to express their level of confidence @xcite , we expect future works to address similar approaches in the aspect of alignment towards the human disagreement distribution. in this way, the reconstructed model distribution with mce and lpe may better capture different interpretations from human individuals, aiding accountability. ## conclusions in this paper, we compare the performance of instruction-following generative llms with other fully fine-tuned smaller models on the fundamental nli task. first, by experimenting on four different nli datasets, we show llms are not performing well in the nli task, considering their touted language comprehension capabilities. further, in agreement with the need for multiple annotations for disagreeable nlp tasks, llms also fail to align with human disagreements in the chaosnli and pk2019 datasets. additional development is needed to capture representative human distributions, as well as to discover key factors to disagreement sources that can influence the llm's answer distribution.