“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation (2020)

Abstract: Despite recent efforts reviewing current human evaluation practices for natural language generation (NLG) research, the lack of reported question wording and potential for framing effects or cognitive biases influencing results has been widely overlooked. In this opinion paper, we detail three possible framing effects and cognitive biases that could be imposed on human evaluation in NLG. Based on this, we make a call for increased transparency for human evaluation in NLG and propose the concept of human evaluation statements. We make several recommendations for design details to report that could potentially influence results, such as question wording, and suggest that reporting pertinent design details can help increase comparability across studies as well as reproducibility of results.

## introduction human evaluation is widely considered the gold standard for evaluating natural language generation (nlg), in part because existing automatic metrics display low correlations with human judgments @xcite @xcite . as a result, human evaluation is frequently used to demonstrate state-of-the-art results for generative tasks. however, this has the potential to be problematic due to the lack of consistency in how human evaluation is carried out @xcite . beyond producing variability in results, this has implications for validity of human evaluation results due to the influence of evaluation design choices. to address this, a number of papers have proposed recommended best practices for different aspects of nlg human evaluation @xcite . however, overlooked have been the issues of transparency and the potential for question framing effects and other cognitive biases influencing results. cognitive biases refer to heuristics that arise in judgment or decision-making @xcite . framing effects @xcite are types of cognitive biases that refer to how something is asked as opposed to what is asked. in the context of natural language generation research, these effects refer to the wording of questions asked and accompanying task descriptions and instructions, as opposed to what the target quality is that is being assessed. in this opinion paper, we demonstrate the lack of transparency in nlg human evaluation through empirically demonstrating the extent to which question wording is not included in evaluation design details, finding that only 15.68% of human evaluation studies in papers we surveyed explicitly reported the actual questions asked. we discuss three types of framing and cognitive biases that could influence results in nlg human evaluation: positive and negative framing, demand characteristics and response bias, and anchoring and adjusting. using concrete examples from studies in humancomputer-interaction and psychology and hypothetical examples for nlg, we demonstrate the importance of including question wording when using human evaluation in nlg. finally, we propose the concept of "human evaluation statements" and suggest a set of design parameters that should be included pertaining to human evaluation study design. ## transparency in human evaluation there is currently no standardized approach or consensus for how human evaluation for nlg should be carried out @xcite . as a result, it is currently very difficult to compare results across different studies due to the variability in evaluation design. past efforts to address this have included overviews of evaluation design practices used during a particular time span @xcite @xcite with corresponding recommendations for best practices (van der @xcite and empirical studies or overviews investigating the effects of different question types and scales @xcite . consistently, these studies have approached variability as a factor impacting the reliability of results. however, yet to be addressed is the lack of transparency in how studies are designed and reported, which has implications for comparability across studies, as well as replicability and validity of results. while transparency has yet to be addressed in human evaluation, transparency of data, models, and automatic evaluations is a growing topic of concern in the machine learning and natural language processing communities. @xcite proposed the usage of "data statements" for mitigating bias and increasing transparency in natural language processing @xcite proposed "datasheets for datasets" for increased data transparency and accountability. transparency in model reporting has also been advocated for. @xcite proposed the usage of "model cards" containing model performance characteristics for transparent model reporting. pertaining to model evaluation, there have been numerous criticisms of task leaderboards (linzen, 2020; rogers, 2019) which has led to calls for transparency through reporting of a more informative suite of metrics @xcite . driving the call for transparency has been the increased attention to issues of reproducibility. crane (2018) identified a number of controllable environmental settings that are widely unreported in question answering research and demonstrated the impact they have on reproducibility of results, including whether or not a model would be considered state-of-the-art. when we consider the impact of environmental variables (crane, 2018), computational budget including number of hyperparameter search trials @xcite , and other factors that can impact results, we can draw comparisons to human evaluation design details that could similarly impact results. we suggest that the design details of human evaluations can be thought of analogously to model hyperparameters, in that careful tuning can directly influence results. it is currently an open question as to what parameters in human evaluation could influence results, but without reporting pertinent details, we cannot begin to make comparisons across studies, or reproduce results. for example, van der @xcite suggested their findings pertaining to sample sizes and demographics in a survey of 89 papers using human evaluation for nlg may not reflect reality, since only 55% of the papers reported the number of participants and 18% reported demographics. an additional design parameter that we believe is largely unreported but could have an immense impact on results is that of the actual wording of questions presented to participants. more specifically, if questions are framed in ways that elicit various cognitive biases such as framing effects, response biases, or anchoring and adjustment effects, results may reflect question design rather than model performance. ## framing effects and cognitive biases framing @xcite refers to how something is asked as opposed to what is asked. in human evaluation for nlg, this would be reflected in the question wording or instructions provided to participants. in this section, we detail three possible framing effects and cognitive biases that could influence the results of human evalation: positive and negative framing, demand characteristics, and anchoring and adjusting. as question wording is extensively not reported in human evaluation in nlg, rather than providing empirical examples we provide hypothetical examples of the forms these effects could take when question wording is not reported. ## human evaluation design statements throughout the previous sections, we have provided examples demonstrating the potential question framing that could elicit human evaluation results for nlg that are biased in favor of a particular model. while these examples may at first glance seem implausible and only possible in cases of conscious (explicit) researcher bias in favor of a particular model, it is important to take into consideration the potential for researchers to possess unconscious (implicit) bias whether due to underlying expectations for a model's performance or due to influences of publication bias. during the peer review process reviewers may default to heuristics to simplify the task of review, including rejecting papers where models do not achieve sota results @xcite . this can implicitly motivate and incentivize researchers to show their model performs best on the gold standard of evaluation for nlg: human evaluation. we use this example to demonstrate the potential for the current lack of evaluation design details, in particular question wording, to leave the door open for results that have been subject to framing effects and bias which threatens the validity of the results. we draw attention to these effects in an effort to both increase researcher awareness to their own evaluation study design, decrease the potential for questions framed in ways in which results reflect question framing rather than actual model performance, and increase the amount of transparency in human evaluation to aid in study replicability and comparability. we also suggest that the results for studies which do not include exact question wording should be viewed through a skeptical lens as though they could contain researcher imparted bias that could significantly impact results. further, we use our demonstration of the potential for framing effects and biases in question wording as support for a call for transparency in human evaluation for nlg through the inclusion of study design details, which can aid in the development of more robust human evaluation guidelines. when guidelines exist that can reduce the complexity and time required to design human evaluation studies, they are used. for the evaluation of paraphrase generation, @xcite included the human evaluation guidelines they used as an appendix, which have since been adopted by other studies @xcite . this example shows that guidelines for human evaluation have value: guidelines make life easier and people often adopt those that are available. as such, we make the case for increased transparency in human evaluation with respect to design details that could potentially influence results. in an effort to take preliminary steps towards human evaluation guidelines, we propose the concept of "human evaluation design statements" akin to data statements @xcite or model cards @xcite . determining what should be included on such statements will require additional input, perspectives, and empirical evidence. as a preliminary effort, we provide a list of design parameters that we believe could influence results and should therefore be included when describing human evaluation design setup: question design: types, scales, wording basic inclusions pertaining to question design are question type and corresponding scales due to the variability that can arise based on these design decisions @xcite . further, as we demonstrated in this paper, question wording also has the potential to influence results. because of the potential for empirical differences due to how questions are framed, it is imperative to report question wording as part of design details, especially in studies where researchers use human evaluation to claim state-of-the-art performance. question presentation: ordering, questions per annotator ordering effects are influences on results that occur based on the order in which a sequence of questions is presented @xcite . as such, reporting question presentation order or balancing increases transparency as well as study comparability and reproducibility. in addition to ordering effects, response fatigue can occur when the quality and integrity of evaluations degrades as participants tire of a task @xcite . due to the possibility of response fatigue effects, statistics regarding the number of questions per annotator should be reported to increase design transparency in terms of potential influences on variability in results. target criteria: definitions it makes intuitive sense that what is actually being measured in human evaluation would influence results, and further that measuring the same or different target criteria in different studies would impact the comparability of the results. however, naming conventions and definitions are inconsistent and may exhibit significant overlap, such as with naturalness, grammaticality, and fluency @xcite . as such, what is being measured should be compared across studies based on definition and the resulting participant understanding of the task, rather than simply based on naming convention: studies may measure the same aspect under different names or different aspects under the same name. studies consistently reporting this detail in human evaluation is also a preliminary step towards agreed upon task definitions. annotators: demographics, background, recruitment, compensation understanding and reporting the details of the human factor in human evaluation is intuitively one of the most important sets of details to include in terms of transparency and potential influence on results. inclusions involve who annotators are in terms of demographics and background, how they were recruited, and whether or not annotators received fair compensation @xcite . as an example impact, annotator familiarity with the target language for a task might largely influence judgments towards biases, fluency, or grammatical correctness. the human factor in human evaluation, our annotators, is central to and interacts with every other detail of study design, and is therefore vital to report. while this list is not comprehensive, we believe these design details could have influences on evaluation results, and as such, are important details to consider and include. ## other considerations one of the factors that could limit the potential for widespread adoption of human evaluation statements that include human evaluation design details is the page limits imposed for many journal and conference papers. one approach to combat this is to include the details of human evaluation in supplementary material that accompanies papers. however, we suggest that many details in human evaluation design are central to understanding the meaningfulness of results, and further suggest that there will need to be community agreed upon guidelines for what details must be included within main papers. we further suggest that a complementary strategy would be the eventual development of comprehensive, agreed upon human evaluation guidelines that could operate similarly to "long-form" data statements @xcite . in this scenario, guidelines could be referenced, summarized briefly, and appended with pertinent additional study details as was proposed with "short-form" data statements @xcite . ## conclusion in this paper, we demonstrate the extent to which including the details of human evaluation is limited in natural language generation. we further demonstrate the need for including design details such as question wording using existing work in psychology and human-computer interaction on framing and cognitive biases, and cite the recent push for transparency with datasets and model details, such as details of hyperparameter tuning, as support for similar efforts to increase transparency in human evaluation. based on these observations, we propose working towards human evaluation statements and make several suggested inclusions, while noting the future need for additional perspectives and direct empirical support.