On Degrees of Freedom in Defining and Testing Natural Language Understanding (2023)

Abstract: Natural language understanding (NLU) studies often exaggerate or underestimate the capabilities of systems, thereby limiting the reproducibility of their findings. These erroneous evaluations can be attributed to the difficulty of defining and testing NLU adequately. In this position paper, we reconsider this challenge by identifying two types of researcher degrees of freedom. We revisit Turing’s original interpretation of the Turing test and reveal that an effective test of NLU does not provide an operational definition; it merely provides inductive evidence that the test subject understands the language sufficiently well to meet stakeholder objectives. In other words, stakeholders are free to arbitrarily define NLU through their objectives. To use the test results as inductive evidence, stakeholders must carefully assess if the interpretation of test scores is valid or not. However, designing and using NLU tests involve other degrees of freedom, such as specifying target skills and defining evaluation metrics. As a result, achieving consensus among stakeholders becomes difficult. To resolve this issue, we propose a validity argument, which is a framework comprising a series of validation criteria across test components. By demonstrating that current practices in NLU studies can be associated with those criteria and organizing them into a comprehensive checklist, we prove that the validity argument can serve as a coherent guideline for designing credible test sets and facilitating scientific communication.

## introduction large-scale pretrained language models, also known as foundation models @xcite , have advanced significantly, leading to systems that are performing increasingly well at various natural language understanding (nlu) tasks and offering real-world applications @xcite @xcite . however, these systems are often erroneously claimed to have human-level understanding @xcite . in other cases, their failures in specific situations are presented as systemic inadequacy, while ignoring their excellent performances in certain tasks @xcite . such exaggerations may stem from unjustified assumptions about the capability of nlu systems. accordingly, researchers have attempted to improve the benchmarking of nlu through discussions to define language understanding @xcite @xcite and practices, such as using auxiliary tasks for sanity check @xcite , decision boundary evaluation @xcite , and dataset sourcing @xcite . such efforts have motivated researchers to reconsider and revise the concept and scientific study of nlu @xcite . nevertheless, we need a more comprehensive guideline for better scientific communication. in this study, we rethink this challenge in terms of researcher degrees of freedom, aiming to reframe the definition and evaluation of nlu and provide a pathway to better benchmarking. we begin by revisiting a recent discussion by @xcite , who define language understanding as the link between linguistic form (i.e., symbolic information) and the communicative intent of the speaker. they support their argument by presenting a thought experiment called the octopus test, which is designed to resemble the turing test. in the octopus test, an intelligent deep-sea octopus that can only use symbolic information (an analogy of language models trained only on textual corpora) tries to mimic a person in conversations. the objective of the test is to ascertain whether the octopus can deceive the other person even when that person is defending themselves against an angry bear. @xcite argue that the octopus cannot achieve language understanding because it cannot pass the test in that situation (as a proof of existence). however, we find that the octopus test may not be a valid thought experiment owing to initial concerns about the turing test itself (e.g., even humans may sometimes fail the test). although we agree that symbol grounding plays an important role, several other features contribute to language understanding, which takes various forms @xcite . here, we see that the degrees of freedom in defining nlu are characterized by the responsedependent interpretation of the turing test (proudfoot, 2020). thus, interpreting the subject's behavior depends on how it appears to a counterpart interrogator in the test, i.e., we cannot use success in the test as a definite criterion of language understanding, which makes it challenging to construct an nlu test with an operational definition. instead, using a test merely enables us to obtain inductive evidence to achieve consensus among stakeholders. we elaborate on this observation by revisiting the turing and octopus tests in sections 2 and 3, respectively. then, we restate the testing of nlu in section 4 considering that its definition is inevitably arbitrary to observers. in short, a test has to assess distinguishable behaviors in a specific domain rather than general nlu to make it easier for stakeholders to arrive at a consensus about the interpretation of the test results. although we can decide what capability we assess in a test, other degrees of freedom are present while designing the test and interpreting its results. this problem can be understood in terms of psychological studies, in which choices arbitrarily taken by researchers heighten the chances of false positive results @xcite . several previously proposed nlu practices facilitate detailed evaluation to lessen unjustified claims about nlu systems. however, they are not sufficiently well-organized to be deployed on a universal framework. therefore, we often choose convenient practices depending on specific situations. consequently, the degrees of freedom that researchers have in evaluating nlu increase, thereby sacrificing the reproducibility of research @xcite . in section 5, we introduce the validity argument, which is a framework used in psychological and educational tests @xcite @xcite , to set out a guideline for designing, conducting, and using nlu tests. we demonstrate that current practices are associated with inference steps in the validity argument and show that it may serve as a comprehensive, coherent guideline for developing or identifying actionable and beneficial practices to construct a better nlu test and use it properly. our major contributions are two-fold: ## reframing the response-dependent interpretation of nlu tests in this section, we investigate how nlu should be practically tested under the response-dependent interpretation of the turing test. considering the three issues outlined in section 3.2, our goal is to formulate a necessary and sufficient condition of language understanding that does not contradict the actions performed in an nlu study. first, to achieve our objective, we summarize the corresponding requirements of the formulation: based on these requirements, we reformulate and elucidate the response-dependent interpretation of language understanding: formulation 4.1. x understands language under the condition c if and only if the subjective probability of an average observer for the hypothesis, x understands language under c, is higher than a threshold, where the hypothesis is supported by evidence obtained by the performance of x on a test under c. according to the three requirements, we elaborate on this formulation as follows. ## validity argument for testing nlu in formulation 4.1, the stakeholders employ certain measures to evaluate the behavior of a test subject and interpret those measures to arrive at a decision. to support the interpretation of the measures (i.e., improve the stakeholders' confidence in the subject's target linguistic behavior), the test has to be well-designed to provide sufficient evidence from various perspectives. despite significant progress, nlu benchmarks often lack the ability to provide substantial evidence @xcite @xcite . to date, useful practices have been proposed to better connect observed scores and intended nlu behavior (e.g., @xcite . however, without guidelines on design and using a test, these practices cannot be organized coherently. as a result, researchers can arbitrarily select practices most suited to their purpose while, intentionally or not, ignoring others. this freedom of choice allows interpretations of test results that are not justified by reliable evidence. such unjustified interpretations are reminiscent of a problem in psychology known as researcher degrees of freedom @xcite . we need to tackle this challenge by developing a unified, comprehensive framework to overview necessary practices that validate the interpretation of test results. in psychological measurements, @xcite proposes the validity argument, which is a theoretical framework that guides the collection of evidence to validate the interpretation and use of test scores (figure 1 ). it decomposes the design, conduct, and use of a test into the seven components where transitions between them are performed as inferences supported by a warrant and its backing that follow toulmin's formulation of arguments @xcite . the six inferences constitute the process of collecting and interpreting evidence that the test designers and users follow. the validity argument has been employed in various fields, including linguistic tests @xcite and clinical exams @xcite , but not in nlu. see appendix b for its background and terminology. in the following section, we apply the validity argument to the evaluation of nlu. we associate current important practices with the inferences of the validity argument, show that the argument serves as a useful guideline to design and use nlu tests, and provide relevant checklist questions here, we take squad @xcite , a question answering (qa) dataset consisting of crowdsourced questions written for wikipedia articles to instantiate a validity argument and identify missing aspects. appendix c elaborates on the checklist and relevant practices. target domain observation observed score expected score construct target score domain definition evaluation test use warrant: observations of the test performance reveal important and representative knowledge, skills, and abilities in the target domain. backing: the tasks in the test require important knowledge and skills that are representative of the abilities in the target domain. warrant: observations of the test performance are evaluated to provide observed scores reflective of the target abilities. backing: rubrics provide evidence of the target abilities, and the statistical characteristics of examples are appropriate for intended test use. ## related work the definition of language understanding has been discussed in various nlp tasks, including symbol grounding @xcite and reading comprehension @xcite . our work is similar to @xcite in that we do not provide concrete definitions and highlight an epistemological perspective. @xcite introduce six aspects to improve the validity of interpreting results in psychological measurements. although @xcite associate these aspects with the requirements of designing nlu datasets, actionable practices are not proposed for these aspects. similarly, @xcite discuss the construct validity in benchmarking ai but do not aim to improve evaluation methods. our validity argument provides a step-by-step guideline for test developers to follow. the concept of researcher degrees of freedom is originally introduced in psychology as a pertinent factor in severe issues such as harking (hypothesizing after the results are known) and publication bias @xcite . in terms of resolving this problem in nlp, preregistration is a potentially promising direction for research @xcite . however, it is not suited to all areas of nlu research because certain explanatory and analytic studies do not begin with a clear hypothesis. nonetheless, clearly stating a research goal, problem definition, data collection method, system statistics, intended use, and potential risks could be the first step towards making credible claims on the capabilities of nlu systems. ## conclusions the prevalence of exaggerated claims about the achievements of foundation models motivates us to reconsider how we define and evaluate nlu. our formulation of nlu using the response-dependent interpretation mitigates the issues of the turing and octopus tests; it stipulates that observers and target conditions, including tasks and abilities, must be specified. however, current practices for creating nlu datasets are yet to be aligned, which provides researchers with the freedom to choose convenient strategies. to organize essential practices using a standard guideline, we introduce the validity ar-gument, which guides stakeholders to collect and interpret evidence for validating that the test subject executes its intended behavior. our proposed checklist helps researchers find relevant practices for benchmarking nlu, but we continually revise it by investigating potential refutation to promote more credible nlu studies.