Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses (2020)

Abstract: Empirical research in Natural Language Processing (NLP) has adopted a narrow set of principles for assessing hypotheses, relying mainly on p-value computation, which suffers from several known issues. While alternative proposals have been well-debated and adopted in other fields, they remain rarely discussed or used within the NLP community. We address this gap by contrasting various hypothesis assessment techniques, especially those not commonly used in the field (such as evaluations based on Bayesian inference). Since these statistical techniques differ in the hypotheses they can support, we argue that practitioners should first decide their target hypothesis before choosing an assessment method. This is crucial because common fallacies, misconceptions, and misinterpretation surrounding hypothesis assessment methods often stem from a discrepancy between what one would like to claim versus what the method used actually assesses. Our survey reveals that these issues are omnipresent in the NLP research community. As a step forward, we provide best practices and guidelines tailored to NLP research, as well as an easy-to-use package for Bayesian assessment of hypotheses, complementing existing tools.

## introduction empirical fields, such as natural language processing (nlp), must follow scientific principles for assessing hypotheses and drawing conclusions from experiments. for instance, suppose we come across the results in the superiority of one system over another? while s 1 has higher accuracy than s 2 in both cases, the gap is moderate and the datasets are of limited size. can this apparent difference in performance be explained simply by random chance, or do we have sufficient evidence to conclude that s 1 is in fact inherently different (in particular, inherently stronger) than s 2 on these datasets? if the latter, can we quantify this gap in inherent strength while accounting for random fluctuation? such fundamental questions arise in one form or another in every empirical nlp effort. researchers often wish to draw conclusions such as: these two conclusions differ in two respects. first, ca claims the two systems are inherently different, while cb goes further to claim a margin of at least 1% between their inherent accuracies. the second, more subtle difference lies in the interpretation of the 95% figure: the 95% confidence expressed in ca is in terms of the space of empirical observations we could have made, given some underlying truth about how the inherent accuracies of s 1 and s 2 relate; while the 95% probability expressed in cb is directly over the space of possible inherent accuracies of the two systems. to support such a claim, one must turn it into a proper mathematical statement that can be validated using a statistical calculation. this in turn brings in additional choices: we can make at least four statistically distinct hypotheses here, each supported by a different statistical evaluation: the inherent accuracy of typical systems. given the empirically observed accuracies, the probability (posterior interval) that the inherent accuracy of s1 exceeds that of s2 by a margin of 1% is at least 95%. (h4) assume a prior belief (a probability distribution) w.r.t. the inherent accuracies of typical systems. given the empirically observed accuracies, the odds increase by a factor of 1.32 (bayes factor) in favor of the hypothesis that the inherent accuracy of s1 exceeds that of s2 by a margin of 1%. as this illustrates, there are multiple ways to formulate empirical hypotheses and support empirical claims. since each hypothesis starts with a different assumption and makes a (mathematically) different claim, it can only be tested with a certain set of statistical methods. therefore, nlp practitioners ought to define their target hypothesis before choosing an assessment method. the most common statistical methodology used in nlp is null-hypothesis significance testing (nhst) which uses p-values @xcite @xcite . hypotheses h1&h2 can be tested with p-value-based methods, which include confidence intervals and operate over the probability space of observations foot_0 ( §2.1 and §2.2). on the other hand, there are often overlooked approaches, based on bayesian inference @xcite , that can be used to assess hypotheses h3&h4 ( §2.3 and §2.4) and have two broad strengths: they can deal more naturally with accuracy margins and they operate directly over the probability space of inherent accuracy (rather than of observations). for each technique reviewed in this work, we discuss how it compares with alternatives and summarize common misinterpretations surrounding it ( §3). for example, a common misconception about p-value is that it represents a probability of the validity of a hypothesis. while desirable, p-values in fact do not provide such a probabilistic interpretation ( §3.2). it is instead through a bayesian analysis of the posterior distribution of the test statistic (inherent accuracy in the earlier example) that one can make claims about the probability space of that statistic, such as h3. we quantify and demonstrate related common malpractices in the field through a manual annotation of 439 acl-2018 conference papers, foot_1 and a survey filled out by 55 nlp researchers ( §4). we highlight surprising findings from the survey, such as the following: while 86% expressed fairto-complete confidence in the interpretation of pvalues, only a small percentage of them correctly answered a basic p-value interpretation question. contributions. this work seeks to inform the nlp community about crucial distinctions between various statistical hypotheses and their corresponding assessment methods, helping move the community towards well-substantiated empirical claims and conclusions. our exposition covers a broader range of methods ( §2) than those included in recent related efforts ( §1.1), and highlights that these methods achieve different goals. our surveys of nlp researchers reveals problematic trends ( §4), emphasizing the need for increased scrutiny and clarity. we conclude by suggesting guidelines for better testing ( §5), as well as providing a toolkit called hybayes (cf. footnote 1) tailored towards commonly used nlp metrics. we hope this work will encourage an improved understanding of statistical assessment methods and effective reporting practices with measures of uncertainty. ## assessment of hypotheses we often wish to draw qualitative inferences based on the outcome of experiments (for example, inferring the relative inherent performance of systems). to do so, we usually formulate a hypothesis that can be assessed through some analysis. suppose we want to compare two systems on a dataset of instances @xmath0, . . . , x n ] with respect to a measure m(s, x) representing the performance of a system s on an instance x. let m(s, x) denote the vector [m(s, x i )] n @xmath1. given systems s 1 , s 2 , define y [m(s 1 , x), m(s 2 , x)] as a vector of observations. 4 in a typical nlp experiment, the goal is to infer some inherent and unknown properties of systems. to this end, a practitioner assumes a probability distribution on the observations y, parameterized by θ, the properties of the systems. in other words, y is assumed to have a distribution foot_3 with unknown parameters θ. in this setting, a hypothesis h is a condition on θ. hypothesis assessment is a way of evaluating the degree to which the observations y are compatible with h. the overall process is depicted in figure 1 . following our running example, we use the task of answering natural language questions @xcite . while our examples are shown for this particular task, all the ideas are applicable to more general experimental settings. for this task, the performance metric m(s, x) is defined as a binary function indicating whether a system s answers a given question x correctly or not. the performance vector m(s, x) captures the system's accuracy on the entire dataset (cf. ## comparisons many aspects influence the choice of an approach to assess significance of hypotheses. this section provides a comparative summary, with details in appendix c and an overall summary in ## current trends and malpractices this section highlights common practices relevant to the our target approaches. to better understand the common practices or misinterpretations in the field, we conducted a survey. we shared the survey among ∼450 nlp researchers (randomly selected from acl'18 proceedings) from which 55 individuals filled out the survey. while similar surveys have been performed in other fields @xcite , this is the first in the nlp community, to the best of our knowledge. here we review the main highlights (see appendix for more details and charts). interpreting p-values. while the majority of the participants have a self-claimed ability to interpret p-values (figure 9f ), many choose its imprecise interpretation "the probability of the observation this extreme happening due to pure chance" (the popular choice) vs. a more precise statement "conditioned on the null hypothesis, the probability of the observation this extreme happening." the use of cis. even though 95% percent of the participants self-claimed the knowledge of cis (figure 9e ), it is rarely used in practice. in an annotation done on acl'18 papers by two of the authors, only 6 (out of 439) papers were found to use cis. ## conclusion using well-founded mechanisms for assessing the validity of hypotheses is crucial for any field that relies on empirical work. our survey indicates that the nlp community is not fully utilizing scientific methods geared towards such assessment, with only a relatively small number of papers using such methods, and most of them relying on p-value. our goal was to review different alternatives, especially a few often ignored in nlp. we surfaced various issues and potential dangers of careless use and interpretations of different approaches. we do not recommend a particular approach. every technique has its own weaknesses. hence, a researcher should pick the right approach according to their needs and intentions, with a proper understanding of the techniques. incorrect use of any technique can result in misleading conclusions. we contribute a new toolkit, hybayes, to make it easy for nlp practitioners to use bayesian assessment in their efforts. we hope that this work provides a complementary picture of hypothesis assessment techniques for the field and encourages more rigorous reporting trends.