The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail (2022)

Abstract: Researchers in NLP often frame and discuss research results in ways that serve to deemphasize the field’s successes, often in response to the field’s widespread hype. Though well-meaning, this has yielded many misleading or false claims about the limits of our best technology. This is a problem, and it may be more serious than it looks: It harms our credibility in ways that can make it harder to mitigate present-day harms, like those involving biased systems for content moderation or resume screening. It also limits our ability to prepare for the potentially enormous impacts of more distant future advances. This paper urges researchers to be careful about these claims and suggests some research directions and communication strategies that will make it easier to avoid or rebut them.

## introduction over the last few years, natural language processing has seen a wave of surprising negative results overturning previously-reported success stories about what our models can do, and showing that widely-used models are surprisingly brittle @xcite @xcite . this shows that many of our standard practices for evaluation and reporting can lead to unrealistically positive initial claims about what we can do. the resulting hype and overclaiming, whether intentional or not, are a problem. they can encourage the reckless deployment of nlp systems in high-stakes settings where they can do significant harm. they also threaten the health and credibility of nlp as a research field, and thereby threaten our ability to influence applied stakeholders or attract funding. fortunately, these results have led to a surge of research and writing that proposes more thorough and cautious practices for the evaluation of model ability @xcite @xcite figure 1 : hype is a problem. the opposite of hype isn't necessarily better. (quoted with permission.) @xcite . while we have only a limited ability to control the public narrative taking place through industry pr and the media, there's reason to be hopeful that we researchers are getting much better at avoiding the worst forms of overconfidence about our systems. less fortunately, this pattern of disappointment seems to have led to many instances of pessimism about model performance that are ungrounded from real empirical results. this leaves room for the research community's consensus about our capabilities to fall short of our actual capabilities. i call this issue underclaiming, for lack of a better term, 1 and argue that it is more dangerous than it might seem. it risks our credibility and thereby limits our ability to influence stakeholders in cases where our current systems are doing real harm. it also limits our ability to accurately forecast and plan for the impacts that may result from the deployment of more capable systems in the future. if we can truly reach near-human-level performance on many of the core problems of nlp, we should expect enormous impacts which will be potentially catastrophic if not planned for. in this paper, i lay out case studies demonstrating four types of underclaiming, focusing especially on writing and citation practices. i then argue that figure 2 : @xcite remains widely cited according to google scholar. the original work pointed out major unexpected limitations in neural networks trained from scratch on the squad reading comprehension task. however, many of these citing works use it to imply that modern pretrained systems-developed more recently than 2017-show these same limitations. ## underclaiming: case studies this paper addresses the phenomenon of scholarly claims that imply state-of-the-art systems are significantly less capable than they actually are. this takes on several forms, including misleading presentations of valid negative results from weak or dated baseline models, misleading claims about the limits of what is conceptually possible with machine learning, and misleading reporting of results on adversarially collected data. ## a word on hype the previous section has laid out some ways in which the mainstream nlp research community makes unjustifiable claims about the limitations of state-of-the-art methods. these claims do not make the opposite phenomenon, hype, any less real or any less harmful. while hype is likely most severe in industry pr and in the media, 7 it is nonetheless still prevalent in the research literature. in one especially clear example, a prominent paper claiming of human parity in machine translation performance @xcite severely overstates what has been accomplished relative to commonsense intuitions about what a human-level translation system would do @xcite @xcite . i do not aim to argue that overclaiming or hype is acceptable or safe. combating hype should be fully compatible with the goals laid out in this paper, and broad-based efforts to improve our practices in evaluation, analysis, writing, and forecasting should help reduce both underclaiming and hype. ## why underclaiming is harmful research papers are generally most useful when they're true and informative. a research field that allows misleading claims to go unchallenged is likely to waste its time solving problems that it doesn't actually have, and is likely to lose credibility with serious funders, reporters, and industry stakeholders. this is the most obvious reason that we should be concerned about underclaiming, but it is not the whole story. this loss of insight and credibility can seriously challenge our ability to anticipate, understand, and manage the impacts of deploying nlp systems. this is especially true of impacts that are contingent on nlp technologies actually working well, which we should expect will become more substantial as time goes on. ## catastrophic risks because these more speculative concerns around advanced artificial intelligence are rarely discussed in the nlp literature, i will here offer a brief overview of that work. recent writing tends to focus on four clusters of hypotheses: unaccountable organizations highly-capable ai is likely to lead to highly-profitable applications, making the institutions that first develop it quite powerful. it is also likely to be able to displace human labor in technical fields to a large extent, increasing the relative value of capital over labor, and making it easier for the leaders of these organiza-tions to take unpopular actions unilaterally. in the longer term, highly-capable ai may also contribute to the effectiveness of persuasion campaigns, further insulating these organizations from outside pressure. these forces could conspire to make the companies or governments that first produce highly-capable ai almost entirely unaccountable, and allowing their decisions to play a major role in the trajectory of humanity as a whole @xcite . alignment and robustness failures even if a system is deployed by an actor with good intentions and substantial oversight, good outcomes are not guaranteed. as ai systems become more capable, they become capable of effecting-directly or indirectly-significant force on the outside world. in these cases, it becomes crucial that they behave in ways that we would endorse, even when they are pushed into unfamiliar new situations. this requires both that the systems be optimized for the right objectives and that the systems actually internalize and generalize those objectives correctly. specifying and using safe objectives, such that aggressively optimizing them does not produce catastrophic outcomes, is difficult @xcite . human preferences are complex, making the problem of specifying an objective that rules out unintended bad behavior non-trivial. goodhart's law foot_7 means that many objectives that serve as good proxies for what we want in in familiar situations can break down in new situations. further, training large models with high precision is difficult. a small flaw in a highly-capable system's learned understanding of its objective can cause catastrophic failures, even if the true intended objective would have been safe @xcite . ## ways to do better the core issue in this paper is one of sloppy communication about results. the most straightforward step that we can take to remedy underclaiming is to simply use the same practices that we already use to avoid overclaiming: the peer-review process already polices overclaiming to a significant extent, and most researchers have learned to be careful about overclaiming in their writing. we should apply high standards of evidence to our own empirical claims and those of others, both in peer-reviewed venues and in more informal scientific communication, even when those claims are negative and cloaked in a frame of individual or field-level modesty. beyond this, there are specific best practices or research directions that can help make these mistakes harder to make: a rule of thumb in light of the issues with negative results on older models discussed in section 2.1, it could be productive to introduce a new heuristic when reviewing or evaluating papers that discuss model failures. 11 in the spirit of the bender rule @xcite , i propose: when describing the failure of a machine learning model on some empirical evaluation, make it clear i. what kind of model has failed, ii. whether the model is significantly less capable than the current state of the art in the domain, and iii. whether the evaluation was deliberately set up to trick that model or another model like it. ## additional related work while much of this paper discusses the state of the nlp literature, a few related works warrant emphasis as starting points for further reading: bender and koller (2020), bender et al. (2021), and raji et al. (2021) discuss the role of hype in driving bad outcomes from the development of language technology. jin et al. (2021) and rogers (2021) offer broader discussion of how to ensure that the net impact of near-future nlp deployments on the world is positive. morris et al. (2020) and hauser et al. (2021) highlight overly strong negative claims in papers analyzing models' robustness to synonym substitution. looking to the longer term, bommasani et al. (2021, §4.9) provides an introduction to the ai risk and ai alignment literature from a perspective that emphasizes nlp and language. welty et al. (2019), linzen (2020), ribeiro et al. (2020), raji et al. (2021), bowman and dahl (2021), and dehghani et al. (2021), among many others, discuss the challenges involved in designing evaluations that yield trustworthy and accurate depictions of the capabilities of ml models. ## conclusion like many research fields that have a tight connection to technological practice, nlp has long struggled to avoid inflated expectations about the capabilities of state-of-the-art tools. this remains a serious issue. however, this paper argues that our attempts to avoid hype often overshoot: instead of merely correcting overly optimistic claims about our capabilities, we replace them with overly pessimistic claims. making misleading claims is generally a bad sign for the health and credibility of a scientific field, and the stakes are high: nlp technologies are implicated in a range of serious real-world harms, and plausible future elaborations of these technologies are potentially much more dangerous still. our ability to mitigate existing harms will depend on our ability to make reliably credible claims about the limitations of our systems. our ability to mitigate future harms will depend on our ability to accurately anticipate, recognize, agree upon, and report upon emerging capabilities. both of these goals are seriously hampered by claims that current technologies are less capable than they in fact are. better evaluation, better tooling for model analysis, and better mechanisms for technical forecasting should all contribute to making these pessimistic claims easier to avoid or debunk. however, this problem is ultimately one of scientific communication, and to solve it fully, we will need to use the tools and norms of science to better police false or misleading claims. the stakes are high.