Reliability Testing for Natural Language Processing Systems (2021)

Abstract: Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.

## introduction rigorous testing is critical to ensuring a program works as intended (functionality) when used under real-world conditions (reliability). hence, it is troubling that while natural language technologies are becoming increasingly pervasive in our everyday lives, there is little assurance that these nlp systems will not fail catastrophically or amplify discrimination against minority demographics when exposed to input from outside the training distribution. recent examples include @xcite agreeing with suggested suicide @xcite , the mistranslation of an innocuous social media post resulting in a minority's arrest @xcite , and biased grading algorithms that can negatively impact a minority student's future @xcite . additionally, a lack of rigorous testing, coupled with machine learning's (ml) implicit assumption of identical training and testing distributions, may inadvertently result in systems that discriminate against minorities, who are often underrepresented in the training data. this can take the form of misrepresentation of or poorer performance for people with disabilities, specific gender, ethnic, age, or linguistic groups @xcite @xcite . amongst claims of nlp systems achieving human parity in challenging tasks such as question answering @xcite , machine translation @xcite , and commonsense inference @xcite , research has demonstrated these systems' fragility to natural and adversarial noise @xcite and out-of-distribution data @xcite . it is also still common practice to equate "testing" with "measuring held-out accuracy", even as datasets are revealed to be harmfully biased @xcite @xcite . many potential harms can be mitigated by detecting them early and preventing the offending model from being put into production. hence, in addition to being mindful of the biases in the nlp pipeline @xcite @xcite and holding creators accountable via audits @xcite , we argue for the need to evaluate an nlp system's reliability in diverse operating conditions. initial research on evaluating out-of-distribution generalization involved manually-designed challenge sets @xcite @xcite , counterfactuals @xcite @xcite , biased sampling @xcite or toolkits for testing if a system has specific capabilities @xcite or robustness to distribution shifts @xcite . however, most of these approaches inevitably overestimate a given system's worst-case performance since they do not mimic the nlp system's adversarial distribution 1 . a promising technique for evaluating worst-case performance is the adversarial attack. however, although some adversarial attacks explicitly focus on specific linguistic levels of analysis @xcite @xcite , many often simply rely on word embeddings or language models for perturbation proposal (see §4). while the latter may be useful to evaluate a system's robustness to malicious actors, they are less useful for dimension-specific testing (e.g., reliability when encountering grammatical variation). this is because they often perturb the input across multiple dimensions at once, which may make the resulting adversaries unnatural. hence, in this paper targeted at nlp researchers, practitioners, and policymakers, we make the case for reliability testing and reformulate adversarial attacks as dimension-specific, worst-case tests that can be used to approximate real-world variation. we contribute a reliability testing framework -doctor -that translates safety and fairness concerns around nlp systems into quantitative tests. we demonstrate how testing dimensions for doc-tor can be drafted for a specific use case. finally, we discuss the policy implications, challenges, and directions for future research on reliability testing. ## terminology definitions let's define key terms to be used in our discussion. ## the case for reliability testing in nlp the accelerating interest in building nlp-based products that impact many lives has led to urgent questions of fairness, safety, and accountability @xcite , prompting research into algorithmic bias @xcite @xcite , robustness (jia and liang, 2017), etc. research is also emerging on best practices for productizing ml: from detailed dataset documentation @xcite , model documentation for highlighting important but often unreported details such as its training data, intended use, and caveats @xcite , and documentation best practices (partnership on ai, 2019), to institutional mechanisms such as auditing @xcite to enforce accountability and red-teaming @xcite to address developer blind spots, not to mention studies on the impact of organizational structures on responsible ai initiatives @xcite . calls for increased accountability and transparency are gaining traction among governments (116th u.s. congress, 2019; nist, 2019; european commission, 2020; smith, 2020; california state legislature, 2020; fda, 2021) and customers increasingly cite ethical concerns as a reason for not engaging ai service providers (eiu, 2020). while there has been significant discussion around best practices for dataset and model creation, work to ensure nlp systems are evaluated in a manner representative of their operational conditions has only just begun. initial work in constructing representative tests focuses on enabling development teams to easily evaluate their models' linguistic capabilities @xcite and accuracy on subpopulations and distribution shifts @xcite . however, there is a clear need for a paradigm that allows experts and stakeholder advocates to collaboratively develop tests that are representative of the practical and ethical concerns of an nlp system's target demographic. we argue that reliability testing, by reframing the concept of adversarial attacks, has the potential to fill this gap. ## adversarial attacks as reliability tests we first give a brief introduction to adversarial attacks in nlp before showing how they can be used for reliability testing. we refer the reader to @xcite for a comprehensive survey. existing work on nlp adversarial attacks perturbs the input at various levels of linguistic analysis: phonology @xcite , orthography @xcite , morphology @xcite , lexicon @xcite , and syntax @xcite . early work did not place any constraints on the attacks and merely used the degradation to a tar- x ← x ∪ c 8: x ← x ∪ {x } 11: focusing on maximally degrading model accuracy overlooks the key feature of adversarial attacks: the ability to find the worst-case example for a model from an arbitrary distribution. many recent attacks perturb the input across multiple dimensions at once, which may make the result unnatural. by constraining our sample perturbations to a distribution modeling a specific dimension of interest, the performance on the generated adversaries is a valid lower bound performance for that dimension. said another way, adversarial attacks can be reframed as interpretable reliability tests if we constrain them to meaningful distributions. this is the key element of our approach as detailed in alg. 1. we specify either an average (lines 5-7) or worse case test (lines 8-10), but conditioned on the data distribution d that models a particular dimension of interest d. the resultant reliability score gauges real-world performance and the worstcase variant returns the adversarial examples that cause worst-case performance. when invariance to input variation is expected, y is equivalent to the source label y. note that by ignoring the averagecase test logic and removing d, we recover the general adversarial attack algorithm. however, the key difference between an adversarial robustness mindset and a testing one is the latter's emphasis on identifying ways in which natural phenomena or ethical concerns can be operationalized as reliability tests. this change in perspective opens up new avenues for interdisciplinary research that will allow researchers and practitioners to have more nuanced discussions about model reliability and can be used to design comprehensive reliability testing regimes. we describe such a framework for interdisciplinary collaboration next. ## a framework for reliability testing we introduce and then describe our general framework, doctor, for testing the reliability of nlp systems. doctor comprises six steps: since it is likely unfeasible to test every possible dimension, stakeholder advocates should be involved to ensure their values and interests are accurately represented and prioritized @xcite , while experts should ensure the dimensions identified can be feasibly tested. a similar approach to that of community juries foot_2 may be taken. we recommend using this question to evaluate the feasibility of operationalizing potential dimensions: "what is the system's performance when exposed to variation along dimension d?". for example, rather than simply "gender", a better-defined dimension would be "gender pronouns". with this understanding, experts and policymakers can then create a set of reliability requirements, comprising the testing dimensions, performance metric(s), and passing thresholds. next, we recommend using the same metrics for held-out, average-case, and worst-case performance for easy comparison. these often vary from task to task and are still a subject of active research @xcite @xcite , hence the question of the right metric to use is beyond the scope of this paper. finally, ethicists, in consultation with the other aforementioned experts and stakeholders, will determine acceptable thresholds for worst-case performance. the system under test must perform above said thresholds when exposed to variation along those dimensions in order to pass. for worst-case performance, we recommend reporting thresholds as relative differences (δ) between the average-case and worst-case performance. these questions may help in applying this step and deciding if specific nlp solutions should even exist @xcite : we note that our framework is general enough to be applied at various levels of organization: within the development team, within the company (compliance team, internal auditor), and within the industry (self-regulation or independent regulator). however, we expect the exact set of dimensions, metrics and acceptable thresholds defined in step 1 to vary depending on the reliability concerns of the actors at each level. for example, independent regulators will be most concerned with establishing minimum safety and fairness standards that all nlp systems used in their industries must meet, while compliance teams may wish to have stricter and more comprehensive standards for brand reasons. developers can use doctor to meet the other two levels of requirements and understand their system's behaviour better with targeted testing. ## from concerns to dimensions we now illustrate how reliability concerns can be converted into concrete testing dimensions (step 1) by considering the scenario of applying automated text scoring to short answers and essays from students in the multilingual population of singapore. we study a second scenario in appendix a. automated text scoring (ats) systems are increasingly used to grade tests and essays @xcite . while they can provide instant feedback and help teachers and test agencies cope with large loads, studies have shown that they often exhibit demographic and language biases, such as scoring african-and indian-american males lower on the gre argument task compared to human graders @xcite . since the results of some tests will affect the futures of the test takers @xcite , the scoring algorithms used must be sufficiently reliable. hence, let us imagine that singapore's education ministry has decided to create a standard set of reliability requirements that all ats systems used in education must adhere to. ## implications for policy there is a mounting effort to increase accountability and transparency around the development and use of nlp systems to prevent them from amplifying societal biases. doctor is highly complementary to the model card approach increasingly adopted foot_4 to surface oft hidden details about nlp models: developers simply need to list the tested dimensions, metrics, and score on each dimension in the model card. crucially, reliability tests can be used to highlight fairness issues in nlp systems by including sensitive attributes for the target population, but it is paramount these requirements reflect local concerns rather than any prescriptivist perspective @xcite . at the same time, the ability to conduct quantitative, targeted reliability testing along specifiable dimensions paves the way for reliability standards to be established, with varying levels of stringency and rigor for different use cases and industries. we envision minimum safety and fairness standards being established for applications that are non-sensitive, not safety-critical, and used in unregulated industries, analogous to standards for household appliances. naturally, applications at greater risks @xcite of causing harm upon failure should be held to stricter standards. policymakers are starting to propose and implement regulations to enforce transparency and accountability in the use of ai systems. for example, the european union's general data protection regulation grants data subjects the right to obtain "meaningful information about the logic involved" in automated decision systems @xcite . the eu is developing aispecific regulation (european commission, 2020): e.g., requiring developers of high-risk ai systems to report their "capabilities and limitations, ... @xcite the conditions under which they can be expected to function as intended". in the u.s., a proposed bill of the state of washington will require public agencies to report "any potential impacts of the automated decision system on civil rights and liberties and potential disparate impacts on marginalized communities" before using automated decision systems (washington state legislature, 2021). one may note that language in the proposed regulation is intentionally vague. there are many ways to measure bias and fairness, depending on the type of model, context of use, and goal of the system. today, companies developing ai systems employ the definitions they believe most reasonable (or perhaps easiest to implement), but regulation will need to be more specific for there to be meaningful compliance. doctor's requirement to explicitly define specific dimensions instead of a vague notion of reliability will help policymakers in this blog.einstein.ai/model-cards-for-ai-model-transparency regard, and can inform the ongoing development of national (nist, 2019) and international standards 7 . while external algorithm audits are becoming popular, testing remains a challenge since companies wishing to protect their intellectual property may be resistant to sharing their code @xcite , and implementing custom tests for each system is unscalable. our approach to reliability testing offers a potential solution to this conundrum by treating nlp systems as black boxes. if reliability tests become a legal requirement, regulatory authorities will be able to mandate independently conducted reliability tests for transparency. such standards, combined with certification programs (e.g., ieee's ethics certification program for autonomous and intelligent systems 8 ), will further incentivize the development of responsible nlp, as the companies purchasing nlp systems will insist on certified systems to protect them from both legal and brand risk. to avoid confusion, we expect certification to occur for individual nlp systems (e.g., an end-to-end question answering system for customer enquiries), rather than for general purpose language models that will be further trained to perform some specific nlp task. while concrete standards and certification programs that can serve this purpose do not yet exist, we believe that they eventually will and hope our paper will inform their development. this multi-pronged approach can help to mitigate nlp's potential harms while increasing public trust in language technology. ## challenges and future directions while doctor is a useful starting point to implement reliability testing for nlp systems, we observe key challenges to its widespread adoption. first, identifying and prioritizing the dimensions that can attest a system's reliability and fairness. the former is relatively straightforward and can be achieved via collaboration with experts (e.g., as part of the u.s. nist's future ai standards (nist, 2019)). the latter, however, is a question of values and power @xcite @xcite , and should be addressed via a code of ethics and ensuring that all stakeholders are adequately represented at the decision table. second, our proposed method of reliability testing may suffer from similar issues plaguing automatic 7 ethicsstandards.org/p7000 8 standards.ieee.org/industry-connections/ecpais.html evaluation metrics for natural language generation @xcite @xcite : due to the tests' synthetic nature they may not fully capture the nuances of reality. for example, if a test's objective were to test an nlp system's reliability when interacting with african american english (aae) speakers, would it be possible to guarantee (in practice) that all generated examples fall within the distribution of aae texts? potential research directions would be to design adversary generation techniques that can offer such guarantees or incorporate human feedback @xcite @xcite . ## conclusion once language technologies leave the lab and start impacting real lives, concerns around safety, fairness, and accountability cease to be thought experiments. while it is clear that nlp can have a positive impact on our lives, from typing autocompletion to revitalizing endangered languages @xcite , it also has the potential to perpetuate harmful stereotypes @xcite , perform disproportionately poorly for underrepresented groups @xcite , and even erase already marginalized communities @xcite . trust in our tools stems from an assurance that stakeholders will remain unharmed, even in the worst-case scenario. in many mature industries, this takes the form of reliability standards. however, for standards to be enacted and enforced, we must first operationalize "reliability". hence, we argue for the need for reliability testing (especially worst-case testing) in nlp by contextualizing it among existing work on promoting accountability and improving generalization beyond the training distribution. next, we showed how adversarial attacks can be reframed as worst-case tests. finally, we proposed a possible paradigm, doctor, for how reliability concerns can be realized as quantitative tests, and discussed how this framework can be used at different levels of organization or industry.