Original: Reliability Testing for Natural Language Processing Systems
Revised: Reliability Testing for Natural Language Processing Systems

Original: Abstract
Revised: Abstract

Original: Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems.
Revised: Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems.

Original: Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments?
Revised: Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments?

Original: To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability.
Revised: To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability.

Original: We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests.
Revised: We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests.

Original: We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.
Revised: We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.

Original:  Introduction
Revised:  Introduction

Original: Rigorous testing is critical to ensuring a program works as intended (functionality) when used under real-world conditions (reliability).
Revised: Rigorous testing is critical to ensuring a program works as intended (functionality) when used under real-world conditions (reliability).

Original: Hence, it is troubling that while natural language technologies are becoming increasingly pervasive in our everyday lives, there is little assurance that these NLP systems will not fail catastrophically or amplify discrimination against minority demographics when exposed to input from outside the training distribution.
Revised: Hence, it is troubling that while natural language technologies are becoming increasingly pervasive in our everyday lives, there is little assurance that these NLP systems will not fail catastrophically or amplify discrimination against minority demographics when exposed to input from outside the training distribution.

Original: Recent examples include GPT-3 (Brown et al., 2020) agreeing with suggested suicide (Rousseau et al., 2020), the mistranslation of an innocuous social media post resulting in a minority’s arrest (Hern, 2017) and biased grading algorithms that can negatively impact a minority student’s future (Feathers, 2019).
Revised: Recent examples include GPT-3 (Brown et al., 2020) agreeing with suggested suicide (Rousseau et al., 2020), the mistranslation of an innocuous social media post resulting in a minority’s arrest (Hern, 2017) and biased grading algorithms that can negatively impact a minority student’s future (Feathers, 2019).

Original: Additionally, a lack of rigorous testing, coupled with machine learning’s
Revised: Additionally, a lack of rigorous testing, coupled with machine learning’s

Original: (ML) implicit assumption of identical training and testing distributions, may inadvertently result in systems that discriminate against minorities, who are often underrepresented in the training data.
Revised: (ML) implicit assumption of identical training and testing distributions, may inadvertently result in systems that discriminate against minorities, who are often underrepresented in the training data.

Original: This can take the form of misrepresentation of or poorer performance for people with disabilities, speciﬁc gender, ethnic, age, or linguistic groups (Hovy and Spruit, 2016; Crawford, 2017; Hutchinson et al., 2020).
Revised: This can take the form of misrepresentation of or poorer performance for people with disabilities, speciﬁc gender, ethnic, age, or linguistic groups (Hovy and Spruit, 2016; Crawford, 2017; Hutchinson et al., 2020).

Original: Amongst claims of NLP systems achieving human parity in challenging tasks such as question answering (Yu et al., 2018), machine translation (Hassan et al., 2018), and commonsense inference (Devlin et al., 2019), research has demonstrated these systems’ fragility to natural and adversarial noise (Goodfellow et al., 2015; Belinkov and Bisk, 2018) and out-of-distribution data (Fisch et al., 2019).
Revised: Amongst claims of NLP systems achieving human parity in challenging tasks such as question answering (Yu et al., 2018), machine translation (Hassan et al., 2018), and commonsense inference (Devlin et al., 2019), research has demonstrated these systems’ fragility to natural and adversarial noise (Goodfellow et al., 2015; Belinkov and Bisk, 2018) and out-of-distribution data (Fisch et al., 2019).

Original: It is also still common practice to equate “testing” with “measuring held-out accuracy”, even as datasets are revealed to be harmfully biased (Wagner et al., 2015; Geva et al., 2019; Sap et al., 2019).
Revised: It is also still common practice to equate “testing” with “measuring held-out accuracy”, even as datasets are revealed to be harmfully biased (Wagner et al., 2015; Geva et al., 2019; Sap et al., 2019).

Original: Many potential harms can be mitigated by detecting them early and preventing the offending model from being put into production.
Revised: Many potential harms can be mitigated by detecting them early and preventing the offending model from being put into production.

Original: Hence, in addition to being mindful of the biases in the NLP pipeline (Bender and Friedman, 2018; Mitchell et al., 2019;
Revised: Hence, in addition to being mindful of the biases in the NLP pipeline (Bender and Friedman, 2018; Mitchell et al., 2019;

Original: Waseem et al., 2021) and holding creators accountable via audits (Raji et al., 2020;
Revised: Waseem et al., 2021) and holding creators accountable via audits (Raji et al., 2020;

Original: Brundage et al., 2020), we argue for the need to evaluate an NLP system’s reliability in diverse operating conditions.
Revised: Brundage et al., 2020), we argue for the need to evaluate an NLP system’s reliability in diverse operating conditions.

Original: Initial research on evaluating out-of-distribution generalization involved manually designed challenge sets (Jia and Liang, 2017; Nie et al., 2020;
Revised: Initial research on evaluating out-of-distribution generalization involved manually designed challenge sets (Jia and Liang, 2017; Nie et al., 2020;

Original: Gardner et al., 2020), counterfactuals (Kaushik et al., 2019; Khashabi et al., 2020; Wu et al., 2021), biased sampling (Søgaard et al., 2021) or toolkits for testing if a system has speciﬁc capabilities (Ribeiro et al., 2020) or robustness to distribution shifts (Goel et al., 2021).
Revised: Gardner et al., 2020), counterfactuals (Kaushik et al., 2019; Khashabi et al., 2020; Wu et al., 2021), biased sampling (Søgaard et al., 2021) or toolkits for testing if a system has speciﬁc capabilities (Ribeiro et al., 2020) or robustness to distribution shifts (Goel et al., 2021).

Original: However, most of these approaches inevitably overestimate a given system’s worst-case performance since they do not mimic the NLP system’s adversarial distribution 1 .
Revised: However, most of these approaches inevitably overestimate a given system’s worst-case performance since they do not mimic the NLP system’s adversarial distribution 1 .

Original: A promising technique for evaluating worst-case performance is the adversarial attack.
Revised: A promising technique for evaluating worst-case performance is the adversarial attack.

Original: However, although some adversarial attacks explicitly focus on speciﬁc linguistic levels of analysis (Belinkov and
Revised: However, although some adversarial attacks explicitly focus on speciﬁc linguistic levels of analysis (Belinkov and

Original: Bisk, 2018; Iyyer et al., 2018; Tan et al., 2020; Eger and Benz, 2020), many often simply rely on word embeddings or language models for perturbation proposal (see §4).
Revised: Bisk, 2018; Iyyer et al., 2018; Tan et al., 2020; Eger and Benz, 2020), many often simply rely on word embeddings or language models for perturbation proposal (see §4).

Original: While the latter may be useful to evaluate a system’s robustness to malicious actors, they are less useful for dimension-speciﬁc testing (e.g., reliability when encountering grammatical variation).
Revised: While the latter may be useful to evaluate a system’s robustness to malicious actors, they are less useful for dimension-speciﬁc testing (e.g., reliability when encountering grammatical variation).

Original: This is because they often perturb the input across multiple dimensions at once, which may make the resulting adversaries unnatural.
Revised: This is because they often perturb the input across multiple dimensions at once, which may make the resulting adversaries unnatural.

Original: Hence, in this paper targeted at NLP researchers, practitioners, and policymakers, we make the case for reliability testing and reformulate adversarial attacks as dimension-speciﬁc , worst-case tests that can be used to approximate real-world variation.
Revised: Hence, in this paper targeted at NLP researchers, practitioners, and policymakers, we make the case for reliability testing and reformulate adversarial attacks as dimension-speciﬁc , worst-case tests that can be used to approximate real-world variation.

Original: We contribute a reliability testing framework —
Revised: We contribute a reliability testing framework —

Original: DOCTOR — that translates safety and fairness concerns around NLP systems into quantitative tests.
Revised: DOCTOR — that translates safety and fairness concerns around NLP systems into quantitative tests.

Original: We demonstrate how testing dimensions for DOCTOR can be drafted for a speciﬁc use case.
Revised: We demonstrate how testing dimensions for DOCTOR can be drafted for a speciﬁc use case.

Original: Finally, we discuss the policy implications, challenges, and directions for future research on reliability testing.
Revised: Finally, we discuss the policy implications, challenges, and directions for future research on reliability testing.

Original:  Terminology Deﬁnitions
Revised:  Terminology Deﬁnitions

Original: Let’s deﬁne key terms to be used in our discussion.
Revised: Let’s deﬁne key terms to be used in our discussion.

Original: NLP system
Revised: NLP system

Original: The entire text processing pipeline built to solve a speciﬁc task, taking raw text as input and producing predictions in the form of labels (classiﬁcation) or text (generation).
Revised: The entire text processing pipeline built to solve a speciﬁc task, taking raw text as input and producing predictions in the form of labels (classiﬁcation) or text (generation).

Original: We exclude raw language models from the discussion since it is unclear how performance, and hence worst-case performance, should be evaluated.
Revised: We exclude raw language models from the discussion since it is unclear how performance, and hence worst-case performance, should be evaluated.

Original: We include NLP systems that use language models internally (e.g.,
Revised: We include NLP systems that use language models internally (e.g.,

Original: BERT-based classiﬁers (Devlin et al., 2019)).
Revised: BERT-based classiﬁers (Devlin et al., 2019)).

Original: Reliability.
Revised: Reliability.

Original: Deﬁned by IEEE (2017) as the “degree to which a system, product or component performs speciﬁed functions under speciﬁed conditions for a speciﬁed period of time”.
Revised: Deﬁned by IEEE (2017) as the “degree to which a system, product or component performs speciﬁed functions under speciﬁed conditions for a speciﬁed period of time”.

Original: We prefer this term over robustness 2 to challenge the NLP community’s common framing of inputs from outside the training distribution as “noisy”.
Revised: We prefer this term over robustness 2 to challenge the NLP community’s common framing of inputs from outside the training distribution as “noisy”.

Original: The notion of reliability requires us to explicitly consider the speciﬁc, diverse, environments (i.e., communities) a system will operate in and is crucial to reducing the negative impact of NLP on the underrepresented.
Revised: The notion of reliability requires us to explicitly consider the speciﬁc, diverse, environments (i.e., communities) a system will operate in and is crucial to reducing the negative impact of NLP on the underrepresented.

Original: Dimension.
Revised: Dimension.

Original: An axis along which variation can occur in the real world.
Revised: An axis along which variation can occur in the real world.

Original: A taxonomy of possible dimensions can be found in Table 1 (Appendix).
Revised: A taxonomy of possible dimensions can be found in Table 1 (Appendix).

Original: Adversarial attack.
Revised: Adversarial attack.

Original: A method of perturbing the input to degrade a target model’s accuracy (Goodfellow et al., 2015).
Revised: A method of perturbing the input to degrade a target model’s accuracy (Goodfellow et al., 2015).

Original: In computer vision, this is achieved by adding adversarial noise to the image, optimized to be maximally damaging to the model.
Revised: In computer vision, this is achieved by adding adversarial noise to the image, optimized to be maximally damaging to the model.

Original: §4 describes how this is done in the NLP context.
Revised: §4 describes how this is done in the NLP context.

Original: Stakeholder.
Revised: Stakeholder.

Original: Someone who is directly or indirectly impacted by the NLP system’s predictions.
Revised: Someone who is directly or indirectly impacted by the NLP system’s predictions.

Original: Actor.
Revised: Actor.

Original: Someone who has inﬂuence over a) the design of an NLP system and its reliability testing regime; b) whether the system is deployed; and c) who it can interact with.
Revised: Someone who has inﬂuence over a) the design of an NLP system and its reliability testing regime; b) whether the system is deployed; and c) who it can interact with.

Original: Within the context of our discussion, actors are likely to be regulators, experts, and stakeholder advocates.
Revised: Within the context of our discussion, actors are likely to be regulators, experts, and stakeholder advocates.

Original: Expert.
Revised: Expert.

Original: Someone who has specialized knowledge, such as ethicists, linguists, domain experts, social scientists, or NLP practitioners.
Revised: Someone who has specialized knowledge, such as ethicists, linguists, domain experts, social scientists, or NLP practitioners.

Original:  The Case for Reliability Testing in NLP
Revised:  The Case for Reliability Testing in NLP

Original: The accelerating interest in building NLP-based products that impact many lives has led to urgent questions of fairness, safety, and accountability (Hovy and Spruit, 2016;
Revised: The accelerating interest in building NLP-based products that impact many lives has led to urgent questions of fairness, safety, and accountability (Hovy and Spruit, 2016;

Original: Bender et al., 2021), prompting research into algorithmic bias (Bolukbasi et al., 2016; Blodgett et al., 2020), explainability (Ribeiro et al., 2016; Danilevsky et al., 2020), robustness (Jia and Liang, 2017), etc.
Revised: Bender et al., 2021), prompting research into algorithmic bias (Bolukbasi et al., 2016; Blodgett et al., 2020), explainability (Ribeiro et al., 2016; Danilevsky et al., 2020), robustness (Jia and Liang, 2017), etc.

Original: Research is also emerging on best practices for productizing
Revised: Research is also emerging on best practices for productizing

Original: ML: from detailed dataset documentation (Bender and Friedman, 2018; Gebru et al., 2018), model documentation for highlighting important but often unreported details such as its training data, intended use, and caveats (Mitchell et al., 2019), and documentation best practices (Partnership on AI, 2019), to institutional mechanisms such as auditing (Raji et al., 2020) to enforce accountability and red-teaming (Brundage et al., 2020) to address developer blind spots, not to mention studies on the impact of organizational structures on responsible
Revised: ML: from detailed dataset documentation (Bender and Friedman, 2018; Gebru et al., 2018), model documentation for highlighting important but often unreported details such as its training data, intended use, and caveats (Mitchell et al., 2019), and documentation best practices (Partnership on AI, 2019), to institutional mechanisms such as auditing (Raji et al., 2020) to enforce accountability and red-teaming (Brundage et al., 2020) to address developer blind spots, not to mention studies on the impact of organizational structures on responsible

Original: AI initiatives (Rakova et al., 2020).
Revised: AI initiatives (Rakova et al., 2020).

Original: Calls for increased accountability and transparency are gaining traction among governments (116th U.S. Congress, 2019; NIST, 2019; European Commission, 2020; Smith, 2020; California
Revised: Calls for increased accountability and transparency are gaining traction among governments (116th U.S. Congress, 2019; NIST, 2019; European Commission, 2020; Smith, 2020; California

Original: State Legislature, 2020; FDA, 2021) and customers increasingly cite ethical concerns as a reason for not engaging AI service providers (EIU, 2020).
Revised: State Legislature, 2020; FDA, 2021) and customers increasingly cite ethical concerns as a reason for not engaging AI service providers (EIU, 2020).

Original: While there has been signiﬁcant discussion around best practices for dataset and model creation, work to ensure NLP systems are evaluated in a manner representative of their operational conditions has only just begun.
Revised: While there has been signiﬁcant discussion around best practices for dataset and model creation, work to ensure NLP systems are evaluated in a manner representative of their operational conditions has only just begun.

Original: Initial work in constructing representative tests focuses on enabling development teams to easily evaluate their models’ linguistic capabilities (Ribeiro et al., 2020) and accuracy on subpopulations and distribution shifts (Goel et al., 2021).
Revised: Initial work in constructing representative tests focuses on enabling development teams to easily evaluate their models’ linguistic capabilities (Ribeiro et al., 2020) and accuracy on subpopulations and distribution shifts (Goel et al., 2021).

Original: However, there is a clear need for a paradigm that allows experts and stakeholder advocates to collaboratively develop tests that are representative of the practical and ethical concerns of an NLP system’s target demographic.
Revised: However, there is a clear need for a paradigm that allows experts and stakeholder advocates to collaboratively develop tests that are representative of the practical and ethical concerns of an NLP system’s target demographic.

Original: We argue that reliability testing , by reframing the concept of adversarial attacks, has the potential to ﬁll this gap.
Revised: We argue that reliability testing , by reframing the concept of adversarial attacks, has the potential to ﬁll this gap.

Original: What is
Revised: What is

Original: reliability testing?
Revised: reliability testing?

Original: Despite the recent advances in neural architectures resulting in breakthrough performance on benchmark datasets, research into adversarial examples and out-of-distribution generalization has found
Revised: Despite the recent advances in neural architectures resulting in breakthrough performance on benchmark datasets, research into adversarial examples and out-of-distribution generalization has found

Original: ML systems to be particularly vulnerable to slight perturbations in the input (Goodfellow et al., 2015) and natural distribution shifts (Fisch et al., 2019).
Revised: ML systems to be particularly vulnerable to slight perturbations in the input (Goodfellow et al., 2015) and natural distribution shifts (Fisch et al., 2019).

Original: While these perturbations are often chosen to maximize model failure, they highlight serious reliability issues for putting ML models into production since they show that these models could fail catastrophically in naturally noisy, diverse, real-world environments (Saria and Subbaswamy, 2019).
Revised: While these perturbations are often chosen to maximize model failure, they highlight serious reliability issues for putting ML models into production since they show that these models could fail catastrophically in naturally noisy, diverse, real-world environments (Saria and Subbaswamy, 2019).

Original: Additionally, bias can seep into the system at multiple stages of the NLP lifecycle (Shah et al., 2020), resulting in discrimination against minority groups (O’Neil, 2016).
Revised: Additionally, bias can seep into the system at multiple stages of the NLP lifecycle (Shah et al., 2020), resulting in discrimination against minority groups (O’Neil, 2016).

Original: The good news, however, is that rigorous testing can help to highlight potential issues before the systems are deployed.
Revised: The good news, however, is that rigorous testing can help to highlight potential issues before the systems are deployed.

Original: The need for rigorous testing in NLP is reﬂected in ACL 2020 giving the Best Paper Award to CheckList (Ribeiro et al., 2020), which applied the idea of behavior testing from software engineering to testing NLP systems.
Revised: The need for rigorous testing in NLP is reﬂected in ACL 2020 giving the Best Paper Award to CheckList (Ribeiro et al., 2020), which applied the idea of behavior testing from software engineering to testing NLP systems.

Original: While invaluable as a ﬁrst step towards the development of comprehensive testing methodology, the current implementation of
Revised: While invaluable as a ﬁrst step towards the development of comprehensive testing methodology, the current implementation of

Original: CheckList may still overestimate the reliability of
Revised: CheckList may still overestimate the reliability of

Original: NLP systems since the individual test examples are largely manually constructed.
Revised: NLP systems since the individual test examples are largely manually constructed.

Original: Importantly, with the complexity and scale of current models, humans cannot accurately determine a model’s adversarial distribution (i.e., the examples that cause model failure).
Revised: Importantly, with the complexity and scale of current models, humans cannot accurately determine a model’s adversarial distribution (i.e., the examples that cause model failure).

Original: Consequently, the test examples they construct are unlikely to be the worst-case examples for the model.
Revised: Consequently, the test examples they construct are unlikely to be the worst-case examples for the model.

Original: Automated assistance is needed.
Revised: Automated assistance is needed.

Original: Therefore, we propose to perform reliability testing , which can be thought of as one component of behavior testing.
Revised: Therefore, we propose to perform reliability testing , which can be thought of as one component of behavior testing.

Original: We categorize reliability tests as average-case tests or the worst-case tests.
Revised: We categorize reliability tests as average-case tests or the worst-case tests.

Original: As their names suggest, average-case and worst-case tests estimate the expected and lower-bound performance, respectively, when the NLP system is exposed to the phenomena modeled by the tests.
Revised: As their names suggest, average-case and worst-case tests estimate the expected and lower-bound performance, respectively, when the NLP system is exposed to the phenomena modeled by the tests.

Original: Average-case tests are conceptually similar to Wu et al.
Revised: Average-case tests are conceptually similar to Wu et al.

Original: (2021)’s counterfactuals, which is contemporaneous work, while worst-case tests are most similar to adversarial attacks (§4).
Revised: (2021)’s counterfactuals, which is contemporaneous work, while worst-case tests are most similar to adversarial attacks (§4).

Original: Our approach parallels boundary value testing in software engineering: In boundary value testing, tests evaluate a program’s ability to handle edge cases using test examples drawn from the extremes of the ranges the program is expected to handle.
Revised: Our approach parallels boundary value testing in software engineering: In boundary value testing, tests evaluate a program’s ability to handle edge cases using test examples drawn from the extremes of the ranges the program is expected to handle.

Original: Similarly, reliability testing aims to quantify the system’s reliability under diverse and potentially extreme conditions.
Revised: Similarly, reliability testing aims to quantify the system’s reliability under diverse and potentially extreme conditions.

Original: This allows teams to perform better quality control of their NLP systems and introduce more nuance into discussions of why and when models fail (§5).
Revised: This allows teams to perform better quality control of their NLP systems and introduce more nuance into discussions of why and when models fail (§5).

Original: Finally, we note that reliability testing and standards are established practices in engineering industries (e.g., aerospace (Nelson,
Revised: Finally, we note that reliability testing and standards are established practices in engineering industries (e.g., aerospace (Nelson,

Original: ; Wilkinson et al., 2016)) and advocate for NL engineering to be at parity with these ﬁelds.
Revised: ; Wilkinson et al., 2016)) and advocate for NL engineering to be at parity with these ﬁelds.

Original: 3. Evaluating worst-case performance in a label-scarce world
Revised: 3. Evaluating worst-case performance in a label-scarce world

Original: A proposed approach for testing robustness to natural and adverse distribution shifts is to construct test sets using data from different domains or writing styles (Miller et al., 2020;
Revised: A proposed approach for testing robustness to natural and adverse distribution shifts is to construct test sets using data from different domains or writing styles (Miller et al., 2020;

Original: Hendrycks et al., 2020), or to use a human vs. model method of constructing challenge sets (Nie et al., 2020; Zhang et al., 2019b).
Revised: Hendrycks et al., 2020), or to use a human vs. model method of constructing challenge sets (Nie et al., 2020; Zhang et al., 2019b).

Original: While they are the gold standard, such datasets are expensive to construct, 4 making it infeasible to manually create worst-case test examples for each NLP system being evaluated.
Revised: While they are the gold standard, such datasets are expensive to construct, 4 making it infeasible to manually create worst-case test examples for each NLP system being evaluated.

Original: Consequently, these challenge sets necessarily overestimate each system’s worst-case performance when the inference distribution differs from the training one.
Revised: Consequently, these challenge sets necessarily overestimate each system’s worst-case performance when the inference distribution differs from the training one.

Original: Additionally, due to their crowdsourced nature, these challenge sets inevitably introduce distribution shifts across multiple dimensions at once, and even their own biases (Geva et al., 2019), unless explicitly controlled for.
Revised: Additionally, due to their crowdsourced nature, these challenge sets inevitably introduce distribution shifts across multiple dimensions at once, and even their own biases (Geva et al., 2019), unless explicitly controlled for.

Original: Building individual challenge sets for each dimension would be prohibitively expensive due to combinatorial explosion, even before having to account for concept drift (Widmer and Kubat, 1996).
Revised: Building individual challenge sets for each dimension would be prohibitively expensive due to combinatorial explosion, even before having to account for concept drift (Widmer and Kubat, 1996).

Original: This coupling complicates efforts to design a nuanced and comprehensive testing regime.
Revised: This coupling complicates efforts to design a nuanced and comprehensive testing regime.

Original: Hence, simulating variation in a controlled manner via reliability tests can be a complementary method of evaluating the system’s out-of-distribution generalization ability.
Revised: Hence, simulating variation in a controlled manner via reliability tests can be a complementary method of evaluating the system’s out-of-distribution generalization ability.

Original:  Adversarial Attacks as Reliability Tests
Revised:  Adversarial Attacks as Reliability Tests

Original: We ﬁrst give a brief introduction to adversarial attacks in NLP before showing how they can be used for reliability testing.
Revised: We ﬁrst give a brief introduction to adversarial attacks in NLP before showing how they can be used for reliability testing.

Original: We refer the reader to
Revised: We refer the reader to

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020b)
Revised: (2020b)

Original: for a comprehensive survey.
Revised: for a comprehensive survey.

Original: Existing work on NLP adversarial attacks perturbs the input at various levels of linguistic analysis: phonology (Eger and Benz, 2020), orthography (Ebrahimi et al., 2018), morphology (Tan et al., 2020), lexicon (Alzantot et al., 2018; Jin et al., 2020), and syntax (Iyyer et al., 2018).
Revised: Existing work on NLP adversarial attacks perturbs the input at various levels of linguistic analysis: phonology (Eger and Benz, 2020), orthography (Ebrahimi et al., 2018), morphology (Tan et al., 2020), lexicon (Alzantot et al., 2018; Jin et al., 2020), and syntax (Iyyer et al., 2018).

Original: Early work did not place any constraints on the attacks and merely used the degradation to a target model’s accuracy as the measure of success.
Revised: Early work did not place any constraints on the attacks and merely used the degradation to a target model’s accuracy as the measure of success.

Original: However, this often resulted in the semantics and expected prediction changing, leading to an over-
Revised: However, this often resulted in the semantics and expected prediction changing, leading to an over-

Original: Require:
Revised: Require:

Original: Data distribution D d = {X , Y} modeling the di- mension of interest d , NLP system M , Source dataset
Revised: Data distribution D d = {X , Y} modeling the di- mension of interest d , NLP system M , Source dataset

Original: X ∼ X , Desired labels Y (cid:48) ∼ Y , Scoring function S .
Revised: X ∼ X , Desired labels Y (cid:48) ∼ Y , Scoring function S .

Original: Ensure: Average- or worst-case examples X (cid:48) , Result r .
Revised: Ensure: Average- or worst-case examples X (cid:48) , Result r .

Original: 1: X (cid:48) ← { ∅ } , r ←2: for x, y (cid:48) in X, Y (cid:48) do 3:
Revised: 1: X (cid:48) ← { ∅ } , r ←2: for x, y (cid:48) in X, Y (cid:48) do 3:

Original: C ← S AMPLE C ANDIDATES ( X ) 4: switch TestType do 5: case AverageCaseTest 6: s ← M EAN ( S ( y (cid:48) , M ( C ))) 7:
Revised: C ← S AMPLE C ANDIDATES ( X ) 4: switch TestType do 5: case AverageCaseTest 6: s ← M EAN ( S ( y (cid:48) , M ( C ))) 7:

Original: X (cid:48) ← X (cid:48) ∪ C 8: case WorstCaseTest 9: x (cid:48) , s ← arg min x c ∈ C S ( y (cid:48) , M ( x c )) 10:
Revised: X (cid:48) ← X (cid:48) ∪ C 8: case WorstCaseTest 9: x (cid:48) , s ← arg min x c ∈ C S ( y (cid:48) , M ( x c )) 10:

Original: X (cid:48) ← X (cid:48) ∪ { x (cid:48) } 11: r ← r + s 12: end for 13: r ← r | X | 14: return X (cid:48) , r estimation of the attack’s success.
Revised: X (cid:48) ← X (cid:48) ∪ { x (cid:48) } 11: r ← r + s 12: end for 13: r ← r | X | 14: return X (cid:48) , r estimation of the attack’s success.

Original: Recent attacks aim to preserve the original input’s semantics.
Revised: Recent attacks aim to preserve the original input’s semantics.

Original: A popular approach has been to substitute words with their synonyms using word embeddings or a language model as a measure of semantic similarity (Alzantot et al., 2018;
Revised: A popular approach has been to substitute words with their synonyms using word embeddings or a language model as a measure of semantic similarity (Alzantot et al., 2018;

Original: Ribeiro et al., 2018; Michel et al., 2019; Ren et al., 2019; Zhang et al., 2019a;
Revised: Ribeiro et al., 2018; Michel et al., 2019; Ren et al., 2019; Zhang et al., 2019a;

Original: Li et al., 2019; Jin et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2020a).
Revised: Li et al., 2019; Jin et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2020a).

Original: Focusing on maximally degrading model accuracy overlooks the key feature of adversarial attacks: the ability to ﬁnd the worst-case example for a model from an arbitrary distribution.
Revised: Focusing on maximally degrading model accuracy overlooks the key feature of adversarial attacks: the ability to ﬁnd the worst-case example for a model from an arbitrary distribution.

Original: Many recent attacks perturb the input across multiple dimensions at once, which may make the result unnatural.
Revised: Many recent attacks perturb the input across multiple dimensions at once, which may make the result unnatural.

Original: By constraining our sample perturbations to a distribution modeling a speciﬁc dimension of interest, the performance on the generated adversaries is a valid lower bound performance for that dimension.
Revised: By constraining our sample perturbations to a distribution modeling a speciﬁc dimension of interest, the performance on the generated adversaries is a valid lower bound performance for that dimension.

Original: Said another way, adversarial attacks can be reframed as interpretable reliability tests if we constrain them to meaningful distributions.
Revised: Said another way, adversarial attacks can be reframed as interpretable reliability tests if we constrain them to meaningful distributions.

Original: This is the key element of our approach as detailed in Alg. 1.
Revised: This is the key element of our approach as detailed in Alg. 1.

Original: We specify either an average (Lines 5–7) or worse case test (Lines 8–10), but conditioned on the data distribution D that models a particular dimension of interest d .
Revised: We specify either an average (Lines 5–7) or worse case test (Lines 8–10), but conditioned on the data distribution D that models a particular dimension of interest d .

Original: The resultant reliability score gauges real-world performance and the worst-case variant returns the adversarial examples that cause worst-case performance.
Revised: The resultant reliability score gauges real-world performance and the worst-case variant returns the adversarial examples that cause worst-case performance.

Original: When invariance to input variation is expected, y (cid:48) is equivalent to the source label y .
Revised: When invariance to input variation is expected, y (cid:48) is equivalent to the source label y .

Original: Note that by ignoring the average-case test logic and removing d , we recover the general adversarial attack algorithm.
Revised: Note that by ignoring the average-case test logic and removing d , we recover the general adversarial attack algorithm.

Original: However, the key difference between an adversarial robustness mindset and a testing one is the latter’s emphasis on identifying ways in which natural phenomena or ethical concerns can be operationalized as reliability tests.
Revised: However, the key difference between an adversarial robustness mindset and a testing one is the latter’s emphasis on identifying ways in which natural phenomena or ethical concerns can be operationalized as reliability tests.

Original: This change in perspective opens up new avenues for interdisciplinary research that will allow researchers and practitioners to have more nuanced discussions about model reliability and can be used to design comprehensive reliability testing regimes.
Revised: This change in perspective opens up new avenues for interdisciplinary research that will allow researchers and practitioners to have more nuanced discussions about model reliability and can be used to design comprehensive reliability testing regimes.

Original: We describe such a framework for interdisciplinary collaboration next.
Revised: We describe such a framework for interdisciplinary collaboration next.

Original:  A Framework for Reliability Testing
Revised:  A Framework for Reliability Testing

Original: We introduce and then describe our general framework, DOCTOR, for testing the reliability of NLP systems.
Revised: We introduce and then describe our general framework, DOCTOR, for testing the reliability of NLP systems.

Original: DOCTOR comprises six steps:
Revised: DOCTOR comprises six steps:

Original: D eﬁne reliability requirements 2.
Revised: D eﬁne reliability requirements 2.

Original: O perationalize dimensions as distributions 3.
Revised: O perationalize dimensions as distributions 3.

Original: C onstruct tests 4.
Revised: C onstruct tests 4.

Original: T est system and report results 5.
Revised: T est system and report results 5.

Original: O bserve deployed system’s behavior 6.
Revised: O bserve deployed system’s behavior 6.

Original: R eﬁne reliability requirements and tests Deﬁning reliability requirements.
Revised: R eﬁne reliability requirements and tests Deﬁning reliability requirements.

Original: Before any tests are constructed, experts and stakeholder advocates should work together to understand the demographics and values of the communities the NLP system will interact with (Friedman and Hendry, 2019) and the system’s impact on their lives.
Revised: Before any tests are constructed, experts and stakeholder advocates should work together to understand the demographics and values of the communities the NLP system will interact with (Friedman and Hendry, 2019) and the system’s impact on their lives.

Original: The latter is also known as algorithmic risk assessment (Ada Lovelace Institute and DataKind UK, 2021).
Revised: The latter is also known as algorithmic risk assessment (Ada Lovelace Institute and DataKind UK, 2021).

Original: There are three critical questions to address: 1)
Revised: There are three critical questions to address: 1)

Original: Along what dimensions should the model be tested? 2) What metrics should be used to measure system performance? 3) What are acceptable performance thresholds for each dimension?
Revised: Along what dimensions should the model be tested? 2) What metrics should be used to measure system performance? 3) What are acceptable performance thresholds for each dimension?

Original: Question 1 can be further broken down into: a) general linguistic phenomena, such as alternative spellings or code-mixing; b) task-speciﬁc quirks, e.g., an essay grading system should not use text length to predict score; c) sensitive attributes, such as gender, ethnicity, sexual orientation, age, or disability status.
Revised: Question 1 can be further broken down into: a) general linguistic phenomena, such as alternative spellings or code-mixing; b) task-speciﬁc quirks, e.g., an essay grading system should not use text length to predict score; c) sensitive attributes, such as gender, ethnicity, sexual orientation, age, or disability status.

Original: This presents an opportunity for interdisciplinary expert collaboration: Linguists are best equipped to contribute to discussions around (a), domain experts to (b), and ethicists and social scientists to (c).
Revised: This presents an opportunity for interdisciplinary expert collaboration: Linguists are best equipped to contribute to discussions around (a), domain experts to (b), and ethicists and social scientists to (c).

Original: However, we recognize that such collaboration may not be feasible for every NLP system being tested.
Revised: However, we recognize that such collaboration may not be feasible for every NLP system being tested.

Original: It is more realistic to expect ethicists to be involved when applying DOCTOR at the company and industry levels, and ethics-trained
Revised: It is more realistic to expect ethicists to be involved when applying DOCTOR at the company and industry levels, and ethics-trained

Original: NLP practitioners to answer these questions within the development team.
Revised: NLP practitioners to answer these questions within the development team.

Original: We provide a taxonomy of potential dimensions in Table 1 (Appendix).
Revised: We provide a taxonomy of potential dimensions in Table 1 (Appendix).

Original: Since it is likely unfeasible to test every possible dimension, stakeholder advocates should be involved to ensure their values and interests are accurately represented and prioritized (Hagerty and
Revised: Since it is likely unfeasible to test every possible dimension, stakeholder advocates should be involved to ensure their values and interests are accurately represented and prioritized (Hagerty and

Original: Rubinov, 2019), while experts should ensure the dimensions identiﬁed can be feasibly tested.
Revised: Rubinov, 2019), while experts should ensure the dimensions identiﬁed can be feasibly tested.

Original: A similar approach to that of community juries 5 may be taken.
Revised: A similar approach to that of community juries 5 may be taken.

Original: We recommend using this question to evaluate the feasibility of operationalizing potential dimensions: “What is the system’s performance when exposed to variation in d ?”
Revised: We recommend using this question to evaluate the feasibility of operationalizing potential dimensions: “What is the system’s performance when exposed to variation in d ?”

Original: where d is the dimension.
Revised: where d is the dimension.

Original: For example, rather than simply “gender”, a better-deﬁned dimension would be “gender pronouns”.
Revised: For example, rather than simply “gender”, a better-deﬁned dimension would be “gender pronouns”.

Original: With this understanding, experts and policymakers can then create a set of reliability requirements , comprising the testing dimensions, performance metric(s), and passing thresholds.
Revised: With this understanding, experts and policymakers can then create a set of reliability requirements , comprising the testing dimensions, performance metric(s), and passing thresholds.

Original: Next, we recommend using the same metrics for held-out, average-case, and worst-case performance for easy comparison.
Revised: Next, we recommend using the same metrics for held-out, average-case, and worst-case performance for easy comparison.

Original: These often vary from task to task and are still a subject of active research (Novikova et al., 2017; Reiter, 2018;
Revised: These often vary from task to task and are still a subject of active research (Novikova et al., 2017; Reiter, 2018;

Original: Kryscinski et al., 2019), hence the question of the right metric to use is beyond the scope of this paper.
Revised: Kryscinski et al., 2019), hence the question of the right metric to use is beyond the scope of this paper.

Original: Finally, ethicists, in consultation with the other aforementioned experts and stakeholders, will determine acceptable thresholds for worst-case performance.
Revised: Finally, ethicists, in consultation with the other aforementioned experts and stakeholders, will determine acceptable thresholds for worst-case performance.

Original: The system under test must perform above said thresholds when exposed to variation along those dimensions in order to pass.
Revised: The system under test must perform above said thresholds when exposed to variation along those dimensions in order to pass.

Original: For worst-case performance, we recommend reporting thresholds as relative differences ( δ ) between the average-case and worst-case performance.
Revised: For worst-case performance, we recommend reporting thresholds as relative differences ( δ ) between the average-case and worst-case performance.

Original: These questions may help in applying this step and deciding if speciﬁc NLP solutions should even exist (Leins et al., 2020):
Revised: These questions may help in applying this step and deciding if speciﬁc NLP solutions should even exist (Leins et al., 2020):

Original: • Who will interact with the NLP system, in what context, and using which language varieties?
Revised: • Who will interact with the NLP system, in what context, and using which language varieties?

Original: • What are the distinguishing features of these varieties compared to those used for training?
Revised: • What are the distinguishing features of these varieties compared to those used for training?

Original: • What is the (short- and long-term) impact on the community’s most underrepresented members if the system performs more poorly for them?
Revised: • What is the (short- and long-term) impact on the community’s most underrepresented members if the system performs more poorly for them?

Original: We note that our framework is general enough to be applied at various levels of organization: within the development team, within the company (compliance team, internal auditor), and within the industry (self-regulation or independent regulator).
Revised: We note that our framework is general enough to be applied at various levels of organization: within the development team, within the company (compliance team, internal auditor), and within the industry (self-regulation or independent regulator).

Original: However, we expect the exact set of dimensions, metrics and acceptable thresholds deﬁned in Stepto vary depending on the reliability concerns of the actors at each level.
Revised: However, we expect the exact set of dimensions, metrics and acceptable thresholds deﬁned in Stepto vary depending on the reliability concerns of the actors at each level.

Original: For example, independent regulators will be most concerned with establishing minimum safety and fairness standards that all NLP systems used in their industries must meet, while compliance teams may wish to have stricter and more comprehensive standards for brand reasons.
Revised: For example, independent regulators will be most concerned with establishing minimum safety and fairness standards that all NLP systems used in their industries must meet, while compliance teams may wish to have stricter and more comprehensive standards for brand reasons.

Original: Developers can use DOCTOR to meet the other two levels of requirements and understand their system’s behaviour better with targeted testing.
Revised: Developers can use DOCTOR to meet the other two levels of requirements and understand their system’s behaviour better with targeted testing.

Original: Operationalizing dimensions.
Revised: Operationalizing dimensions.

Original: While the abstractness of dimensions allows people who are not NLP practitioners to participate in drafting the set of reliability requirements, there is no way to test NLP systems using fuzzy concepts.
Revised: While the abstractness of dimensions allows people who are not NLP practitioners to participate in drafting the set of reliability requirements, there is no way to test NLP systems using fuzzy concepts.

Original: Therefore, every dimension the system is to be tested along must be operationalizable as a distribution from which perturbed examples can be sampled in order for NLP practitioners to realize them as tests.
Revised: Therefore, every dimension the system is to be tested along must be operationalizable as a distribution from which perturbed examples can be sampled in order for NLP practitioners to realize them as tests.

Original: Since average-case tests attempt to estimate a system’s expected performance in its deployed environment, the availability of datasets that reﬂect real-world distributions is paramount to ensure that the tests themselves are unbiased.
Revised: Since average-case tests attempt to estimate a system’s expected performance in its deployed environment, the availability of datasets that reﬂect real-world distributions is paramount to ensure that the tests themselves are unbiased.

Original: This is less of an issue for worst-case tests; the tests only needs to know which perturbations that are possible, but not how frequently they occur in the real world.
Revised: This is less of an issue for worst-case tests; the tests only needs to know which perturbations that are possible, but not how frequently they occur in the real world.

Original: Figuring out key dimensions for different classes of NLP tasks and exploring ways of operationalizing them as reliability tests are also promising directions for future research.
Revised: Figuring out key dimensions for different classes of NLP tasks and exploring ways of operationalizing them as reliability tests are also promising directions for future research.

Original: Such research would help NLP practitioners and policymakers deﬁne reliability requirements that can be feasibly implemented.
Revised: Such research would help NLP practitioners and policymakers deﬁne reliability requirements that can be feasibly implemented.

Original: Constructing tests.
Revised: Constructing tests.

Original: Next, average- and worstcase tests are constructed (Alg. 1).
Revised: Next, average- and worstcase tests are constructed (Alg. 1).

Original: Average-case tests can be data-driven and could take the form of manually curated datasets or model-based perturbation generation (e.g., PolyJuice (Wu et al., 2021)), while worst-case tests can be rule-based (e.g., Morpheus (Tan et al., 2020)) or model-based (e.g., BERT-Attack (Li et al., 2020a)).
Revised: Average-case tests can be data-driven and could take the form of manually curated datasets or model-based perturbation generation (e.g., PolyJuice (Wu et al., 2021)), while worst-case tests can be rule-based (e.g., Morpheus (Tan et al., 2020)) or model-based (e.g., BERT-Attack (Li et al., 2020a)).

Original: We recommend constructing tests that do not require access to the NLP model’s parameters (black-box assumption); this not only yields more system-agnostic tests, but also allows for (some) tests to be created independently from the system development team.
Revised: We recommend constructing tests that do not require access to the NLP model’s parameters (black-box assumption); this not only yields more system-agnostic tests, but also allows for (some) tests to be created independently from the system development team.

Original: If the black-box assumption proves limiting, the community can establish a standard set of items an NLP system should export for testing purposes, e.g., network gradients if the system uses a neural model.
Revised: If the black-box assumption proves limiting, the community can establish a standard set of items an NLP system should export for testing purposes, e.g., network gradients if the system uses a neural model.

Original: Regardless of assumption, keeping the regulators’ test implementations separate and hidden from the system developers is critical for stakeholders and regulators to trust the results.
Revised: Regardless of assumption, keeping the regulators’ test implementations separate and hidden from the system developers is critical for stakeholders and regulators to trust the results.

Original: This separation also reduces overﬁtting to the test suite.
Revised: This separation also reduces overﬁtting to the test suite.

Original: Testing systems.
Revised: Testing systems.

Original: A possible model for test ownership is to have independently implemented tests at the three levels of organization described above (team-level, company-level, industry-level).
Revised: A possible model for test ownership is to have independently implemented tests at the three levels of organization described above (team-level, company-level, industry-level).

Original: At the development team level, reliability tests would be used to diagnose weaknesses with the goal of improving the NLP system for a speciﬁc use case and set of target users.
Revised: At the development team level, reliability tests would be used to diagnose weaknesses with the goal of improving the NLP system for a speciﬁc use case and set of target users.

Original: Since the resulting worstcase examples are constrained along speciﬁc dimensions, comparing those that successfully induced model failure to the perturbed and source examples that did not may yield greater insights into a model’s inner workings compared to unconstrained adversarial examples.
Revised: Since the resulting worstcase examples are constrained along speciﬁc dimensions, comparing those that successfully induced model failure to the perturbed and source examples that did not may yield greater insights into a model’s inner workings compared to unconstrained adversarial examples.

Original: Studying how modiﬁcations (to the architecture, training data and process) affect the system’s reliability along each dimension will also give engineers insight into the factors affecting system reliability.
Revised: Studying how modiﬁcations (to the architecture, training data and process) affect the system’s reliability along each dimension will also give engineers insight into the factors affecting system reliability.

Original: These tests should be executed and updated regularly during system development according to standard software engineering best practices such as Agile (Beck et al., 2001).
Revised: These tests should be executed and updated regularly during system development according to standard software engineering best practices such as Agile (Beck et al., 2001).

Original: In a company, red teams are internal teams tasked with ﬁnding security vulnerabilities in computer software or systems developed by the company.
Revised: In a company, red teams are internal teams tasked with ﬁnding security vulnerabilities in computer software or systems developed by the company.

Original: Brundage et al.
Revised: Brundage et al.

Original: (2020) propose to apply the concept of red teaming to surface ﬂaws in an AI system’s safety and security.
Revised: (2020) propose to apply the concept of red teaming to surface ﬂaws in an AI system’s safety and security.

Original: In companies that maintain multiple NLP systems, we propose employing similar specialized teams to build and maintain reliability tests that ensure their NLP systems adhere to company-level reliability standards.
Revised: In companies that maintain multiple NLP systems, we propose employing similar specialized teams to build and maintain reliability tests that ensure their NLP systems adhere to company-level reliability standards.

Original: These tests will be more general than the ones developed by engineering teams due to their wider scope, while the reliability standards may be created and maintained by compliance teams or the red teams themselves.
Revised: These tests will be more general than the ones developed by engineering teams due to their wider scope, while the reliability standards may be created and maintained by compliance teams or the red teams themselves.

Original: Making these standards available for public scrutiny and ensuring their products meet them will enable companies to build trust with their users.
Revised: Making these standards available for public scrutiny and ensuring their products meet them will enable companies to build trust with their users.

Original: To ensure all NLP systems meet company-level reliability standards, these reliability tests should be executed as a part of regular internal audits (Raji et al., 2020), investigative audits after an incident, and before major releases (especially if it is the system’s ﬁrst release or if the architecture or training data received a major update).
Revised: To ensure all NLP systems meet company-level reliability standards, these reliability tests should be executed as a part of regular internal audits (Raji et al., 2020), investigative audits after an incident, and before major releases (especially if it is the system’s ﬁrst release or if the architecture or training data received a major update).

Original: They may also be regularly executed on randomly chosen production systems and trigger an alert upon failure.
Revised: They may also be regularly executed on randomly chosen production systems and trigger an alert upon failure.

Original: At the independent regulator level, reliability tests would likely be carried out during product certiﬁcation (e.g., ANSI/ISO certiﬁcation) and external audits.
Revised: At the independent regulator level, reliability tests would likely be carried out during product certiﬁcation (e.g., ANSI/ISO certiﬁcation) and external audits.

Original: These industry-level reliability standards and tests may be developed in a similar manner to the company-level ones.
Revised: These industry-level reliability standards and tests may be developed in a similar manner to the company-level ones.

Original: However, we expect them to be more general and less comprehensive than the latter, analogous to minimum safety standards such as IEC 60335-1 (IEC, 2020).
Revised: However, we expect them to be more general and less comprehensive than the latter, analogous to minimum safety standards such as IEC 60335-1 (IEC, 2020).

Original: Naturally, high risk applications and NLP systems used in regulated industries should comply with more stringent requirements (European Commission, 2021).
Revised: Naturally, high risk applications and NLP systems used in regulated industries should comply with more stringent requirements (European Commission, 2021).

Original: Our proposed framework is also highly compatible with the use of model cards (Mitchell et al., 2019) for auditing and transparent reporting (Raji et al., 2020).
Revised: Our proposed framework is also highly compatible with the use of model cards (Mitchell et al., 2019) for auditing and transparent reporting (Raji et al., 2020).

Original: In addition to performance on taskrelated metrics, model cards surface information and assumptions about a machine learning system and training process that may not be readily available otherwise.
Revised: In addition to performance on taskrelated metrics, model cards surface information and assumptions about a machine learning system and training process that may not be readily available otherwise.

Original: When a system has passed all tests and is ready to be deployed, its average and worstcase performance on all tested dimensions can be included as an extra section on the accompanying model card.
Revised: When a system has passed all tests and is ready to be deployed, its average and worstcase performance on all tested dimensions can be included as an extra section on the accompanying model card.

Original: In addition, the perturbed examples generated during testing and their labels ( x (cid:48) , y (cid:48) ) can be stored for audit purposes or examined to ensure that the tests are performing as expected.
Revised: In addition, the perturbed examples generated during testing and their labels ( x (cid:48) , y (cid:48) ) can be stored for audit purposes or examined to ensure that the tests are performing as expected.

Original: Observing and Reﬁning requirements.
Revised: Observing and Reﬁning requirements.

Original: Since it is unlikely for teams to get everything right initially, it is crucial to regularly monitor the systems’ impact post-launch and add, update, or re-prioritize dimensions and thresholds accordingly.
Revised: Since it is unlikely for teams to get everything right initially, it is crucial to regularly monitor the systems’ impact post-launch and add, update, or re-prioritize dimensions and thresholds accordingly.

Original: Monitoring for large scale deployments can be done via community juries, in which stakeholders who will be likely impacted (or their advocates) give feedback on the pain points and raise concerns about potential negative effects.
Revised: Monitoring for large scale deployments can be done via community juries, in which stakeholders who will be likely impacted (or their advocates) give feedback on the pain points and raise concerns about potential negative effects.

Original: Smaller teams without the resources to organize community juries can set up avenues for affected stakeholders to give feedback, raise concerns, and seek remediation.
Revised: Smaller teams without the resources to organize community juries can set up avenues for affected stakeholders to give feedback, raise concerns, and seek remediation.

Original:  From Concerns to Dimensions
Revised:  From Concerns to Dimensions

Original: We now illustrate how reliability concerns can be converted into concrete testing dimensions (Step 1) by considering the scenario of applying automated text scoring to short answers and essays from students in the multilingual population of Singapore.
Revised: We now illustrate how reliability concerns can be converted into concrete testing dimensions (Step 1) by considering the scenario of applying automated text scoring to short answers and essays from students in the multilingual population of Singapore.

Original: We study a second scenario in Appendix A. Automated Text Scoring (ATS) systems are increasingly used to grade tests and essays (Markoff, 2013;
Revised: We study a second scenario in Appendix A. Automated Text Scoring (ATS) systems are increasingly used to grade tests and essays (Markoff, 2013;

Original: Feathers, 2019).
Revised: Feathers, 2019).

Original: While they can provide instant feedback and help teachers and test agencies cope with large loads, studies have shown that they often exhibit demographic and language biases, such as scoring African-
Revised: While they can provide instant feedback and help teachers and test agencies cope with large loads, studies have shown that they often exhibit demographic and language biases, such as scoring African-

Original: and Indian-American males lower on the GRE Argument task compared to human graders (Bridgeman et al., 2012;
Revised: and Indian-American males lower on the GRE Argument task compared to human graders (Bridgeman et al., 2012;

Original: Ramineni and
Revised: Ramineni and

Original: Williamson, 2018).
Revised: Williamson, 2018).

Original: Since the results of some tests will affect the futures of the test takers (Salaky, 2018), the scoring algorithms used must be sufﬁciently reliable.
Revised: Since the results of some tests will affect the futures of the test takers (Salaky, 2018), the scoring algorithms used must be sufﬁciently reliable.

Original: Hence, let us imagine that Singapore’s education ministry has decided to create a standard set of reliability requirements that all ATS systems used in education must adhere to.
Revised: Hence, let us imagine that Singapore’s education ministry has decided to create a standard set of reliability requirements that all ATS systems used in education must adhere to.

Original: Linguistic landscape.
Revised: Linguistic landscape.

Original: A mix of language varieties are used in Singapore: a prestige English variety, a colloquial English variety, three other ofﬁcial languages (Chinese, Malay, and Tamil), and a large number of other languages.
Revised: A mix of language varieties are used in Singapore: a prestige English variety, a colloquial English variety, three other ofﬁcial languages (Chinese, Malay, and Tamil), and a large number of other languages.

Original: English is the lingua franca , with ﬂuency in the prestige variety correlating with socioeconomic status (Vaish and Tan, 2008).
Revised: English is the lingua franca , with ﬂuency in the prestige variety correlating with socioeconomic status (Vaish and Tan, 2008).

Original: A signiﬁcant portion of the population does not speak English at home.
Revised: A signiﬁcant portion of the population does not speak English at home.

Original: Subjects other than languages are taught in English.
Revised: Subjects other than languages are taught in English.

Original: Stakeholder impact.
Revised: Stakeholder impact.

Original: The key stakeholders affected by ATS systems would be students in schools and universities.
Revised: The key stakeholders affected by ATS systems would be students in schools and universities.

Original: The consequences of lower scores could be life-altering for the student who is unable to enroll in the major of their choice.
Revised: The consequences of lower scores could be life-altering for the student who is unable to enroll in the major of their choice.

Original: At the population level, biases in an ATS system trained on normally sampled data would unfairly discriminate against already underrepresented groups.
Revised: At the population level, biases in an ATS system trained on normally sampled data would unfairly discriminate against already underrepresented groups.

Original: Additionally, biases against disﬂuent or ungrammatical text when they are not the tested attributes would result in discrimination against students with a lower socioeconomic status or for whom English is a second language.
Revised: Additionally, biases against disﬂuent or ungrammatical text when they are not the tested attributes would result in discrimination against students with a lower socioeconomic status or for whom English is a second language.

Original: Finally, NLP systems have also been known to be overly sensitive to alternative spellings (Belinkov and Bisk, 2018).
Revised: Finally, NLP systems have also been known to be overly sensitive to alternative spellings (Belinkov and Bisk, 2018).

Original: When used to score subject tests, this could result in the ATS system unfairly penalizing dyslexic students (Coleman et al., 2009).
Revised: When used to score subject tests, this could result in the ATS system unfairly penalizing dyslexic students (Coleman et al., 2009).

Original: Since education is often credited with enabling social mobility, 6 unfair grading may perpetuate sys- temic discrimination and increase social inequality.
Revised: Since education is often credited with enabling social mobility, 6 unfair grading may perpetuate sys- temic discrimination and increase social inequality.

Original: Dimension.
Revised: Dimension.

Original: We can generally categorize written tests into those that test for content correctness (e.g., essay questions in a history test), and those that test for language skills (e.g., proper use of grammar).
Revised: We can generally categorize written tests into those that test for content correctness (e.g., essay questions in a history test), and those that test for language skills (e.g., proper use of grammar).

Original: While there are tests that simultaneously assess both aspects, modern ATS systems often grade them separately (Ke and Ng, 2019) and we treat each aspect as a separate test here.
Revised: While there are tests that simultaneously assess both aspects, modern ATS systems often grade them separately (Ke and Ng, 2019) and we treat each aspect as a separate test here.

Original: When grading students on content correctness, we would expect the ATS system to ignore linguistic variation and sensitive attributes as long as they do not affect the answer’s validity.
Revised: When grading students on content correctness, we would expect the ATS system to ignore linguistic variation and sensitive attributes as long as they do not affect the answer’s validity.

Original: Hence, we would expect variation in these dimensions to have no effect on scores: answer length, language/vocabulary simplicity, alternative spellings/misspellings of non-keywords, grammatical variation, syntactic variation (especially those resembling transfer from a ﬁrst language), and proxies for sensitive attributes.
Revised: Hence, we would expect variation in these dimensions to have no effect on scores: answer length, language/vocabulary simplicity, alternative spellings/misspellings of non-keywords, grammatical variation, syntactic variation (especially those resembling transfer from a ﬁrst language), and proxies for sensitive attributes.

Original: On the other hand, we would expect the system to be able to differentiate proper answers from those trying to game the test with gibberish (Chin, 2020; Ding et al., 2020).
Revised: On the other hand, we would expect the system to be able to differentiate proper answers from those trying to game the test with gibberish (Chin, 2020; Ding et al., 2020).

Original: When grading students on language skills, however, we would expect ATS systems to be only sensitive to the relevant skill.
Revised: When grading students on language skills, however, we would expect ATS systems to be only sensitive to the relevant skill.

Original: For example, when assessing grammar use, we would expect the system to be sensitive to grammatical errors (from the perspective of the language variety the student is expected to use), but not to the other dimensions mentioned above (e.g., misspellings).
Revised: For example, when assessing grammar use, we would expect the system to be sensitive to grammatical errors (from the perspective of the language variety the student is expected to use), but not to the other dimensions mentioned above (e.g., misspellings).

Original: Actors.
Revised: Actors.

Original: Relevant experts include teachers of the subjects the ATS systems will be used in, linguists, and computer scientists.
Revised: Relevant experts include teachers of the subjects the ATS systems will be used in, linguists, and computer scientists.

Original: The stakeholders (students) may be represented by their student unions (at the university level) or focus groups comprising a representative sample of the student population.
Revised: The stakeholders (students) may be represented by their student unions (at the university level) or focus groups comprising a representative sample of the student population.

Original:  Implications for Policy
Revised:  Implications for Policy

Original: As we alluded to in §3, there is a mounting effort to increase accountability and transparency around the development and use of NLP systems to prevent them from amplifying societal biases.
Revised: As we alluded to in §3, there is a mounting effort to increase accountability and transparency around the development and use of NLP systems to prevent them from amplifying societal biases.

Original: DOCTOR is highly complementary to the model card approach increasingly adopted 7 to surface oft hidden details about NLP models: Developers simply need to list the tested dimensions, metrics, and score on each dimension in the model card.
Revised: DOCTOR is highly complementary to the model card approach increasingly adopted 7 to surface oft hidden details about NLP models: Developers simply need to list the tested dimensions, metrics, and score on each dimension in the model card.

Original: Crucially, reliability tests can be used to highlight fairness issues in NLP systems by including sensitive attributes for the target population, but it is paramount these requirements reﬂect local concerns rather than any prescriptivist perspective (Sambasivan et al., 2021).
Revised: Crucially, reliability tests can be used to highlight fairness issues in NLP systems by including sensitive attributes for the target population, but it is paramount these requirements reﬂect local concerns rather than any prescriptivist perspective (Sambasivan et al., 2021).

Original: At the same time, the ability to conduct quantitative, targeted reliability testing along speciﬁable dimensions paves the way for reliability standards to be established, with varying levels of stringency and rigor for different use cases and industries.
Revised: At the same time, the ability to conduct quantitative, targeted reliability testing along speciﬁable dimensions paves the way for reliability standards to be established, with varying levels of stringency and rigor for different use cases and industries.

Original: We envision minimum safety and fairness standards being established for applications that are non-sensitive, not safety-critical, and used in unregulated industries, analogous to standards for household appliances.
Revised: We envision minimum safety and fairness standards being established for applications that are non-sensitive, not safety-critical, and used in unregulated industries, analogous to standards for household appliances.

Original: Naturally, applications at greater risks (Li et al., 2020b) of causing harm upon failure should be held to stricter standards.
Revised: Naturally, applications at greater risks (Li et al., 2020b) of causing harm upon failure should be held to stricter standards.

Original: Policymakers are starting to propose and implement regulations to enforce transparency and accountability in the use of AI systems.
Revised: Policymakers are starting to propose and implement regulations to enforce transparency and accountability in the use of AI systems.

Original: For example, the European
Revised: For example, the European

Original: Union’s General Data Protection Regulation grants data subjects the right to obtain “meaningful information about the logic involved” in automated decision systems (EU, 2016).
Revised: Union’s General Data Protection Regulation grants data subjects the right to obtain “meaningful information about the logic involved” in automated decision systems (EU, 2016).

Original: The EU are developing additional, AI-speciﬁc regulation (European
Revised: The EU are developing additional, AI-speciﬁc regulation (European

Original: Commission, 2020) for example, requiring developers of high-risk AI systems to report their “capabilities and limitations, ...
Revised: Commission, 2020) for example, requiring developers of high-risk AI systems to report their “capabilities and limitations, ...

Original: [and] the conditions under which they can be expected to function as intended”.
Revised: [and] the conditions under which they can be expected to function as intended”.

Original: In the U.S., a proposed bill will require
Revised: In the U.S., a proposed bill will require

Original: Washington State’s public agencies to submit a report describing “any potential impacts of the automated decision system on civil rights and liberties and potential disparate impacts on marginalized communities” before using automated decision systems (Washington State Legislature, 2021).
Revised: Washington State’s public agencies to submit a report describing “any potential impacts of the automated decision system on civil rights and liberties and potential disparate impacts on marginalized communities” before using automated decision systems (Washington State Legislature, 2021).

Original: One may note that language in the proposed regulation is intentionally vague.
Revised: One may note that language in the proposed regulation is intentionally vague.

Original: There are many ways to measure bias and fairness depending on the type of model, context of use, and goal of the system.
Revised: There are many ways to measure bias and fairness depending on the type of model, context of use, and goal of the system.

Original: Today, companies developing AI systems employ the deﬁnitions they believe most reasonable (or perhaps easiest to implement) but regulation will need to be more speciﬁc for there to be meaningful compliance.
Revised: Today, companies developing AI systems employ the deﬁnitions they believe most reasonable (or perhaps easiest to implement) but regulation will need to be more speciﬁc for there to be meaningful compliance.

Original: DOCTOR’s requirement to explicitly deﬁne speciﬁc dimensions instead of a vague notion of “reliability” will help policymakers in this regard, and can inform the ongoing development of national (NIST, 2019) and international standards 8 .
Revised: DOCTOR’s requirement to explicitly deﬁne speciﬁc dimensions instead of a vague notion of “reliability” will help policymakers in this regard, and can inform the ongoing development of national (NIST, 2019) and international standards 8 .

Original: While external algorithm audits are becoming popular, testing remains a challenge since companies wishing to protect their intellectual property may be resistant to sharing their code (Johnson, 2021), and implementing custom tests for each system is unscalable.
Revised: While external algorithm audits are becoming popular, testing remains a challenge since companies wishing to protect their intellectual property may be resistant to sharing their code (Johnson, 2021), and implementing custom tests for each system is unscalable.

Original: Our approach to reliability testing offers a potential solution to this conundrum by treating NLP systems as black-boxes.
Revised: Our approach to reliability testing offers a potential solution to this conundrum by treating NLP systems as black-boxes.

Original: If reliability tests become a legal requirement, regulatory authorities will be able to mandate independently conducted reliability tests for transparency.
Revised: If reliability tests become a legal requirement, regulatory authorities will be able to mandate independently conducted reliability tests for transparency.

Original: Such standards, combined with certiﬁcation programs (e.g., IEEE’s Ethics Certiﬁcation Program for Autonomous and Intelligent Systems 9 ), will further incentivize the development of responsible NLP, as the companies purchasing NLP systems will insist on certiﬁed systems to protect them from both legal and brand risk.
Revised: Such standards, combined with certiﬁcation programs (e.g., IEEE’s Ethics Certiﬁcation Program for Autonomous and Intelligent Systems 9 ), will further incentivize the development of responsible NLP, as the companies purchasing NLP systems will insist on certiﬁed systems to protect them from both legal and brand risk.

Original: To avoid confusion, we expect certiﬁcation to occur for individual NLP systems (e.g., an end-to-end question answering system for customer enquiries) rather than for general purpose language models that will be further trained to perform some speciﬁc NLP task.
Revised: To avoid confusion, we expect certiﬁcation to occur for individual NLP systems (e.g., an end-to-end question answering system for customer enquiries) rather than for general purpose language models that will be further trained to perform some speciﬁc NLP task.

Original: While concrete standards and certiﬁcation programs that can serve this purpose do not yet exist, we believe that they eventually will and hope our paper will inform their development.
Revised: While concrete standards and certiﬁcation programs that can serve this purpose do not yet exist, we believe that they eventually will and hope our paper will inform their development.

Original: This multi-pronged approach can help to mitigate NLP’s potential harms while increasing public trust in language technology.
Revised: This multi-pronged approach can help to mitigate NLP’s potential harms while increasing public trust in language technology.

Original:  Challenges and Future Directions
Revised:  Challenges and Future Directions

Original: While we believe our framework to be a useful starting point for introducing reliability testing in NLP systems, we observe some challenges that need to be overcome before it can become ubiquitous.
Revised: While we believe our framework to be a useful starting point for introducing reliability testing in NLP systems, we observe some challenges that need to be overcome before it can become ubiquitous.

Original: First, identifying and prioritizing the dimensions that can attest a system’s reliability and fairness.
Revised: First, identifying and prioritizing the dimensions that can attest a system’s reliability and fairness.

Original: The former is relatively straightforward and can be achieved via collaboration with experts (e.g., as part of NIST’s future AI standards (NIST, 2019)).
Revised: The former is relatively straightforward and can be achieved via collaboration with experts (e.g., as part of NIST’s future AI standards (NIST, 2019)).

Original: The latter, however, is a question of values and power (Noble, 2018; Mohamed et al., 2020; Leins et al., 2020), and should be addressed by having a code of ethics and ensuring that the stakeholders are adequately represented at the decision table.
Revised: The latter, however, is a question of values and power (Noble, 2018; Mohamed et al., 2020; Leins et al., 2020), and should be addressed by having a code of ethics and ensuring that the stakeholders are adequately represented at the decision table.

Original: Second, our proposed method of reliability testing may suffer from similar issues plaguing automatic evaluation metrics for natural language generation (Novikova et al., 2017; Reiter, 2018;
Revised: Second, our proposed method of reliability testing may suffer from similar issues plaguing automatic evaluation metrics for natural language generation (Novikova et al., 2017; Reiter, 2018;

Original: Kryscinski et al., 2019): due to the tests’ synthetic nature they may not fully capture the nuances of reality.
Revised: Kryscinski et al., 2019): due to the tests’ synthetic nature they may not fully capture the nuances of reality.

Original: For example, if a test’s objective were to test an NLP system’s reliability when interacting with African American English (AAE) speakers, would it be possible to guarantee (in practice) that all generated examples will fall within the distribution of text that is considered AAE?
Revised: For example, if a test’s objective were to test an NLP system’s reliability when interacting with African American English (AAE) speakers, would it be possible to guarantee (in practice) that all generated examples will fall within the distribution of text that is considered AAE?

Original: Potential research directions would be to design adversary generation techniques that can offer such guarantees or incorporate human feedback (Nguyen et al., 2017; Kreutzer et al., 2018; Stiennon et al., 2020).
Revised: Potential research directions would be to design adversary generation techniques that can offer such guarantees or incorporate human feedback (Nguyen et al., 2017; Kreutzer et al., 2018; Stiennon et al., 2020).

Original:  Conclusion
Revised:  Conclusion

Original: Once natural language technologies move out of the lab and start impacting real lives, concerns around safety, fairness, and accountability cease to be mere thought experiments.
Revised: Once natural language technologies move out of the lab and start impacting real lives, concerns around safety, fairness, and accountability cease to be mere thought experiments.

Original: While it is clear that
Revised: While it is clear that

Original: NLP can have a positive impact on our lives, from helping us type faster (autocomplete) to revitalizing endangered languages (Zhang et al., 2020a), there is also no denying its potential for perpetuating harmful stereotypes (Bolukbasi et al., 2016;
Revised: NLP can have a positive impact on our lives, from helping us type faster (autocomplete) to revitalizing endangered languages (Zhang et al., 2020a), there is also no denying its potential for perpetuating harmful stereotypes (Bolukbasi et al., 2016;

Original: Sap et al., 2019), performing disproportionately poorly for underrrepresented groups (Hern, 2017;
Revised: Sap et al., 2019), performing disproportionately poorly for underrrepresented groups (Hern, 2017;

Original: Bridgeman et al., 2012), and even erasing already marginalized communities (Bender et al., 2021).
Revised: Bridgeman et al., 2012), and even erasing already marginalized communities (Bender et al., 2021).

Original: Trust in our tools stems from an assurance that we will remain unharmed, even in the worst-case scenario.
Revised: Trust in our tools stems from an assurance that we will remain unharmed, even in the worst-case scenario.

Original: In many mature industries, this takes the form of reliability standards.
Revised: In many mature industries, this takes the form of reliability standards.

Original: However, for standards to be enacted and enforced, we must ﬁrst operationalize “reliability”.
Revised: However, for standards to be enacted and enforced, we must ﬁrst operationalize “reliability”.

Original: Hence in this paper, we argued for the need for reliability testing, especially worst-case testing, in NLP by contextualizing it among existing work on promoting accountability and improving generalization beyond the training distribution.
Revised: Hence in this paper, we argued for the need for reliability testing, especially worst-case testing, in NLP by contextualizing it among existing work on promoting accountability and improving generalization beyond the training distribution.

Original: Next, we showed how adversarial attacks can be reframed as worst-case tests.
Revised: Next, we showed how adversarial attacks can be reframed as worst-case tests.

Original: Finally, we proposed a possible paradigm, DOCTOR, for how reliability concerns can be realized as quantitative tests, and discussed how this framework can be used at different levels of organization or industry.
Revised: Finally, we proposed a possible paradigm, DOCTOR, for how reliability concerns can be realized as quantitative tests, and discussed how this framework can be used at different levels of organization or industry.

Original: Acknowledgements
Revised: Acknowledgements

Original: Samson is supported by Salesforce and Singapore’s
Revised: Samson is supported by Salesforce and Singapore’s

Original: Economic Development Board under the Industrial
Revised: Economic Development Board under the Industrial

Original: Postgraduate Programme.
Revised: Postgraduate Programme.

Original: Araz is grateful for the support provided by the Centre of Trusted Internet and Community, National University of Singapore through the project CTIC-RP-20-02.
Revised: Araz is grateful for the support provided by the Centre of Trusted Internet and Community, National University of Singapore through the project CTIC-RP-20-02.

Original: Broader Impact
Revised: Broader Impact

Original: Much like how we expect to not be exposed to harmful electric shocks when using electrical appliances, we should expect some minimum levels of safety and fairness for the NLP systems we interact with in our everyday lives.
Revised: Much like how we expect to not be exposed to harmful electric shocks when using electrical appliances, we should expect some minimum levels of safety and fairness for the NLP systems we interact with in our everyday lives.

Original: As mentioned in §1, §3, and §7, standards and regulations for AI systems are in the process of being developed for this purpose, especially for applications deemed “high-risk”, e.g., healthcare (European Commission, 2020).
Revised: As mentioned in §1, §3, and §7, standards and regulations for AI systems are in the process of being developed for this purpose, especially for applications deemed “high-risk”, e.g., healthcare (European Commission, 2020).

Original: Reliability testing, and our proposed framework, is one way to approach the problem of enacting enforceable standards and regulations.
Revised: Reliability testing, and our proposed framework, is one way to approach the problem of enacting enforceable standards and regulations.

Original: However, the ﬂip side of heavily regulating every single application of NLP is that it may slow down innovation.
Revised: However, the ﬂip side of heavily regulating every single application of NLP is that it may slow down innovation.

Original: Therefore, it is important that the level of regulation for a particular application is proportionate to its potential for harm (Daten
Revised: Therefore, it is important that the level of regulation for a particular application is proportionate to its potential for harm (Daten

Original: Ethik Kommission, 2019).
Revised: Ethik Kommission, 2019).

Original: Our framework can be adapted to different levels of risk by scaling down the implementation of some steps (e.g., the method and depth in which stakeholder consultation happens or the comprehensiveness of the set of testing dimensions) for low-risk applications.
Revised: Our framework can be adapted to different levels of risk by scaling down the implementation of some steps (e.g., the method and depth in which stakeholder consultation happens or the comprehensiveness of the set of testing dimensions) for low-risk applications.

Original: Finally, it is important to ensure that any tests, standards, or regulations developed adequately represents the needs of the most vulnerable stakeholders, instead of constructing them in a prescriptivist manner (Hagerty and Rubinov, 2019).
Revised: Finally, it is important to ensure that any tests, standards, or regulations developed adequately represents the needs of the most vulnerable stakeholders, instead of constructing them in a prescriptivist manner (Hagerty and Rubinov, 2019).

Original: Hence,
Revised: Hence,

Original: DOCTOR places a strong emphasis on involving stakeholder advocates and analyzing the impact of an application of NLP on the target community.
Revised: DOCTOR places a strong emphasis on involving stakeholder advocates and analyzing the impact of an application of NLP on the target community.

Original: Appendix A Testing Dimensions: Detecting Violent Content on Social Media
Revised: Appendix A Testing Dimensions: Detecting Violent Content on Social Media

Original: In this second case study, we apply DOCTOR for measuring the reliability of a violent content detection system for English social media posts.
Revised: In this second case study, we apply DOCTOR for measuring the reliability of a violent content detection system for English social media posts.

Original: Although we limit this discussion to the U.S., this is a growing, global problem (Laub, 2019) that can lead to deadly outcomes (Rajagopalan et al., 2018).
Revised: Although we limit this discussion to the U.S., this is a growing, global problem (Laub, 2019) that can lead to deadly outcomes (Rajagopalan et al., 2018).

Original: In this hypothetical use case, the system may automatically remove violent content or alert content moderators to potential violations of the social media company’s acceptable use policy.
Revised: In this hypothetical use case, the system may automatically remove violent content or alert content moderators to potential violations of the social media company’s acceptable use policy.

Original: Moderators can decide if speciﬁc content should be removed, and if necessary, notify law enforcement to avert pending violence (e.g., threats against individuals, planned violent events).
Revised: Moderators can decide if speciﬁc content should be removed, and if necessary, notify law enforcement to avert pending violence (e.g., threats against individuals, planned violent events).

Original: It should be noted that social media platforms have broad latitude (Klonick, 2018) to develop their own policies for acceptable content and how they handle it as a result of the 1996 Communications Decency Act 10 .
Revised: It should be noted that social media platforms have broad latitude (Klonick, 2018) to develop their own policies for acceptable content and how they handle it as a result of the 1996 Communications Decency Act 10 .

Original: In this scenario, the compliance ofﬁcer of the company developing the system is responsible for making sure it does not discriminate against speciﬁc user demographics.
Revised: In this scenario, the compliance ofﬁcer of the company developing the system is responsible for making sure it does not discriminate against speciﬁc user demographics.

Original: Research has shown that hate speech can lead to hateful actions (Marsters, 2019).
Revised: Research has shown that hate speech can lead to hateful actions (Marsters, 2019).

Original: In many cases, individuals posted their intents online prior to committing violence (Cohen et al., 2014).
Revised: In many cases, individuals posted their intents online prior to committing violence (Cohen et al., 2014).

Original: When identifying content to remove and especially when involving law enforcement, it is important distinguish between "Hunters" (those who act) and "Howlers" (those who do not).
Revised: When identifying content to remove and especially when involving law enforcement, it is important distinguish between "Hunters" (those who act) and "Howlers" (those who do not).

Original: This is to avoid wrongly detaining individuals have no intention of committing violence, even if their words are not defensible.
Revised: This is to avoid wrongly detaining individuals have no intention of committing violence, even if their words are not defensible.

Original: Between the two extremes, posters may harass, stalk, dox, or otherwise abuse victims from a distance, therefore it is still necessary to ﬂag, remove, and potentially track or document violent content.
Revised: Between the two extremes, posters may harass, stalk, dox, or otherwise abuse victims from a distance, therefore it is still necessary to ﬂag, remove, and potentially track or document violent content.

Original: Linguistic landscape.
Revised: Linguistic landscape.

Original: In this case study, we focus on solely on English speakers, but we acknowledge that the actual linguistic landscape is much more complex (over 350 languages).
Revised: In this case study, we focus on solely on English speakers, but we acknowledge that the actual linguistic landscape is much more complex (over 350 languages).

Original: Posters on social media may speak English as their ﬁrst language or as a second language and they often codeswitch and code-mix.
Revised: Posters on social media may speak English as their ﬁrst language or as a second language and they often codeswitch and code-mix.

Original: General American English is used for business purposes in the U.S. but there are other frequently used language varieties including African American English (AAE), Cajun Vernacu- lar English, and three different Latino (Hispanic) Vernacular Englishes.
Revised: Standard American English is used for business purposes in the U.S. but there are other frequently used language varieties including African American English (AAE), Cajun Ver- nacular English, and three different Latino (Hispanic) Vernacular Englishes.

Original: Stakeholder Impact.
Revised: Stakeholder Impact.

Original: The key stakeholders that will be impacted are those most often facing violent threats online: minorities, women, immigrants, and the LGBTQ community (Amnesty International, 2018; Ganesh, 2018;
Revised: The key stakeholders that will be impacted are those most often facing violent threats online: minorities, women, immigrants, and the LGBTQ community (Amnesty International, 2018; Ganesh, 2018;

Original: Davidson et al., 2019; Wakeﬁeld, 2020).
Revised: Davidson et al., 2019; Wakeﬁeld, 2020).

Original: Additionally, anyone that posts content on the social media site is a stakeholder.
Revised: Additionally, anyone that posts content on the social media site is a stakeholder.

Original: Unfortunately, the very communities that are often receiving violent posts, are often wrongly ﬂagged as posting toxic content themselves due to racial biases present in the training data (Sap et al., 2019;
Revised: Unfortunately, the very communities that are often receiving violent posts, are often wrongly ﬂagged as posting toxic content themselves due to racial biases present in the training data (Sap et al., 2019;

Original: Davidson et al., 2019).
Revised: Davidson et al., 2019).

Original: Given the risk of harm to victims if the system misses violent posts from hunters or misidentiﬁes content as violent and notiﬁes law enforcement, it is critical the right level of false positives and false negatives is achieved in ﬂagging content.
Revised: Given the risk of harm to victims if the system misses violent posts from hunters or misidentiﬁes content as violent and notiﬁes law enforcement, it is critical the right level of false positives and false negatives is achieved in ﬂagging content.

Original: Dimensions.
Revised: Dimensions.

Original: There are two tasks under consideration here: identifying violent content and identifying Hunters who “truly intend to use lethal violence” (Marsters, 2019).
Revised: There are two tasks under consideration here: identifying violent content and identifying Hunters who “truly intend to use lethal violence” (Marsters, 2019).

Original: In the ﬁrst task, the system is looking for content that negatively targets a socially deﬁned group.
Revised: In the ﬁrst task, the system is looking for content that negatively targets a socially deﬁned group.

Original: Additionally, the content includes not only hate speech (e.g., profanity, epithets, vulgarity) but also incites others to hatred or violence.
Revised: Additionally, the content includes not only hate speech (e.g., profanity, epithets, vulgarity) but also incites others to hatred or violence.

Original: Since content written in AAE has been shown to be ﬂagged as toxic more often (Sap et al., 2019; Davidson et al., 2019), we must ensure that the system is reliable when encountering dialectal variation.
Revised: Since content written in AAE has been shown to be ﬂagged as toxic more often (Sap et al., 2019; Davidson et al., 2019), we must ensure that the system is reliable when encountering dialectal variation.

Original: Additionally, due to the casual environment of social media, multilingual speakers often code-switch and code-mix.
Revised: Additionally, due to the casual environment of social media, multilingual speakers often code-switch and code-mix.

Original: Hence, we expect variation in these dimensions to have no effect on the system’s predictions: alternative spellings, morphosyntactic variation, word choice, code-mixing, idioms, and references to and manifestations of sensitive attributes and their proxies.
Revised: Hence, we expect variation in these dimensions to have no effect on the system’s predictions: alternative spellings, morphosyntactic variation, word choice, code-mixing, idioms, and references to and manifestations of sensitive attributes and their proxies.

Original: However, we must expect the system to be sensitive to in-group and out-group usage of reclaimed slurs so that the in-group usage does not result in a ﬂag while out-group usage result in ﬂagged posts.
Revised: However, we must expect the system to be sensitive to in-group and out-group usage of reclaimed slurs so that the in-group usage does not result in a ﬂag while out-group usage result in ﬂagged posts.

Original: When identifying hunters, we may expect the system to be sensitive to uses of ﬁrst person pronouns, certainty adverbs, negative evaluative adjectives, and modiﬁers (Marsters, 2019).
Revised: When identifying hunters, we may expect the system to be sensitive to uses of ﬁrst person pronouns, certainty adverbs, negative evaluative adjectives, and modiﬁers (Marsters, 2019).

Original: However, in order to avoid unfairly penalizing vernacular
Revised: However, in order to avoid unfairly penalizing vernacular

Original: English speakers we should expect the system’s predictions to be equally unaffected by variation in the dimensions listed for the other task.
Revised: English speakers we should expect the system’s predictions to be equally unaffected by variation in the dimensions listed for the other task.

Original: [Table] [Table]
Revised: [Table] [Table]