Original: Reliability Testing for Natural Language Processing Systems
Revised: Reliability Testing for Natural Language Processing Systems

Original: Abstract
Revised: Abstract

Original: Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems.
Revised: Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems.

Original: Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments?
Revised: Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments?

Original: To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability.
Revised: To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability.

Original: We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests.
Revised: We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests.

Original: We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.
Revised: We argue that reliability testing — with an emphasis on interdisciplinary collaboration — will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.

Original:  Introduction
Revised:  Introduction

Original: Rigorous testing is critical to ensuring a program works as intended (functionality) when used under real-world conditions (reliability).
Revised: Rigorous testing is critical to ensuring a program works as intended (functionality) when used under real-world conditions (reliability).

Original: Hence, it is troubling that while natural language technologies are becoming increasingly pervasive in our everyday lives, there is little assurance that these NLP systems will not fail catastrophically or amplify discrimination against minority demographics when exposed to input from outside the training distribution.
Revised: Hence, it is troubling that while natural language technologies are becoming increasingly pervasive in our everyday lives, there is little assurance that these NLP systems will not fail catastrophically or amplify discrimination against minority demographics when exposed to input from outside the training distribution.

Original: Recent examples include GPT-3 (Brown et al., 2020) agreeing with suggested suicide (Rousseau et al., 2020), the mistranslation of an innocuous social media post resulting in a minority’s arrest (Hern, 2017) and biased grading algorithms that can negatively impact a minority student’s future (Feathers, 2019).
Revised: Recent examples include GPT-3 (Brown et al., 2020) agreeing with suggested suicide (Rousseau et al., 2020), the mistranslation of an innocuous social media post resulting in a minority’s arrest (Hern, 2017) and biased grading algorithms that can negatively impact a minority student’s future (Feathers, 2019).

Original: Additionally, a lack of rigorous testing, coupled with machine learning’s
Revised: Additionally, a lack of rigorous testing, coupled with machine learning’s

Original: (ML) implicit assumption of identical training and testing distributions, may inadvertently result in systems that discriminate against minorities, who are often underrepresented in the training data.
Revised: (ML) implicit assumption of identical training and testing distributions, may inadvertently result in systems that discriminate against minorities, who are often underrepresented in the training data.

Original: This can take the form of misrepresentation of or poorer performance for people with disabilities, speciﬁc gender, ethnic, age, or linguistic groups (Hovy and Spruit, 2016; Crawford, 2017; Hutchinson et al., 2020).
Revised: This can take the form of misrepresentation of or poorer performance for people with disabilities, speciﬁc gender, ethnic, age, or linguistic groups (Hovy and Spruit, 2016; Crawford, 2017; Hutchinson et al., 2020).

Original: Amongst claims of NLP systems achieving human parity in challenging tasks such as question answering (Yu et al., 2018), machine translation (Hassan et al., 2018), and commonsense inference (Devlin et al., 2019), research has demonstrated these systems’ fragility to natural and adversarial noise (Goodfellow et al., 2015; Belinkov and Bisk, 2018) and out-of-distribution data (Fisch et al., 2019).
Revised: Amongst claims of NLP systems achieving human parity in challenging tasks such as question answering (Yu et al., 2018), machine translation (Hassan et al., 2018), and commonsense inference (Devlin et al., 2019), research has demonstrated these systems’ fragility to natural and adversarial noise (Goodfellow et al., 2015; Belinkov and Bisk, 2018) and out-of-distribution data (Fisch et al., 2019).

Original: It is also still common practice to equate “testing” with “measuring held-out accuracy”, even as datasets are revealed to be harmfully biased (Wagner et al., 2015; Geva et al., 2019; Sap et al., 2019).
Revised: It is also still common practice to equate “testing” with “measuring held-out accuracy”, even as datasets are revealed to be harmfully biased (Wagner et al., 2015; Geva et al., 2019; Sap et al., 2019).

Original: Many potential harms can be mitigated by detecting them early and preventing the offending model from being put into production.
Revised: Many potential harms can be mitigated by detecting them early and preventing the offending model from being put into production.

Original: Hence, in addition to being mindful of the biases in the NLP pipeline (Bender and Friedman, 2018; Mitchell et al., 2019;
Revised: Hence, in addition to being mindful of the biases in the NLP pipeline (Bender and Friedman, 2018; Mitchell et al., 2019;

Original: Waseem et al., 2021) and holding creators accountable via audits (Raji et al., 2020;
Revised: Waseem et al., 2021) and holding creators accountable via audits (Raji et al., 2020;

Original: Brundage et al., 2020), we argue for the need to evaluate an NLP system’s reliability in diverse operating conditions.
Revised: Brundage et al., 2020), we argue for the need to evaluate an NLP system’s reliability in diverse operating conditions.

Original: Initial research on evaluating out-of-distribution generalization involved manually designed challenge sets (Jia and Liang, 2017; Nie et al., 2020;
Revised: Initial research on evaluating out-of-distribution generalization involved manually designed challenge sets (Jia and Liang, 2017; Nie et al., 2020;

Original: Gardner et al., 2020), counterfactuals (Kaushik et al., 2019; Khashabi et al., 2020; Wu et al., 2021), biased sampling (Søgaard et al., 2021) or toolkits for testing if a system has speciﬁc capabilities (Ribeiro et al., 2020) or robustness to distribution shifts (Goel et al., 2021).
Revised: Gardner et al., 2020), counterfactuals (Kaushik et al., 2019; Khashabi et al., 2020; Wu et al., 2021), biased sampling (Søgaard et al., 2021) or toolkits for testing if a system has speciﬁc capabilities (Ribeiro et al., 2020) or robustness to distribution shifts (Goel et al., 2021).

Original: However, most of these approaches inevitably overestimate a given system’s worst-case performance since they do not mimic the NLP system’s adversarial distribution 1 .
Revised: However, most of these approaches inevitably overestimate a given system’s worst-case performance since they do not mimic the NLP system’s adversarial distribution 1 .

Original: A promising technique for evaluating worst-case performance is the adversarial attack.
Revised: A promising technique for evaluating worst-case performance is the adversarial attack.

Original: However, although some adversarial attacks explicitly focus on speciﬁc linguistic levels of analysis (Belinkov and
Revised: However, although some adversarial attacks explicitly focus on speciﬁc linguistic levels of analysis (Belinkov and

Original: Bisk, 2018; Iyyer et al., 2018; Tan et al., 2020; Eger and Benz, 2020), many often simply rely on word embeddings or language models for perturbation proposal (see §4).
Revised: Bisk, 2018; Iyyer et al., 2018; Tan et al., 2020; Eger and Benz, 2020), many often simply rely on word embeddings or language models for perturbation proposal (see §4).

Original: While the latter may be useful to evaluate a system’s robustness to malicious actors, they are less useful for dimension-speciﬁc testing (e.g., reliability when encountering grammatical variation).
Revised: While the latter may be useful to evaluate a system’s robustness to malicious actors, they are less useful for dimension-speciﬁc testing (e.g., reliability when encountering grammatical variation).

Original: This is because they often perturb the input across multiple dimensions at once, which may make the resulting adversary unnatural.
Revised: This is because they often perturb the input across multiple dimensions at once, which may make the resulting adversary unnatural.

Original: We hence reformulate adversarial attacks as dimension-speciﬁc , worst-case tests that can be used to approximate real-world variation.
Revised: We hence reformulate adversarial attacks as dimension-speciﬁc , worst-case tests that can be used to approximate real-world variation.

Original: We contribute a reliability testing framework — DOCTOR — that translates safety and fairness concerns of
Revised: We contribute a reliability testing framework — DOCTOR — that translates safety and fairness concerns of

Original: NLP systems into quantitative tests.
Revised: NLP systems into quantitative tests.

Original: We demonstrate how testing dimensions for DOCTOR can be drafted for a speciﬁc use case.
Revised: We demonstrate how testing dimensions for DOCTOR can be drafted for a speciﬁc use case.

Original: Finally, we discuss the policy implications, challenges, and directions for future research for reliability testing.
Revised: Finally, we discuss the policy implications, challenges, and directions for future research for reliability testing.

Original:  Terminology Deﬁnitions
Revised:  Terminology Deﬁnitions

Original: Let’s deﬁne key terms to be used in our discussion.
Revised: Let’s deﬁne key terms to be used in our discussion.

Original: NLP system.
Revised: NLP system.

Original: The entire text processing pipeline built to solve a speciﬁc task, taking raw text as input and producing predictions in the form of labels (classiﬁcation) or text (generation).
Revised: The entire text processing pipeline built to solve a speciﬁc task, taking raw text as input and producing predictions in the form of labels (classiﬁcation) or text (generation).

Original: We exclude raw language models from the discussion since it is unclear how performance, and hence worst-case performance, should be evaluated.
Revised: We exclude raw language models from the discussion since it is unclear how performance, and hence worst-case performance, should be evaluated.

Original: We include NLP systems that use language models internally (e.g.,
Revised: We include NLP systems that use language models internally (e.g.,

Original: BERT-based classiﬁers (Devlin et al., 2019)).
Revised: BERT-based classiﬁers (Devlin et al., 2019)).

Original: Reliability.
Revised: Reliability.

Original: Deﬁned by IEEE (2017) as the “degree to which a system, product or component performs speciﬁed functions under speciﬁed conditions for a speciﬁed period of time”.
Revised: Deﬁned by IEEE (2017) as the “degree to which a system, product or component performs speciﬁed functions under speciﬁed conditions for a speciﬁed period of time”.

Original: We prefer this term over robustness 2 to challenge the NLP community’s common framing of inputs from outside the training distribution as “noisy”.
Revised: We prefer this term over robustness 2 to challenge the NLP community’s common framing of inputs from outside the training distribution as “noisy”.

Original: The notion of reliability requires us to explicitly consider the speciﬁc, diverse, environments (i.e., communities) a system will operate in and is crucial to reducing the negative impact of NLP on the underrepresented.
Revised: The notion of reliability requires us to explicitly consider the speciﬁc, diverse, environments (i.e., communities) a system will operate in and is crucial to reducing the negative impact of NLP on the underrepresented.

Original: Dimension.
Revised: Dimension.

Original: An axis along which variation can occur in the real world.
Revised: An axis along which variation can occur in the real world.

Original: A taxonomy of possible dimensions can be found in Table 1 (Appendix).
Revised: A taxonomy of possible dimensions can be found in Table 1 (Appendix).

Original: Adversarial attack.
Revised: Adversarial attack.

Original: A method of perturbing the input to degrade a target model’s accuracy (Goodfellow et al., 2015).
Revised: A method of perturbing the input to degrade a target model’s accuracy (Goodfellow et al., 2015).

Original: In computer vision, this is achieved by adding adversarial noise to the image, optimized to be maximally damaging to the model.
Revised: In computer vision, this is achieved by adding adversarial noise to the image, optimized to be maximally damaging to the model.

Original: §4 describes how this is done in the NLP context.
Revised: §4 describes how this is done in the NLP context.

Original: Stakeholder.
Revised: Stakeholder.

Original: Someone who is directly or indirectly impacted by the NLP system’s predictions.
Revised: Someone who is directly or indirectly impacted by the NLP system’s predictions.

Original: Actor.
Revised: Actor.

Original: Someone who has inﬂuence over a) the design of an NLP system and its reliability testing regime; b) whether the system is deployed; and c) who it can interact with.
Revised: Someone who has inﬂuence over a) the design of an NLP system and its reliability testing regime; b) whether the system is deployed; and c) who it can interact with.

Original: Within the context of our discussion, actors are likely to be regulators, experts, and stakeholder advocates.
Revised: Within the context of our discussion, actors are likely to be regulators, experts, and stakeholder advocates.

Original: Expert.
Revised: Expert.

Original: Someone who has specialized knowledge, such as ethicists, linguists, domain experts, social scientists, or NLP practitioners.
Revised: Someone who has specialized knowledge, such as ethicists, linguists, domain experts, social scientists, or NLP practitioners.

Original:  The Case for Reliability Testing in NLP
Revised:  The Case for Reliability Testing in NLP

Original: The accelerating interest in building NLP-based products that impact many lives has led to urgent questions of fairness, safety, and accountability (Hovy and Spruit, 2016;
Revised: The accelerating interest in building NLP-based products that impact many lives has led to urgent questions of fairness, safety, and accountability (Hovy and Spruit, 2016;

Original: Bender et al., 2021), prompting research into algorithmic bias (Bolukbasi et al., 2016; Blodgett et al., 2020), explainability (Ribeiro et al., 2016; Danilevsky et al., 2020), robustness (Jia and Liang, 2017), etc.
Revised: Bender et al., 2021), prompting research into algorithmic bias (Bolukbasi et al., 2016; Blodgett et al., 2020), explainability (Ribeiro et al., 2016; Danilevsky et al., 2020), robustness (Jia and Liang, 2017), etc.

Original: Research is also emerging on best practices for productizing
Revised: Research is also emerging on best practices for productizing

Original: ML: from detailed dataset documentation (Bender and Friedman, 2018; Gebru et al., 2018), model documentation for highlighting important but often unreported details such as its training data, intended use, and caveats (Mitchell et al., 2019), and documentation best practices (Partnership on AI, 2019), to institutional mechanisms such as auditing (Raji et al., 2020) to enforce accountability and red-teaming (Brundage et al., 2020) to address developer blind spots, not to mention studies on the impact of organizational structures on responsible
Revised: ML: from detailed dataset documentation (Bender and Friedman, 2018; Gebru et al., 2018), model documentation for highlighting important but often unreported details such as its training data, intended use, and caveats (Mitchell et al., 2019), and documentation best practices (Partnership on AI, 2019), to institutional mechanisms such as auditing (Raji et al., 2020) to enforce accountability and red-teaming (Brundage et al., 2020) to address developer blind spots, not to mention studies on the impact of organizational structures on responsible

Original: AI initiatives (Rakova et al., 2020).
Revised: AI initiatives (Rakova et al., 2020).

Original: Calls for increased accountability and transparency are gaining traction among governments (116th U.S. Congress, 2019; NIST, 2019; European Commission, 2020; Smith, 2020; California
Revised: Calls for increased accountability and transparency are gaining traction among governments (116th U.S. Congress, 2019; NIST, 2019; European Commission, 2020; Smith, 2020; California

Original: State Legislature, 2020; FDA, 2021) and customers increasingly cite ethical concerns as a reason for not engaging AI service providers (EIU, 2020).
Revised: State Legislature, 2020; FDA, 2021) and customers increasingly cite ethical concerns as a reason for not engaging AI service providers (EIU, 2020).

Original: While there has been signiﬁcant discussion around best practices for dataset and model creation, work to ensure NLP systems are evaluated in a manner representative of their operational conditions has only just begun.
Revised: While there has been signiﬁcant discussion around best practices for dataset and model creation, work to ensure NLP systems are evaluated in a manner representative of their operational conditions has only just begun.

Original: Initial work in constructing representative tests focuses on enabling development teams to easily evaluate their models’ linguistic capabilities (Ribeiro et al., 2020) and accuracy on subpopulations and distribution shifts (Goel et al., 2021).
Revised: Initial work in constructing representative tests focuses on enabling development teams to easily evaluate their models’ linguistic capabilities (Ribeiro et al., 2020) and accuracy on subpopulations and distribution shifts (Goel et al., 2021).

Original: However, there is a clear need for a paradigm that allows experts and stakeholder advocates to collaboratively develop tests that are representative of the practical and ethical concerns of an NLP system’s target demographic.
Revised: However, there is a clear need for a paradigm that allows experts and stakeholder advocates to collaboratively develop tests that are representative of the practical and ethical concerns of an NLP system’s target demographic.

Original: We argue that reliability testing , by reframing the concept of adversarial attacks, has the potential to ﬁll this gap.
Revised: We argue that reliability testing , by reframing the concept of adversarial attacks, has the potential to ﬁll this gap.

Original: 3. What is reliability testing?
Revised: 3. What is reliability testing?

Original: Despite the recent advances in neural architectures resulting in breakthrough performance on benchmark datasets, research into adversarial examples and out-of-distribution generalization has found
Revised: Despite the recent advances in neural architectures resulting in breakthrough performance on benchmark datasets, research into adversarial examples and out-of-distribution generalization has found

Original: ML systems to be particularly vulnerable to slight perturbations in the input (Goodfellow et al., 2015) and natural distribution shifts (Fisch et al., 2019).
Revised: ML systems to be particularly vulnerable to slight perturbations in the input (Goodfellow et al., 2015) and natural distribution shifts (Fisch et al., 2019).

Original: While these perturbations are often chosen to maximize model failure, they highlight serious reliability issues for putting ML models into production since they show that these models could fail catastrophically in naturally noisy, diverse, real-world environments (Saria and Subbaswamy, 2019).
Revised: While these perturbations are often chosen to maximize model failure, they highlight serious reliability issues for putting ML models into production since they show that these models could fail catastrophically in naturally noisy, diverse, real-world environments (Saria and Subbaswamy, 2019).

Original: Additionally, bias can seep into the system at multiple stages of the NLP lifecycle (Shah et al., 2020), resulting in discrimination against minority groups (O’Neil, 2016).
Revised: Additionally, bias can seep into the system at multiple stages of the NLP lifecycle (Shah et al., 2020), resulting in discrimination against minority groups (O’Neil, 2016).

Original: The good news, however, is that rigorous testing can help to highlight potential issues before the systems are deployed.
Revised: The good news, however, is that rigorous testing can help to highlight potential issues before the systems are deployed.

Original: The need for rigorous testing in NLP is reﬂected in ACL 2020 giving the Best Paper Award to CheckList (Ribeiro et al., 2020), which applied the idea of behavior testing from software engineering to testing NLP systems.
Revised: The need for rigorous testing in NLP is reﬂected in ACL 2020 giving the Best Paper Award to CheckList (Ribeiro et al., 2020), which applied the idea of behavior testing from software engineering to testing NLP systems.

Original: While invaluable as a ﬁrst step towards the development of comprehensive testing methodology, the current implementation of
Revised: While invaluable as a ﬁrst step towards the development of comprehensive testing methodology, the current implementation of

Original: CheckList may still overestimate the reliability of
Revised: CheckList may still overestimate the reliability of

Original: NLP systems since the individual test examples are largely manually constructed.
Revised: NLP systems since the individual test examples are largely manually constructed.

Original: Importantly, with the complexity and scale of current models, humans cannot accurately determine a model’s adversarial distribution (i.e., the examples that cause model failure).
Revised: Importantly, with the complexity and scale of current models, humans cannot accurately determine a model’s adversarial distribution (i.e., the examples that cause model failure).

Original: Consequently, the test examples they construct are unlikely to be the worst-case examples for the model.
Revised: Consequently, the test examples they construct are unlikely to be the worst-case examples for the model.

Original: Automated assistance is needed.
Revised: Automated assistance is needed.

Original: Therefore, we propose to perform reliability testing , which can be thought of as one component of behavior testing.
Revised: Therefore, we propose to perform reliability testing , which can be thought of as one component of behavior testing.

Original: We categorize reliability tests as average-case tests or the worst-case tests.
Revised: We categorize reliability tests as average-case tests or the worst-case tests.

Original: As their names suggest, average-case and worst-case tests estimate the expected and lower-bound performance, respectively, when the NLP system is exposed to the phenomena modeled by the tests.
Revised: As their names suggest, average-case and worst-case tests estimate the expected and lower-bound performance, respectively, when the NLP system is exposed to the phenomena modeled by the tests.

Original: Average-case tests are conceptually similar to Wu et al.
Revised: Average-case tests are conceptually similar to Wu et al.

Original: (2021)’s counterfactuals, which is contemporaneous work, while worst-case tests are most similar to adversarial attacks (§4).
Revised: (2021)’s counterfactuals, which is contemporaneous work, while worst-case tests are most similar to adversarial attacks (§4).

Original: Our approach parallels boundary value testing in software engineering: In boundary value testing, tests evaluate a program’s ability to handle edge cases using test examples drawn from the extremes of the ranges the program is expected to handle.
Revised: Our approach parallels boundary value testing in software engineering: In boundary value testing, tests evaluate a program’s ability to handle edge cases using test examples drawn from the extremes of the ranges the program is expected to handle.

Original: Similarly, reliability testing aims to quantify the system’s reliability under diverse and potentially extreme conditions.
Revised: Similarly, reliability testing aims to quantify the system’s reliability under diverse and potentially extreme conditions.

Original: This allows teams to perform better quality control of their NLP systems and introduce more nuance into discussions of why and when models fail (§5).
Revised: This allows teams to perform better quality control of their NLP systems and introduce more nuance into discussions of why and when models fail (§5).

Original: Finally, we note that reliability testing and standards are established practices in engineering industries (e.g., aerospace (Nelson, 2003; Wilkinson et al., 2016)) and advocate for NL engineering to be at parity with these ﬁelds.
Revised: Finally, we note that reliability testing and standards are established practices in engineering industries (e.g., aerospace (Nelson, 2003; Wilkinson et al., 2016)) and advocate for NL engineering to be at parity with these ﬁelds.

Original: 3. Evaluating worst-case performance in a label-scarce world
Revised: 3. Evaluating worst-case performance in a label-scarce world

Original: A proposed approach for testing robustness to natural and adverse distribution shifts is to construct test sets using data from different domains or writing styles (Miller et al., 2020;
Revised: A proposed approach for testing robustness to natural and adverse distribution shifts is to construct test sets using data from different domains or writing styles (Miller et al., 2020;

Original: Hendrycks et al., 2020), or to use a human vs. model method of constructing challenge sets (Nie et al., 2020; Zhang et al., 2019b).
Revised: Hendrycks et al., 2020), or to use a human vs. model method of constructing challenge sets (Nie et al., 2020; Zhang et al., 2019b).

Original: While they are the gold standard, such datasets are expensive to construct, 4 making it infeasible to manually create worst-case test examples for each NLP system being evaluated.
Revised: While they are the gold standard, such datasets are expensive to construct, 4 making it infeasible to manually create worst-case test examples for each NLP system being evaluated.

Original: Consequently, these challenge sets necessarily overestimate each system’s worst-case performance when the inference distribution differs from the training one.
Revised: Consequently, these challenge sets necessarily overestimate each system’s worst-case performance when the inference distribution differs from the training one.

Original: Additionally, due to their crowdsourced nature, these challenge sets inevitably introduce distribution shifts across multiple dimensions at once, and even their own biases (Geva et al., 2019), unless explicitly controlled for.
Revised: Additionally, due to their crowdsourced nature, these challenge sets inevitably introduce distribution shifts across multiple dimensions at once, and even their own biases (Geva et al., 2019), unless explicitly controlled for.

Original: Building individual challenge sets for each dimension would be prohibitively expensive due to combinatorial explosion, even before having to account for concept drift (Widmer and Kubat, 1996).
Revised: Building individual challenge sets for each dimension would be prohibitively expensive due to combinatorial explosion, even before having to account for concept drift (Widmer and Kubat, 1996).

Original: This coupling complicates efforts to design a nuanced and comprehensive testing regime.
Revised: This coupling complicates efforts to design a nuanced and comprehensive testing regime.

Original: Hence, simulating variation in a controlled manner via reliability tests can be a complementary method of evaluating the system’s out-of-distribution generalization ability.
Revised: Hence, simulating variation in a controlled manner via reliability tests can be a complementary method of evaluating the system’s out-of-distribution generalization ability.

Original:  Adversarial Attacks as Reliability Tests
Revised:  Adversarial Attacks as Reliability Tests

Original: We ﬁrst give a brief introduction to adversarial attacks in NLP before showing how they can be used for reliability testing.
Revised: We ﬁrst give a brief introduction to adversarial attacks in NLP before showing how they can be used for reliability testing.

Original: We refer the reader to
Revised: We refer the reader to

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020b)
Revised: (2020b)

Original: for a comprehensive survey.
Revised: for a comprehensive survey.

Original: Existing work on NLP adversarial attacks perturbs the input at various levels of linguistic analysis: phonology (Eger and Benz, 2020), orthography (Ebrahimi et al., 2018), morphology (Tan et al., 2020), lexicon (Alzantot et al., 2018; Jin et al., 2020), and syntax (Iyyer et al., 2018).
Revised: Existing work on NLP adversarial attacks perturbs the input at various levels of linguistic analysis: phonology (Eger and Benz, 2020), orthography (Ebrahimi et al., 2018), morphology (Tan et al., 2020), lexicon (Alzantot et al., 2018; Jin et al., 2020), and syntax (Iyyer et al., 2018).

Original: Early work did not place any constraints on the attacks and merely used the degradation to a target model’s accuracy as the measure of success.
Revised: Early work did not place any constraints on the attacks and merely used the degradation to a target model’s accuracy as the measure of success.

Original: However, this often resulted in the semantics and expected prediction changing, leading to an overestimation of the attack’s success.
Revised: However, this often resulted in the semantics and expected prediction changing, leading to an overestimation of the attack’s success.

Original: Recent attacks aim to preserve the original input’s semantics.
Revised: Recent attacks aim to preserve the original input’s semantics.

Original: A popular approach has been to substitute words with their synonyms using word embeddings or a language model as a measure of semantic similarity (Alzantot et al., 2018;
Revised: A popular approach has been to substitute words with their synonyms using word embeddings or a language model as a measure of semantic similarity (Alzantot et al., 2018;

Original: Ribeiro et al., 2018; Michel et al., 2019; Ren et al., 2019; Zhang et al., 2019a;
Revised: Ribeiro et al., 2018; Michel et al., 2019; Ren et al., 2019; Zhang et al., 2019a;

Original: Li et al., 2019; Jin et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2020a).
Revised: Li et al., 2019; Jin et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2020a).

Original: Focusing on maximally degrading model accuracy overlooks the key feature of adversarial attacks: the ability to ﬁnd the worst-case example for a model from an arbitrary distribution.
Revised: Focusing on maximally degrading model accuracy overlooks the key feature of adversarial attacks: the ability to ﬁnd the worst-case example for a model from an arbitrary distribution.

Original: Many recent attacks perturb the input across multiple dimensions at once, which may make the result unnatural.
Revised: Many recent attacks perturb the input across multiple dimensions at once, which may make the result unnatural.

Original: By constraining our sample perturbations to a distribution modeling a speciﬁc dimension of interest, the performance on the generated adversaries is a valid lower bound performance for that dimension.
Revised: By constraining our sample perturbations to a distribution modeling a speciﬁc dimension of interest, the performance on the generated adversaries is a valid lower bound performance for that dimension.

Original: Said another way, adversarial attacks can be reframed as interpretable reliability tests if we constrain them to meaningful distributions.
Revised: Said another way, adversarial attacks can be reframed as interpretable reliability tests if we constrain them to meaningful distributions.

Original: This is the key element of our approach as detailed in Alg. 1.

Original: We specify either an average (Lines 5–7) or worse case test (Lines 8–10), but conditioned on the data distribution D that models

Original: Require:
Revised: Require:

Original: Data distribution D d = {X , Y} modeling the di- mension of interest d , NLP system M , Source dataset
Revised: Data distribution D d = {X , Y} modeling the di- mension of interest d , NLP system M , Source dataset

Original: X ∼ X , Desired labels Y (cid:48) ∼ Y , Scoring function S .
Revised: X ∼ X , Desired labels Y (cid:48) ∼ Y , Scoring function S .

Original: Ensure: Average- or worst-case examples X (cid:48) , Result r .
Revised: Ensure: Average- or worst-case examples X (cid:48) , Result r .

Original: 1: X (cid:48) ← { ∅ } , r ←2: for x, y (cid:48) in X, Y (cid:48) do 3:
Revised: 1: X (cid:48) ← { ∅ } , r ←2: for x, y (cid:48) in X, Y (cid:48) do 3:

Original: C ← S AMPLE C ANDIDATES ( X ) 4: switch TestType do 5: case AverageCaseTest 6: s ← M EAN ( S ( y (cid:48) , M ( C ))) 7:
Revised: C ← S AMPLE C ANDIDATES ( X ) 4: switch TestType do 5: case AverageCaseTest 6: s ← M EAN ( S ( y (cid:48) , M ( C ))) 7:

Original: X (cid:48) ← X (cid:48) ∪ C 8: case WorstCaseTest 9: x (cid:48) , s ← arg min x c ∈ C S ( y (cid:48) , M ( x c )) 10:
Revised: X (cid:48) ← X (cid:48) ∪ C 8: case WorstCaseTest 9: x (cid:48) , s ← arg min x c ∈ C S ( y (cid:48) , M ( x c )) 10:

Original: X (cid:48) ← X (cid:48) ∪ { x (cid:48) } 11: r ← r + s 12: end for 13: r ← r | X | 14: return X (cid:48) , r a particular dimension of interest d .
Revised: X (cid:48) ← X (cid:48) ∪ { x (cid:48) } 11: r ← r + s 12: end for 13: r ← r | X | 14: return X (cid:48) , r This is the key element of our approach as detailed in Alg. 1.

Revised: We specify either an average (Lines 5–7) or worse case test (Lines 8–10), but conditioned on the data distribution D that models a particular dimension of interest d .

Original: The resultant reliability score gauges real-world performance and the worst-case variant returns the adversarial examples that cause worst-case performance.
Revised: The resultant reliability score gauges real-world performance and the worst-case variant returns the adversarial examples that cause worst-case performance.

Original: When invariance to input variation is expected, y (cid:48) is equivalent to the source label y .
Revised: When invariance to input variation is expected, y (cid:48) is equivalent to the source label y .

Original: Note that by ignoring the average-case test logic and removing d , we recover the general adversarial attack algorithm.
Revised: Note that by ignoring the average-case test logic and removing d , we recover the general adversarial attack algorithm.

Original: However, the key difference between an adversarial robustness mindset and a testing one is the latter’s emphasis on identifying ways in which natural phenomena or ethical concerns can be operationalized as reliability tests.
Revised: However, the key difference between an adversarial robustness mindset and a testing one is the latter’s emphasis on identifying ways in which natural phenomena or ethical concerns can be operationalized as reliability tests.

Original: This change in perspective opens up new avenues for interdisciplinary research that will allow researchers and practitioners to have more nuanced discussions about model reliability and can be used to design comprehensive reliability testing regimes.
Revised: This change in perspective opens up new avenues for interdisciplinary research that will allow researchers and practitioners to have more nuanced discussions about model reliability and can be used to design comprehensive reliability testing regimes.

Original: We describe such a framework for interdisciplinary collaboration next.
Revised: We describe such a framework for interdisciplinary collaboration next.

Original:  A Framework for Reliability Testing
Revised:  A Framework for Reliability Testing

Original: We introduce and then describe our general framework, DOCTOR, for testing the reliability of NLP systems.
Revised: We introduce and then describe our general framework, DOCTOR, for testing the reliability of NLP systems.

Original: DOCTOR comprises six steps:
Revised: DOCTOR comprises six steps:

Original: D eﬁne reliability requirements 2.
Revised: D eﬁne reliability requirements 2.

Original: O perationalize dimensions as distributions 3.
Revised: O perationalize dimensions as distributions 3.

Original: C onstruct tests 4.
Revised: C onstruct tests 4.

Original: T est system and report results 5.
Revised: T est system and report results 5.

Original: O bserve deployed system’s behavior 6.
Revised: O bserve deployed system’s behavior 6.

Original: R eﬁne reliability requirements and tests Deﬁning reliability requirements.
Revised: R eﬁne reliability requirements and tests Deﬁning reliability requirements.

Original: Before any tests are constructed, experts and stakeholder advocates should work together to understand the demographics and values of the communities the NLP system will interact with (Friedman and Hendry, 2019) and the system’s impact on their lives.
Revised: Before any tests are constructed, experts and stakeholder advocates should work together to understand the demographics and values of the communities the NLP system will interact with (Friedman and Hendry, 2019) and the system’s impact on their lives.

Original: The latter is also known as algorithmic risk assessment (Ada Lovelace Institute and DataKind UK, 2021).
Revised: The latter is also known as algorithmic risk assessment (Ada Lovelace Institute and DataKind UK, 2021).

Original: There are three critical questions to address: 1)
Revised: There are three critical questions to address: 1)

Original: Along what dimensions should the model be tested? 2) What metrics should be used to measure system performance?  What are acceptable performance thresholds for each dimension?
Revised: Along what dimensions should the model be tested? 2) What metrics should be used to measure system performance? 3) What are acceptable performance thresholds for each dimension?

Original: Question 1 can be further broken down into: a) general linguistic phenomena, such as alternative spellings or code-mixing; b) task-speciﬁc quirks, e.g., an essay grading system should not use text length to predict score; c) sensitive attributes, such as gender, ethnicity, sexual orientation, age, or disability status.
Revised: Question 1 can be further broken down into: a) general linguistic phenomena, such as alternative spellings or code-mixing; b) task-speciﬁc quirks, e.g., an essay grading system should not use text length to predict score; c) sensitive attributes, such as gender, ethnicity, sexual orientation, age, or disability status.

Original: This presents an opportunity for interdisciplinary expert collaboration: Linguists are best equipped to contribute to discussions around (a), domain experts to (b), and ethicists and social scientists to (c).
Revised: This presents an opportunity for interdisciplinary expert collaboration: Linguists are best equipped to contribute to discussions around (a), domain experts to (b), and ethicists and social scientists to (c).

Original: However, we recognize that such collaboration may not be feasible for every NLP system being tested.
Revised: However, we recognize that such collaboration may not be feasible for every NLP system being tested.

Original: It is more realistic to expect ethicists to be involved when applying DOCTOR at the company and industry levels, and ethics-trained
Revised: It is more realistic to expect ethicists to be involved when applying DOCTOR at the company and industry levels, and ethics-trained

Original: NLP practitioners to answer these questions within the development team.
Revised: NLP practitioners to answer these questions within the development team.

Original: We provide a taxonomy of potential dimensions in Table 1 (Appendix).
Revised: We provide a taxonomy of potential dimensions in Table 1 (Appendix).

Original: Since it is likely unfeasible to test every possible dimension, stakeholder advocates should be involved to ensure their values and interests are accurately represented and prioritized (Hagerty and
Revised: Since it is likely unfeasible to test every possible dimension, stakeholder advocates should be involved to ensure their values and interests are accurately represented and prioritized (Hagerty and

Original: Rubinov, 2019), while experts should ensure the dimensions identiﬁed can be feasibly tested.
Revised: Rubinov, 2019), while experts should ensure the dimensions identiﬁed can be feasibly tested.

Original: A similar approach to that of community juries may be taken 5 .
Revised: A similar approach to that of community juries may be taken 5 .

Original: We recommend using this question to evaluate the feasibility of operationalizing potential dimensions: “What is the system’s performance when exposed to variation in d ?”
Revised: We recommend using this question to evaluate the feasibility of operationalizing potential dimensions: “What is the system’s performance when exposed to variation in d ?”

Original: where d is the dimension.
Revised: where d is the dimension.

Original: For example, rather than simply “gender”, a better-deﬁned dimension would be “gender pronouns”.
Revised: For example, rather than simply “gender”, a better-deﬁned dimension would be “gender pronouns”.

Original: With this understanding, experts and policymakers can then create a set of reliability requirements , comprising the testing dimensions, performance metric(s), and passing thresholds.
Revised: With this understanding, experts and policymakers can then create a set of reliability requirements , comprising the testing dimensions, performance metric(s), and passing thresholds.

Original: Next, we recommend using the same metrics for held-out, average-case, and worst-case performance for easy comparison.
Revised: Next, we recommend using the same metrics for held-out, average-case, and worst-case perfor- mance for easy comparison.

Original: These often vary from task to task and are still a subject of active research (Novikova et al., 2017; Reiter, 2018;
Revised: These often vary from task to task and are still a subject of active research (Novikova et al., 2017; Reiter, 2018;

Original: Kryscinski et al., 2019), hence the question of the right met- ric to use is beyond the scope of this paper.
Revised: Kryscinski et al., 2019), hence the question of the right metric to use is beyond the scope of this paper.

Original: Finally, ethicists, in consultation with the other aforementioned experts and stakeholders, will determine acceptable thresholds for worst-case performance.
Revised: Finally, ethicists, in consultation with the other aforementioned experts and stakeholders, will determine acceptable thresholds for worst-case performance.

Original: The system under test must perform above said thresholds when exposed to variation along those dimensions in order to pass.
Revised: The system under test must perform above said thresholds when exposed to variation along those dimensions in order to pass.

Original: For worst-case performance, we recommend reporting thresholds as relative differences ( δ ) between the average-case and worst-case performance.
Revised: For worst-case performance, we recommend reporting thresholds as relative differences ( δ ) between the average-case and worst-case performance.

Original: These questions may help in applying this step and deciding if speciﬁc NLP solutions should even exist (Leins et al., 2020):
Revised: These questions may help in applying this step and deciding if speciﬁc NLP solutions should even exist (Leins et al., 2020):

Original: • Who will interact with the NLP system, in what context, and using which language varieties?
Revised: • Who will interact with the NLP system, in what context, and using which language varieties?

Original: • What are the distinguishing features of these varieties compared to those used for training?
Revised: • What are the distinguishing features of these varieties compared to those used for training?

Original: • What is the (short- and long-term) impact on the community’s most underrepresented members if the system performs more poorly for them?
Revised: • What is the (short- and long-term) impact on the community’s most underrepresented members if the system performs more poorly for them?

Original: We note that our framework is general enough to be applied at various levels of organization: within the development team, within the company (compliance team, internal auditor), and within the industry (self-regulation or independent regulator).
Revised: We note that our framework is general enough to be applied at various levels of organization: within the development team, within the company (compliance team, internal auditor), and within the industry (self-regulation or independent regulator).

Original: However, we expect the exact set of dimensions, metrics and acceptable thresholds deﬁned in Stepto vary depending on the reliability concerns of the actors at each level.
Revised: However, we expect the exact set of dimensions, metrics and acceptable thresholds deﬁned in Stepto vary depending on the reliability concerns of the actors at each level.

Original: For example, independent regulators will be most concerned with establishing minimum safety and fairness standards that all NLP systems used in their industries must meet, while compliance teams may wish to have stricter and more comprehensive standards for brand reasons.
Revised: For example, independent regulators will be most concerned with establishing minimum safety and fairness standards that all NLP systems used in their industries must meet, while compliance teams may wish to have stricter and more comprehensive standards for brand reasons.

Original: Developers can use DOCTOR to meet the other two levels of requirements and understand their system’s behaviour better with targeted testing.
Revised: Developers can use DOCTOR to meet the other two levels of requirements and understand their system’s behaviour better with targeted testing.

Original:  dimensions.
Revised: Operationalizing dimensions.

Original: While the abstractness of dimensions allows people who are not NLP practitioners to participate in drafting the set of reliability requirements, there is no way to test NLP systems using fuzzy concepts.
Revised: While the abstractness of dimensions allows people who are not NLP practitioners to participate in drafting the set of reliability requirements, there is no way to test NLP systems using fuzzy concepts.

Original: Therefore, every dimension the system is to be tested along must be operationalizable as a distribution from which perturbed examples can be sampled in order for NLP practitioners to realize them as tests.
Revised: Therefore, every dimension the system is to be tested along must be operationalizable as a distribution from which perturbed examples can be sampled in order for NLP practitioners to realize them as tests.

Original: Since average-case tests attempt to estimate a system’s expected performance in its deployed environment, the availability of datasets that reﬂect real-world distributions is paramount to ensure that the tests themselves are unbiased.
Revised: Since average-case tests attempt to estimate a system’s expected performance in its deployed environment, the availability of datasets that reﬂect real-world distributions is paramount to ensure that the tests themselves are unbiased.

Original: This is less of an issue for worst-case tests; the tests only needs to know which perturbations that are possible, but not how frequently they occur in the real world.
Revised: This is less of an issue for worst-case tests; the tests only needs to know which perturbations that are possible, but not how frequently they occur in the real world.

Original: Figuring out key dimensions for different classes of NLP tasks and exploring ways of operationalizing them as reliability tests are also promising directions for future research.
Revised: Figuring out key dimensions for different classes of NLP tasks and exploring ways of operationalizing them as reliability tests are also promising directions for future research.

Original: Such research would help NLP practitioners and policymakers deﬁne reliability requirements that can be feasibly implemented.
Revised: Such research would help NLP practitioners and policymakers deﬁne reliability requirements that can be feasibly implemented.

Original: Constructing tests.
Revised: Constructing tests.

Original: Next, average- and worstcase tests are constructed (Alg. 1).
Revised: Next, average- and worstcase tests are constructed (Alg. 1).

Original: Average-case tests can be data-driven and could take the form of manually curated datasets or model-based perturbation generation (e.g., PolyJuice (Wu et al., 2021)), while worst-case tests can be rule-based (e.g., Morpheus (Tan et al., 2020)) or model-based (e.g., BERT-Attack (Li et al., 2020a)).
Revised: Average-case tests can be data-driven and could take the form of manually curated datasets or model-based perturbation generation (e.g., PolyJuice (Wu et al., 2021)), while worst-case tests can be rule-based (e.g., Morpheus (Tan et al., 2020)) or model-based (e.g., BERT-Attack (Li et al., 2020a)).

Original: We recommend constructing tests that do not need access to the NLP model’s parameters (black-box assumption); this not only yields more system-agnostic tests, but also allows for (some) tests to be created independently from the system development team.
Revised: We recommend constructing tests that do not need access to the NLP model’s parameters (black-box assumption); this not only yields more system-agnostic tests, but also allows for (some) tests to be created independently from the system development team.

Original: If the black-box assumption proves limiting, the community can establish a standard set of items an NLP system should export for testing purposes, e.g., network gradients if the system uses a neural model.
Revised: If the black-box assumption proves limiting, the community can establish a standard set of items an NLP system should export for testing purposes, e.g., network gradients if the system uses a neural model.

Original: Regardless of assumption, keeping the regulators’ test implementations separate and hidden from the system developers is critical for stakeholders and regulators to trust the results.
Revised: Regardless of assumption, keeping the regulators’ test implementations separate and hidden from the system developers is critical for stakeholders and regulators to trust the results.

Original: This separation also reduces overﬁtting to the test suite.
Revised: This separation also reduces overﬁtting to the test suite.

Original: Testing systems.
Revised: Testing systems.

Original: A possible model for test ownership is to have independently implemented tests at the three levels of organization described above.
Revised: A possible model for test ownership is to have independently implemented tests at the three levels of organization described above.

Original: At the development team level, reliability tests would be used to diagnose weaknesses with the goal of improving the NLP system for a speciﬁc use case and set of target users.
Revised: At the development team level, reliability tests would be used to diagnose weaknesses with the goal of improving the NLP system for a speciﬁc use case and set of target users.

Original: Since the resulting worst-case examples are constrained along speciﬁc dimensions, comparing those that successfully induced model failure to the perturbed and source examples that did not may yield greater insights into a model’s inner workings compared to unconstrained adversarial examples.
Revised: Since the resulting worst-case examples are constrained along speciﬁc dimensions, comparing those that successfully induced model failure to the perturbed and source examples that did not may yield greater insights into a model’s inner workings compared to unconstrained adversarial examples.

Original: Studying how modiﬁcations (to the architecture, training data and process) affect the system’s reliability along each dimension will also give engineers insight into the factors affecting system reliability.
Revised: Studying how modiﬁcations (to the architecture, training data and process) affect the system’s reliability along each dimension will also give engineers insight into the factors affecting system reliability.

Original: These tests should be executed and updated regularly during system development, as in standard software testing.
Revised: These tests should be executed and updated regularly during system development, as in standard software testing.

Original: At the compliance team level, reliability tests would be built by a specialized team, similar to a red team (Brundage et al., 2020), to ensure all NLP systems deployed adhere to company-level reliability standards.
Revised: At the compliance team level, reliability tests would be built by a specialized team, similar to a red team (Brundage et al., 2020), to ensure all NLP systems deployed adhere to company-level reliability standards.

Original: These tests should be executed as a part of regular internal audits (Raji et al., 2020), investigative audits after an incident, and before major releases (especially if it is the system’s ﬁrst release or if the architecture or training data received a major update).
Revised: These tests should be executed as a part of regular internal audits (Raji et al., 2020), investigative audits after an incident, and before major releases (especially if it is the system’s ﬁrst release or if the architecture or training data received a major update).

Original: They can also be executed on randomly chosen production systems on a regular basis, triggering an alert upon failure.
Revised: They can also be executed on randomly chosen production systems on a regular basis, triggering an alert upon failure.

Original: At the independent regulator level, reliability tests would likely be carried out during product certiﬁcation (e.g., ANSI/ISO certiﬁcation) and external audits.
Revised: At the independent regulator level, reliability tests would likely be carried out during product certiﬁcation (e.g., ANSI/ISO certiﬁcation) and external audits.

Original: Our proposed framework is highly compatible with the use of model cards (Mitchell et al., 2019) for auditing and transparent reporting (Raji et al., 2020).
Revised: Our proposed framework is highly compatible with the use of model cards (Mitchell et al., 2019) for auditing and transparent reporting (Raji et al., 2020).

Original: When a system has passed all tests and is ready to be deployed, its average and worst-case performance on all tested dimensions can be included as extra information on the accompanying model card.
Revised: When a system has passed all tests and is ready to be deployed, its average and worst-case performance on all tested dimensions can be included as extra information on the accompanying model card.

Original: In addition, the perturbed examples generated during testing and their labels ( x (cid:48) , y (cid:48) ) can be stored for audit purposes or examined to ensure that the tests are performing as expected.
Revised: In addition, the perturbed examples generated during testing and their labels ( x (cid:48) , y (cid:48) ) can be stored for audit purposes or examined to ensure that the tests are performing as expected.

Original: Observing and Reﬁning requirements.
Revised: Observing and Reﬁning requirements.

Original: Since it is unlikely for teams to get everything right initially, it is crucial to regularly monitor the systems’ impact post-launch and add, update, or re-prioritize dimensions and thresholds accordingly.
Revised: Since it is unlikely for teams to get everything right initially, it is crucial to regularly monitor the systems’ impact post-launch and add, update, or re-prioritize dimensions and thresholds accordingly.

Original: Monitoring can be done via community juries, or even something as simple as making feedback and remediation forms available to affected stakeholders.
Revised: Monitoring can be done via community juries, or even something as simple as making feedback and remediation forms available to affected stakeholders.

Original:  From Concerns to Dimensions
Revised:  From Concerns to Dimensions

Original: We now illustrate how reliability concerns can be converted into concrete testing dimensions (Step 1) by considering the scenario of applying automated text scoring to short answers and essays from students in the multilingual population of Singapore.
Revised: We now illustrate how reliability concerns can be converted into concrete testing dimensions (Step 1) by considering the scenario of applying automated text scoring to short answers and essays from students in the multilingual population of Singapore.

Original: We study a second scenario in Appendix A. Automated Text Scoring (ATS) systems are increasingly used to grade tests and essays (Markoff, 2013;
Revised: We study a second scenario in Appendix A. Automated Text Scoring (ATS) systems are increasingly used to grade tests and essays (Markoff, 2013;

Original: Feathers, 2019).
Revised: Feathers, 2019).

Original: While they can provide instant feedback and help teachers and test agencies cope with large loads, studies have shown that they often exhibit demographic and language biases, such as scoring African-
Revised: While they can provide instant feedback and help teachers and test agencies cope with large loads, studies have shown that they often exhibit demographic and language biases, such as scoring African-

Original: and Indian-American males lower on the GRE Argument task compared to human graders (Bridgeman et al., 2012;
Revised: and Indian-American males lower on the GRE Argument task compared to human graders (Bridgeman et al., 2012;

Original: Ramineni and
Revised: Ramineni and

Original: Williamson, 2018).
Revised: Williamson, 2018).

Original: Since the results of some tests will affect the futures of the test takers (Salaky, 2018), the scoring algorithms used must be sufﬁciently reliable.
Revised: Since the results of some tests will affect the futures of the test takers (Salaky, 2018), the scoring algorithms used must be sufﬁciently reliable.

Original: Hence, let us imagine that Singapore’s education ministry has decided to create a standard set of reliability requirements that all ATS systems used in education must adhere to.
Revised: Hence, let us imagine that Singapore’s education ministry has decided to create a standard set of reliability requirements that all ATS systems used in education must adhere to.

Original: Linguistic landscape.
Revised: Linguistic landscape.

Original: A mix of language varieties are used in Singapore: a prestige English variety, a colloquial English variety, three other ofﬁcial languages (Chinese, Malay, and Tamil), and a large number of other languages.
Revised: A mix of language varieties are used in Singapore: a prestige English variety, a colloquial English variety, three other ofﬁcial languages (Chinese, Malay, and Tamil), and a large number of other languages.

Original: English is the lingua franca , with ﬂuency in the prestige variety correlating with socioeconomic status (Vaish and Tan, 2008).
Revised: English is the lingua franca , with ﬂuency in the prestige variety correlating with socioeconomic status (Vaish and Tan, 2008).

Original: A signiﬁcant portion of the population does not speak English at home.
Revised: A signiﬁcant portion of the population does not speak English at home.

Original: Subjects other than languages are taught in English.
Revised: Subjects other than languages are taught in English.

Original: Stakeholder impact.
Revised: Stakeholder impact.

Original: The key stakeholders affected by ATS systems would be students in schools and universities.
Revised: The key stakeholders affected by ATS systems would be students in schools and universities.

Original: The consequences of lower scores could be life-altering for the student who is unable to enroll in the major of their choice.
Revised: The consequences of lower scores could be life-altering for the student who is unable to enroll in the major of their choice.

Original: At the population level, biases in an ATS system trained on normally sampled data would unfairly discriminate against already underrepresented groups.
Revised: At the population level, biases in an ATS system trained on normally sampled data would unfairly discriminate against already underrepresented groups.

Original: Additionally, biases against disﬂuent or ungrammatical text when they are not the tested attributes would result in discrimination against students with a lower socioeconomic status or for whom English is a second language.
Revised: Additionally, biases against disﬂuent or ungrammatical text when they are not the tested attributes would result in discrimination against students with a lower socioeconomic status or for whom English is a second language.

Original: Finally, NLP systems have also been known to be overly sensitive to alternative spellings (Belinkov and Bisk, 2018).
Revised: Finally, NLP systems have also been known to be overly sensitive to alternative spellings (Belinkov and Bisk, 2018).

Original: When used to score subject tests, this could result in the ATS system unfairly penalizing dyslexic students (Coleman et al., 2009).
Revised: When used to score subject tests, this could result in the ATS system unfairly penalizing dyslexic students (Coleman et al., 2009).

Original: Since education is often credited with enabling social mobility, 6 unfair grading may perpetuate systemic discrimination and increase social inequality.
Revised: Since education is often credited with enabling social mobility, 6 unfair grading may perpetuate systemic discrimination and increase social inequality.

Original: Dimension.
Revised: Dimension.

Original: We can generally categorize written tests into those that test for content correctness (e.g., essay questions in a history test), and those that test for language skills (e.g., proper use of grammar).
Revised: We can generally categorize written tests into those that test for content correctness (e.g., essay questions in a history test), and those that test for language skills (e.g., proper use of grammar).

Original: While there are tests that simultaneously assess both aspects, modern ATS systems often grade them separately (Ke and Ng, 2019) and we treat each aspect as a separate test here.
Revised: While there are tests that simultaneously assess both aspects, modern ATS systems often grade them separately (Ke and Ng, 2019) and we treat each aspect as a separate test here.

Original: When grading students on content correctness, we would expect the ATS system to ignore linguistic variation and sensitive attributes as long as they do not affect the answer’s valid- ity.
Revised: When grading students on content correctness, we would expect the ATS system to ignore linguistic variation and sensitive attributes as long as they do not affect the answer’s validity.

Original: Hence, we would expect variation in these dimensions to have no effect on scores: answer length, language/vocabulary simplicity, alternative spellings/misspellings of non-keywords, grammatical variation, syntactic variation (especially those resembling transfer from a ﬁrst language), and proxies for sensitive attributes.
Revised: Hence, we would expect variation in these dimensions to have no effect on scores: answer length, language/vocabulary simplicity, alternative spellings/misspellings of non-keywords, grammatical variation, syntactic variation (especially those resembling transfer from a ﬁrst language), and proxies for sensitive attributes.

Original: On the other hand, we would expect the system to be able to differentiate proper answers from those trying to game the test with gibberish (Chin, 2020; Ding et al., 2020).
Revised: On the other hand, we would expect the system to be able to differentiate proper answers from those trying to game the test with gibberish (Chin, 2020; Ding et al., 2020).

Original: When grading students on language skills, however, we would expect ATS systems to be only sensitive to the relevant skill.
Revised: When grading students on language skills, however, we would expect ATS systems to be only sensitive to the relevant skill.

Original: For example, when assessing grammar use, we would expect the system to be sensitive to grammatical errors (from the perspective of the language variety the student is expected to use), but not to the other dimensions mentioned above (e.g., misspellings).
Revised: For example, when assessing grammar use, we would expect the system to be sensitive to grammatical errors (from the perspective of the language variety the student is expected to use), but not to the other dimensions mentioned above (e.g., misspellings).

Original:  Implications for Policy
Revised:  Implications for Policy

Original: As we alluded to in §3, there is a mounting effort to increase accountability and transparency around the development and use of NLP systems to prevent them from amplifying societal biases.
Revised: As we alluded to in §3, there is a mounting effort to increase accountability and transparency around the development and use of NLP systems to prevent them from amplifying societal biases.

Original: DOCTOR is highly complementary to the model card approach increasingly adopted 7 to surface oft hidden details about NLP models: Developers simply need to list the tested dimensions, metrics, and score on each dimension in the model card.
Revised: DOCTOR is highly complementary to the model card approach increasingly adopted 7 to surface oft hidden details about NLP models: Developers simply need to list the tested dimensions, metrics, and score on each dimension in the model card.

Original: Crucially, reliability tests can be used to highlight fairness issues in NLP systems by including sensitive attributes for the target population, but it is paramount these requirements reﬂect local concerns rather than any prescriptivist perspective (Sambasivan et al., 2021).
Revised: Crucially, reliability tests can be used to highlight fairness issues in NLP systems by including sensitive attributes for the target population, but it is paramount these requirements reﬂect local concerns rather than any prescriptivist perspective (Sambasivan et al., 2021).

Original: At the same time, the ability to conduct quantitative, targeted reliability testing along speciﬁable dimensions paves the way for reliability standards to be established, with varying levels of stringency and rigor for different use cases and industries.
Revised: At the same time, the ability to conduct quantitative, targeted reliability testing along speciﬁable dimensions paves the way for reliability standards to be established, with varying levels of stringency and rigor for different use cases and industries.

Original: We envision minimum safety and fairness standards being established for applications that are non-sensitive, not safety-critical, and used in unregulated industries, analogous to standards for household appliances.
Revised: We envision minimum safety and fairness standards being established for applications that are non-sensitive, not safety-critical, and used in unreg- ulated industries, analogous to standards for household appliances.

Original: Naturally, applications at greater risks (Li et al., 2020b) of causing harm upon failure should be held to stricter standards.
Revised: Naturally, applications at greater risks (Li et al., 2020b) of causing harm upon failure should be held to stricter standards.

Original: Policymakers are starting to propose and implement regulations to enforce transparency and accountability in the use of AI systems.
Revised: Policymakers are starting to propose and implement regulations to enforce transparency and accountability in the use of AI systems.

Original: For example, the European
Revised: For example, the European

Original: Union’s General Data Protection Regulation grants data subjects the right to obtain “meaningful information about the logic involved” in automated decision systems (EU, 2016).
Revised: Union’s General Data Protection Regulation grants data subjects the right to obtain “meaningful information about the logic involved” in automated decision systems (EU, 2016).

Original: The EU are developing additional, AI-speciﬁc regulation European
Revised: The EU are developing additional, AI-speciﬁc regulation European

Original: Commission (2020) for example, requiring developers of high-risk AI systems to report their “capabilities and limitations, ...
Revised: Commission (2020) for example, requiring developers of high-risk AI systems to report their “capabilities and limitations, ...

Original: [and] the conditions under which they can be expected to function as intended”.
Revised: [and] the conditions under which they can be expected to function as intended”.

Original: In the U.S., a proposed bill will require
Revised: In the U.S., a proposed bill will require

Original: Washington State’s public agencies to submit a report describing “any potential impacts of the automated decision system on civil rights and liberties and potential disparate impacts on marginalized communities” before using automated decision systems (Washington State Legislature, 2021).
Revised: Washington State’s public agencies to submit a report describing “any potential impacts of the automated decision system on civil rights and liberties and potential disparate impacts on marginalized communities” before using automated decision systems (Washington State Legislature, 2021).

Original: One may note that language in the proposed regulation is intentionally vague.
Revised: One may note that language in the proposed regulation is intentionally vague.

Original: There are many ways to measure bias and fairness depending on the type of model, context of use, and goal of the system.
Revised: There are many ways to measure bias and fairness depending on the type of model, context of use, and goal of the system.

Original: Today, companies developing AI systems employ the deﬁnitions they believe most reasonable (or perhaps easiest to implement) but regulation will need to be more speciﬁc for there to be meaningful compliance.
Revised: Today, companies developing AI systems employ the deﬁnitions they believe most reasonable (or perhaps easiest to implement) but regulation will need to be more speciﬁc for there to be meaningful compliance.

Original: DOCTOR’s requirement to explicitly deﬁne speciﬁc dimensions instead of a vague notion of “reliability” will help policymakers in this regard, and can inform the ongoing development of national (NIST, 2019) and international standards 8 .
Revised: DOCTOR’s requirement to explicitly deﬁne speciﬁc dimensions instead of a vague notion of “reliability” will help policymakers in this regard, and can inform the ongoing development of national (NIST, 2019) and international standards 8 .

Original: While external algorithm audits are becoming popular, testing remains a challenge since companies wishing to protect their intellectual property may be resistant to sharing their code (Johnson, 2021), and implementing custom tests for each system is unscalable.
Revised: While external algorithm audits are becoming popular, testing remains a challenge since companies wishing to protect their intellectual property may be resistant to sharing their code (Johnson, 2021), and implementing custom tests for each system is unscalable.

Original: Our approach to reliability testing offers a potential solution to this conundrum by treating NLP systems as black-boxes.
Revised: Our approach to reliability testing offers a potential solution to this conundrum by treating NLP systems as black-boxes.

Original: If reliability tests become a legal requirement, regulatory authorities will be able to mandate independently conducted reliability tests for transparency.
Revised: If reliability tests become a legal requirement, regulatory authorities will be able to mandate independently conducted reliability tests for transparency.

Original: Such standards, combined with certiﬁcation programs (e.g., IEEE’s Ethics Certiﬁcation Program for Autonomous and Intelligent Systems 9 ), will further incentivize the development of responsible NLP, as the companies purchasing NLP systems will insist on certiﬁed systems to protect them from both legal and brand risk.
Revised: Such standards, combined with certiﬁcation programs (e.g., IEEE’s Ethics Certiﬁcation Program for Au- tonomous and Intelligent Systems 9 ), will further incentivize the development of responsible NLP, as the companies purchasing NLP systems will insist on certiﬁed systems to protect them from both legal and brand risk.

Original: These combined inﬂuences will reduce NLP’s societal harms while increasing the public’s trust in language technology.
Revised: These combined inﬂuences will reduce NLP’s societal harms while increasing the public’s trust in language technology.

Original:  Challenges and Future Directions
Revised:  Challenges and Future Directions

Original: While we believe our framework to be a useful starting point for introducing reliability testing in NLP systems, we observe some challenges that need to be overcome before it can become ubiquitous.
Revised: While we believe our framework to be a useful starting point for introducing reliability testing in NLP systems, we observe some challenges that need to be overcome before it can become ubiquitous.

Original: First, identifying and prioritizing the dimensions that can attest a system’s reliability and fairness.
Revised: First, identifying and prioritizing the dimensions that can attest a system’s reliability and fairness.

Original: The former is relatively straightforward and can be achieved via collaboration with experts (e.g., as part of NIST’s future AI standards (NIST, 2019)).
Revised: The former is relatively straightforward and can be achieved via collaboration with experts (e.g., as part of NIST’s future AI standards (NIST, 2019)).

Original: The latter, however, is a question of values and power (Noble, 2018; Mohamed et al., 2020; Leins et al., 2020), and should be addressed by having a code of ethics and ensuring that the stakeholders are adequately represented at the decision table.
Revised: The latter, however, is a question of values and power (Noble, 2018; Mohamed et al., 2020; Leins et al., 2020), and should be addressed by having a code of ethics and ensuring that the stakeholders are adequately represented at the decision table.

Original: Second, our proposed method of reliability testing may suffer from similar issues plaguing automatic evaluation metrics for natural language generation (Novikova et al., 2017; Reiter, 2018;
Revised: Second, our proposed method of reliability testing may suffer from similar issues plaguing automatic evaluation metrics for natural language generation (Novikova et al., 2017; Reiter, 2018;

Original: Kryscinski et al., 2019): due to the tests’ synthetic nature they may not fully capture the nuances of reality.
Revised: Kryscinski et al., 2019): due to the tests’ synthetic nature they may not fully capture the nuances of reality.

Original: For example, if a test’s objective were to test an NLP system’s reliability when interacting with African American English (AAE) speakers, would it be possible to guarantee (in practice) that all generated examples will fall within the distribution of text that is considered AAE?
Revised: For example, if a test’s objective were to test an NLP system’s reliability when interacting with African American English (AAE) speakers, would it be possible to guarantee (in practice) that all generated examples will fall within the distribution of text that is considered AAE?

Original: Potential research directions would be to design adversary generation techniques that can offer such guarantees or incorporate human feedback (Nguyen et al., 2017; Kreutzer et al., 2018; Stiennon et al., 2020).
Revised: Potential research directions would be to design adversary generation techniques that can offer such guarantees or incorporate human feedback (Nguyen et al., 2017; Kreutzer et al., 2018; Stiennon et al., 2020).

Original:  Conclusion
Revised:  Conclusion

Original: Once natural language technologies move out of the lab and start impacting real lives, concerns around safety, fairness, and accountability cease to be mere thought experiments.
Revised: Once natural language technologies move out of the lab and start impacting real lives, concerns around safety, fairness, and accountability cease to be mere thought experiments.

Original: While it is clear that
Revised: While it is clear that

Original: NLP can have a positive impact on our lives, from helping us type faster (autocomplete) to revitalizing endangered languages (Zhang et al., 2020a), there is also no denying its potential for perpetuating harmful stereotypes (Bolukbasi et al., 2016;
Revised: NLP can have a positive impact on our lives, from helping us type faster (autocomplete) to revitalizing endangered languages (Zhang et al., 2020a), there is also no denying its potential for perpetuating harmful stereotypes (Bolukbasi et al., 2016;

Original: Sap et al., 2019), performing disproportionately poorly for underrrepresented groups (Hern, 2017;
Revised: Sap et al., 2019), performing disproportionately poorly for underrrepresented groups (Hern, 2017;

Original: Bridgeman et al., 2012), and even erasing already marginalized communities (Bender et al., 2021).
Revised: Bridgeman et al., 2012), and even erasing already marginalized communities (Bender et al., 2021).

Original: Trust in our tools stems from an assurance that we will remain unharmed, even in the worst-case scenario.
Revised: Trust in our tools stems from an assurance that we will remain unharmed, even in the worst-case scenario.

Original: In many mature industries, this takes the form of reliability standards.
Revised: In many mature industries, this takes the form of reliability standards.

Original: However, for standards to be enacted and enforced, we must ﬁrst op- erationalize “reliability”.
Revised: However, for standards to be enacted and enforced, we must ﬁrst operationalize “reliability”.

Original: Hence in this paper, we argued for the need for reliability testing, especially worst-case testing, in NLP by contextualizing it among existing work on promoting accountability and improving generalization beyond the training distribution.
Revised: Hence in this paper, we argued for the need for reliability testing, especially worst-case testing, in NLP by contextualizing it among existing work on promoting accountability and improving generalization beyond the training distribution.

Original: Next, we showed how adversarial attacks can be reframed as worst-case tests.
Revised: Next, we showed how adversarial attacks can be reframed as worst-case tests.

Original: Finally, we proposed a possible paradigm, DOCTOR, for how reliability concerns can be realized as quantitative tests, and discussed how this framework can be used at different levels of organization or industry.
Revised: Finally, we proposed a possible paradigm, DOCTOR, for how reliability concerns can be realized as quantitative tests, and discussed how this framework can be used at different levels of organization or industry.

Original:  Broader Impact
Revised:  Broader Impact

Original: Much like how we expect to not be exposed to harmful electric shocks when using electrical appliances, we should expect some minimum levels of safety and fairness for the NLP systems we interact with in our everyday lives.
Revised: Much like how we expect to not be exposed to harmful electric shocks when using electrical appliances, we should expect some minimum levels of safety and fairness for the NLP systems we interact with in our everyday lives.

Original: As mentioned in §1, §3, and §7, standards and regulations for AI systems are in the process of being developed for this purpose, especially for applications deemed “high-risk”, e.g., healthcare (European Commission, 2020).
Revised: As mentioned in §1, §3, and §7, standards and regulations for AI systems are in the process of being developed for this purpose, especially for applications deemed “high-risk”, e.g., healthcare (European Commission, 2020).

Original: Reliability testing, and our proposed framework, is one way to approach the problem of enacting enforceable standards and regulations.
Revised: Reliability testing, and our proposed framework, is one way to approach the problem of enacting enforceable standards and regulations.

Original: However, the ﬂip side of heavily regulating every single application of NLP is that it may slow down innovation.
Revised: However, the ﬂip side of heavily regulating every single application of NLP is that it may slow down innovation.

Original: Therefore, it is important that the level of regulation for a particular application is proportionate to its potential for harm (Daten
Revised: Therefore, it is important that the level of regulation for a particular application is proportionate to its potential for harm (Daten

Original: Ethik Kommission, 2019).
Revised: Ethik Kommission, 2019).

Original: Our framework can be adapted to different levels of risk by scaling down the implementation of some steps (e.g., the method and depth in which stakeholder consultation happens or the comprehensiveness of the set of testing dimensions) for low-risk applications.
Revised: Our framework can be adapted to different levels of risk by scaling down the implementation of some steps (e.g., the method and depth in which stakeholder consultation happens or the comprehensiveness of the set of testing dimensions) for low-risk applications.

Original: Finally, it is important to ensure that any tests, standards, or regulations developed adequately represents the needs of the most vulnerable stakeholders, instead of constructing them in a prescriptivist manner (Hagerty and Rubinov, 2019).
Revised: Finally, it is important to ensure that any tests, standards, or regulations developed adequately represents the needs of the most vulnerable stakeholders, instead of constructing them in a prescriptivist manner (Hagerty and Rubinov, 2019).

Original: Hence,
Revised: Hence,

Original: DOCTOR places a strong emphasis on involving stakeholder advocates and analyzing the impact of an application of NLP on the target community.
Revised: DOCTOR places a strong emphasis on involving stakeholder advocates and analyzing the impact of an application of NLP on the target community

Original: Advances
Revised: . Acknowledgements

Revised: Samson is supported by Salesforce and Singapore’s Economic Development Board under its Industrial Postgraduate Programme.

Original:  Neural Information Processing 
Revised: Advances in Neural Information Processing Systems

Original: Processing

Original: Suzhou
Revised: Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. - ference the Paciﬁc of the Association Computational Linguistics ternational

Original: - ation
Revised: Conference Language Processing , pages 447–459, Suzhou, China.

Original:  Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: Daten Ethik Kommission.
Revised: Daten Ethik Kommission.

Original: About ML.
Revised: About ML

Revised: Technical report, Data Ethics Commission of the Federal

Revised: Government (Germany).

Revised: Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber.

Revised: Racial hate speech abusive detection pages

Revised: 35, Florence, Italy.

Original: Techni- pages 4171–4186, Minneapolis, Minnesota.
Revised: Association for Computational Linguistics. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Toutanova. , pages 4171–4186, Minneapolis, Minnesota.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: Yuning Ding, Brian Riordan, Andrea Horbach, Aoife
Revised: Yuning Ding, Brian Riordan, Andrea Horbach, Aoife

Original: Cahill, and Torsten Zesch.
Revised: Cahill, and Torsten Zesch.

Original: Don’t
Revised: Don’t

Original: “nswvt nvakgxpm” surprising vulnera bility adversarial
Revised: “nswvtnvakgxpm” ad

Original:  Barcelona, Spain ( tional
Revised: 892, Barcelona, Spain (Online).

Revised: International Committee on Computational Linguistics.

Original: Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel
Revised: Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel

Original: Stanovsky, Sameer Singh, and Matt Gardner.
Revised: Stanovsky, Sameer Singh, and Matt Gardner.

Revised: DROP:

Revised: A reading comprehension benchmark requiring discrete reasoning over paragraphs.

Revised: In Proceedings of the 2019 Conference of the North American

Original:  Minneapolis, 
Revised: Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume(Long and Short Papers) , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics. Michael Dunn.

Revised: Gender determined dialect variation.

Revised: In The expression of gender , pages 39–68.

Original:  Gruyter.
Revised: De Gruyter.

Original: Javid Ebrahimi, Anyi Rao, Daniel Lowd,  Dejing
Revised: Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing

Original: Compu tational Linguistics (Volume 2: Short Papers) , pages 31–36, Melbourne, Australia.
Revised: Dou. 2018. HotFlip: White-box adversarial examples for text classiﬁcation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 31–36, Melbourne, Australia.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: Steffen Eger and Yannik Benz. 2020.
Revised: Steffen Eger and Yannik Benz. 2020.

Original: From hero to zéroe: A benchmark of low-level adversarial attacks.
Revised: From hero to zéroe: A benchmark of low-level adversarial attacks.

Original: In Proceedings of the 1st Conference of the
Revised: In Proceedings of the 1st Conference of the

Original: Asia-Paciﬁc Chapter of the Association for Computational Linguistics and the 10th International Joint
Revised: Asia-Paciﬁc Chapter of the Association for Computational Linguistics and the 10th International Joint

Original: Conference on Natural Language Processing , pages 786–803, Suzhou, China.
Revised: Conference on Natural Language Processing , pages 786–803, Suzhou, China.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: EIU.
Revised: EIU.

Original: Staying ahead of the curve: The business case for responsible AI.
Revised: Staying ahead of the curve: The business case for responsible AI.

Original: Technical report,
Revised: Technical report,

Original: The
Revised: The

Original: Economist Intelligence Unit.
Revised: Economist Intelligence Unit.

Original: EU. 2016.
Revised: EU. 2016.

Original: General data protection regulation.
Revised: General data protection regulation.

Original: European Commission. 2020.
Revised: European Commission. 2020.

Original: On artiﬁcial intelligence - a European approach to excellence and trust.
Revised: On artiﬁcial intelligence - a European approach to excellence and trust.

Original: Technical report, European Commission.
Revised: Technical report, European Commission.

Original: FDA.
Revised: FDA.

Original: Artiﬁcial intelligence/machine learning (ai/ml)-based software as a medical device (samd) action plan.
Revised: Artiﬁcial intelligence/machine learning (ai/ml)-based software as a medical device (samd) action plan.

Original: Technical report, U.S. Food & Drug
Revised: Technical report, U.S. Food & Drug

Original: Administration.
Revised: Administration.

Original: Todd Feathers.
Revised: Todd Feathers.

Original: Flawed algorithms are grading millions of students’ essays.
Revised: Flawed algorithms are grading millions of students’ essays.

Original: Vice .
Revised: Vice .

Original: Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019.
Revised: Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019.

Original: MRQAshared task: Evaluating generalization in reading comprehension
Revised: MRQAshared task: Evaluating generalization in reading comprehension.

Original:  Proceedings shop Machine  pages 1– Hong Kong, China.
Revised: In Proceedings of the 2nd Workshop on Machine Reading for Question Answering , pages 1–13, Hong Kong, China.

Original: Association for
Revised: Association for

Original: Computational Linguistics.
Revised: Computational Linguistics.

Original: Batya Friedman and David G Hendry.
Revised: Batya Friedman and David G Hendry.

Original: sitive Shaping  Moral - ination
Revised: Value Sensitive Design: Shaping Technology with Moral Imagination .

Original: MIT Press.
Revised: MIT Press.

Original: Bharath Ganesh.
Revised: Bharath Ganesh.

Original: The ungovernability of  hate culture.
Revised: The ungovernability of digital hate culture.

Original: Columbia Journal of International Affairs .
Revised: Columbia Journal of International Affairs .

Original: Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan
Revised: Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan

Original: Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi,
Revised: Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi,

Original: Dheeru Dua, Yanai Elazar, Ananth Gottumukkala,
Revised: Dheeru Dua, Yanai Elazar, Ananth Gottumukkala,

Original: Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco,
Revised: Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco,

Original: Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer
Revised: Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer

Original: Singh, Noah A. Smith, Sanjay Subramanian, Reut
Revised: Singh, Noah A. Smith, Sanjay Subramanian, Reut

Original: Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou.
Revised: Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou.

Original: Evaluating models’local decision boundaries 
Revised: Evaluating models’ decision boundaries sets

Original:  Findings of  for Computational Linguistics EMNLP  pages 1307–1323, Online.
Revised: In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1307–1323, Online.

Original: Association for Computational
Revised: Association for Computational

Original: Linguistics.
Revised: Linguistics.

Original: Siddhant Garg and Goutham Ramakrishnan. 2020.
Revised: Siddhant Garg and Goutham Ramakrishnan. 2020.

Original: BAE: BERT-based adversarial examples for text classiﬁcation.
Revised: BAE: BERT-based adversarial examples for text classiﬁcation.

Original: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , Online. Association for Computational Linguistics.
Revised: In Proceedings of the 2020 Confer

Original: Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford Mor
Revised: 9th

Original: , Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Revised: Joint Confer ence

Original: Processingand the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP) , pages 1161–1166, Hong Kong, China.
Revised: Processing (EMNLPIJCNLP) , pages 1161–1166, Hong Kong, China.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics

Original: Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong annd Mohit Bansal, and Christopher Ré. 2021. Robustness gym: Unifying the nlp evaluation landscape. arXiv preprint arXiv:2101.04840 .
Revised: Pretrained transformers improve out-of-distribution robustness

Original: Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversar ial examples. In 3rd International Conference on
Revised: 58th

Original: Learning Representations , San Diego, California. Alexa Hagerty and Igor Rubinov. 2019. Global ai ethics: a review of the social impacts and ethical implications of artiﬁcial intelligence. arXiv preprint arXiv:1907.07892 .
Revised: Association

Original: Hany Hassan, Anthony Aue, Chang Chen, Vishal

Original: Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018.
Revised: Computational

Original: Achieving human parity automatic chinese to english news translation arXiv preprint arXiv:1803.05567 .

Original: Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2744–2751, Online.
Revised: Online.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: Alex Hern. 2017.
Revised: Hern

Original: Facebook translates ‘good morning’ into ‘attack them’, leading to arrest.
Revised: Facebook translates ‘good morning’ into ‘attack them’, leading to arrest.

Original: The Guardian .
Revised: The Guardian .

Original: Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari.
Revised: Dirk Hovy Bianchi and Fornaciari

Original: “yousound just likeyour father” commercial machine translation systems include stylistic biases.
Revised: “you just like father” com mercial machine translation systems include stylistic biases

Original: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages1686–1690, Online.
Revised: 58th Annual - ing the Association for Computational Linguistics pages Online.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: Dirk Hovy and Shannon L Spruit.

Original: - sociation
Revised: tational sociation

Original: ISO/IEC/IEEE International standard - systems and software engineering–vocabulary.
Revised: ISO/IEC/IEEE International standard - systems and software engineering–vocabulary

Original: ISO/IEC/IEEE 24765:2017(E)
Revised: ISO/IEC/IEEE Iyyer

Original: , pages 1–541.

Original:  Wieting
Revised: John Wieting

Original: Gimpel
Revised: Kevin

Revised: Adversarial example generation with syntactically controlled paraphrase networks

Original: Zettlemoyer cal
Revised: In Empiri cal

Original: Copenhagen
Revised: Natural Language pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.

Original: Di Jin Zhijing Jin Joey Tianyi Zhou,  Peter
Revised: Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter

Original: Szolovits. 
Revised: Szolovits. 2020.

Original: Is BERT reallyrobust?
Revised: Is BERT really

Original: A strong baseline  language  on text classiﬁcation and entailment.
Revised: A strong baseline for natural language attack on text classiﬁcation and entailment.

Original: In The Thirty- AAAI Conference on Artiﬁcial Intelligence AAAI 
Revised: AAAI Conference on Artiﬁcial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of

Original: The Artiﬁcial Intelligence Conference, IAAI 
Revised: Artiﬁcial Intelligence Conference, IAAI 2020,

Original: The
Revised: The

Original: Tenth AAAI Symposium on Educational Advances in Artiﬁcial Intelligence, EAAI 2020, New York, NY,
Revised: Tenth AAAI Symposium on Educational Advances in Artiﬁcial Intelligence, EAAI 2020, New York, NY,

Original: USA, February 7-12, 2020 , pages 8018–8025.
Revised: USA, February 7-12, 2020 , pages 8018–8025.

Original: AAAI
Revised: AAAI

Original: Press.
Revised: Press.

Original: Khari Johnson 
Revised: Khari Johnson need to succeed

Revised: VentureBeat .

Original: Divyansh Kaushik, Eduard Hovy, and Zachary Lipton
Revised: Divyansh Kaushik, Eduard Hovy, and Zachary Lipton.

Revised: Zixuan Ke and Vincent

Original: Daniel Khashabi, Tushar Khot and Ashish Sabharwal
Revised: Daniel Khashabi, Tushar Khot, and Ashish Sabharwal.

Original: Shirish Keskar pages 540– 551, Hong Kong, China.
Revised: tion

Original: Association for Computational Linguistics.
Revised: Natural Processing (EMNLP) , Online tics.

Original: Zachary Laub.
Revised: Cann Neural summarization the Conference Conference

Original: on social media:
Revised: IJCNLP)

Original: Global comparisons.
Revised: Hong Kong, China

Original: Council on Foreign Relations .
Revised: Association Computa tional Linguistics.

Original: Kobi Leins, Jey Han Lau, and Timothy Baldwin. 2020.
Revised: Kobi Leins, Jey Han Lau and Timothy Baldwin

Original: Give me convenience and give her death: Who should decide what uses of NLP are appropriate, and on what basis?
Revised: Give me convenience and give her death: Who should decide what uses of NLP are appropriate, and on what basis?

Original: In Proceedings of the 58th Annual
Revised: In Proceedings of the 58th Annual

Original: Meeting of the Association for Computational Linguistics , pages 2908–2913, Online. Association for
Revised: Meeting of the Association for Computational Linguistics Online

Original: Computational Linguistics.
Revised: Computational Linguistics.

Original: Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li,and Ting
Revised: Jinfeng Li, Shouling Ji, Tianyu Du Bo Li,

Original: Wang.
Revised: Wang

Original: Textbugger: Generating adversarial text against real-world applications.

Original: In 26th Annual Network and Distributed System Security Symposium .
Revised: In 26th nual Network and Distributed System Security Symposium .

Original: Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020a.
Revised: Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue and Xipeng Qiu

Original: Bert-attack: Adversarial attack against BERT using BERT.
Revised: BERT BERT the

Original: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , Online. Association for
Revised: Nat

Original: ComputationalLinguistics.
Revised: Computational

Original: Yanwei Li, Araz Taeihagh, Martin de Jong, and Andreas Klinke. 2020b.
Revised: Yanwei Li, Araz Taeihagh, Martin de Jong, and Andreas Klinke. 2020b

Original: Towarda commonly shared public policy perspective for analyzing risk coping strategies. Risk Analysis . John Markoff. Essay-grading software offers professors a break.
Revised: Toward

Original: The New York Times . Alexandria Marsters. When Hate Speech Leads to Hateful Actions: A Corpus and Discourse Analytic Approach to Linguistic Threat Assessment of Hate Speech .
Revised: Nguyen

Original: Ph.D. thesis, Georgetown University,
Revised: Hal Daumé III

Original: Washington, D.C. Paul Michel, Xian Li, Graham Neubig, and Juan Pino.
Revised: Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.

Original: On evaluation of adversarial perturbations for sequence-to-sequence models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for ComputationalLinguistics:
Revised: Adversarial NLI new for natural language understanding 58th Annual Meeting of the Association for Computational

Original: Human Language Technologies, Volume 1 (Long and Short Papers) , pages3103–3114, Minneapolis, Minnesota. Association for Computational Linguistics. John Miller, Karl Krauth, Benjamin Recht, and Ludwig
Revised: Linguistics , pages

Original: Schmidt.
Revised: 4901, Online.

Original: The effect of natural distribution Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason and Douwe Kiela
Revised: Association for Computational Linguistics.

Original: NIST
Revised: NIST

Revised: U.S in ai federal engagement in developing technical standards and related tools.

Original:  Institute 
Revised: Technical report, National Institute of Standards and Technology. Saﬁya Umoja Noble. Algorithms of oppression:

Revised: How search engines reinforce racism .

Revised: NYU Press.

Original: Jekaterina Novikova, Ondˇrej Dušek, Amanda Cercas Curry, and Verena Rieser we big
Revised: Jekaterina Novikova, Ond ˇ rej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017.

Revised: Why we need new evaluation metrics for NLG

Revised: In Proceedings of the 2017 Conference on Empirical Methods in

Revised: Natural Language Processing pages

Original: Partnership on AI
Revised: Copenhagen, Denmark. Association for Computational Linguistics. Cathy O’Neil Weapons of math destruction:

Revised: How big data increases inequality and threatens democracy .

Revised: Crown.

Revised: Partnership on AI. 2019.

Original: About ML
Revised: About ML.

Original: Technical 
Revised: Technical report,

Original: Partnership on AI.
Revised: Partnership on AI.

Original: Barbara Plank.
Revised: Barbara Plank.

Original: What to do about non-standard (or non-canonical) language  nlp
Revised: What to do about non-standard (or non-canonical) language in nlp

Original: Proceedings  the 13th Conference on Natural Language Processing (KONVENS 2016) .
Revised: Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016) .

Original: Megha Rajagopalan, Lam Thuy Vo, and Aung Naing
Revised: Megha Rajagopalan, Lam Thuy Vo, and Aung Naing

Original: Soe.
Revised: Soe.

Original: How Facebook failedthe Rohingya in
Revised: How Facebook failed

Original: Myanmar. BuzzFeed News . Inioluwa Deborah Raji, Andrew Smart, Rebecca N
Revised: Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020.

Original: White, Margaret Mitchell, Timnit Gebru, Ben
Revised: Where responsible ai meets reality: Practitioner perspectives on enablers for shifting organizational practices. arXiv preprint arXiv:2006.12358 .

Original: Hutchinson, Jamila Smith-Loud, Daniel Theron, and 
Revised: Chaitanya Ramineni and David Williamson.

Original: Parker Barnes. 2020. Closing the ai accountability gap: Deﬁning an end-to-end framework for internal algorithmic auditing.
Revised: Understanding score rater® mographically

Original: In Proceedings of the
Revised: ETS Research Report Series

Original: Conference on Fairness, Accountability, and Transparency , pages 33–44. Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020. Where responsible ai meets reality: Practitioner perspectives on enablers for shifting organizational practices.
Revised: Suchi Saria and Adarsh Subbaswamy. 2019.

Original: arXiv
Revised: Safe and reliable machine learning ( ACM - ence

Original: Anders Søgaard, Sebastian Ebert, Joost Bastings, and 
Revised: Deven Santosh Shah, Andrew , and Dirk Hovy. 2020.

Revised: Predictive in natural processing models:

Original: Katja Filippova. 2021.
Revised: A conceptual framework and overview. In Proceedings of the 58th Annual Meeting of the , pages 5248–5264, Online. Association for Computational Linguistics.

Revised: Andrew Smith.

Revised: Using artiﬁcial intelligence and algorithms.

Revised: Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2021.

Original: We need to talk about random splits.
Revised: We need to talk about random splits.

Original: In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics .
Revised: In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 1823–1832, Online.

Original: Association for Computational
Revised: Association for Computational

Original: Linguistics.
Revised: Linguistics.

Original: Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M
Revised: Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M

Original: Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford,
Revised: Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford,

Original: Dario Amodei, and Paul Christiano. 2020.
Revised: Dario Amodei, and Paul Christiano. 2020.

Original: Learning to summarize from human feedback.
Revised: Learning to summarize from human feedback.

Original: arXiv preprint arXiv:2009.01325 .
Revised: arXiv preprint arXiv:2009.01325 .

Original: Samson Tan and Shaﬁq Joty. 2021.
Revised: Samson Tan and Shaﬁq Joty. 2021.

Original: Code-Mixing on
Revised: Code-Mixing on

Original: Sesame Street: Dawn of the adversarial polyglots.
Revised: Sesame Street: Dawn of the adversarial polyglots.

Original: In Proceedings of the 2021 Conference of the North
Revised: In Proceedings of the 2021 Conference of the North

Original: American Chapter of the Association for Computational Linguistics: Human Language Technologies , Mexico City, Mexico.
Revised: American Chapter of the Association for Computational Linguistics: Human Language Technologies , Mexico City, Mexico.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: Samson Tan, Shaﬁq Joty, Min-Yen Kan, and Richard
Revised: Samson Tan, Shaﬁq Joty, Min-Yen Kan, and Richard

Original: Socher. 2020.
Revised: Socher. 2020.

Original: It’s morphin’ time!
Revised: It’s morphin’ time!

Original: Combating linguistic discrimination with inﬂectional perturbations.
Revised: Combating linguistic discrimination with inﬂectional perturbations.

Original: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2920–2935, Online.
Revised: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2920–2935, Online.

Original: Association for Computational Linguistics.
Revised: Association for Computational Linguistics.

Original: Deborah Tannen. 1991.
Revised: Deborah Tannen. 1991.

Original: You just don’t understand:
Revised: You just don’t understand:

Original: Women and men in conversation .
Revised: Women and men in conversation .

Original: Ballantine books
Revised: Ballantine books

Original: New York.
Revised: New York.

Original: Deborah Tannen et al.
Revised: Deborah Tannen et al.

Original: Conversational style:
Revised: Conversational style:

Original: Analyzing talk among friends .
Revised: Analyzing talk among friends .

Original: Oxford University
Revised: Oxford University

Original: Press.
Revised: Press

Original: Viniti Vaish and Teck Kiang Tan.

Original: Language and social class: Linguistic capital in singapore

Original: In Annual Meeting of the American Educational Research

Original: Association .

Original: American Educational Research Association.

Original: Wagner,

Original: Garcia, Mohsen Jadidi, and

Original: It’s man’s assessing gender inequality in an online encyclopedia.

Original: In Proceedings of the International AAAI Conference on Web and Social Media volume 9.

Original: Lily Wakeﬁeld. 2020.

Original: Queer people are being forced off social media by trolling and online abuse, sear arXiv:2101.11974 .

Original: Li.

Original: 2019a.

Original: Generating ﬂuent adversarial examples for natural languages.

Original: In Proceedings of the 57th Annual Meeting of the Association for Computational

Original: Linguistics , pages 5564–5569, Florence, Italy.

Original: Association for Computational Linguistics.

Original: Shiyue Zhang, Benjamin Frey, and Mohit Bansal.

Original: 2020a.

Original: ChrEn: Cherokee-English machine translation for endangered language revitalization.

Original: In

Original: Proceedings of the 2020 Conference on Empirical

Original: Methods in Natural Language Processing (EMNLP) , pages 577–595, Online.

Original: Association for Computational Linguistics.

Original: Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li.

Original: 2020b.

Original: Adversarial attacks on deep-learning models in natural language processing: A survey.

Original: ACM Transactions on Intelligent Systems and Technology , 11(3).

Original: Yuan Zhang, Jason Baldridge, and Luheng He. 2019b.

Original: PAWS: Paraphrase adversaries from word scrambling.

Original: In Proceedings of the 2019 Conference of the North American Chapter of the Association for

Original: Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 1298–1308, Minneapolis, Minnesota.

Original: Association for Computational Linguistics.

Original: Appendix A Testing Dimensions: Detecting Violent Content on Social Media
Revised: Appendix A Testing Dimensions: Detecting Violent Content on Social Media

Original: In this second case study, we apply DOCTOR for measuring the reliability of a violent content detection system for English social media posts.
Revised: In this second case study, we apply DOCTOR for measuring the reliability of a violent content detection system for English social media posts.

Original: Although we limit this discussion to the U.S., this is a growing, global problem (Laub, 2019) that can lead to deadly outcomes (Rajagopalan et al., 2018).
Revised: Although we limit this discussion to the U.S., this is a growing, global problem (Laub, 2019) that can lead to deadly outcomes (Rajagopalan et al., 2018).

Original: In this hypothetical use case, the system may automatically remove violent content or alert content moderators to potential violations of the social media company’s acceptable use policy.
Revised: In this hypothetical use case, the system may automatically remove violent content or alert content moderators to potential violations of the social media company’s acceptable use policy.

Original: Moderators can decide if speciﬁc content should be removed, and if necessary, notify law enforcement to avert pending violence (e.g., threats against individuals, planned violent events).
Revised: Moderators can decide if speciﬁc content should be removed, and if necessary, notify law enforcement to avert pending violence (e.g., threats against individuals, planned violent events).

Original: It should be noted that social media platforms have broad latitude (Klonick, 2018) to develop their own policies for acceptable content and how they handle it as a result of the 1996 Communications Decency Act 10 .
Revised: It should be noted that social media platforms have broad latitude (Klonick, 2018) to develop their own policies for acceptable content and how they handle it as a result of the 1996 Communications Decency Act 10 .

Original: In this scenario, the compliance ofﬁcer of the company developing the system is responsible for making sure it does not discriminate against speciﬁc user demographics.
Revised: In this scenario, the compliance ofﬁcer of the company developing the system is responsible for making sure it does not discriminate against speciﬁc user demographics.

Original: Research has shown that hate speech can lead to hateful actions (Marsters, 2019).
Revised: Research has shown that hate speech can lead to hateful actions (Marsters, 2019).

Original: In many cases, individuals posted their intents online prior to committing violence (Cohen et al., 2014).
Revised: In many cases, individuals posted their intents online prior to committing violence (Cohen et al., 2014).

Original: When identifying content to remove and especially when involving law enforcement, it is important distinguish between "Hunters" (those who act) and "Howlers" (those who do not).
Revised: When identifying content to remove and especially when involving law enforcement, it is important distinguish between "Hunters" (those who act) and "Howlers" (those who do not).

Original: This is to avoid wrongly detaining individuals have no intention of committing violence, even if their words are not defensible.
Revised: This is to avoid wrongly detaining individuals have no intention of committing violence, even if their words are not defensible.

Original: Between the two extremes, posters may harass, stalk, dox, or otherwise abuse victims from a distance, therefore it is still necessary to ﬂag, remove, and potentially track or document violent content.
Revised: Between the two extremes, posters may harass, stalk, dox, or otherwise abuse victims from a distance, therefore it is still necessary to ﬂag, remove, and potentially track or document violent content.

Original: Linguistic landscape.
Revised: Linguistic landscape.

Original: In this case study, we focus on solely on English speakers, but we acknowledge that the actual linguistic landscape is much more complex (over 350 languages).
Revised: In this case study, we focus on solely on English speakers, but we acknowledge that the actual linguistic landscape is much more complex (over 350 languages).

Original: Posters on social media may speak English as their ﬁrst language or as a second language and they often codeswitch and code-mix.
Revised: Posters on social media may speak English as their ﬁrst language or as a second language and they often codeswitch and code-mix.

Original: General American English is used for business purposes in the U.S. but there are other frequently used language varieties including
Revised: General American English is used for business purposes in the U.S. but there are other frequently used language varieties including

Original: African American English (AAE), Cajun Vernacu- lar English, and three different Latino (Hispanic)
Revised: African American English (AAE), Cajun Vernacu- lar English, and three different Latino (Hispanic)

Original: Vernacular Englishes.
Revised: Vernacular Englishes.

Original: Stakeholder Impact.
Revised: Stakeholder Impact.

Original: The key stakeholders that will be impacted are those most often facing violent threats online: minorities, women, immigrants, and the LGBTQ community (Amnesty International, 2018; Ganesh, 2018;
Revised: The key stakeholders that will be impacted are those most often facing violent threats online: minorities, women, immigrants, and the LGBTQ community (Amnesty International, 2018; Ganesh, 2018;

Original: Davidson et al., 2019; Wakeﬁeld, 2020).
Revised: Davidson et al., 2019; Wakeﬁeld, 2020).

Original: Additionally, anyone that posts content on the social media site is a stakeholder.
Revised: Additionally, anyone that posts content on the social media site is a stakeholder.

Original: Unfortunately, the very communities that are often receiving violent posts, are often wrongly ﬂagged as posting toxic content themselves due to racial biases present in the training data (Sap et al., 2019;
Revised: Unfortunately, the very communities that are often receiving violent posts, are often wrongly ﬂagged as posting toxic content themselves due to racial biases present in the training data (Sap et al., 2019;

Original: Davidson et al., 2019).
Revised: Davidson et al., 2019).

Original: Given the risk of harm to victims if the system misses violent posts from hunters or misidentiﬁes content as violent and notiﬁes law enforcement, it is critical the right level of false positives and false negatives is achieved in ﬂagging content.
Revised: Given the risk of harm to victims if the system misses violent posts from hunters or misidentiﬁes content as violent and notiﬁes law enforcement, it is critical the right level of false positives and false negatives is achieved in ﬂagging content.

Original: Dimensions.
Revised: Dimensions.

Original: There are two tasks under consideration here: identifying violent content and identifying Hunters who “truly intend to use lethal violence” (Marsters, 2019).
Revised: There are two tasks under consideration here: identifying violent content and identifying Hunters who “truly intend to use lethal violence” (Marsters, 2019).

Original: In the ﬁrst task, the system is looking for content that negatively targets a socially deﬁned group.
Revised: In the ﬁrst task, the system is looking for content that negatively targets a socially deﬁned group.

Original: Additionally, the content includes not only hate speech (e.g., profanity, epithets, vulgarity) but also incites others to hatred or violence.
Revised: Additionally, the content includes not only hate speech (e.g., profanity, epithets, vulgarity) but also incites others to hatred or violence.

Original: Since content written in AAE has been shown to be ﬂagged as toxic more often (Sap et al., 2019; Davidson et al., 2019), we must ensure that the system is reliable when encountering dialectal variation.
Revised: Since content written in AAE has been shown to be ﬂagged as toxic more often (Sap et al., 2019; Davidson et al., 2019), we must ensure that the system is reliable when encountering dialectal variation.

Original: Additionally, due to the casual environment of social media, multilingual speakers often code-switch and code-mix.
Revised: Additionally, due to the casual environment of social media, multilingual speakers often code-switch and code-mix.

Original: Hence, we expect variation in these dimensions to have no effect on the system’s predictions: alternative spellings, morphosyntactic variation, word choice, code-mixing, idioms, and references to and manifestations of sensitive attributes and their proxies.
Revised: Hence, we expect variation in these dimensions to have no effect on the system’s predictions: alternative spellings, morphosyntactic variation, word choice, code-mixing, idioms, and references to and manifestations of sensitive attributes and their proxies.

Original: However, we must expect the system to be sensitive to in-group and out-group usage of reclaimed slurs so that the in-group usage does not result in a ﬂag while out-group usage result in ﬂagged posts.
Revised: However, we must expect the system to be sensitive to in-group and out-group usage of reclaimed slurs so that the in-group usage does not result in a ﬂag while out-group usage result in ﬂagged posts.

Original: When identifying hunters, we may expect the system to be sensitive to uses of ﬁrst person pronouns, certainty adverbs, negative evaluative adjectives, and modiﬁers (Marsters, 2019).
Revised: When identifying hunters, we may expect the system to be sensitive to uses of ﬁrst person pronouns, certainty adverbs, negative evaluative adjectives, and modiﬁers (Marsters, 2019).

Original: However, in order to avoid unfairly penalizing vernacular
Revised: However, in order to avoid unfairly penalizing vernacular

Original: English speakers we should expect the system’s predictions to be equally unaffected by variation in the dimensions listed for the other task.
Revised: English speakers we should expect the system’s predictions to be equally unaffected by variation in the dimensions listed for the other task.

Original: [Table] [Table]
Revised: [Table] [Table]