Handling and Presenting Harmful Text in NLP Research (2022)

Abstract: Text data can pose a risk of harm. However, the risks are not fully understood, and how to handle, present, and discuss harmful text in a safe way remains an unresolved issue in the NLP community. We provide an analytical framework categorising harms on three axes: (1) the harm type (e.g., misinformation, hate speech or racial stereotypes); (2) whether a harm is sought as a feature of the research design if explicitly studying harmful content (e.g., training a hate speech classifier), versus unsought if harmful content is encountered when working on unrelated problems (e.g., language generation or part-of-speech tagging); and (3) who it affects, from people (mis)represented in the data to those handling the data and those publishing on the data. We provide advice for practitioners, with concrete steps for mitigating harm in research and in publication. To assist implementation we introduce HarmCheck – a documentation standard for handling and presenting harmful text in research.

## introduction text data can cause harm in a range of ways. first, different content creates different risks of harm. for instance, misinformation can contaminate the information landscape and polarise groups @xcite ; hate speech and abusive language can pollute online communities and inflict long-lasting trauma on its victims @xcite ; negative social stereotypes and misrepresentations of individuals or groups can perpetuate historical injustice and lead to unjust allocation of opportunities or resources @xcite . second, different research settings affect how harms are encountered. some harmful content is sought when researchers deliberately investigate phenomena such as hate speech, extremism or misinformation. in other cases, researchers are working in seemingly unrelated domains (e.g., language generation, part-of-speech tagging or semantic search) but may still encounter unsought harmful content, especially if the data are scraped from internet sources @xcite @xcite . third, different groups are harmed by text content during the research process and may suffer immediate, representational or vicarious harms. these groups include data subjects (i.e., people represented in the data); data handlers and researchers, (i.e., those who collect, annotate or audit the data, and produce research outputs) @xcite @xcite ; and readers and reviewers (i.e., those who read research outputs). each of these complexities -what the source of harm is, how one is harmed and who is harmed -presents concerning ethical and methodological challenges that need to be addressed for the nlp field to advance in a responsible and equitable manner. if left unaddressed, inadequate safeguarding can stifle healthy research practice around harmful content. careless presentation of examples in academic papers perpetuates negative portrayals of data subjects and could distress readers, while incautious research protocols inflict an emotional toll on data handlers, annotators and researchers. furthermore, if not explicitly stated, researchers risk being misconstrued as aligning with the views expressed in harmful content. despite these concerns, avoiding any research that bears a risk of harm is untenable because harmful content is a feature of many datasets @xcite . bringing text harms to light and communcating their features via research is a critical first step to tackling them. we address these unresolved challenges, and provide both an analytical framework that describes the sources of harm, who they affect and how; and a set of guidelines which outline the responsibilities that researchers have, to others and to themselves, when handling and presenting harmful text. to ease adoption of our recommendations, we present harmcheck -a checklist for transparent, responsible, and reflective reporting of text harms in research. while algorithm audits and the risks of harm arising from model deployment have received relatively widespread attention (for a summary, see @xcite , there is significantly less research documenting the risks that arise pre-deployment, i.e., during the research design, dataset processing and publication stages. the novel contribution of our work is to establish a common ground and encourage better protocols for continuing empirical research on harmful text in a safe and proportionate manner. note: any paper that presents harmful content should include a clear content warning in the introduction or abstract at least a page before any examples are shown. content warning: this document discusses examples of harmful content (hate, abuse, misinformation and negative stereotypes). the authors do not support the use of harmful language, nor any of the harmful representations quoted below. ## guidelines for handling and presenting harmful text in areas of academic research outside of nlp, there are well-established practices for reducing the risk and severity of harm that researchers experience. people working in a chemistry lab are protected by safety protocols for working with hazardous materials, such as poisonous gases or radioactive substances. analogously, there should be protocols for reducing the risk of harm from hazardous materials in language. this section outlines ways to reduce the risk of harm from text data. we present our guidance in chronological order of the contact points that arise during research and practice. ## harmcheck: a checklist for handling and presenting harmful text in recent years, there has been a movement towards the responsible and transparent documentation of research artefacts @xcite @xcite , as well as encouraging best practices for ethical nlp research @xcite . some conferences now require that authors fill in a responsible nlp checklist to accompany their submission. 3 in a similar vein, we encourage standardised documentation of harmful content contained in research outputs. to this end, we present harmcheck, a simple checklist that works as both a standalone piece of documentation and as an addendum to existing documentation standards. it is intended for people specifically researching sought harmful content. each section is designed to be filled in as a statement (such as in a data statement @xcite or model card @xcite ), with some sections being more appropriate for different harm types. researchers should focus on the areas that are most relevant to their research. harmcheck should not be seen as a compliance form, but as a starting point for a broader conversation about the risk of harm through research. harmcheck is not exhaustive of all possible harmful text or ways of handling them. transparency through documentation should be seen primarily as an opportunity for reflecting on and scrutinising research, rather than a box-ticking exercise. although we provide a list of starter questions for 3 https://github.com/acl-org/ responsiblenlpresearch each section to guide researchers, they should not be seen as a definitive or complete set for assessing harm in all cases. in appendix a, we demonstrate how harmcheck can be used by providing completed harm statement examples for three types of harmful text: hate speech, misinformation and negative stereotypes. ## conclusion harmful content can be contained in datasets, and encountered whether one is looking for it or not. various forms of harmful text risk inflicting serious negative effects on many groups of people -prior to, during and after the research process. some professional and academic areas have established protocols for dealing with their inherent hazards and harms -and we have now proposed such a protocol for nlp research. the harm caused by textual content does not emerge in a social, cultural or historical vacuumdata embeds and perpetuates social norms, historical injustices, and uneven power dynamics. as such, technical fixes from individual researchers or adopted protocols cannot fully address this problem. instead, it requires acknowledging and contesting unjust systems and their manifestations in text data, envisioning what worldview the data should (or should not) represent, and eventually working towards making such visions a reality. we do not suggest that the nlp community can alone bear the weight of responsibility for countering the deep-rooted historical, cultural and societal issues in-grained in language data -this remains an unresolved problem which requires systemic change from multidisciplinary perspectives. at the same time, while tackling harmful content in datasets requires broader systemic change, adopting better practices in discussion and presentation can incrementally contribute to such change. thus, another unresolved problem in nlp is how the production, sharing and consumption of research itself can be approached in a way that encourages awareness and mitigation of harms. this paper has identified the problems and risks that emerge when handling and presenting harmful text. it also provides practical solutions, including the introduction of harmcheck. discussions about the harm caused by and through nlp research, as well as other computational and data-intensive areas, are still at an early stage, despite the progress made in recent times. as such, the arguments and recommendations we have presented here are only the start of a larger conversation about risks and harms in our field. we hope that delineating and describing them opens a broad dialogue in the nlp community towards responsible, safe and ethical research on harmful text.