Ethics consideration sections in natural language processing papers (2022)

Abstract: In this paper, we present the results of a manual classification of all ethical consideration sections for ACL 2021. We also compare how many papers had an ethics consideration section per track and per world region in ACL 2021. We classified papers according to the ethical issues covered (research benefits, potential harms, and vulnerable groups affected) and whether the paper was marked as requiring ethics review by at least one reviewer. Moreover, we discuss recurring obstacles we have observed (highlighting some interesting texts we found along the way) and conclude with three suggestions. We think that this paper may be useful for anyone who needs to write — or review — an ethics section and would like to get an overview of what others have done.

## introduction the first conference of the association for computational linguistics (acl) to include an ethics advisory committee was the one organized by its north american chapter (naacl) in 2021. since then all acl conferences have had one, including all chapter conferences, and the largest conferences acl-ijcnlp and emnlp. the acl 2021 webpage call for papers says the following. 1 authors will be allowed extra space after the 8th page for a broader impact statement or other discussion of ethics. note that though the ethical consideration paragraph is not mandatory, authors of papers working with sensitive data or on sensitive tasks that do not sufficiently discuss these issues may receive a conditional acceptance recommendation. this paper examines the current state of the nlp research community's discussion of the ethical impact of its work, as reflected in its publications. we first pose five empirical questions, and then explore 1 source https://2021.aclweb.org/calls/papers/ possible obstacles to the discussion. the paper is organized as follows. section 2 briefly reviews related work, giving references relevant to several areas of nlp. in section 3 we explore the following five questions regarding ethical consideration sections in acl 2021 papers: ## related previous work since the first acl workshop on ethics in natural language processing in 2017, the acl has initiated regular discussions of various aspects of ethics and the social impacts of nlp. ethical guidelines have been proposed for broad research areas such as machine translation @xcite and natural language generation @xcite and for more specialized topics, for example, text simplification @xcite . resources exist for nlp in general @xcite , for ai in the community @xcite , and for considerations relevant when designing shared tasks or benchmarks @xcite . there are many references relevant to writing a good ecs. we have no space to discuss them, but we note the following references relevant to responsible data usage @xcite @xcite @xcite @xcite , crowdsourcing @xcite @xcite @xcite @xcite , biases @xcite , language diversity @xcite @xcite @xcite , rigorous and meaningful evaluation @xcite @xcite , environmental impact @xcite @xcite @xcite , and human harms and values @xcite @xcite . ## methodology and empirical results in this section we first describe how we answered the questions posed in the introduction, and then present and discuss our findings. ## overcoming obstacles in this section we reflect on the obstacles facing the discussion of ethical issues by the nlp community. perhaps the most basic question to pose here is whether ethical issues can usefully be discussed in ecss in research papers. in our experience, there are many people in the nlp community who are not keen on the idea. sometimes this seems to come from worries about doing it well: i'm a computer scientist -what do i know about ethics? or perhaps my english is ok for writing technical papers, but no way could i write about benefits and harms. but we have also heard frustrations about the very idea of writing ecss: they are a waste of time, all that has to be said already has been said, that there simply is nothing serious to say here (when i asked my supervisor about this, he laughed). in short: it is nothing but a waste of valuable research time. but many authors do have interesting things to say. the following is from an ecs on the track interpretability and analysis of models for nlp @xcite reflecting on vulnerable groups (see boldface) and containing critical evaluations of harms and benefits: there are several ethical reasons to study lms with limited pretraining data. training massive lms like roberta from scratch comes with non-trivial environmental costs @xcite , and they are expensive to train, limiting contributions to pretraining research from scientists in lower-resource contexts. by evaluating lms with limited pretraining, we demonstrate that smaller lms match massive ones in performance in many respects. we also identify a clear gap in our knowledge regarding why extensive pretraining is effective. answering this question could lead to more efficient pretraining and ultimately reduce environmental costs and make nlp more accessible. here is an example of an ecs in the ethics in nlp track (blodgett et al., 2021) reflecting on potential harms when it is hard to see them. work concerning the fairness, transparency, or ethics of computational systems is often taken to be inherently beneficial with little to no potential for harm, and thus often (paradoxically) fails to examine its limitations or possible unintended negative consequences @xcite . and yet, our work is not without risks either; we risk discouraging the type of work we actually want to encourage, and dissuading practitioners from using existing benchmarks to test their models. the next example shows that it is possible to discuss the benefits of the research without overselling it in the dialogue and interactive system track track @xcite . considerable additional work is needed to determine what are appropriate levels of support for systems to provide or that can be expected from systems, but our work provides a cautious, yet concrete, step towards developing systems capable of reasonably modest levels of support. the corpus we construct can also provide examples to enable future work that probes the ethical extent to which systems can or should provide support. in addition to these broader ethical considerations, we have sought to ethically conduct this study, including by transparently communicating with crowdworkers about data use and study intent, compensating workers at a reasonable hourly wage, and obtaining study approval from the institutional review board. over the years we have also heard many statements to the effect that my work is completely theoretical -it raises no ethical issues at all. thus we also found the next ecs insightful. it is from a paper that links theoretical work on transfer learning with risks for language diversity @xcite , and the authors made the following point: we discuss the limitations and challenges in utilizing universal parsers to benefit the pretrained models. among the negative aspects of our work is the lack of explanation why some languages get more benefits over others due to universal syntax knowledge incorporation. however, as mentioned earlier, another type of ecs caught our attention: what we called 'disclaimers'. these sometimes draw on factual statements-for example, noting that the project leading to the research had been classified as no risk-but sometimes rely on more subjective judgements, for example, that the work raises no ethical issues not already raised by established nlp applications. we would immediately like to emphasize that there is nothing wrong with disclaimers. the fact that a review board has classified research as (say) low risk may be worth noting-though to say only that seems unlikely to raise many interesting ethical issues. what is potentially concerning, however, is that this type of ecs could become a template, perhaps something to teach new phd students (you write the abstract like so, the acknowledgments like so, and the ecs like so-got it?). all well and good-but if folding ecss into the structure of a research paper becomes a mechanical exercise, this could tame their potential. this leads to tough questions. what are ecss meant to achieve? to encourage self-reflection among researchers? to signal compliance with institutionalized norms of best practice? how can ecss best be used to reach these goals? whatever the answers, we think it will be enlightening to track the different types of ecss (as we did in this paper) in future acl events. ## three concrete suggestions we conclude here with three concrete suggestions. first (for organizers) we suggest that they give authors the option of asking their paper to be reviewed by an ethics committee. one of our findings is that ethics review can be an educative process in which authors may gain a better understanding of the impact of their work. second (again for organizers) all the acl 2021 papers are available in the anthology with their videos, but almost none mention ethical considerations; we propose that subsequent conferences offer (optional) extra video time for ethical commentary. thirdly, in october 2021 the acl established a stable ethics committee to provide consistency between conferences. one of its goals is to build the acl ethics union bibliography. this is a public list moderated by the current acl ethics committee; you issue a pull request against the repository to have your suggestions discussed before they are approved for integration with the list. 6 any contributor with an approved suggestion can add their name to the list of contributors to the bibliography. contributions can be assigned a topic tag in the areas we discussed in section 2. we suggest that all acl members check it out! ## conclusions we believe our paper may be useful for authors writing an ethics section -and perhaps even for reviewers of such sections -who would like to get an overview of what others have done, including what might be addressed (benefits, harms, vulnerable groups). this paper provides a dive into the contents of ethical consideration sections (ecs) in acl proceedings papers from last year. specifically, the paper compares the percentages of papers containing ecs from different countries and the contents of these ecs. we also discuss the types of obstacles facing ecs writers, and provide concrete suggestions for authors, reviewers, and organizers. we wrote this paper because we think it is important to inspire the community to think about the social impacts of their work. we value diverse perspectives -so we do not supply cut-and-dried solutions to the issues we raise. rather, we have tried to make ethical discussions more meaningful by drawing attention to what has already been done in ecss. the statistics we reported are simple and basic, and doubtless more detailed analyses will be made as more data becomes available. but our main goal here was to raise these issues clearly and directly -and as soon as possible -to the nlp community.