Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis? (2020)

Abstract: As part of growing NLP capabilities, coupled with an awareness of the ethical dimensions of research, questions have been raised about whether particular datasets and tasks should be deemed off-limits for NLP research. We examine this question with respect to a paper on automatic legal sentencing from EMNLP 2019 which was a source of some debate, in asking whether the paper should have been allowed to be published, who should have been charged with making such a decision, and on what basis. We focus in particular on the role of data statements in ethically assessing research, but also discuss the topic of dual use, and examine the outcomes of similar debates in other scientific disciplines.

## introduction nlp tools are increasingly being deployed in the wild with potentially profound societal implications. alongside the rise in technical capabilities has been a growing awareness of the moral obligation of the field to self-assess issues including: dataset and system bias @xcite , dataset ethics @xcite , and dual use @xcite . more recently, there has also been vigorous debate on whether it is ethical for the community to work on certain topics or data types. this paper aims to investigate this issue, focused around the examination of a paper recently published at emnlp 2019 on automatic prison term prediction by @xcite . specifically, the paper in question proposes a neural model which performs structured prediction of the individual charges laid against an individual, and the prison term associated with each, which can provide an overall prediction of the prison term associated with the case. this model was constructed using a large-scale dataset of real-world chinese court cases. the primary question we attempt to address in this paper is on what basis a given paper satisfies basic ethical requirements for publication, in addition to examining the related question of who should make this judgement. note that our intention is in no way to victimise the authors of the paper in question, but rather to use it as a test case to objectively ground an ethical assessment. the authors did highlight potential ethical concerns of its application, but missed the point that there are data ethics issue in the first place. note also that, given the topic of the paper, we will focus somewhat on nlp applications in the legal domain, but the majority of the findings/recommendations generalise and will be of equal relevance to other domains. ## comparable concerns in the biological sciences we have consultations, which of the inventions and experiences which we have discovered shall be published, and which not; and take all an oath of secrecy for the concealing of those which we think fit to keep secret; though some of those we do reveal sometime to the state, and some not. ## who decides who decides? questions associated with who decides what should be published are not only legal, as illustrated in fouchier's work, but also fundamentally philosophical. how should values be considered and reflected within a community? what methodologies should be used to decide what is acceptable and what is not? who assesses the risk of dual use, misuse or potential weaponisation? and who decides that potential scientific advances are so socially or morally repugnant that they cannot be permitted? how do we balance competing interests in light of complex systems @xcite . much like nuclear, chemical and biological scientists in times past, computer scientists are increasingly being questioned about the potential applications, and long-term impact, of their work, and should at the very least be attuned to the issues and trained to perform a basic ethical self-assessment. ## moving forward given all of the above, what should have been the course of action for the paper in question? it is important to note that the only mentions of research integrity/ethics in the call for papers relate to author anonymisation, dual submissions, originality, and the veracity of the research, meaning that there was no relevant mechanism for reviewers or pc chairs to draw on in ruling on the ethics of this or any other submission. a recent innovation in this direction has been the adoption of the acm code of ethics by the association for computational linguistics, and explicit requirement in the emnlp 2020 calls for papers for conformance with the code: 10 where a paper may raise ethical issues, we ask that you include in the paper an explicit discussion of these issues, which will be taken into account in the review process. we reserve the right to reject papers on ethical grounds, where the authors are judged to have operated counter to the code of ethics, or have inadequately addressed legitimate ethical concerns with their work this is an important first step, in providing a structure for the program committee to assess a paper for ethical compliance, and potentially reject it in cases of significant concerns. having said this, the acm code of ethics is (deliberately) abstract in its terms, with relevant principles which would guide an assessment of the paper in question including: 1.2 avoid harm; 1.4 be fair and take action not to discriminate; 1.6 respect privacy; 2.6 perform work only in areas of competence; and 3.1 ensure that the public good is the central concern during all professional computing work. in each of these cases, the introspection present in a clearlyarticulated data statement would help ameliorate potential concerns. what could an ethics assessment for acl look like? would an ethics statement for acl be enough to address all concerns? as argued above, it is not clear that acl should attempt to position itself as ethical gatekeeper, or has the resources to do so. and even if acl could do so, and wanted to do so, the efficacy of ethics to answer complex political and societal challenges needs to be questioned @xcite . there certainly seems to be an argument for a requirement that papers describing new datasets are accompanied by a data statement or datasheet of some form (e.g. as part of the supplementary material, to avoid concerns over this using up valuable space in the body of the paper). this still leaves the question of what to do with pre-existing datasets: should they all be given a free pass; or should there be a requirement for a data statement to be retrospectively completed? the gdpr provides some protection for the use of data, but its scope and geographic reach are limited. further, the term "anonymised" is often a misnomer as even data that is classified by governments and other actors as "anonymous" can often easily be reidentified @xcite . what about code and model releases? should there be a requirement that code/model releases also be subject to scrutiny for possible misuse, e.g. via a central database/registry? as noted above, there are certainly cases where even if there are no potential issues with the dataset, the resulting model can potentially be used for harm (e.g. gpt-2). one could consider this as part of an extension of data statements, in requiring that all code/model releases associated with acl papers be accompanied with a structured risk assessment of some description, and if risk is found to exist, some management plan be put in place. looking to other scientific disciplines that have faced similar issues in the past may provide some guidance for our future. finally, while we have used one particular paper as a case study throughout this paper, our intent was in no way to name and shame the authors, but rather to use it as a case study to explore different ethical dimensions of research publications, and attempt to foster much broader debate on this critical issue for nlp research.