Thorny Roses: Investigating the Dual Use Dilemma in Natural Language Processing (2023)

Abstract: Dual use, the intentional, harmful reuse of technology and scientific artefacts, is an ill-defined problem within the context of Natural Language Processing (NLP). As large language models (LLMs) have advanced in their capabilities and become more accessible, the risk of their intentional misuse becomes more prevalent. To prevent such intentional malicious use, it is necessary for NLP researchers and practitioners to understand and mitigate the risks of their research. Hence, we present an NLP-specific definition of dual use informed by researchers and practitioners in the field. Further, we propose a checklist focusing on dual-use in NLP, that can be integrated into existing conference ethics-frameworks. The definition and checklist are created based on a survey of NLP researchers and practitioners.

## introduction as usage of nlp artefacts (e.g., code, data, and models) increases in research and commercial efforts, it becomes more important to examine their rightful use and potential misuse. the association for computing machinery code of ethics 2 , which has been adopted by acl 3 , states that "[c]omputing professionals should consider whether the results of their efforts (...) will be used in socially responsible ways". new regulation in the european union also requires producers of llms to outline foreseeable misuses and mitigation techniques @xcite . however, without guidance on how to assess the impacts on society, researchers and practitioners (henceforth professionals) are illequipped to consider social responsibility in deciding on tasks to work on or managing resulting while very few research projects have malicious motivations, some are reused to harm any but particularly marginalised groups of society. this presents a crucial gap in the ethics of artificial intelligence and nlp on malicious reuse, or dual use, which has been particularly absent in literature. concerns of dual use of artificial intelligence (ai) have been discussed by prior work (e.g., @xcite @xcite @xcite . however, nlp technologies are rarely included in such considerations. as llms are being incorporated into a wide range of consumer-facing products, the dual use considerations become increasingly critical for research and deployment. for instance, a online mental health services company has used chatgpt to respond to people seeking advice and aid, raising concerns of informed consent, and impact on vulnerable users @xcite . to address such misuse of llms, it is crucial to support researchers in making decisions to limit the dual use of their work. we propose a checklist that can be implemented by researchers to guide them in the decision making. if integrated into a paper, the checklist further serves to elucidate the considerations made and approaches taken to mitigate dual use to other professionals. we keep the checklist concise, such that it can readily be integrated into existing checklists for ethics used by conferences within nlp. the checklist is developed based on a mixed-methods survey. in the survey of nlp professionals, we find that the majority of our respondents do not spend a significant amount of time considering dual use concerns arising from their work and that institutional support often is inadequate. in summary, our paper contributes: (a) a definition of dual use for nlp artefacts; (b) a survey of nlp professionals documenting their concerns of misuse of nlp artefacts; and (c) a checklist and outlines of methods for mitigating risks of dual use. ## dual use in nlp nlp artefacts, such as llms, can have severe impacts on society, even when they are not developed with malicious intent. addressing the issue of malicious use, the european union defines dual use as: "goods, software and technology that can be used for both civilian and military applications". 4 however, similarly to many other definitions of dual use, this definition only transfers to a subset of nlp projects. from the field of neuroscience, @xcite suggest that the current military use and export control lens for dual use is not adequate for scientists and policy-makers to anticipate harms and recommend investigating the political, security, intelligence, and military domains of application. drawing from dual use research in life sciences, @xcite highlight the need for considerations in ai in light of nlp models as sophisticated text generators. we argue that the field of nlp is at a crossroadon one hand, artefacts have become more accessible. on the other, the increased access to resources limits the control over their use. while availability and open access can facilitate reproducible science, there is a need to develop practices that prevent the malicious use of artefacts. discussing the social impacts of nlp, @xcite describe dual use as "unintended consequences of research". however, this definition leaves malicious reuse out of scope. @xcite argue that dual use definitions should neither be too narrow or too broad in scope, as that results in limiting what can be considered or make mitigating risks of dual use unmanageable. @xcite further posit that the context of dual use is critical in determining its implications and context-specific threats and risks should be specified. therefore, in our work, we present a nlp specific definition of dual use, and an associated checklist that is based on the insights from our survey of the field. our definition is founded in a classification of harms @xcite . ## survey of nlp professionals in order to develop a checklist, we survey members of the nlp community on their stances towards misuse of research artefacts through an anonymous online form. @xcite conduct a survey of nlp professionals' perspectives on general topics, e.g. potential decline in industry interest and promising research directions, dual use is notably absent from consideration. indeed, only one question alludes to the issue: "is it unethical to build easily-misusable systems?" to which 59% respond in the affirmative, indicating that professionals are responsible for minimising misuse within their practice. we similarly conduct an anonymous survey of nlp professionals, seeking detailed insights into their views on dual use concerns. our survey sought to elicit participants understanding of dual use, their perception of harms arising from it, and how to mitigate such harms. our adapted survey questions were shaped by feedback on a preliminary version shared researchers within the same group, at different levels of seniority and discussions with colleagues in academia and industry. the survey was open to the nlp community, widely, from april until the end of june 2022 and advertised on twitter, professional mailing lists and relevant reddit communities. 5 the survey was conducted on the limesurvey 6 platform, and approved by the university of copenhagen ethics board under number 504-0313/22-5000. the survey contained a total of 23 questions, which consist of a mixture of multiple-choice questions, free-form text answers, and likert scales (see appendix b for the survey setup). we inductively code all free-form text answers (see ## prevention of dual use in prior work, there are a set of approaches that address the problem of mitigating the misuse of research. in the following, we present four approaches mentioned in the context of machine learning at large; forbidden knowledge, ethics review boards, education of researchers, and guidelines/checklists. ## conclusion in this paper, we discussed the topic of dual use, the intentional, harmful misuse of research artefacts, in the context of nlp research. dual use has been overlooked in the field of nlp, a gap this paper aims to close. in order to gauge the current level of consideration for dual use in the field of nlp, we conducted a survey among nlp professionals. our survey results revealed that a majority of the participants expressed concerns regarding the potential misuse of their work. they identified several potential dangers associated with their work, including the oppression of (marginalised) groups in society, surveillance, and manipulation. however, they reported taking limited steps toward preventing such misuse. based on the participants' inputs on a provided definition of dual use, we propose a definition of dual use that is more appropriate for researchers in the field of nlp. we discuss existing proposals to address dual use in machine learning and other research fields. many of these approaches need institutional structures to be realised on a large scale. despite the participants' concerns about the potential for misuse of their work in nlp, there appears to be a lack of institutional support for addressing these concerns fully. it is imperative for structured and comprehensive initiatives to be implemented to address this issue. utilising the participants' insights, we have developed a dual use checklist for nlp professionals. this checklist, which can be integrated with existing ethics checklists, addresses both the potential dual use of scientific artefacts and measures to prevent such dual use. in conclusion, the dual use of nlp artefacts is a critical concern that has received limited attention in the field. our paper strives to raise awareness and initiate a conversation among nlp professionals to address this important and overlooked issue.