NLP Security and Ethics, in the Wild (2025)

Abstract: As NLP models are used by a growing number of end-users, an area of increasing importance is NLP Security (NLPSec): assessing the vulnerability of models to malicious attacks and developing comprehensive countermeasures against them. While work at the intersection of NLP and cybersecurity has the potential to create safer NLP for all, accidental oversights can result in tangible harm (e.g., breaches of privacy or proliferation of malicious models). In this emerging field, however, the research ethics of NLP have not yet faced many of the long-standing conundrums pertinent to cybersecurity, until now. We thus examine contemporary works across NLPSec, and explore their engagement with cybersecurity’s ethical norms. We identify trends across the literature, ultimately finding alarming gaps on topics like harm minimization and responsible disclosure. To alleviate these concerns, we provide concrete recommendations to help NLP researchers navigate this space more ethically, bridging the gap between traditional cybersecurity and NLP ethics, which we frame as “white hat NLP”. The goal of this work is to help cultivate an intentional culture of ethical research for those working in NLP Security.

## introduction securing large language models (llms) and nlp technology in general has not been a priority until recently. yet mass adoption of this technology has led to deployment in contexts where security failures present a risk to individuals, organizations, and society at large-demonstrated by, inter alia, llm-assisted identity fraud @xcite , phishing campaigns @xcite , and automated influence operations @xcite . if the latest nlp technologies are to withstand an increasing barrage of threats, nlp practitioners must now educate themselves on cybersecurity and cybersecurity practitioners must educate themselves on nlp. yet in this process, as one culture of research adapts to another, there is potential for long-standing intra-community norms to be lost in translation, including ethical norms. in interdisciplinary research, an accidental oversight of ethical norms from one field can risk reintroducing previously resolved ethical dilemmas. discussions around ethical research conduct have long been a concern for both cybersecurity @xcite @xcite @xcite and for nlp @xcite @xcite @xcite @xcite , but given that these disciplines have historically been disjoint fields, the specific ethical and sociocultural norms of both fields have developed in separate silos. to better understand how interdisciplinary nlpsec has adapted to the ethical norms and values across both disciplines, we examine a set of peer-reviewed nlpsec publications from nlp venues to gauge the compliance with norms in cybersecurity. we find that several principles regarded by the cybersecurity community as best practices have not been widely adopted in nlpsec research, despite measures in the nlp peer-review process to improve research practices. simultaneously, we find that nlp ethical norms regarding lower-resourced languages are at risk of being overlooked in nlpsec. to help save nlpsec the potential growing pains of reinventing the wheel, in this work, we seek to address this issue through an interdisciplinary conversation about ethics, with the goal of cultivating a culture of white hat nlp. ethical nlp security (nlpsec) in cybersecurity, a ''white hat'' hacker is typically a professional hired by a company with the specific purpose of maintaining or increasing the existing security of a computer system. white hat hackers, foot_1 referred to more generally as ''ethical hackers'', are keen to engage with vendors-either their employer or a third party-whose products they have found flaws in. they aim to get security issues fixed quickly and release details of their findings to the public so defenders can evaluate their own environments and prioritize concomitant patching and mitigation efforts. in contrast, a ''black hat'' hacker represents a cybercriminal, and somewhere in between is the ''gray hat'' hacker, who infiltrates others' computer systems without permission, with the intention of enhancing security @xcite . this is generally mapped to similar activity in the context of llms, where llm red teaming is defined as a limit-seeking, manual, and non-malicious activity @xcite . outside the strict boundaries of ethical hackers, gray hat hackers rely on their own moral compass, which can lead to potentially precarious situations, like introducing the risk of authorized system intrusions @xcite . alarmingly, much of the contemporary work in nlpsec is arguably similar to the above gray hat scenario (figure 1 ). due to the distributed and decentralized nature of the research landscape, most nlpsec researchers will necessarily lack a mandate from organizations to investigate security vulnerabilities inherent to models. researchers also lack direct accountability to those most affected by the security breaches they study. while the acm code of ethics foot_2 provides overarching guidance, it naturally cannot provide exact guidance for every ethical conundrum, leaving individual researchers to rely on their own moral compass, like a gray hat hacker. it is in this light that we aim to extend the framework of ''white hat'' to nlp. we thus define the scope of white hat nlpsec to consist of works which are intentionally and carefully grounded in the established ethical best practices of both cybersecurity and nlp. contributions we examine the current culture of research ethics in nlpsec through a survey of 80 peer-reviewed works across nlp venues, measuring their compliance with typical ethical practices from cybersecurity. to start, we introduce these ethical practices, and touch upon their relevance to works in nlpsec (section 2). we then describe our selection process for papers included in the survey, and our annotation process identifying compliance with these cybersecurity ethical best practices (section 3). concretely, we find that ethical norms like harm minimization and responsible disclosure are not widely adopted in nlpsec (section 4). we identify several challenges of adopting ethical norms from both cybersecurity and nlp to the field of nlpsec, and we provide concrete advice to help researchers engage in research more ethically than previously (section 5). while the recommendations presented in this work cannot provide solutions to all potential ethical conundrums latent to nlpsec, the discussions in this work aim to highlight the urgency for discussion on research ethics in nlpsec and serve as a conversation starter. ## the culture of ethics in cybersecurity within any field, some research may have negative consequences, and foreseeable risks generally serve as a guiding force for calibrating ethical norms. this holds particularly true for cybersecurity, as a field that is largely engaged with preempting and outmaneuvering criminals while protecting the public. as the field is naturally situated in a sensitive context, cybersecurity's research culture has unsurprisingly evolved to be one that often faces difficult ethical discussions @xcite . in this section, we will introduce three ethical norms commonly expected of works across cybersecurity. for each norm, we further describe potential nuances, borrowing relevant trolley problems directly from @xcite . our aim is to familiarize readers with these concepts, demonstrate how best practices in cybersecurity continue to be developed, and briefly touch upon how these concepts are relevant to works in nlpsec. ## methodology we examine 80 pertinent, peer-reviewed works across nlpsec for common trends and themes pertaining to discussions on research ethics. all papers were manually gathered from the acl anthology, 4 by querying keywords associated with common attack types (i.e., ''security'' + {''adversarial'', ''backdoor'', ''data recreation'', ''inversion'', ''instance encoding''}) to ensure they fall within the scope of nlpsec. we specifically did not include ''attack'' or ''defense'' in our keyword search, to avoid influencing the results. for each keyword search, we examined the relevance-sorted list of results. as terms like ''adversarial'' or ''inversion'' can be used in a wide variety of contexts beyond nlpsec, we first review each paper title, and keep those which are obviously relevant to nlpsec; where relevance is unclear from the title alone, we further scan the abstract to determine relevance. papers dated between 2019 and 2023 were obtained from the anthology in january 2024 (@xmath0. to include papers published in 2024, we used the same procedure in november 2024, following the publication of the emnlp 2024 proceedings (@xmath1. accordingly, all 80 papers for this study are guaranteed to be relevant and peer-reviewed. while a substantial number of publications in nlpsecparticularly those concerning attack methods-are published in preprint archives like arxiv, such papers do not necessarily go through peer review (e.g., @xcite . we intentionally limit ourselves to peer-reviewed publications to ensure rigorous publications whose claims have been vetted, rather than considering preprints whose claims we must take at face value. additionally, publications in major conferences and journals are obliged to abide by the acm code of ethics; as such, we assume good intent by the authors to meet a standard of ethics that is acceptable to the scientific community. for each of the 80 papers, we manually annotate the following: attack scenario. (values: adversarial, backdoor, or data reconstruction attack). this is coded in accordance with the keyword which was used to retrieve the paper. ## nlpsec survey results despite nlpsec's position as an interdisciplinary field, we find that the works in our survey have largely failed to adopt ethical norms of cybsersecurity research. ## discussion while nlpsec is an interdisciplinary field with an indisputable connection to cybersecurity, there are cases where the parallels between these fields diverge. in this section, we explore some areas where the analogy between nlpsec and cybersecurity fails, with the goal of illuminating the urgent need for a broader conversation about ethics in nlpsec. to help initiate such a conversation, we conclude with some concrete recommendations for nlpsec researchers, towards adopting better practices for more ethical nlpsec research. ## conclusion in the burgeoning field of nlpsec, most works consider scenarios where a malicious attacker seeks to undermine a system's intended behavior, with the goal of causing harm. research output in nlpsec thus stands to be highly consequential in the face of mass-adoption of language technologies such as llms, and its relevance to public safety necessitates heightened scrutiny when it comes to best practices for ethical research. given nlpsec's position as a truly interdisciplinary field, practitioners in this space can benefit from the rich traditions of research ethics from both cybersecurity and nlp. in this work, however, we find that nlpsec works published in nlp venues generally fall short of the ethical standards set by cybersecurity (section 4), signaling a higher-level disconnect between nlp and cybersecurity practitioners for work in this area. this failure to inherit ethical best practices can arise from a variety factors (section 5), but largely stem from the differences between traditional cybersecurity and nlpsec, which underscores the limitations of the ''white versus black hat'' paradigm of cybersecurity as applied to nlpsec. still, we argue that the repercussions of the current research patterns are grim: works in nlpsec may benefit would-be attackers more than the public (section 2), with dire consequences for everyone, but especially for already-marginalized communities (section 5.2). by highlighting these problems and exploring their nuances, this work aims to persuade the field of the urgency of the present situation and to spark a much-needed conversation across the field of nlpsec. to kick off this conversation, we provide some concrete recommendations to help practitioners transition from gray hat to white hat nlp.