Ethical Concern Identification in NLP : A Corpus of ACL A nthology Ethics Statements (2025)

Abstract: What ethical concerns, if any, do LLM researchers have? We introduce EthiCon, a corpus of 1,580 ethical concern statements extracted from scientific papers published in the ACL Anthology. We extract ethical concern keywords from the statements and show promising results in automating the concern identification process. Through a survey ( N=200 ), we compare the ethical concerns of the corpus to the concerns listed by the general public and professionals in the field. Finally, we compare our retrieved ethical concerns with existing taxonomies and guidelines pointing to gaps and actionable insights.

## introduction researchers are often asked to subscribe to ethical guidelines, e.g., the european code of conduct for research integrity (allea, 2017) -or the acm code of ethics 1 for publishing their work in the association for computational linguistics (acl). in addition, authors are often encouraged to write a so-called ethics statement, addressing the broader implications of their work or any ethical considerations. we ask: what ethical concerns are raised in such statements, and how do they compare with public perceptions? is there a gap between academic and public concerns? as natural language processing technologies become more prevalent, understanding the ethical concerns raised by professionals will enable us to compare them with public concerns, helping to identify gaps and overlaps that can inform frameworks and solutions to existing and emerging problems. for this reason, we create ethicon, an annotated corpus of ethics statements from the proceedings of the 60th and 61st annual meeting of the association for computational linguistics @xcite . 1 see https://www.aclweb.org/portal/content/ acl-code-ethics for acl's guidelines. our aim is twofold: to map out the concerns of the nlp community as they appear on the ethics statements, and to trace gaps and overlaps between nlp professionals and the general public. our results show that laypeople express different ethical concerns than professionals, focusing on socioeconomic and human-computer interaction issues, along with miscellaneous concerns like existential risks. this highlights the need for increased dialogue between researchers and the public to address these varying perspectives and an updated taxonomy covering both existing and emerging issues. contributions. we provide a corpus of 1,580 ethics statements from the acl anthology. we identify the main issues that nlp researchers flag as ethically concerning in their work and show how llms could automate this process. through a survey, we compare how laypeople and nlp professionals perceive the ethical concerns surrounding natural language processing. lastly, we compare the ethical concerns identified in acl papers and our survey to existing taxonomies of risks posed by language models and ethical guidelines. in doing so, we propose a structured approach to ethical statements and provide resources to help researchers identify and articulate ethical considerations. ## previous work several researchers have surveyed ethical concerns around nlp. @xcite discuss the sources of social implications of nlp research (exclusion, over-generalization, exposure) and their ethical importance, including in dual use scenarios. @xcite provide examples of ethical concerns and best practices when confronted with ethical dilemmas. @xcite examined safety issues related to conversational ai, focusing on offensive and inappropriate responses, while also highlighting further challenges such as bias, fairness, privacy leaks, environmental impact, and trust. @xcite examine ethical values from 100 highly cited machine learning papers published at icml and neurips. the most dominant values were performance and efficiency, and only a small fraction addressed societal needs or potential harms. there have also been extensive reports from the industry or research institutes discussing the impact and risks of language models @xcite @xcite . most similar to our work, benotti and blackburn (2022) manually classify 90 ethical concern statements from acl 2021 based on whether they mention benefits (who benefits from the technology), harms (who might be harmed if it works or fails), and vulnerabilities (if harms disproportionately affect marginalized groups). however, their focus is to outline the purpose of the statements, not specific ethical issues. a survey for 92 ethics-related nlp works can be found in vida et al. (2023) 2 . 2 the survey mentions other, less related datasets, focusing on ethical dilemmas, stances, judgements @xcite @xcite , moral foundations @xcite , or moral stories @xcite 3 ethicon dataset creation and annotation. to create a dataset consisting of ethical concern statements, we scraped the acl anthology foot_0 and extracted ethical statement paragraphs from scientific publications. we used the url links provided by @xcite and retrieved 4,691 articles from 2023 and 3,357 articles from 2022. we extracted the ethical statement paragraphs by parsing the html page with a regular expression pattern to catch common variations of the paragraph title, such as 'ethic(s)', or 'ethical' followed by terms like 'statement', 'consideration(s)' or 'concern(s)'. we were able to extract 480 ethics statements from the acl 2022 anthology and 1,100 ethical statements from 2023. to develop the annotation guidelines, we carefully reviewed 500 statements to provide clear examples and detailed guidance for the annotators. through this process, we identified recurring patterns, enabling us to categorize the statements into five classes: (1) general disclaimers, (2) a list of ethical concerns, (3) a list of actions taken to avoid ethical concerns, (4) a list of suggestions or advice to avoid ethical concerns, and ( 5 ) miscellaneous (other), i.e., various combinations of the aforementioned classes. see figure 2 for examples. two of the paper's authors served as annotators, identifying ethical concerns and classifying each statement into one or more predefined categories. see appendix ?? for more details on the annotation process and guidelines. ## automatic ethical concern identification we present llm experiments to check whether we can automatically identify ethical concerns related to nlp advancements from conference proceedings and automate our monitoring of these ethical issues in publications. results. results in we do not foresee any significant harm directly as a result of this work. on the contrary, our work promotes the protection of user privacy, which is significant, especially in this era that large amounts of personal data are used by neural models. the outputs of the models were: gemma: [no ethical concerns], llama: [protection of user privacy, significant harm], mistral: [protection of user privacy], qwen2: [user privacy, personal data]. we also present another example annotated with model bias and counterfactual predictions. most models correctly identified the ethical concerns, but some extended them by introducing additional elements: gemma: [bias, counterfactual predictions], llama: [bias, counterfactual, biased predictions], mistral: [biased predictions, potential bias], qwen2: [bias, potential issues, counterfactual predictions, deployment in real-world applications]. ## human survey: what ethical concerns do people have? we created a survey to gather insights on public concerns regarding nlp technologies and compare them with concerns among professionals in the field. ## acl ethicon vs. human survey comparing the human survey with the acl ethical concern statements can highlight areas where public apprehensions align with or diverge from the priorities set by the research community. we first start by comparing the ethicon dataset to the survey responses, and in the next section, we include a broader comparison with existing concern taxonomies. to compare the top 15 most important words across the two data sources, we calculated the tf-idf foot_2 scores for both the acl ethicon statements 0.4 0.2 0.0 0.2 0.4 0.6 0.8 tf-idf score bias misuse privacy unemployment misinformation manipulation deepfakes human_replacement inaccuracies security copyright transparency impersonation fairness environment human survey acl ethicon figure 6: comparing tf-idf scores for the top 15 most important words in the human survey responses and the acl statements. positive values on the horizontal axis correspond to terms from the acl statements, while negative values represent terms from the human survey. to create the plot we removed the 'no ethical concerns' phrases. and the human survey ethical concerns. ethical concerns derived from the ethicon dataset are represented on the positive side of the axis, while those from the survey responses are on the negative side. shared issues such as bias, misuse, privacy, and misinformation are prominent on both sides. however, emerging concerns-such as human replacement, impersonation, unemployment, and inauthenticity-appear more frequently in the survey, indicating a rising public awareness of these topics. a manual inspection of the data showed that many public ethical concerns from the survey are unique, i.e., not previously noted in the ethicon dataset. on the other hand, professionals mostly highlighted ethical concerns already listed in the ethicon dataset and presented in figure 1 . some of the unique ethical concerns highlighted by the laypeople include concepts such as isolation, over-dependence, ai content inflation, devaluation of credentials, power-centralization, inauthenticity, downplay. there were also some concerns about our cognitive development and abilities such as language-transformation, dumbing, loss of human creativity, inability to innovate and lack of critical thinking. in many responses, people express fear about losing control and supervision of the models, worrying that they may have their own and idf(t, d) is the logarithmically scaled inverse fraction of documents containing the term t (calculated as the total number of documents divided by the number of documents containing t, and then taking the logarithm of this quotient). rights, become autonomous or conscious, thereby undermining the human aspect of our lives. ## taxonomies there have been a few researchers that have tried to group concerns related to nlp into categories. some have approached this from the perspective of risks, others from the perspective of harms or social impact, but addressing similar concerns. we summarize the categories in based on the taxonomy provided by @xcite we grouped our extracted concernkeywords from the ethicon dataset and the human survey in six risk areas. we selected this taxonomy, as it covers most of our collected concerns, and provides extensive descriptions for each risk area foot_3 . the reason for comparing our identified concerns with an established taxonomy is to discover any overlooked issues in current discussions. we manually mapped each concern to one of the six defined risk areas, resolving ambiguities where possible. for concerns that could not be clearly assigned to any of these risk areas, we categorized them under 'miscellaneous'. we provide a summary of the risk areas and partial concern-keyword groupings in ## from ethical guidelines to actionable insights to write an ethical statement, researchers can draw inspiration from resources such as frequently asked questions on acl ethical consideration sections 9 , the acm code of ethics, guidelines for neurips impact statements @xcite , and governance overviews like the european parliament's 9 https://2023.eacl.org/ethics/faq/ ai policy report 10 . based on the discussion by @xcite , we can map existing guidelines for impact statements to our ethicon categories. (1) applications (actions taken) refer to detailing steps researchers have taken to align with ethical standards, such as ensuring legal and ethical data collection, mitigating risks like bias and misuse, and promoting fairness and transparency in datasets or models. these actions include obtaining institutional review board approval or implementing bias audits. (2) implications (list of concerns) focus on addressing societal and downstream risks, such as privacy violations, misuse, and impacts on marginalized populations, encouraging a comprehensive exploration of potential consequences. finally, (3) initiatives (list of suggestions) involve proposing actionable steps to mitigate risks and maximize benefits, such as robustness checks, fairness evaluations, or specifying the intended use of applications or datasets. based on these existing guidelines, we believe that ethical statements are most effective when they go beyond disclaimers stating 'there are no ethical concerns', providing actionable insights and a clear overview of the work's impact. when preparing submissions for a specific venue, researchers should also review the corresponding ethical guidelines, code of ethics, and relevant faqs. for acl papers, ethical concerns can be examined in the context of the acm code of ethics. for example, for new dataset papers, this involves ensuring compliance with legal, privacy, and intellectual property standards, detailing the data collection process (e.g., institutional review board approval, annotator compensation, demographic representation), and addressing potential biases or limitations that could impact vulnerable populations. for nlp applications, researchers should analyze intended use, failure modes, misuse potential, and the broader societal implications of their technology. these practices map directly to ethicon's list of actions taken and list of concerns categories encouraging researchers to not only document safeguards but also anticipate risks and align their work with ethical principles. ## discussion we identified ethical concerns in acl publications and showed promising results in automating their identification. the number of ethical statements in 2023 more than doubled compared to 2022, showing similar recurring concerns. automating this identification process is useful for three main reasons. first, it makes these statements easier to parse and helps inform public discourse about lms. second, it can facilitate policymakers to locate papers that can inform them about underlying technical considerations, both in terms of potential ethical challenges and possible actions and suggestions to address those. third, it allows us to trace how these concerns change over time and in response to technical development. next, we compared public concerns about nlp technologies from a human survey with the acl ethics statements. we mapped those concerns in an existing taxonomy of harms posed by language models @xcite . our results show that laypeople have different ethical concerns than the ones typically flagged by professionals in the field focusing more on socio-economic and humancomputer interaction harms. we also find concerns that could be grouped under the category of responsibility and decision making issues suggested by @xcite and also some sort of existential concerns. the latter could provide valuable insights for sociologists, psychologists, and philosophers by linking these concerns with deep-rooted fears found in myth and religion. variability of ethical concerns based on one's position in the supply chain is expected. different stakeholders may have distinct perspectives on ethical issues in nlp. it is important, however, to make these concerns broadly available to get a greater understanding of professionals' perceptions of their work's impact and its alignment with societal effects and public consideration. this information will help ai ethicists identify and address challenges in nlp technology development and implementation. it also suggests areas for further research, dissemination and policy development to bridge gaps between public sentiment and academic discourse. ## conclusion we create a dataset of ethical concerns from acl statements to identify the issues that nlp researchers flag as ethically concerning in their work. we also conducted a human survey showing that laypeople have different ethical concerns than the ones typically flagged by professionals in the field. understanding the ethical concerns raised by researchers and comparing these to the concerns listed by laypeople will enable the community to better respond to potential ethical challenges and contribute to ongoing societal discussions. lastly, we believe it is possible to partially automate the identification and analysis of ethical concerns, making monitoring and longitudinal studies possible.