Regulation and NLP ( R eg NLP ): Taming Large Language Models (2023)

Abstract: The scientific innovation in Natural Language Processing (NLP) and more broadly in artificial intelligence (AI) is at its fastest pace to date. As large language models (LLMs) unleash a new era of automation, important debates emerge regarding the benefits and risks of their development, deployment and use. Currently, these debates have been dominated by often polarized narratives mainly led by the AI Safety and AI Ethics movements. This polarization, often amplified by social media, is swaying political agendas on AI regulation and governance and posing issues of regulatory capture. Capture occurs when the regulator advances the interests of the industry it is supposed to regulate, or of special interest groups rather than pursuing the general public interest. Meanwhile in NLP research, attention has been increasingly paid to the discussion of regulating risks and harms. This often happens without systematic methodologies or sufficient rooting in the disciplines that inspire an extended scope of NLP research, jeopardizing the scientific integrity of these endeavors. Regulation studies are a rich source of knowledge on how to systematically deal with risk and uncertainty, as well as with scientific evidence, to evaluate and compare regulatory options. This resource has largely remained untapped so far. In this paper, we argue how NLP research on these topics can benefit from proximity to regulatory studies and adjacent fields. We do so by discussing basic tenets of regulation, and risk and uncertainty, and by highlighting the shortcomings of current NLP discussions dealing with risk assessment. Finally, we advocate for the development of a new multidisciplinary research space on regulation and NLP (RegNLP), focused on connecting scientific knowledge to regulatory processes based on systematic methodologies.

## introduction the development of large language models (llms) is at its fastest pace to date. in the past years alone, llms have seen considerable advancement across a multitude of languages and types of data, with models such as gpt-3.5 @xcite , @xcite , llama @xcite , and palm-2 @xcite demonstrating unprecedented capabilities across a broad collection of natural language processing (nlp) tasks. 1 these innovations have led to rapid shifts in various applications such as open-domain search, coding, e-commerce and education. for example, state-of-the-art llms already power conversational search engines (e.g. openai chatgpt, bing chat, and google bard), coding assistants (e.g. openai codex and github copilot), product recommender systems (e.g. alibaba tongyi and salesforce com-mercegpt) and educational assistants (e.g. khanmigo) inter alia. as with most technologies, the development and use of llms do not come without concerns. researchers are rightfully worried that while this technology may be transformative, its societal implications might be higher than its benefits @xcite . concerns have been raised es-pecially around ethics @xcite , bias @xcite , safety @xcite and environmental impact @xcite @xcite . the unsupervised use of llms has already led to widely-publicized examples of professional negligence. the all too public fiasco of the lawyer who used chatgpt for a court brief and unknowingly included made-up case law references was taken by many as an example of the dangers of immersing daily professional activities in generative ai. 2 the public debate around the future of ai, and implicitly nlp, is very complex and multi-layered. while the debate seems to converge on the point of calling for regulation to control the unwanted effects of these technologies, foot_2 different regulatory directions are proposed by the various stakeholders involved in this debate. the current public discourse has been dominated by two groups. on the one hand, proponents of ai existential risks (the ai safety movement) foot_3 that include technology ceos and ai researchers have been publishing open letters 5,6,7 and regularly meet with regulators to warn about catastrophic scenarios around general ai, proposing industry-friendly solutions @xcite . on the other hand, the ai ethics movement @xcite , mostly reflects the voice of researchers from various disciplines as well as civil society activists. they have raised compelling alarm bells with respect to the risks posed by llms @xcite @xcite . the ai ethics movement has also offered guidance to regulators such as the european parliament on ai governance, arguing for e.g. broad definitions of general purpose ai. 8 yet while nlp research increasingly focuses on llm regulation, it remains generally detached from prior work on regulation studies. instead, the often intense public conflicts in this space are nudging regulators towards reactionary public relations activities rather than the collection of scientific expertise representing broader parts of the nlp and ai communities. for instance, right after the most recent letter on existential risks, the us and eu agreed to develop an ai code of conduct 'within weeks'. 9 similarly, the uk has announced it will be holding a global summit on ai safety, foot_9 and the us congress has been taking evidence from a wide array of industry executives, in what has been described as a global race to regulate ai. 11 with legacy and social media considerably increasing the visibility of these often polarizing debates, there is a real danger of regulatory capture by visible voices in industry and academia alike on scientific views that might not necessarily be representative of the 'silent majority' of nlp researchers. regulatory capture is the process by which regulation is directed away from the public interest and towards the interests of specific groups @xcite . against this background, we call on the nlp community to familiarize itself with regulation studies. we argue that this can lead to a clearer vision about how nlp as a field can properly participate in ai governance not only as an object of regulation, but also as a source of scientific knowledge that can benefit individuals, societies and markets alike. in this paper, we contribute to the existing debate relating to the future of nlp by discussing the benefits of interfacing nlp research with regulation studies in a systematic way. this view is based on two main ideas: ## regulation: a short introduction why do we regulate? calls to regulate llms and ai are everywhere, to the extent that overusing the term 'regulation' is trivializing its meaning. so what exactly is regulation and why do we rely on it? historically, regulation was defined by reference to state intervention in the economy. @xcite defined regulation as 'a sustained and focused control exercised by a public agency over activities that are valued by the community'. over the last decades, regulation has evolved and it has increasingly acquired a hybrid character as both public and private actors may issue rules that shape social behavior. in this paper, we draw on black's definition of regulation as an organized and intentional attempt to manage another person's behavior so as to solve a collective problem @xcite . this is done through a combination of rules or norms which come together with means for their implementation and enforcement @xcite . regulation includes different types of regulatory instruments (e.g. laws) such as traditional topdown or command-and-control regulations. recent examples in digital public policy include the laws issued by the european union, such as the digital services act, or india's law banning tiktok. however, beyond these public regulatory instruments, there is an array of private or hybrid instruments @xcite . some of these are qualified as 'soft' regulation because they cannot be enforced in court but they remain relevant and they effectively shape the behavior of the industry. examples include the eu's ethics guidelines for trustworthy ai 12 , but also codes of conduct initiated by industry itself. 13 over the last decades, different theories of regulation have helped us understand either on normative or empirical accounts why we should regulate. they generally reflect various hypotheses about 'why regulation emerges, which actors contribute to that emergence and typical patterns of interaction between regulatory actors' @xcite . public interest and private interest theories are two well-known examples @xcite . according to public interest theories, regulation is used by law-makers to serve a broad public interest, seeking to regulate in the most efficient way possible. regulators assume that markets need 'a helping hand' because when unhindered, they will fail. information asymmetries are one of the market failures that regulation seeks to address. at the same time, public interest theories of regulation are normative and prescriptive: on the one hand, they assume that benevolent state regulators ought always to use regulation to advance the public interest; on the other, they also advise on how to achieve this goal. in contrast, private interest theories are not primarily concerned with normative justifications for regulation. rather, they are prescriptive accounts of the complex dynamics between different market actors, stakeholders and public officials situated in a given socio-economic, political and cultural time and place. they explain, for example, why regulation may fail to pursue the public interest but they do not offer prescriptions on how address to such problems. private interest theories assume that regulation emerges from the actions of individuals or groups motivated to maximize their self-interest. ## llms: risk and uncertainty the need to bridge nlp research with regulation studies is especially important in the discussion of risks. new and emerging technologies are typically accompanied by risk and uncertainty. the regulation of technological change and innovation is highly complex, as innovation remains an elusive concept hard to define, measure, and thus regulate. in the past years, the question of risks arising out of nlp developments such as llms has been increasingly embraced in computer science literature. one strand of this literature is reflected by the theme of algorithmic unfairness. this theme emerged at the intersection of discrimination law and automated decision-making, and includes questions relating to fairness and machine learning in general @xcite , as well as specific examples of algorithmic bias risks in nlp @xcite @xcite , computer vision @xcite , multimodal models @xcite , as well as privacy risks @xcite . another strand of this literature looks at llms from a more holistic perspective, raising concerns about their size vis-a-vis a broader number of risks for e.g. the environment, bias, representation or hate speech @xcite @xcite . this theme does not only include risks in commercial applications, but also risks arising out of the mere scientific development of technology. this literature has raised important concerns relating to the immediate and longer-term implications around the advancement of machine learning and nlp. however, when positioned in the regulatory context, we can observe conceptual clashes with frameworks which have been traditionally relied upon in public policy and risk regulation. one such framework is the field of risk and uncertainty. put simply, 'risk is the situation under which the decision outcomes and their probabilities of occurrences are known to the decision-maker, and uncertainty is the situation under which such information is not available to the decision-maker' @xcite . in more technical terms, 'risk is the probability of an event multiplied by its impact, and uncertainty reflects the accuracy with which a risk can be assessed' @xcite . as a field, risk and uncertainty has made considerable contributions to the development of risk regulation, most notably in relation to environmental regula-tion @xcite . it is important that policy-makers have a concrete quantification of risk @xcite , in order to determine the adequate level of risk associated with various public policies. in addition, risk determination and management have important economic consequences, specifically for determining 'what level of expenditure in reducing risk is proportionate to the risk itself' @xcite . especially in the case of technologies that easily transcend physical borders, societies and economies, determining risk and uncertainty is a complex undertaking, even for scientists. some of the factors that make it difficult to assess risk and uncertainty include the complexity of the technology itself, as well as the information asymmetry underlying commercial practices. llms are humongous (billion-parameters-sized) transformerbased @xcite ) models, which have been initially pre-trained as standard language models @xcite in a vast quantity of text data (mostly web scrapes) and have been also further optimized to follow instructions @xcite and user alignment @xcite with reinforcement learning from human feedback (rlhf) @xcite . particularly, ai alignment has been a controversial topic, since it implies a broad consensus on what sort of values (standards) ai should align with @xcite . as such, llms are complex technologies where in addition to risk, we also deal with considerable uncertainty, which can be, among others, descriptive (e.g. relating to the variables defining a system), or related to measurement (e.g. uncertainty about the value of the variables in a system) @xcite . llms as the new gmos? yet llms are neither the first nor the last technology development posing concerns about wide-spread risks. as an illustration, in the 1970s and 1980s, in-vitro fertilization (ivf) was a demonized scientific development considered inhumane, which led to nothing short of a large-scale moral panic @xcite . in the 2000s, concerns around genetically modified organisms (gmos) dominated media coverage in european and north american countries, in what was deemed a 'superstorm' of moral panic and new risk discourses @xcite . these are only two examples of risk narratives that were amplified by media coverage in ways that overshadowed important scientific expertise. yet through regulation supporting scientific advancement, their use today has become mundane as part of solving considerable society problems such as infertility or food availability. these comparisons by no means imply that earlier biotechnological innovations pose the same levels of risk as llms or should entail the same level of regulation. however, it is important to learn from our past experiences with technology how to distinguish between moral panics and real problems that need scientific solutions. researchers are starting to develop concrete methodologies for the auditing of llms and related nlp technologies @xcite ), foot_13 as well as dealing with particular risks such as environmental impact @xcite . these contributions are much needed, as they can be translated into concrete measurements of risk and uncertainty, and further lead to the development of policy options in risk management. however, these initiatives are so far too few, as no cohesive scientific approach exists on the assessment of the risk and uncertainty posed by llms. to date, even the most comprehensive overviews of llm risks @xcite lack basic methodological practices such as the systematic retrieval of information from the disciplines of inquiry @xcite . in some cases, strong projections about risk impact are made without any scientific rigor whatsoever @xcite . similarly, @xcite note that while many technical approaches, including approaches related to 'mathematical criteria for "safety" or "fairness" have started to emerge, 'their systematic organization and prioritization remains unclear and contested. as a result, the lack of systematization and methodological integrity in scientific work around llm risks contributes to a credibility crisis which may impact regulation and governance directly. ## scientific expertise, social media and regulatory capture regulatory capture as a source of evidence for policy-makers, scientific expertise has increasingly played a central role in regulation @xcite . in some supranational governance contexts, scientific expertise is called upon in procedures that often require a certain level of transparency. this is the case of the call for public comments which we have discussed above as part of the eu's 'better regulation' agenda. however, in the past decade, the rise of science communication on social media has somewhat changed the interaction between policy-makers and scientists (van dijck and alinejad, 2020). a lot of the public debate between stakeholders from industry, academia and policy relating to llm risks is had on social media platforms such as twitter. this can pose a regulatory capture problem. regulatory capture occurs when the regulator advances the interests of the industry it is supposed to regulate, or of special interest groups rather than pursuing the public interest @xcite . regulatory capture fits within the private interest theories we explored in section 2, and often refers to the influence exercised by industry over regulation processes @xcite ). the most recent example is openai's white paper suggesting narrow regulatory interpretations for general purpose high-risk ai systems to european regulators 16 . similarly, multiple technology executives of large companies using llms, such as huggingface or openai have been testifying before the us congress to propose industryfriendly interpretations of ai risks. this is happening in a context of existing concerns around the industry orchestration of research agendas in nlp @xcite , and science @xcite in general. however, regulation can also be captured by special interest groups from civil society, and increasingly, academia. in this meaning, regulatory capture has a cultural or value-driven dimension that encompasses 'intellectual, ideological, or political forms of dominance' @xcite . in a landscape where special interest groups are increasingly represented by popular science communicators, a lot of questions arise in relation to the power exercised by the rising impact of science influencers, whether from academic, journalism and industry environments @xcite . the rise of science influencers traditionally, science communication has followed a conventional model dominated by professional actors gatekeeping information (e.g. scientists, journalists and government). social media has led to the creation of a networked model of science communication, with underlying socio-technical and political power shifts (van dijck and alinejad, 2020). science influencers are rooted in this development, as well as the broader rise of social media influencers @xcite . they also bring with them additional complexities. they may emerge from a scientific background, but may use their platforms both for professional as well as personal self-disclosure @xcite . in doing so, they also become political influencers who 'harness their digital clout to promote political causes and social issues' @xcite . to signal the social media influence of such public opinion leaders, some media outlets even rank them for awards purposes, 17 or profile them vis-a-vis state of the art scientific expertise. 18 this development is especially important since some social media platforms are more relevant for regulators than others. a recent report published by oxford university's reuters institute shows that twitter is the platform politicians pay most attention to across all studied markets. 19 with this in mind, the polarized narratives around llm risks unfolding on social media pose the danger that scientific expertise is only partially represented in public debates, in spite of the promises of speech democratization expected from these platforms in the context of science communication. while followers and engagement may be a measure of popularity with some communities and stakeholders, it raises concerns relating to the role popularity metrics and algorithmic amplification on social media may have in representing scientific or industry consensus before policy-makers. as @xcite put it, there is a popularity bias that means 'attention tends to beget attention'. in other words, 'the more contacts you have and make, the more valuable you become, because people think you are popular and hence want to connect with you @xcite . how exactly scientific popularity influences regulation still needs to be explored in greater detail, particularly as a novel example of potential regulatory capture. what we know so far is that social media influencers can be highly effective in relying on authenticity and para-social relations for persuasion purposes @xcite . in this context, popularity determines power relationships within social media networks that may capture regulatory processes in two ways. first, by exercising persuasion over policy-makers as audiences through visibility and popularity. simply put, not all research that is available in a given field is presented on social media. under the premise of basing policy on scientific evidence, politicians may rely on research that gains visibility due to amplification by scientific influencers, particularly when social media popularity is doubled by the brand power of prestigious academic institutions. second, by amplifying polarized debates that may trigger policy options which are not sufficiently informed through transparent and collective processes of evidence gathering. if multiple ai research groups are vocal on social media about the future of ai research, this fuels a race towards ai regulation. this can take away from the thoroughness that is necessary in collecting evidence for such a complex field. ## regulation and nlp (regnlp): a new field the danger of regulatory capture, taken together with the lack of systematization in the identification and measurement of risk and uncertainty around llms, calls for a cohesive scientific agenda and strategy. in 2023, the world counts 8 billion humans foot_19 , and further digitalization and automation are not only unstoppable, but also absolutely necessary. technological innovation is currently vital in the governance of our society a concrete proposal we advocate for is the creation of a new field of scientific inquiry which we call regulation and nlp (regnlp). regnlp has three essential features which we discuss below. ## conclusion llms reflect a momentous development in nlp research. as they unleash a new era of automation, it is important to understand their risks and how these risks can be controlled. while eager to engage with regulatory matters, nlp research on llm risks has so far been disjointed from other fields which are of direct interest, such as regulation studies. in particular, the field of risk and uncertainty has been conceptualizing and discussing scientific risks for decades. in this paper, we introduced these two areas of study and explained why it would be beneficial for nlp research to consider them in greater depth. in doing so, we also raised concerns relating to the fact that a lot of scientific debates on nlp risks are taking place on social media. this may lead to regulatory capture, or in other words the exercise of influence over law-makers, who are notoriously active on social media platforms. to tackle these issues, we propose a new multidisciplinary area of scientific inquiry at the intersection of regulation and nlp (regnlp), aimed at the development of a systematic approach for the identification and measurement of risks arising out of llms and nlp technology more broadly.