Fighting Cyber-malice: A Forensic Linguistics Approach to Detecting AI -generated Malicious Texts (2024)

Abstract: Technology has long been used for criminal purposes, but the technological developments of the last decades have allowed users to remain anonymous online, which in turn increased the volume and heterogeneity of cybercrimes and made it more difficult for law enforcement agencies to detect and fight them. However, as they ignore the very nature of language, cybercriminals tend to overlook the potential of linguistic analysis to positively identify them by the language that they use. Forensic linguistics research and practice has therefore proven reliable in fighting cybercrime, either by analysing authorship to confirm or reject the law enforcement agents’ suspicions, or by sociolinguistically profiling the author of the cybercriminal communications to provide the investigators with sociodemographic information to help guide the investigation. However, large language models and generative AI have raised new challenges: not only has cybercrime increased as a result of AI-generated texts, but also generative AI makes it more difficult for forensic linguists to attribute the authorship of the texts to the perpetrators. This paper argues that, although a shift of focus is required, forensic linguistics plays a core role in detecting and fighting cybercrime. A focus on deep linguistic features, rather than low-level and purely stylistic elements, has the potential to discriminate between human- and AI-generated texts and provide the investigation with vital information. We conclude by discussing the foreseeable future limitations, especially resulting from the developments expected from language models.

## introduction technology has long been used for criminal purposes, either by allowing users to replicate online crimes that they would otherwise commit in the 'physical' world, or by powering new forms of crime that do not exist outside virtual worlds, and hence are cyber-dependent. however, the recent technological developments have allowed users to remain-or perceive themselves as remaininganonymous online, thus increasing the volume and heterogeneity of cybercrimes and making it more difficult for law enforcement agencies (leas) to detect and fight them. some types of cybercrime, such as cyberterrorism, easily attract the lea's attention; conversely, other types receive comparatively little attention, e.g. cyberbullying, cyberstalking, cyber-harassment, cyber-extortion, phishing or scamming, among others. the pervasiveness of these cybercriminal activities adds to the already dire challenges. leas have overlooked one relevant aspect in the fight against cybercrime: as cybercriminals ignore that they can be identified by the language that they speak and write, they tend to use language that enables their positive identification. by conducting forensic authorship analyses, forensic linguists have devised reliable methods to investigate and give evidence in cybercriminal cases. nevertheless, large language models (llms) and generative artificial intelligence (ai) raise new challenges for cybercriminal investigations: not only has cybercrime increased as a result of ai, but also generative ai makes authorship attribution of cybercriminal texts more difficult. since llms generate texts based on probabilistic models, each text is taken to be unique and free from individual features of authorship, which, in extreme cases, has the potential to prevent the identification of cybercriminals. this article revisits forensic linguistic approaches to cybercriminal investigations in the light of llms and generative ai. its aim is two-fold: (1) to discuss the features that can discriminate between human-and ai-generated texts in forensic contexts; and (2) to understand the anatomy of toxic and malicious aigenerated texts. these will provide new insights for the investigation of cybercriminal activities. the article is structured as follows. section 2 discusses cybercrime, online toxicity and artificial intelligence and is followed by section 3, which discusses the fight against cybercrime. section 5 briefly describes the data and methodology adopted. section 6 presents the results of the analysis and discusses the findings related to cybercriminal texts and toxic and malicious texts. the article ends with the conclusions, in section 7, and an indication of limitations and future work in section 8. ## cybercrime, online toxicity and artificial intelligence the most recent technological developments, especially since the launch of openai's @xcite , have drawn more attention to the use of generative ai systems for cybercriminal practices, given the augmentation of security risks that those systems enable @xcite . although technology has long been used for criminal purposes, the nature of cybercriminal activities has become increasingly sophisticated, which demands constant reconceptualisation and, consequently, terminological and legal adjustments. the very term 'cybercrime' has gone through different definitions. early approaches tended to describe it as 'computer crime', 'computer-related crime', 'crime by computer' @xcite or as 'harmful behaviour that is somehow related to a computer' @xcite . consequently, cybercrimes tended to be typified within the same categories as ordinary, 'real-world' crimes, except that they took place online @xcite . the sophistication of cybercrime has revealed, however, that simply adopting regular countercriminal practices is not sufficient to counter cybercriminal activities @xcite , which led to a broad consensus that cybercriminal activities can be divided into two main categories: cyber-dependent crimes and cyber-enabled crimes @xcite . cyber-dependent crimes target computers, networks or other technological systems, so the existence of technology is a requirement. these include, e.g., hacking, malware, or denial of service (dos) attacks. conversely, cyberenabled crimes are those that can be perpetrated offline, including, e.g., stalking, bullying, illegal content sharing or child sexual abuse. however, whether they can be treated as traditional offline crimes is doubtful, since their scale and anonymisation potential can be largely extended by the online environment @xcite . more recent official approaches define cyber-crime as a 'borderless issue' that can include crimes specific to the internet (e.g., attacks against information systems or phishing), online fraud and forgery (including identity theft, phishing, spam and malicious code), or illegal online content sharing (e.g. child pornography material, incitement to racial hatred or terrorism, and glorification of violence) foot_0 . the borderless nature of cybercriminal activities remains one of the major challenges in the fight against cybercrime: not only are leas required to exchange information across borders, they are also required to ensure that the evidence produced is admissible in different jurisdictions (european union agency for criminal justice cooperation, 2022). due to common anonymisation and stealth technologies currently available and to the ease with which fake online profiles can be created, any user can easily become a potential cybercriminal anywhere. remaining (or perceiving oneself as remaining) anonymous online triggers the volume of cyberthreats, consequently making it virtually impossible to investigate and act against all existing cases. therefore, it can be very difficult to positively identify the cybercriminals, especially when they resort to crimeware-as-a-service (also known as cybercrime-as-a-service, caas). caas enables criminals to perpetrate complex cybercriminal attacks, even when they lack the technical skills, by using products and services provided by other sophisticated cybercriminal groups or individuals. the main challenge of caas is that because the cybercriminal means and infrastructure are shared among multiple perpetrators, leas struggle to attribute the crime to a group or particular individual @xcite . caas is an illustrative example of how cybercriminals are usually a step ahead of law enforcement in their command of technology, but it also provides a forecast of how ai can be used to promote cybercriminal activities. although ai is not a new field of computer science @xcite , it has attracted general attention in 2022, after the launch of chatgpt, which offered widespread access to generative ai tools. as a general-purpose tool that combines the potential of computers, large datasets and sets of instructions, ai is perceived as being able to perform tasks usually associated with humans, e.g. reasoning, learning, decision-making and problem-solving. its general purpose potential allows it to be used to perform different tasks, including for malicious and toxic purposes. common uses in cybercriminal contexts include, among others: streamlining existing types of attacks, to circumvent the protection offered by computer software; devising new forms of attacks, by manipulating or creating fake data to impersonate other users or generate confusion; or, more importantly, automating and scaling attacks, by machinegenerating large-scale attacks with little effort. the simple fact that ai embeds the knowledge of millions of users enables cybercriminal and malicious users to undertake all sorts of illicit activities, including producing deepfakes, cracking passwords, automating and enhancing hacking activities, or planting malicious code to compromise organisational software or hardware @xcite . ai has fuelled cyber-enabled crimes, most of which victimise individuals who commonly refrain from resorting to legal action. as common users are given the power to generate text using ai, they discover new ways to produce toxic and malicious contents to harm others or themselves. ## the fight against cybercrime according to the world economic forum (citing security magazine), in 2023 ca. 2,200 cyberattacks were reported per day, i.e. more than 800,000 attacks per year, and many more cases may remain unreported. the exponential increase in the volume of both cyber-dependent and cyber-enabled crimes has called for better and more efficient cybercriminal detection methods and tools. this demand has been addressed mostly via the development of sophisticated computational systems for repairing or early preventing cyber-attacks. most systems have focused on cyber-dependent crimes, as these are the large proportion of reported cybercrimes and, moreover, tend to be perpetrated more systematically against corporations or organisations' systems to cause disruption, spread ransom demands, or get hold of users' personal and often sensitive data, including bank or health details, usernames and passwords. the five most high-profile cases identified by the world economic forum foot_1 in 2023 are: theft of us state department records (at least 60,000 emails were taken by hackers from the outlook accounts of us state department personnel); the cyberattack against the digital protection firm dark-beam (which exposed 3.8 billion records, including emails and passwords); royal mail's ransomware attack (which demanded a ransom of $80 million to enable handling international parcels); moveit data theft (a vulnerability in the file transfer software was exploited to steal personal and corporate data, thus affecting an estimated number of 2,000+ organisations and 60 million individuals); and indonesia's stolen passport records (which involved the theft of passport data of 34 million indonesian citizens by a hacktivist, and subsequent sell on the dark web, and which has originated a number of scams and identity fraud). understandably, while attention has been diverted to cybercrimes perpetrated mostly against systems, the seriousness of cyber-enabled crimes, especially those against individuals, has been neglected. consequently, every day millions of people are victims of cyber-bullying, cyber-staling, cyber-harassment, cyber-extortion, phishing, scamming, cyber-trespass, illegal access to personal data, illegal content sharing or child pornography, among others. all these forms of crime are highly pervasive, since they can be committed by virtually anyone, anywhere in the world, regardless of whether the perpetrators are known to the victim, or whether the attacks are systematic. as has been posited, the simple perception that one can remain anonymous online suffices to give criminals the (false) impression that they can go unpunished for their unlawful activities @xcite and this encourages more focused, rather than widespread, attacks. conversely, other forms of cybercrime, such as cyber-extortion, phishing, scamming, cyber-trespass, illegal access to personal data, illegal content sharing or child pornography, tend to be more widespread, targeting general users, unknown to the perpetrators. in both instances, attacks typically target silent victims, who either know the perpetrator and may consider pressing charges against them, or, at most, act only when e.g. they fall victims of scams involving their bank accounts. in so doing, they frequently neglect the seriousness of other types of cybercriminal activities, including illegal access to personal data, or 'petty crimes' such as 'post scams', which are usually overlooked by the victims because they do not have apparent serious implications, other than small sums of money. altogether, these factors reveal the complexity of understanding, typifying, and fighting against cybercrime. the increasing volume of attacks, the constantly evolving types of cybercriminal activities, the lack of human resources to fight them, and the sophisticated technological developments make it difficult to efficiently counter it @xcite . the developments in ai have furthered these complexities, by fuelling criminal and toxic activities online @xcite . in addition to data breach incidents deriving unintentionally from using generative ai systems @xcite , these systems help perpetrators generate their threatening or toxic communications instantly, more easily, and with a degree of truthfulness that deceives the victims by making them believe that the messages are genuine. 47.4% of all internet traffic in 2022 originated in bots, while human traffic decreased to its lowest in eight years @xcite . although not all traffic generated by bots is malicious, bad bot traffic is on the rise, and accounted for 27.7% of all global website traffic in 2021 for account takeover, scraping, and scalping @xcite . at the same time, the report concludes, bots are becoming increasingly sophisticated and designed to evade bot detection tools. generative ai adds another layer of complexity when handling cyber-enabled crimes, which target especially individual users: the generation of seemingly human texts with the speed and the breadth of automated systems. although ai systems lack the ability to produce mental processes, the behaviour of a physical system can be successfully simulated without having the internal structure of the entity that it models @xcite . therefore, the very nature of generative ai, by building upon llms, imitates natural language generation by humans @xcite and even amplifies it. therefore, ai-generated text successfully tricks even native speakers of a language into believing that artificially generated texts were produced by humans. this is largely because to lay, non-professional speakers and writers, artificially produced texts tend to be exempt from spelling, grammar and punctuation mistakes, which gives the reader or listener the false impression that they are high-quality texts. that makes fighting against cybercrime and addressing risks in processing digital information particularly difficult (velasco, 2022). however, since a large proportion of cybercriminal and online toxic activities (espe-cially those that are cyber-enabled to target end users) involve language production, forensic linguistic analysis plays a core role in cybercriminal investigations @xcite . therefore, whereas cybersecurity and computer forensics are of little use in some instances of cybercrime, linguistic analyses are pivotal to detect, prevent and fight against it. ## the forensic linguistics potential forensic linguistics, the branch of linguistics applied to forensic contexts, has traditionally been defined in a broad as subsuming three different areas: (i) the study of the written language of the law; (ii) the study of interaction in the legal process; and (iii) the analysis of language as evidence @xcite . forensic linguistic analysis, and especially forensic authorship analysis and its sibling sociolinguistic profiling, are particularly robust in the detection and investigation of cybercriminal communications, malicious and toxic contents online. authorship analysis is one of the most visible applications of forensic linguistics. it consists of establishing the most likely author of a forensic text whose authorship is disputed, from a pool of suspect authors @xcite . in less common scenarios, it can also establish whether a suspect can be confirmed or otherwise rejected as the author of a questioned text. authorship analysis builds upon the concept of idiolect, i.e., the principle that every speaker of a language has a version of the language that they speak or write, which results in distinctive and idiosyncratic choices in texts @xcite . by being provided with the questioned texts and samples of texts known to have been written by the suspects, forensic linguists qualitatively establish the most likely author of the questioned text based on the author's internal consistency and on their distinctiveness when compared to other authors @xcite . this investigation typically involves a small pool of suspects @xcite (typically, three or four), since, for forensic linguists, it is very difficult to establish the most likely author from a large number of suspects. the qualitative approaches can be of limited usefulness in cybercriminal contexts, where the pool of suspects can be large and an identification of specific suspects may not exist. in this case, linguists are commonly provided with the questioned texts and are asked to establish sociolin-guistic features of the possible author(s), including age range, sex/gender, level of education, socioeconomic status, or their native language/language variety, among others @xcite . sociolinguistic profiling has the potential to provide leas with elements of the sociolinguistic features of the speakers or writers that enable them to direct the investigation to specific groups of individuals sharing those features @xcite . from a computational perspective, both authorship analysis and authorship profiling have been approached as a classification problem @xcite . by employing stylometric approaches @xcite @xcite , computational methods have a significant potential, especially because they are immune to fatigue, apply analyses systematically @xcite and can provide precision and recall rates, which may be appealing to courts for their potential to establish the known error rates. nevertheless, they tend to miss the fine-grained linguistic information required to make theoretically grounded decisions and offer linguistic explanations for the phenomena analysed. an appropriate approach to detecting and analysing cybercriminal communications therefore requires a unified approach to the linguistic individual @xcite , which identifies consistent and distinctive features of an author's language, but also offers explanations for such consistency and distinctiveness. this is even more relevant when analysing authorship of texts produced, in whole or in part, by generative ai. if, on the one hand, generative ai produces highly patterned texts based on how the probabilistic lms operate, on the other, those huge volumes of language data were collected from millions of speakers, so some diversity and sparsity would be expected from the data. one can thus speculate that, while such individual contributions are evident in the data, it is the nature of the lms that standardises the data and secures its regularity. ## conclusions the technological developments of the last decades triggered cybercriminal, toxic and malicious activities. many of these contents revolve around language use. as cybercriminals ignore the very nature of language, they tend to overlook the potential of forensic linguistic analysis to positively identify them by the language that they speak or write, via authorship analysis and sociolinguistic profiling. these applications have produced significant progress in the fight against cybercrime. however, the most recent technological developments, especially related to llms and to the massive use of generative ai, raised significant challenges to law enforcement agents and forensic linguists alike, since they not only make it easier and faster to perpetrate cybercrimes, but also make it more difficult to attribute the authorship of the texts to the perpetrators. forensic linguistics will continue to play a core role in detecting and fighting cybercrime, notwithstanding the need to shift the focus of analysis. firstly, forensic linguistic approaches allow properly discriminating between human-and aigenerated texts. contrary to ai-generated text detection tools, whose predominant stylometric approaches may result in a large volume of false positives and false negatives, forensic linguistic approaches provide robust information at all linguistics levels to discriminate between human-and ai-generated texts. subsequently, depending on whether the cybercriminal communications are produced by humans or by machines, forensic linguistics can establish the best methods to profile the originator of the message. additionally, forensic linguistics plays a core role in analysing the ai generation of malicious and toxic contents. this knowledge will not only contribute to inform forensic linguistics detection methods and techniques, but also help detect vulnerabilities in generative ai systems. as has been demonstrated, despite the security filters devised by current ai systems, those filters can be easily circumvented by simple prompts, and which are reminiscent of @xcite .