B ad R ock at S em E val-2024 Task 8: D istil BERT to Detect Multigenerator, Multidomain and Multilingual Black-Box Machine-Generated Text (2024)

Abstract: The rise of Large Language Models (LLMs) has brought about a notable shift, rendering them increasingly ubiquitous and readily accessible. This accessibility has precipitated a surge in machine-generated content across diverse platforms encompassing news outlets, social media platforms, question-answering forums, educational platforms, and even academic domains. Recent iterations of LLMs, exemplified by entities like ChatGPT and GPT-4, exhibit a remarkable ability to produce coherent and contextually relevant responses across a broad spectrum of user inquiries. The fluidity and sophistication of these generated texts position LLMs as compelling candidates for substituting human labor in numerous applications. Nevertheless, this proliferation of machine-generated content has raised apprehensions regarding potential misuse, including the dissemination of misinformation and disruption of educational ecosystems. Given that humans marginally outperform random chance in discerning between machine-generated and human-authored text, there arises a pressing imperative to develop automated systems capable of accurately distinguishing machine-generated text. This pursuit is driven by the overarching objective of curbing the potential misuse of machine-generated content. Our manuscript delineates the approach we adopted for participation in this competition. Specifically, we detail the use of a DistilBERT model for classifying each sample in the test set provided. Our submission is able to reach an accuracy equal to 0.754 in place of the worst result obtained at the competition that is equal to 0.231.

## introduction large language models (llms) are increasingly pervasive and readily accessible, leading to a surge in machine-generated content across a multitude of platforms @xcite . llms have demonstrated an impressive ability to generate highly fluent responses to diverse user queries. the eloquent nature of these generated texts renders llms appealing candidates for replacing human labour across various scenarios. however, this widespread adoption has sparked concerns regarding the potential misuse of such texts, including the dissemination of misinformation in journalistic contexts and disruptions within educational systems @xcite . the increasing adoption of transformer-based architectures in academic research has also been bolstered by various methodologies showcased at semeval 2024. these methodologies tackle diverse tasks and yield noteworthy findings. for instance, at the task 2 @xcite , where to address the challenge of identifying the inference relation between a plain language statement and clinical trial reports is used t5 @xcite ; task 4 @xcite and task 10 ( @xcite where is employed a mistral 7b model to detect persuasion techniques in memes @xcite and to perform emotion recognition in conversation (erc) within hindi-english codemixed conversations respectively @xcite . despite human evaluators marginally outperforming random chance in distinguishing between machine-generated and human-written text @xcite , the need for automatic methods to detect machine-generated content has become increasingly urgent. this necessity prompted the organizers of task 8 at semeval-2024 to focus on developing such methods with the aim of mitigating potential misuse. previous efforts in detecting machine-generated text have been made. for instance, @xcite devised methods to discern whether a text was generated by chatgpt or authored by a human across various domains. however, these endeavours primarily concentrated on the outputs of chatgpt. the ruatd shared task 2022 tackled artificial text in russian, spanning models for paraphrase generation, text simplification, text summarization, and machine translation @xcite . however, their emphasis was on models fine-tuned for specific tasks or domains, which differs from the focus of the task 8. while @xcite detected outputs of various llms such as gpt-2, opt-2.7, neo-2.7, gpt-j, and neox, it's pertinent to note that these models have become obsolete with the advent of gpt-3 and even gpt-4. the task 8 hosted at semeval 2024 was built upon the previous work discussed in @xcite . to address these objectives, there is an ongoing demand for automated tools capable of extracting and categorizing data, facilitating the classification with recent nlp models. recent advancements in the machine and deep learning architectures have spurred heightened interest in natural language processing (nlp). substantial endeavours have been directed towards devising techniques for the automated identification and categorization of textual content accessible on the internet today. in the literature, to perform text classification tasks, several strategies have already been proposed. in the last fifteen years, some of the most successful strategies have been based on svm @xcite , on convolutional neural network (cnn) @xcite , on graph neural network (gnn) @xcite , on ensemble models @xcite and, recently, on transformers @xcite . participants in semeval-2024 task 8 could compete for three subtasks better described in the rest of this paper. however, our team participated in the first subtask only. the first subtask (i.e., subtask a) is the binary human-written vs. machine-generated text classification one: participants are tasked with determining, based on a given full text, whether it is human-written or machine-generated. there are two tracks for subtask a: monolingual (only english sources) and multilingual. the subsequent sections of the paper are structured as follows: section 2 offers background information on task 6, held at semeval-2024. in section 3, we outline the approach introduced in this study. section 4 delves into the specifics of the experimental setup employed to reproduce our findings. the outcomes of the official task and relevant discussions are presented in section 5. finally, sec-tion 6 concludes our study and suggests avenues for future research. we make all the code publicly available and reusable on github foot_0 . ## background the task 8 hosted at semeval-2024 @xcite is made of three different subtasks. machine-generated text classification: given a full text, determine whether it is human-written or machine-generated. there are two tracks for subtask a: monolingual (only english sources) and multilingual. an example from the original dataset -better described in @xcite ) -is the one that follows. given the prompt: while the original human-written version is available online 2 , the output produced by a chat-gpt 3.5 turbo was: "william whitehouse was a 19th-century british engineer and inventor who made significant contributions to the field of hydraulics. he was born in 1825 in birmingham, england, where he grew up and eventually became an apprentice at the soho foundry. this apprenticeship instilled in him a passion for steam engines and hydraulics that would shape his future career. after completing his apprenticeship, whitehouse worked for the birmingham water works company as an engineer. it was during this time that he began experimenting with hydraulic machinery and invented several innovative devices. in 1850, he developed the first hydraulic press, which was able to exert immense amounts of pressure and became widely used in industrial manufacturing processes. he also created hydraulic lifts for raising and lowering heavy loads and hydraulic cranes for lifting and moving heavy objects. whitehouse's inventions revolutionized manufacturing and construction industries and helped to increase productivity and efficiency. in addition to hydraulic machinery, he also worked on steam engines and locomotives, improving their performance and reliability. he was regarded as an expert in his field and wrote several papers on hydraulic engineering that were widely read and respected. whitehouse's achievements did not go unnoticed by the scientific community, and he was awarded the royal society's prestigious telford medal in 1875. he also became a fellow of the royal society and was elected to the institution of civil engineers and the institution of mechanical engineers. despite his success, whitehouse remained humble and dedicated to creating innovative and practical solutions to engineering problems. he passed away in 1905, leaving behind a legacy of groundbreaking hydraulic inventions that continue to play a vital role in modern manufacturing and construction industries." ## system overview even if it has already been proved that the transformers are not necessarily the best option for any text classification task @xcite , depending on the goal, some strategies like domainspecific fine-tuning @xcite , or data augmentation @xcite @xcite ) can be beneficial for the considered task. however, to address the task 8 hosted at semeval-2024 we employed a zero-shot learning strategy @xcite , making use of distilbert @xcite , finetuned on the sst-2 dataset @xcite . distilbert, akin to its larger counterparts (i.e., bert), exhibits commendable performance across a diverse array of tasks when fine-tuned. while prior research predominantly delved into distillation techniques for crafting task-specific models, the distillation approach in this case harnesses knowledge distillation during the pre-training phase. distilbert demonstrate the feasibility of reducing the size of a bert model by 40%, while retaining 97% of its language understanding prowess and achieving 60% increase in speed. to harness the inductive biases inherent in larger models during pre-training, a triple loss mechanism is introduced with this model. this mechanism combines language modelling, distillation, and cosine-distance losses. the compact, expedited, and resource-efficient model not only streamlines the pre-training process but also showcases its potential for on-device computations through a proof-of-concept experiment and comparative ondevice analysis. the stanford sentiment treebank stands as the inaugural corpus equipped with fully labeled parse trees, facilitating comprehensive exploration of the compositional effects of sentiment in language. it comprises 11,855 individual sentences culled from film reviews. leveraging the stanford parser, the corpus encompasses a total of 215,154 unique phrases, each annotated by three human evaluators. this novel dataset affords an opportunity to delve into the intricacies of sentiment analysis and capture nuanced linguistic phenomena. numerous examples within the corpus exhibit distinct compositional structures. the granularity and breadth of this dataset are poised to empower the community in training compositional models grounded in supervised and structured machine learning methodologies. while extant datasets primarily focus on document and chunk labelling, there remains a pressing need to enhance sentiment capture from concise remarks, such as those found in twitter data. utilizing distilbert trained on the sst stanford dataset for detecting human or ai-generated text holds significant promise due to its nuanced understanding of sentiment and context. by leveraging distilbert's fine-grained sentiment analysis capabilities, coupled with its proficiency in discerning contextual nuances, the model we used is supposed to effectively distinguish between human-generated and ai-generated text. the sst dataset, annotated for human sentiments classification task, enables distilbert to grasp the subtleties of human language, making it adept at identifying deviations indicative of ai-generated content. moreover, fine-tuning distilbert on this dataset enhances its sensitivity to linguistic cues that differentiate human-authored texts from those generated by ai algorithms, thereby offering a robust solution for text authenticity verification in various applications, including misinformation detection, content moderation, and forensic linguistics. in this study, we employed a fine-tuning approach to enhance the performance of distilbert, initially trained on the sst dataset, for the task of distinguishing between human and ai-generated text. the fine-tuning process involved training the model for three epochs on the provided training set, utilizing a portion of the data for validation. specifically, we partitioned 20% of the training set samples to form a validation set, crucial for assessing the model's performance and preventing overfitting. after completing the fine-tuning process, we systematically evaluated the model's performance across the three epochs on the validation set. subsequently, we selected the tuned version of the model that exhibited superior performance, as determined by its validation set accuracy. this validation methodology ensures the reliability and generalization capability of the fine-tuned distil-bert model for the targeted task of differentiating between human and ai-generated text. in a recent study @xcite , has been shown that the contribution of preprocessing for text classification tasks is generally not impactful when using transformers. more specifically, the best combination of preprocessing strategies is not very different from doing no preprocessing at all in the case of transformers. for these reasons, and to keep our system highly fast and computationally light, we have not performed any preprocessing on the text. ## experimental setup we implemented our model on google colab. the library we used comes from huggingface foot_2 and is the uncased version of distilbert specifically trained on the above-mentioned sst2 dataset foot_3 . we did perform a three-epochs additional fine-tuning, before generating the prediction on the unlabelled test set. this model is versatile and can serve as a foundational tool for topic classification tasks. while it can function as a raw model for masked language modelling or next sentence prediction, its primary utility lies in its adaptability for finetuning on downstream tasks. users can explore the model hub to discover fine-tuned versions tailored for specific tasks beyond its original scope. as already mentioned, all of our code is available on github. ## results given the binary nature of the classification task, the organizers proposed accuracy as the evaluation metric to be considered for the final ranking. the accuracy is defined in the equation 1. where tp stands for the number of correctly predicted right answers, fp stands for the number of wrongly predicted right answers, and fn stands for the right answers wrongly predicted as wrong answers. ## conclusion this paper presents the application of a distilbertmodel for addressing the task 8 at semeval-2024. in the table are shown the results obtained by the first three teams, by the last one and by our approach. in parentheses is reported the position in the official final ranking. our approach is not ranked in the official final ranking, but the score obtained ranks between the positions 68 and 69. for our submission, we decided to fine-tune a pretrained transformer. the model was used to perform a sequence classification task to detect if a piece of text is written by a human or by a generative model. the task is challenging, and there is still opportunity for improvement, as can be noted looking at the final ranking. possible alternative approaches to our can include utilizing the fewshot capabilities or also the use of other models like llama and t5, eventually using further data, or directly integrating other samples from the training and from the development sets. further improvements could be obtained with a fine-tuning and modelling the problem as a text classification task. furthermore, given the interesting results recently provided on a plethora of tasks, also other few-shot learning @xcite @xcite or data augmentation strategies @xcite @xcite ) could be employed to improve the results. looking at the final ranking, our simple approach exhibits some room for improvements. however, it is worth notice that it has required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by google colab.