UMUT eam at S em E val-2024 Task 8: Combining Transformers and Syntax Features for Machine-Generated Text Detection (2024)

Abstract: These working notes describe the UMUTeam’s participation in Task 8 of SemEval-2024 entitled “Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection”. This shared task aims at identifying machine-generated text in order to mitigate its potential misuse. This shared task is divided into three subtasks: Subtask A, a binary classification task to determine whether a given full-text was written by a human or generated by a machine; Subtask B, a multi-class classification problem to determine, given a full-text, who generated it. It can be written by a human or generated by a specific language model; and Subtask C, mixed human-machine text recognition. We participated in Subtask B, using an approach based on fine-tuning a pre-trained model, such as RoBERTa, combined with syntactic features of the texts. Our system placed 23rd out of a total of 77 participants, with a score of 75.350%, outperforming the baseline.

## introduction in the area of natural language generation (nlg), advances such as gpt-2 @xcite , gpt-3 @xcite and instructgpt @xcite have provided support for various writing tasks. the widespread adoption of large language models (llms) such as chat-gpt and gpt-4 @xcite has led to an increase in machine-generated content across various platforms, including news, social media, education and science. while these models produce remarkably fluid responses, concerns have arisen about their potential to spread misinformation and disrupt established systems. concerns remain about their misuse, particularly in scenarios such as academic dishonesty and scientific research, where ai-generated content may be presented as original work. the emergence of ai-generated scientific texts raises ethical and integrity concerns in academic publishing, requiring tools or models to distinguish between human-generated and ai-generated content @xcite . efforts to detect ai-generated text have primarily involved fine-tuning pre-trained models and developing detection systems. recent studies have presented datasets and methods specifically designed for the detection of ai-generated scientific documents. however, challenges remain in achieving high performance and interpretability across different domains and models @xcite . for this reason, task 8 of semeval @xcite , entitled "multigenerator, multidomain, and multilingual black-box machine-generated text detection", aims at identifying automatic systems for the detection of machine-generated text in order to mitigate its potential misuse. to this end, the task is divided into three subtasks that address two text generation paradigms: (1) full text, where a text is considered to be entirely written by a human or generated by a machine; and (2) mixed text, where a machine-generated text is refined by a human, or a text written by a human is paraphrased by a machine. this shared task is divided into three subtasks: determine whether a given full-text was authored by a human or generated by a machine. it offers two tracks: monolingual (english source only) and multilingual. in this competition, the umuteam participated only in the subtask b with an approach based on fine-tuning a pre-trained model such as roberta combined with syntactic features of the text. syntax features of the text refer to the writing style, such as token-level features (e.g. word length, part of speech, function word frequency and stop word ratio) and sentence-level features (e.g. sentence length). during our experiments, we found that the syntactic features of texts can complement and improve the performance of pre-trained transformerbased models and that roberta is more suitable for this type of task. the rest of this paper is organized as follows. first, section 2 provides a summary of important details about the shared task setup. second, section 3 gives an overview of our system. section 4 presents the specific details of our systems. section 5 discusses the results of the experiments, and finally, the conclusions are presented in section 6. ## background recent advances in ai technology, particularly in the field of natural language processing (nlp), have led to the emergence of many models capable of generating natural language using llms. these can produce remarkably fluent responses, and this has led to an increase in machine-generated content across multiple domains and platforms, including news, social media, education, and science. llms face several technical and social challenges as they advance in nlp tasks. recent research has shown that pre-trained llms can not only learn linguistic knowledge, but also reason about large amounts of acquired knowledge @xcite . however, llms have other problems, such as hallucination, producing texts that contain information or details that are not based on reality or are completely invented; and asserting falsehoods as facts, which means that they can involuntarily produce texts that present false information as true. the latest generative llms, such as gpt-3, are capable of producing highly fluent text, but they can produce inaccurate, toxic or unhelpful content. some researchers have explored the use of reinforcement learning from human feedback (rlhf) @xcite to adjust language models to better match user intent. chatgpt, one of ope-nai's models based on gpt-3 and trained with rlhf, performs well in conversations with humans, demonstrating an understanding of user instructions and generating useful, reliable, honest and harmless text content. therefore, a growing number of studies have been conducted to analyze, recognize and identify text generated by ai, especially text generated by gpt. current research focuses on two main areas: human behavior for recognizing text generated by ai and recognition models for identifying text generated by llms. for example, in @xcite , an approach was proposed to determine whether a text (in english and chinese) was generated by chatgpt or written by a human across different domains, while in @xcite , a model was developed to identify whether toefl essays were written by humans or generated by chatgpt on a small dataset (126 essays for each). there are other studies that focus on detecting fake information or fake news generated by llms. for example, in @xcite , the grover model was proposed to generate and detect examples of fake news. after the release of gpt-2, ope-nai proposed the gpt-2 generated text detector, which achieved a high f1 score. this detector was fine-tuned based on roberta in a binary text classification format. in addition, many studies also use various data augmentation techniques to improve model performance in the classification task through external data that complements the model or simply increases the training set @xcite . in paper @xcite , an approach was proposed to detect text generated by language models using different text features such as writing style, coherence, consistency, and argument logistics. the model with only syntax features (writing style) achieved the best result. for this shared task, we used a fine-tuning approach of transformer-based models such as roberta to create a detector for text generated by different llms. unlike other existing studies on llm-generated text detection, we have concatenated syntactic features during the fine-tuning process to improve its performance. the model evaluated for subtask b is roberta @xcite , a model based on transformers, which was pre-trained on a large corpus of english data with masked language model (mlm) goal. for this task, we evaluated the base version. figure 1 shows the architecture of our system. first, we extracted the syntactic features of the texts using the syntactic feature extractor and encoded the texts into a vector containing the dense representation of all the information contained in the text by the pre-trained models, i.e., the last hidden state of the model with text as input. second, once the vector and syntactic features were obtained, we normalized the syntactic feature values and concatenated them with the text vector. third, the fine-tuning process is performed, and a sequence classification layer is added on top of the pre-trained model. this layer takes the sequence representation generated by the pre-trained model and performs a classification based on the labels of the specific classification task. finally, a performance evaluation is performed using the validation set. ## experimental setup for subtask b, we used the dataset provided by the organizers, which consists of two subsets: training and validation. figure 2 shows the distribution of the training and validation sets. we can see that both the training and validation sets are balanced and that there are a total of 5 types of texts generated by different llms or by humans. the types are: davinci, bloomz, human, chatgpt, dolly and cohere. we used the following fine-tuning hyperparameters: a batch size of 16 for both training and validation, 10 epochs, a learning rate of 2e-5, and a weight decay of 0.01. during training, we used macro-f1 as a reference. macro-f1 is a measure used to evaluate the performance of a model in a multi-class classification problem. it calculates the average f1 score for each class individually, and then averages these scores to obtain an overall score. the macro f1 score assigns equal weight to each class, regardless of its size or distribution in the data set. this means that all classes are equally important in the final scoring metric. ## results in the in order to perform an error analysis and to observe the behavior of our model in predicting different classes of texts, we have generated the confusion matrix for our model based on the test set, as shown in figure 3 . our analysis shows that our model has a strong predictive performance for texts generated by bloomz, dolly, chatgpt and davinci, reaching accuracies above 90%. however, when it comes to human-generated texts, our model shows a 27.73% tendency to misclassify them as generated by the dolly model. in particular, when predicting texts generated by cohere, our model tends to misclassify them as generated by davinci at a rate of 70%, leading to a decrease in overall accuracy. ## conclusion this paper describes the participation of the umuteam in the 8th shared task of semeval 2024, focused on the identification of automatic systems for the recognition of machine-generated text in order to mitigate its potential misuse. the task consisted of three subtasks: subtask a, a binary classification task to determine whether a given full-text was written by a human or generated by a machine; subtask b, a multi-class classification problem to determine, given a full-text, who gen-4 700 erated it. it can be written by a human or generated by a specific language model; and subtask c, mixed human-machine text recognition. in this shared task, we participated in subtask b, using a approach based on fine-tuning a roberta pretrained model with syntactic features of texts. in terms of results, our system achieved the 23rd position with a score of 75.350%, outperforming the baseline. due to our line of research, we will evaluate our system on texts containing figurative language (garcía-díaz and valencia-garcía, 2022) and financial language (garcía-díaz et al., 2023a). on the one hand, the ambiguity and creativity of figurative language poses a challenge to the recognition of automatically generated text, as llms may have difficulty replicating the creative nuances of human-generated content. on the other hand, the recognition of automatically generated financial and business text is challenging due to specialized vocabulary and complex technical concepts. ideally, llms must have deep domain-specific understanding to produce accurate content that requires regulatory compliance and accuracy, which requires careful review and validation against authoritative sources.