iimas NLP at S em E val-2024 Task 8: Unveiling structure-aware language models for automatic generated text identification (2024)

Abstract: Large language models (LLMs) are artificial intelligence systems that can generate text, translate languages, and answer questions in a human-like way. While these advances are impressive, there is concern that LLMs could also be used to generate fake or misleading content. In this work, as a part of our participation in SemEval-2024 Task-8, we investigate the ability of LLMs to identify whether a given text was written by a human or by a specific AI. We believe that human and machine writing style patterns are different from each other, so integrating features at different language levels can help in this classification task. For this reason, we evaluate several LLMs that aim to extract valuable multilevel information (such as lexical, semantic, and syntactic) from the text in their training processing. Our best scores on Sub- taskA (monolingual) and SubtaskB were 71.5% and 38.2% in accuracy, respectively (both using the ConvBERT LLM); for both subtasks, the baseline (RoBERTa) achieved an accuracy of 74%.

## introduction large language models (llms) have become widely available and easily accessible, leading to an increase in machine-generated content across diverse platforms including question-and-answer forums, social media platforms, educational resources, and academic settings. recent advancements in llm technology, exemplified by models like chatgpt and gpt-4, produce coherent responses to a vast majority of user inquiries, making them increasingly appealing for replacing human labor in various applications. however, this accessibility has raised concerns about potential misuse, such as generating fake news, financial services industry, legal domain, and disruptions in educational settings. given the challenge humans face in distinguishing between machine-generated and human-written text, there is a pressing need to develop automated systems capable of identifying machine-generated content to mitigate the risks associated with its misuse. motivated by these challenges, semeval-2024 task-8 @xcite offers three subtasks over two paradigms of text generation: (1) full text when a considered text is entirely written by a human or generated by a machine; and ( 2 ) mixed text when a machine-generated text is refined by a human or a human-written text paraphrased by a machine. these three subtasks are composed in the following way: subtask a is a binary classification task that focuses on identity if a given text was written by a human or a machine; it is split into monolingual (english) and multilingual (arabic, russian, chinese, etc). subtask b is a multi-class classification task that aims to identify which specific llm generates a given text among six different known options: human-made, chatgpt, cohere, davinci, bloomz, and dolly. finally, subtask c, given a mixed text, where the first part is humanwritten and the second part is machine-generated, determines the boundary, where the change occurs. we tackled two of these three subtasks: subtask a (monolingual) and subtask b. we applied fine-tuning of four llms (described in the following section) that included structural information in their pre-training. these models have proven their efficiency in multiple natural language understanding (nlu) tasks, such as question-answer entailment, paraphrasing, and textual similarity. we aim to test the efficiency in machine-text detection by comparing the results of given baselines for each subtask (a and b) with our fine-tuning llms with different approaches for the implementation of structural information. our scores show a modest performance related to the final ranking (especially in subtask b), but, based on the analysis of the results we observe that all of these llms used in this research, struggle 1 1110 to classify human text, meanwhile, they achieve a good performance classifying machine text. this paper is structured as follows: section 2 summarizes related works on machine text generation. section 3 describes the dataset used for the task. section 4 presents the system overview and the experimental setup. section 5 and 6 shows the results and conclusions, respectively. ## related work in recent years, many interesting shared tasks that related to the automatic detection of ai-generated text. besides the semeval task8, one of the most popular and challenging tasks called autextification: automated text identification @xcite , aims to address the detection of content created by text generation models in english and spanish. to mention a few interesting research works related to autextification-2023, the system titled "i've seen things you machines wouldn't believe: measuring content predictability to identify automatically generated text" @xcite achieves the best performance among the submissions in subtask 1 (differentiating between humanand machine-generated text), both for english and spanish. their model focuses on assessing the "predictability" of given text by multiple llms, leveraging features related to grammatical accuracy, word frequency, and linguistic patterns, along with a fine-tuned llm representation. another remarkable work titled "generative ai text classification using ensemble llm approaches" @xcite , proposes an ensemble neural model that leverages probabilities generated by different pretrained llms as features for a traditional machine learning (tml) classifier (their model ranked in first place in subtask 2 for english and spanish). on the other hand, pre-training llms with structural information enrich the learning process with contextual and syntactic cues. these cues encompass sentence structure, paragraph organization, grammatical rules, and broader linguistic patterns. fine-tuning llms with such structural knowledge enhances their ability to both comprehend and generate text that adheres to human-like writing styles and conventions. this approach has been explored in multiple ways; so now we briefly describe the approach taken by the models we used: ernie model @xcite implements an implicit knowledge of syntactic information through multiple levels of masking (token, phrase, and entity level). span-bert model @xcite masks random spans of contiguous tokens and trains to predict every token for each span instead of just masking and predicting each token. convbert model @xcite substitutes attention blocks for span-based dynamic convolutions capable of storing structural information in the generated kernels. finally, xlnet @xcite , this llm does not corrupt the text with masking but rather utilizes all the multiple permutations of tokens in a given sentence during the training process. ## dataset the data provided for semeval task 8 is an extension of the m4 dataset @xcite . this is a large-scale benchmark, which is a multigenerator, multi-domain, and multi-lingual corpus for machine-generated text detection. this extensive m4 corpus encompasses texts from various domains, including news articles, programming code, and fictional narratives. additionally, the m4 corpus incorporates texts in numerous languages, such as english, spanish, and chinese. this diversity in both domain and language coverage contributes to the effectiveness of m4 in effectively identifying machine-generated text (see figure 1 ). for machine generation, it prompts the following multilingual llms: gpt-4, chatgpt, gpt3.5 (tex-davinci-003), cohere, and dolly-v2. the models are asked to write articles given a title (wikipedia), abstracts given a paper title (arxiv), peer reviews based on the title and the abstract of a paper (peerread), news briefs based on a title (news), also to summarize wikipedia articles (arabic), and to answer questions (reddit). 2 1111 our system evaluates different llms that integrate features at different language levels (such as lexical, semantic, and syntactic) with the idea of extracting human and machine writing style patterns and being able to distinguish text from each other. for this reason, we applied a fine-tuning process using the four llms mentioned before: ernie foot_0 , spanbert foot_1 , convbert foot_2 , and xlnet foot_3 (using the hugging face library) for subtask a monolingual and subtask b. starting with the data partition process, we used the same partition proposed by the organizers in the baseline code for both tasks. the training dataset was split into the train (80%) and validation (20%) for the fine-tuning process and the development dataset was used to measure the accuracy of each model with unknown data. finally, the test dataset was only used to rank the models and verify the results. afterwards, in the fine-tuned process we tried with different hyperparameters on batch size (16, 32), learning rates (2e-5, 5e-5), random seed (0, 42), epochs (3, 5), and a weight decay of 0.01. along with these params configurations, we used the trainer, automodel, and autotokenizer classes from the transformers. each sequence was padded and truncated at 512 after tokenization due to the constraints of some of the models we used (most of them had a limit in the allowed length of the input sequence). these hyperparameters were chosen based on empirical experiments and hyperparameter tuning to achieve the best performance on our validation dataset. for the evaluation we computed macro-f1, micro-f1, and accuracy scores; being the last ones used by those organized to evaluate the final ranking. finally, in the test process, the output predictions for the model (logits) serve as an input for a softmax function and then apply an argmax function in order to get the final prediction class. ## results after the fine-tuning process using the training dataset, we measured the performance of each llms on the test set (development set provided) for subtask a (monolingual) and subtask b (using the respective training and test data provided). for subtask a (monolingual), we submitted our two best prediction results to the codabench platform: ernie and convbert llms. on the other hand, on the other hand, figure 2 and figure 3 finally, we would like to mention that, due to some technical issues in our servers, we did not submit the models with the best model scores in the validation stages for both subtasks. for this reason, 4 1113 we reported different scores in the final submission compared to our scores in the test evaluation (postsubmission, with gold labels). ## conclusion we applied a fine-tuning process using four llms: ernie, spanbert, convbert, and xlnet. in general, this llm aims to extract lexical, semantic, and syntactic information from the text. we obtained comparable results with the baselines reported (initially), but, below compared to those in the first positions. for future work, it could be interesting to prove more llms that focus on multilevel language and stylistic features; also apply a more robust finetuning process to evaluate more hyperparameters; and finally try a different approach based on text graph called graph neural networks.