Team MGTD 4 ADL at S em E val-2024 Task 8: Leveraging (Sentence) Transformer Models with Contrastive Learning for Identifying Machine-Generated Text (2024)

Abstract: This paper outlines our approach to SemEval-2024 Task 8 (Subtask B), which focuses on discerning machine-generated text from human-written content, while also identifying the text sources, i.e., from which Large Language Model (LLM) the target text is generated. Our detection system is built upon Transformer-based techniques, leveraging various pre-trained language models (PLMs), including sentence transformer models. Additionally, we incorporate Contrastive Learning (CL) into the classifier to improve the detecting capabilities and employ Data Augmentation methods. Ultimately, our system achieves a peak accuracy of 76.96% on the test set of the competition, configured using a sentence transformer model integrated with CL methodology.

## introduction the emergence of sophisticated large language models (llms) has significantly blurred the lines between human-written and machine-generated texts, prompting an urgent need for systems capable of accurately distinguishing between the diverse sources. in response, our team participated in semeval-2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection, as defined by @xcite . this task aims to identify the origin of texts across various languages and domains, addressing critical concerns around the misuse of llms. we focused on subtask b, which involves classifying english texts by their generative sources. this task adopted a fine-grained label set, for distinguishing not only between human-written and machine-generated texts, but also among texts generated by different machines. our system leveraged transformer-based pre-trained lan- † corresponding author. ## background and related work the introduction of the m4 dataset by @xcite offers a comprehensive landscape for evaluating detection techniques across various generators, domains, and languages. the research done on the m4 dataset underscores the difficulties in generalizing detection across different domains and generators, highlighting the limitations of current approaches. data for subtask b of semeval-2024 task 8, focusing on the detection of human-written against machine-generated texts from multiple generators across monolingual (english) contexts, is derived from the original m4 dataset. the dataset comprises 71,027 training and 3,000 development/test samples, distributed across multiple sources -wikipedia, reddit, arxiv, and wikihow -with the testing data focused on the out-of-domain peer-read domain. the task demands the identification of text origins, whether human or machinegenerated by models. this underscores the necessity for systems that are adept at handling multiclass and out-of-domain classification challenges. in response to these challenges, our approach builds upon the insights from prior work. @xcite , for instance, applied linguistic-and transformer-based method to detecting the author-1 1711 ship of text. we also considered the methods used in the m4 dataset paper to compare with. ## system overview having outlined the urgency and relevance of distinguishing machine-generated text, we now describe our transformer-based approach to tackle this issue. our system tackles machine-generated text detection by carefully selecting a suite of transformer models (roberta base , roberta large @xcite , gpt-2 small @xcite , xlnet-base @xcite ), sentence transformer models (all-mpnet-base-v1 foot_0 , allmpnet-base-v2 foot_1 , all-roberta-large-v1 foot_2 ), and integrating two different contrastive learning techniques, namely supervised contrastive learning (scl) @xcite and dual contrastive learning (dualcl) @xcite , alongside data augmentation strategies, inspired by @xcite . initially, we conducted hyperparameter tuning across both transformer and sentence transformer models in their base forms with trivial cross-entropy (ce) loss to identify optimal configurations. subsequently, we refined our model selection to gpt-2 small, roberta base , roberta large , and allroberta-large-v1, based on performance metrics on the enriched test set, which is described in section 4.1, further experimenting with combination of contrastive learning technique variants to enhance detection accuracy. our approach culminates in the additional application of data augmentations aiming to improve robustness and generalizability. ## results and discussions in our analysis of the performance of four transformer models as our baseline on the task of detecting machine-generated text, distinct variations in accuracy underscore the impact of model design and size on effectiveness, as shown in ## conclusion in conclusion, our paper presents a comprehensive approach to semeval-2024 task 8 (subtask b), focusing on the detection of machine-generated text and its attribution to specific large language models (llms). leveraging transformer-based methods, pre-trained language models (plms), contrastive learning (cl), and data augmentation techniques, we have developed a robust detection system achieving a peak accuracy of 74.69%. our findings underscore the effectiveness of integrat-ing cl into the classification process and highlight the strength of leveraging diverse plms for improved performance in discerning between human and machine-generated text.