HU at S em E val-2024 Task 8 A : Can Contrastive Learning Learn Embeddings to Detect Machine-Generated Text? (2024)

Abstract: This paper describes our system developed for SemEval-2024 Task 8, “Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection.” Machine-generated texts have been one of the main concerns due to the use of large language models (LLM) in fake text generation, phishing, cheating in exams, or even plagiarizing copyright materials. A lot of systems have been developed to detect machine-generated text. Nonetheless, the majority of these systems rely on the text-generating model. This limitation is impractical in real-world scenarios, as it’s often impossible to know which specific model the user has used for text generation. In this work, we propose a single model based on contrastive learning, which uses ~40% of the baseline’s parameters (149M vs. 355M) but shows a comparable performance on the test dataset (21st out of 137 participants). Our key finding is that even without an ensemble of multiple models, a single base model can have comparable performance with the help of data augmentation and contrastive learning.

## introduction in recent years, natural language processing (nlp) has been totally dependent on deep learning rather than statistical machine learning. with multi-task learning @xcite , attentionbased transformers @xcite , and the use of reinforcement learning in nlp @xcite , it has been used in our day-to-day life from mathematical calculations @xcite to email writing. but with huge help, it has also been used to generate fake news @xcite , to plagiarize copyright materials @xcite , and also to cheat in exams or assignments @xcite . humans can identify machine-generated text only at the chance level @xcite . there has been a dire need to develop a system to detect machine-generated text. though a lot of works @xcite @xcite @xcite @xcite @xcite have already been deployed for detecting machine-generated text, with the current development of llms, most of the systems are failing to find out which one is humangenerated vs. machine-generated (mostly due to the improvement of coherency, fluency and usage of real-world dataset @xcite ). in this context, the task "multigenerator, multidomain, and multilingual black-box machine-generated text detection" provides a dataset for training models to classify machine-generated texts. the shared task consists of three sub-tasks: binary classification (machine vs. human), multi-class classification (which model/human generated this?), and span detection (which part of the text is machinegenerated?). a detailed description of the task can be found in the shared task paper @xcite . in this paper, we describe our final submission on subtask a (binary classification). there were two big challenges of this task: first, five different models have been used to generate the machinegenerated text. @xcite has shown that the best defense for machine-generated text is the model itself that was used for generation. however, in reality, there is a massive surge in large language models (llms), each with its own unique style of text generation. the challenge in this particular subtask has heightened due to the utilization of five different llms. this complexity demands a versatile, model-agnostic architecture capable of detecting text generated by llms in a generalized manner. second, following the previous challenge, the organizers have employed a different model for generating the validation and test datasets compared to those used in the training set. this implies that the text was drawn from a completely distinct distribution. as a result, participants must develop a generalized model capable of performing effectively regardless of the specific model used in the text generation process. in response to the key challenges, we have investigated the performance of contrastive learning for this particular task. contrastive learning has been used as a valuable technique across various domains, including text embedding @xcite , document embedding @xcite , event embedding (roy dipta et al., 2023), vision @xcite and language-vision learning @xcite . notably, unlike the majority of submissions in any shared task like competition, our final submission utilized a single model to classify the machine-generated texts rather than an ensemble of multiple models. hence, our contributions to this paper are as follows, 1. we proposed a novel data augmentation technique, which nearly makes the data x times bigger (x is the number of models used for data augmentation). ## related works in this section, we will provide the prior works that have been done in the realm of machinegenerated text detection ( §2.1) and contrastive learning ( §2.1). ## experimental setup the following sections are used to describe the technical details behind our data augmentation technique ( §4.1), encoder ( §4.2), classifier head ( §4.3) and hyperparameters ( §4.4). ## results in this section, we report our results on subtask a and discuss our analysis. our evaluation is based on the accuracy metric, but we have provided the micro and macro-f1 for better comparison. all the results are averaged on 3 runs with 3 different random seeds. ## conclusion & future work in this work, we introduce our contrastive learningbased system, which shows a comparable performance. we demonstrate that a model with half the parameters and without an ensemble of large models or hand-engineered features can show a comparable performance, which requires more exploration in this field. for future work, the use of recent prompt-based models foot_8 can be used for data augmentation. also, the effect of more advanced contrastive loss, i.e., triplet loss @xcite or infonce loss @xcite , need to be explored.