CLUL ab- U of A at S em E val-2024 Task 8: Detecting Machine-Generated Text Using Triplet-Loss-Trained Text Similarity and Text Classification (2024)

Abstract: Detecting machine-generated text is a critical task in the era of large language models. In this paper, we present our systems for SemEval-2024 Task 8, which focuses on multi-class classification to discern between human-written and maching-generated texts by five state-of-the-art large language models. We propose three different systems: unsupervised text similarity, triplet-loss-trained text similarity, and text classification. We show that the triplet-loss trained text similarity system outperforms the other systems, achieving 80% accuracy on the test set and surpassing the baseline model for this subtask. Additionally, our text classification system, which takes into account sentence paraphrases generated by the candidate models, also outperforms the unsupervised text similarity system, achieving 74% accuracy.

## introduction the rapid evolution of large language models (llms) has significantly impacted the dynamics of information exchange, blurring the lines between human and machine-generated text. stateof-the-art llms are available to the public on a large scale, allowing users to generate humanlike text with minimal effort. this advancement poses a dual-edged sword: while offering unprecedented capabilities in generating human-like text, it also raises critical concerns about privacy @xcite , ethics @xcite , and misinformation @xcite @xcite -especially given the llms' tendency to produce plausible yet factually baseless content, known as hallucinations @xcite . distinguishing between human and machine authorship has thus emerged as a major challenge, bearing implications for content credibility and ethical standards in digital communication. as a response to the need for effective detection methods that can discern the origin of text in this new landscape, the semeval-2024 task 8 @xcite presents an exciting challenge of ai-generated text detection over three different subtasks: subtask a: binary human-written vs. machine-generated text classification, subtask b: multi-way machine-generated text classification, and subtask c: human-machine mixed text detection. in this paper, we work on subtask b, which focuses on multi-class classification to distinguish between human-written and machine-generated text by five state-of-the-art llms. these models are chatgpt, text-davinci-003, llama @xcite , cohere, dolly-v2 @xcite , and bloomz @xcite . we propose three different systems to address this task: unsupervised text similarity, triplet-losstrained text similarity, and text classification. we show that the triplet-loss-trained text similarity system outperforms the other systems, achieving 80% accuracy on the test set and surpassing the baseline model for this subtask. additionally, our text classification system, which takes into account sentence paraphrases generated by the candidate models, also outperforms the unsupervised text similarity system, achieving 74% accuracy. however, the unsupervised text similarity system performs poorly, achieving only 29% accuracy on the test set. we note that the latter is the only system that we submitted to the task, and the other systems are post-evaluation improvements. the main contributions of this paper are: ## background and related work recent research has resulted in significant advancements in natural language generation (nlg) models @xcite and generative pre-trained transformer (gpt) models @xcite however, with potential threats posed by these models, research on identifying machinegenerated text has also surged @xcite . initially, methods employing traditional machine learning models such as logistic regression were proposed @xcite . nevertheless, the limitation of the machine learning model, which requires extensive re-training @xcite , and the rise of the pre-trained transformer models, have prompted researchers to adopt the large models. relatively smaller language models such as roberta @xcite have achieved state-of-the-art performance across various domains including social media, news articles, and online reviews @xcite @xcite . in addition to this, other approaches based on contrastive learning and similarity metrics @xcite have also emerged. such research efforts continue with the ongoing evolution and adoption of text-generative models. ## dataset we work with m4 @xcite , a dataset for semeval-2024 task 8, which consists of 71,027 data samples for the training set, 3,000 data samples for the development set, and 18,000 data samples for the test set. each sample is labeled with one of the six labels: human, chatgpt, davinci, cohere, bloomz, or dolly. figure 1 shows an example of the given dataset, which consists of id, text, model, label, and source. @xcite prompted these models to write a passage given some information from the source. the sources of the texts are diverse, including wikipedia, wikihow @xcite , reddit, arxiv, and peer-read @xcite . ## system overview in this section, we provide a comprehensive overview of the three approaches we explored: unsupervised text similarity, triplet-loss-trained text similarity, and text classification. ## conclusion in this paper, we presented several different systems for semeval 2024 task 8's text classification between human and five distinct machines. our submitted model, which relied on unsupervised embeddings coupled with cosine similarity, was poor at handling the diverse writing styles over the same topics that were present in the data, resulting in low classification scores. our text classification approaches and our triplet-trained text similarity approach all outperformed the simple unsupervised model. the triplet loss learning especially improved performance over the submitted model, with its pretraining allowing it to better maximize the distinctions between texts. for future work, we plan to adapt our systems to other classification tasks. we also plan to explore other methods for training the triplet loss model, such as using a larger model or using a different loss function. additionally, using a larger dataset for the text classification models could improve their performance.