Team AT at S em E val-2024 Task 8: Machine-Generated Text Detection with Semantic Embeddings (2024)

Abstract: This study investigates the detection of machine-generated text using several semantic embedding techniques, a critical issue in the era of advanced language models. Different methodologies were examined: GloVe embeddings, N-gram embedding models, Sentence BERT, and a concatenated embedding approach, against a fine-tuned RoBERTa baseline. The research was conducted within the framework of SemEval-2024 Task 8, encompassing tasks for binary and multi-class classification of machine-generated text.

## introduction in the burgeoning field of natural language processing (nlp), the distinction between human and machine-generated text is becoming an area of critical importance, particularly with the rise of advanced language models capable of producing text that closely mimics human writing. the advent of such technology poses a dual-faceted challenge: while it opens new frontiers for automation and assistance, it also necessitates robust detection mechanisms to prevent misuse and uphold information credibility. this research centers on the application of semantic embeddings to detect machinegenerated text. semantic embeddings offer a nuanced approach to understanding and representing the meaning encapsulated within text, providing a fertile ground for discriminating between the subtleties of human and ai-authored content. this study contributes to this domain by evaluating the efficacy of various semantic embedding techniques in the context of semeval-2024 task 8's @xcite challenges, which include the detection of machinegenerated text across multiple generators and domains. in this study, i concentrated on the application of semantic embeddings, examining and contrasting approaches such as glove and sentence bert. i developed classifiers for the task of classifying machine-generated text as part of semeval-2024 task 8. specifically, my efforts were directed towards subtask a (monolingual) and subtask b, which involve the binary classification of machinegenerated text and multi-class classification of machine-generated text, respectively. ## related work the identification and analysis of machinegenerated text have become an increasingly pertinent field of study within the realm of natural language processing (nlp). previous research has primarily focused on detecting text authored by specific language models @xcite or within narrow domains @xcite . the latest iteration of this exploration is represented in the work by semeval-2024 task 8 @xcite , aiming at detecting text generated by a variety of models across multiple domains and languages, thus expanding the scope of investigation significantly beyond the existing literature. early approaches, such as those by @xcite , utilized basic statistical features and machine learning models for text classification tasks, providing a foundation for subsequent research. advancements were made by @xcite , who proposed a sophisticated embedding technique known as glove, which captures global word cooccurrence statistics (bullinaria and levy, 2007) to generate word representations. this technique has been widely adopted for its robustness in capturing semantic nuances. the introduction of transformer-based @xcite models, particularly bert @xcite and its variants, has revolutionized the field, as demonstrated by reimers and gurevych 2019 with the adaptation of bert for sentencelevel embeddings (sbert). these models have significantly outperformed traditional embeddings and n-gram models in various nlp tasks due to their deep contextual understanding and adaptability to different tasks and domains. moreover, roberta @xcite (a robustly optimized bert pretraining approach) refines the bert model's training methodology to substantially improve performance across a spectrum of nlp benchmarks. more recently, large language models (llms) have revolutionized text generation, achieving human-like proficiency across diverse writing tasks. as llms like chatgpt @xcite and its successors become more adept at generating coherent and contextually relevant narratives, the importance of distinguishing between machine-generated and human-produced text grows, primarily to ensure transparency and mitigate the spread of misinformation. consequently, developing robust detection methods for machine-generated text is crucial in maintaining the integrity of information and upholding trust in digital communications. ## methods in this study, i explored four semantic embedding methods to evaluate against the fine-tuned roberta baseline provided by the task coordinators @xcite . the methods employed encompass the glove @xcite embedding method, the training n-gram embedding method, sentence bert method, and the concatenated embedding method. in this section, i will present the methodologies applied to address subtask a (monolingual) and subtask b. their primary distinction lies in the extraction of text features. ## experimental results in this section, i will present and analyze the experimental outcomes derived from the implementation of my methodologies. ## conclusion in conclusion, this research has contributed to the field of detecting machine-generated text by exploring the efficacy of various semantic embedding methodologies. the results present the performance of several pre-trained semantic embeddings, like glove and sbert, on the tasks of machinegenerated text detection in semeval-2024 task 8.