Team Innovative at S em E val-2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection (2024)

Abstract: With the widespread adoption of large language models (LLMs), such as ChatGPT and GPT-4, in various domains, concerns regarding their potential misuse, including spreading misinformation and disrupting education, have escalated. The need to discern between human-generated and machine-generated text has become increasingly crucial. This paper addresses the challenge of automatic text classification with a focus on distinguishing between human-written and machine-generated text. Leveraging the robust capabilities of the RoBERTa model, we propose an approach for text classification, termed as RoBERTa hybrid, which involves fine-tuning the pre-trained Roberta model coupled with additional dense layers and softmax activation for authorship attribution. In this paper, we present an approach that leverages Stylometric features, hybrid features, and the output probabilities of a fine-tuned RoBERTa model. Our method achieves a test accuracy of 73% and a validation accuracy of 89%, demonstrating promising advancements in the field of machine-generated text detection. These results mark significant progress in the domain of machine-generated text detection, as evidenced by our 74th position on the leaderboard for Subtask-A of SemEval-2024 Task 8.

## introduction semeval-2024 task 8 @xcite is centered on the detection of machine-generated text across multiple generators, domains, and languages. this detection is crucial for mitigating the risks associated with the potential misuse of large language models (llms), which have advanced capabilities in generating multilingual human-like texts. in this task, the goal is to differentiate between machinegenerated and human-authored texts, addressing concerns regarding the authenticity and trustworthiness of textual content in various contexts and languages. the rapid advancement of deep learning technologies has ushered in a new era where the boundaries between human-generated and machinegenerated artifacts are increasingly blurred. this evolution is epitomized by the emergence of deepfakes, which convincingly mimic genuine human actions, and the widespread adoption of natural language generation (nlg) systems, particularly those leveraging neural language models. these developments have led to the creation of neural texts that bear striking resemblances to humanauthored content, posing significant challenges in distinguishing between the two. traditionally, authorship attribution uchendu et al. (2020) within the realm of natural language processing (nlp) focused on accurately attributing text to its true human author. however, with the advent of neural language generation (nlg) techniques @xcite capable of producing human-quality open-ended texts, the attribution landscape has expanded to encompass authorship by humans, machines, or a combination thereof. as the quality of machine-generated texts continues to improve, the lines between human and machine-generated text become increasingly indistinct, exacerbating the challenge of differentiation. moreover, the potential for misuse of these technologies, including the generation of misinformation, fake reviews, and political propaganda at scale @xcite , underscores the critical need for effective methods to discern neural texts from human-authored content-a problem known as neural text detection (ntd) @xcite , which is a sub-problem of the broader authorship attribution domain. in this paper, we present an approach named roberta hybrid, tailored to address the challenge of distinguishing between human and machinegenerated texts. this method utilizes the finetuning of a pre-trained roberta language model, enhanced with additional layers for text classifica-tion, in evaluating the performance of pre-trained language models for text differentiation tasks. we specifically target the task of automated detection of machine-generated text, recognizing the growing importance of discerning between human-authored and artificially generated content in today's digital landscape. in this paper, we present an approach named roberta hybrid, tailored to address the challenge of distinguishing between human and machinegenerated texts. this method utilizes the finetuning of a pre-trained roberta language model, enhanced with additional layers for text classification, in evaluating the performance of pre-trained language models for text differentiation tasks. we specifically target the task of automated detection of machine-generated text, recognizing the growing importance of discerning between human-authored and artificially generated content in today's digital landscape. we secured the rank 74 on the leaderboard for subtask-a ## related work and background subtask a of task-8 in the semeval challenge @xcite utilized a monolingual dataset, focusing on distinguishing between human-written and machine-generated text. the dataset primarily employed english as its medium. each instance in the dataset is labeled as either 1 (indicating machine-generated text, specifically generated by chat-gpt) or 0 (indicating human-written text). the subtask a dataset consists of three subsets: training, development, and testing datasets. the training dataset contains 119,757 samples, while the development dataset comprises 5,000 samples. the testing dataset includes 34,272 samples. in the training dataset, there are 63,351 samples labeled as class 0 and 56,406 samples labeled as class 1. conversely, the development dataset consists of 2,500 samples, all of which are labeled. within the dataset, there are columns denoted as "model" and "source." the "model" column specifies which model generated a particular text, while the "source" column indicates the origin of the text. previous research has explored various approaches to authorship attribution (aa), aiming to accurately identify the authors of texts, which is crucial in fields such as forensic linguistics, plagiarism detection, and content analysis. traditional methods have relied on stylometric features, which capture the distinctive writing style of individual authors based on linguistic patterns and characteristics. however, recent advancements in natural language processing (nlp) have introduced transformer-based models like bert (bidirectional encoder representations from transformers) that have demonstrated state-of-the-art performance across a range of nlp tasks. fabien et al. (2020) showcased the effectiveness of bert for text classification tasks, highlighting its ability to extract semantic and syntactic information from text. however, there has been a lack of systematic exploration into the performance of fine-tuned pretrained language models, specifically for authorship attribution. the introduction of bertaa addresses this gap by fine-tuning bert with a dense layer and softmax activation specifically for authorship attribution. the incorporation of bert allows for the utilization of semantic and syntactic information encoded in text representations, potentially improving the accuracy of authorship attribution systems. furthermore, bertaa integrates stylometric features, which capture lexical and structural characteristics of text, and hybrid features, which combine character-level n-grams, enhancing the model's ability to capture both content-related and stylistic aspects of authorship. however, stylometric classifiers encounter challenges when tasked with accurately determining the authorship of human versus neural texts. @xcite noted that certain stylometric classifiers were surpassed in performance by deep learning-based models. furthermore, research findings, as cited in @xcite , revealing stylometry's inability to identify neural misinformation underscore the necessity for alternative methodologies to address the authorship attribution (aa) task within the context of neural text generation (ntd). consequently, researchers have increasingly embraced and refined deep learning-based approaches for distinguishing between neural and human-generated text. these approaches can be further classified into three main categories: glove-based, energy-based, and transformer-based attribution models. language models often exhibit a lack of syntactic and lexical diversity, characterized by the repetition of the same expressions and a limited use of synonyms and references. this behavior fröhling and zubiaga (2021) can be approximated using named entities (ne) and properties of coreference chains, along with shifts in part-of-speech (pos) distributions between human and machine-generated text. features based on netags, coreference chains, and pos distributions fröhling and zubiaga (2021) can effectively capture the differences in syntactic and lexical diversity @xcite between human and machine-generated text. repetitiveness: machine-generated text is prone to repetitiveness @xcite , often overusing frequent words and exhibiting highly parallel sentence structures. features such as the share of stop-words, unique words, and words from "toplists" can highlight the lack of diversity in machinegenerated text. additionally, measures of n-gram overlap in consecutive sentences can reveal patterns of lexical and syntactic repetition, further distinguishing between human and machine-generated text. lack of coherence: a significant challenge in machine-generated text is the lack of coherence @xcite , particularly over longer sentences and paragraphs. coherence can be assessed through the development of entities and the tracking of their appearance and grammatical roles across the text. features based on entity grids and transition frequencies @xcite between consecutive sentences can capture the coherence or lack thereof in machine-generated text. by incorporating these features into automated detection methods, researchers aim to develop robust and accessible tools for distinguishing between human and machine-generated text, thereby mitigating the risks associated with language model abuse. ## system overview the experiments conducted encompassed subtask a within the monolingual track. subtask a posed a binary classification challenge, aimed at discriminating between human-generated text and text produced by the machine (chatgpt). in addressing subtask a, a stylometric classifier was developed to exploit diverse stylistic attributes, encompassing text length, word count, average word length, count of short words, proportion of digits and capital letters, frequencies of individual characters and digits, hapax-legomena (a measure of text richness), and the frequency of 12 punctuation marks. these stylometric features were employed in training a logistic regression model. furthermore, hybrid features, incorporating the 100 most frequent character-level bi-grams and trigrams, were integrated. logistic regression was applied for classification using these hybrid features as well. the ultimate model adopted a hybrid strategy, whereby output probabilities from the roberta classifier, the stylometric classifier, and the hybrid features classifier were concatenated. this concatenated output underwent classification using an additional logistic regression model. we chose roberta due to its robust performance in natural language understanding tasks, its ability to handle a wide range of text data, and its pre-training on large-scale corpora, which helps capture nuanced linguistic patterns. to refine the roberta hybrid model, class probabilities derived from the stylometric features and those obtained from fine-tuning the roberta model were concatenated separately for both the training and test datasets. additionally, the probabilities derived from training a logistic regression model on hybrid features were integrated into the hybrid model. ## conclusion in this paper, we have presented an innovative approach, termed the roberta hybrid model, for the task of detecting machine-generated text. leveraging the robust capabilities of the roberta model, we fine-tuned it coupled with additional dense layers and softmax activation for authorship attribution. our method incorporates a hybrid of stylometric features, character-level n-grams, and the output probabilities of a fine-tuned roberta model, achieving significant advancements in machine-generated text detection. experimental results demonstrate the effectiveness of our approach, with a validation accuracy of 89% and a test accuracy of 73%. although these results do not surpass the baseline methods, they highlight the potential of our approach in addressing the challenges posed by machine-generated text across diverse domains. moving forward, our work opens avenues for further research in enhancing the accuracy and robustness of machine-generated text detection systems. future efforts may focus on exploring additional feature representations, optimizing model architectures, and addressing the challenges posed by monolingual machine-generated text. our efforts in enhancing machine-generated text detection we have tried to contribute to the broader objective of safeguarding the integrity and credibility of online content.