FI Group at S em E val-2024 Task 8: A Syntactically Motivated Architecture for Multilingual Machine-Generated Text Detection (2024)

Abstract: In this paper, we present the description of our proposed system for Subtask A - multilingual track at SemEval-2024 Task 8, which aims to classify if text has been generated by an AI or Human. Our approach treats binary text classification as token-level prediction, with the final classification being the average of token-level predictions. Through the use of rich representations of pre-trained transformers, our model is trained to selectively aggregate information from across different layers to score individual tokens, given that each layer may contain distinct information. Notably, our model demonstrates competitive performance on the test dataset, achieving an accuracy score of 95.8%. Furthermore, it secures the 2nd position in the multilingual track of Subtask A, with a mere 0.1% behind the leading system.

## introduction the evolution and widespread adoption of generative pre-trained transformers, notably with the release of chatgpt have significantly influenced the landscape of digital communication and content creation. while these advancements herald a new era of efficiency and creativity, enabling applications ranging from sophisticated writing aids to advanced conversational agents, they simultaneously introduce significant challenges and ethical concerns. in fact, the proliferation of ai-generated texts has raised alarm over issues like the dissemination of misinformation, the facilitation of academic fraud, and the potential erosion of trust in digital content. this underscores the urgent requirement for robust solutions to identify ai-generated content, safeguarding the integrity of information while embracing the benefits of ai advancements. in this paper, we aim to develop a reliable detection system by participating in the semeval task 8 on machine-generated text detection. this task is notable for its complexity, as it involves multi-generator, multidomain, and multilingual text, making it a highly challenging endeavor. furthermore, the evaluation is conducted on unseen domains and languages, establishing it as a robust benchmark for evaluating ai text detectors. this requires the model to effectively generalize across different domains and languages. we focus our efforts on the binary detection, which aims to determine whether a text has been generated by an ai or not. to tackle this challenge, we propose a syntactically motivated architecture. our approach is primarily inspired by the realization that texts generated by ai and humans are semantically similar, as they are derived from comparable topical distributions. hence, we argue that the distinction between them lies in their syntax and writing style. typically, transformer-based text classification relies on information from the last layer for classification. however, our model takes a different approach by dynamically aggregating information from all layers of the transformer (a.k. multi-layer fusion @xcite . this method is intentionally designed to harness the diverse linguistic information present at various levels of the transformer, as noted in previous studies @xcite @xcite . these studies reveal the uneven distribution of linguistic features across the transformer's architecture, with syntactic details predominantly in the initial layers and complex semantic information in the deeper layers. by utilizing insights from all layers, our model aims to capture the entire range of linguistic cues, enhancing its capability to accurately differentiate between human and ai-generated content. additionally, our model moves beyond the standard practice of using just the [cls] token for classification in bert-based classifiers. it applies sequence labeling to classify each token in the text as either human or ai. we believe that this approach enables the capture of more complex phrasal structures, which helps in more effectively distinguishing the style and syntax of a text. our proposed model obtains competitive performance on the test leaderboard of the shared task subtask a, securing the 2nd best position on binary multilingual detection, using a much smaller model than other approaches often using finetuned llms. ## related works since the introduction of large-scale pre-trained models like gpt-3, capable of generating highquality text, the detection of machine-generated text has attracted considerable interest. the most common and straightforward strategy for addressing this task involves training models on a labeled dataset comprising both human and ai-generated text. this approach is utilized by well-known models such as the openai ai text detector and commercial models such as gptzero @xcite . while these models achieve strong indomain results, they often require labeled datasets from a wide range of sources and domains to achieve generalization. an alternative approach involves zero-shot detectors, which do not necessitate any model training. for example, dna-gpt @xcite assesses n-gram divergence between the continuation distribution of re-prompted text and the original text for making predictions, while detect-gpt @xcite employs a curvature-based criterion to determine if a passage is generated by a specific llm. ## preliminary study in this section, we detail a preliminary study that provided essential insights, guiding us towards our final model design. motivation our aim was to assess the efficacy of semantic embeddings, particularly sentence-bert, in differentiating between machine-generated and human-authored texts. figure 1 illustrates the embeddings of texts from both humans and various language models, visualized using sentence-bert embeddings @xcite and umap for dimensionality reduction @xcite . ## architecture in this section, we provide a detailed overview of our proposed model's architecture illustrated in figure 2 . ## conclusion in this paper, we presented our system submitted to semeval-2024 task 8 for detecting human-written and machine-generated text, achieving 2nd place for subtask a on multilingual texts. our system relies on a hierarchical fusion strategy that adaptively fuses representations from transformer's layers, with a focus on syntactic rather than semantic information. by leveraging syntactic features, particularly through sequence labeling, we captured more phrasal structures of text, thereby enhancing our ability to distinguish text styles and syntax. our system achieved robust performance across diverse unseen domains and languages, demonstrating its adaptability and generalization capability, notably considering that we used a smaller model compared to other proposed systems often reliant on fine-tuned llms.