Mast Kalandar at S em E val-2024 Task 8: On the Trail of Textual Origins: R o BERT a- B i LSTM Approach to Detect AI -Generated Text (2024)

Abstract: Large Language Models (LLMs) have showcased impressive abilities in generating fluent responses to diverse user queries. However, concerns regarding the potential misuse ofsuch texts in journalism, educational, and academic contexts have surfaced. SemEval 2024introduces the task of Multigenerator, Multidomain, and Multilingual Black-Box MachineGenerated Text Detection, aiming to developautomated systems for identifying machinegenerated text and detecting potential misuse. In this paper, we i) propose a RoBERTaBiLSTM based classifier designed to classifytext into two categories: AI-generated or human ii) conduct a comparative study of ourmodel with baseline approaches to evaluate itseffectiveness. This paper contributes to the advancement of automatic text detection systemsin addressing the challenges posed by machinegenerated text misuse. Our architecture ranked46th on the official leaderboard with an accuracy of 80.83 among 125.

## introduction the task of classifying text as either ai-generated or human-generated holds significant importance in the field of natural language processing (nlp). it addresses the growing need to distinguish between content created by artificial intelligence models and that generated by human authors, a distinction crucial for various applications such as content moderation, misinformation detection, and safeguarding against ai-generated malicious content. this task is outlined in the task overview paper by @xcite , emphasizing its relevance and scope in the nlp community. our system employs a hybrid approach combining deep learning techniques with feature engineering to tackle the classification task effectively. specifically, we leverage a bilstm (bidirectional long short-term memory) @xcite neural network in conjunction with roberta @xcite , a pre-trained language representation model, to capture both sequential and contextual information from the input sentences. this hybrid architecture enables our system to effectively capture nuanced linguistic patterns and semantic cues for accurate classification. participating in this task provided valuable insights into the capabilities and limitations of our system. quantitatively, our system achieved competitive results, ranking 46 relative to other teams in terms of accuracy and f1 score. qualitatively, we observed that our system struggled with distinguishing between sentences generated by ai models trained on specific domains or datasets with highly similar linguistic patterns. we have released the code for our system on github foot_0 , facilitating transparency and reproducibility in our approach. ## related works in the field of detecting machine-generated text, numerous methodologies and models have been examined. a distinguished methodology is the application of the roberta classifier, which enhances the roberta language model through fine-tuning for the specific purpose of identifying machinegenerated text. the proficiency of pre-trained classifiers like roberta in this domain has been affirmed through various studies, including those conducted by @xcite and additional research by @xcite @xcite @xcite . concurrently, the xlm-r classifier exploits the multilingual training of the xlm-roberta model to effectively recognize machine-generated text in various languages, as demonstrated by @xcite . alternatively, the exploration of logistic regression models that incorporate gltr (giant language model test room) features has been undertaken. these models strive to discern subtleties in text generation methodologies by analyzing token probabilities and distribution entropy, as investigated by @xcite . furthermore, detection efforts have utilized stylometric and nela (news landscape) features, which account for a broad spectrum of linguistic and structural characteristics, including syntactic, stylistic, affective, and moral dimensions, as reported by @xcite and @xcite . additionally, proprietary frameworks like gptzero, devised by princeton university, focus on indicators such as perplexity and burstiness to analyze texts for machine-generated content identification. although the specific technical details are sparingly disclosed, the reported effectiveness of gptzero in identifying outputs from various ai language models highlights its significance in the ongoing development of machine-generated text detection strategies @xcite @xcite . ## system overview based on our observation (see 7), we discovered that language modeling task encodes the various features required for detection of ai written text. so we used pretrained roberta in most of our architectures so exploit this power of language models. ## results we tested our models on various models on the test set. the results can be viewed in (table : 3). ## conclusion in conclusion, our bilstm+roberta model effectively tackled the task, achieving competitive results, thanks to its deep learning and pre-trained language model. while a similar model with unfrozen roberta boasted higher precision, its complexity came at the cost of increased parameters. impressively, our model ranked 46th out of 125 competition entries ( moving forward, there are several avenues for future work to explore. firstly, further experimentation with different model architectures, including alternative combinations of encoders and classifiers, could potentially yield improvements in performance. additionally, fine-tuning hyperparameters and exploring advanced techniques for model optimization may enhance the robustness and generalization capabilities of our system. furthermore, incorporating additional contextual information or domain-specific knowledge could potentially augment the model's understanding and performance on specific tasks. overall, our findings contribute to the ongoing research efforts in natural language processing and provide valuable insights for future developments in this domain.