RFBES at S em E val-2024 Task 8: Investigating Syntactic and Semantic Features for Distinguishing AI -Generated and Human-Written Texts (2024)

Abstract: Nowadays, the usage of Large Language Models (LLMs) has increased, and LLMs have been used to generate texts in different languages and for different tasks. Additionally, due to the participation of remarkable companies such as Google and OpenAI, LLMs are now more accessible, and people can easily use them. However, an important issue is how we can detect AI-generated texts from human-written ones. In this article, we have investigated the problem of AI-generated text detection from two different aspects: semantics and syntax. Finally, we presented an AI model that can distinguish AI-generated texts from human-written ones with high accuracy on both multilingual and monolingual tasks using the M4 dataset. According to our results, using a semantic approach would be more helpful for detection. However, there is a lot of room for improvement in the syntactic approach, and it would be a good approach for future work.

## introduction large language models (llms) are widely used. they are easily accessible, and people can use them by passing their queries to chatbots to generate their desired texts for several purposes and, more importantly, in different languages. although llms have their own advantages and simplify the text generation process for humans, they have increased concerns about the misuse of this technology for adversarial purposes such as generating hallucinations, misinformation, disinformation, and fake news. furthermore, improper use of llms can cause disruption in students' learning process. this issue has led to research on detecting aigenerated texts versus human-written ones, and a number of articles have investigated this classification task. however, the main focus of the presented works has mainly been on the semantic aspect of this text classification task. in this article, we have investigated this issue using two different approaches to consider both the semantic and syntactic aspects of texts. to achieve this aspiration, we have developed two different models for both syntactic and semantic-based analysis to apply * equal contribution them to both multilingual and monolingual datasets. in this way, we used the m4 article's dataset @xcite for both multilingual and monolingual tasks. for the syntax analysis of this task, we have developed an attention-based long short-term memory(lstm) model to cover the complexities related to long sentences and the relationship between different parts of a sentence, and regarding the semantic analysis of this task, we have developed a transformer-based model. according to the results, our systems have performed better than m4's provided baseline in the multilingual task, and in the monolingual task, our results are really close to m4's provided baseline. in the end, we have provided our results and compared the results of both these models with each other and with previous works in this area. ## background today, due to the remarkable advancements in natural language processing (nlp), models like chatgpt, @xcite , gemini (formerly known as bard) @xcite , and others have reached a point where they can generate texts that closely resemble human writing. consequently, the task of identifying texts generated by ai has become increasingly important. this task holds significant value across various domains, including content moderation, plagiarism detection, and ensuring transparency in ai-generated content. the approaches for this task can be categorized into three categories: (1) deep learning-based detection, (2) statistical discrepancy detection, and (3) watermark-based detection. deep learning-based models can be formulated as a classification task where the input is a text that can be generated by either a human or an ai. the model is trained with labeled data, where each text is assigned a label indicating whether it was generated by ai or by a human. this allows the model to learn patterns and features that can accurately classify texts based on their origin. these methods are susceptible to adversarial attacks, which can manipulate the input text to deceive the model's classification. however, deep learning-based models generally demonstrate good performance on the training data distribution @xcite . statistical discrepancy detection methods first learn the patterns of ai-generated and human-written texts separately. then, they identify statistical discrepancies between these patterns to distinguish between the two. by analyzing various linguistic features, such as word frequencies, sentence structures, or syntactic patterns, these methods can detect differences that arise from the distinct nature of ai-generated and human-written texts. some tools like gptzero @xcite use perplexity (how well a language model predicts the next word based on the previous ones) and burstiness (variations in sentence length) to assess whether the text is ai-generated or human-written. the idea of watermarking initially emerged from the field of computer vision and has since been applied to nlp @xcite . this method involves embedding a hidden "watermark" during the text generation process with the objective of identifying text generated by a specific language model. in the context of black-box language models, @xcite utilize this watermarking method to detect and identify text generated by such models. in the semeval2024 task 8, @xcite , our attention was directed towards the multilingual and monolingual tracks of subtask a. this subtask was designed for binary classification to distinguish between texts written by humans and those generated by machines. in terms of the dataset employed for this task, we utilized the multilingual dataset provided in the m4 article. the human-written texts in this dataset were collected from diverse sources spanning different domains. these sources include wikipedia (march 2022 version), wikihow, reddit (eli5), arxiv, and peer-read for english. for chinese, the texts were sourced from baike and web question answering (qa). additionally, texts from news sources were included for urdu and indonesian, while for russian, texts were obtained from ruatd. for arabic, the texts were collected from arabic wikipedia. for the monolingual section, we have used the english corpora. in this dataset, ai-generated texts leverage multilingual llms such as chatgpt, textdavinci-003, llama, flant5, cohere, dolly-v2, and bloomz. these models undertake diverse tasks, including creating wikipedia articles from titles and abstracts (from arxiv), generating peer reviews from titles and abstracts (peerread), answering questions from platforms like reddit and baike/web qa, and composing news briefs based on the title. this dataset contains 122k human-machine parallel data in total, with 101k for english, 9k for chinese, 9k for russian, 9k for urdu, 9k for indonesian, and 9k for arabic, respectively @xcite . for our experiment, we used the english corpora of this dataset in the monolingual track. for the multilingual track, we utilized the whole dataset, which contains human-written and ai-generated texts from six different languages: english, arabic, chinese, indonesian, russian, and urdu. as it is evident, our model's input is text documents, and its output is a single label that specifies whether the given text is human-written or ai-generated. ## method to classify texts as either ai-generated or humanwritten, we have examined two crucial aspects: semantics and syntax. our analysis of these aspects, which is detailed below, aims to identify distinctive features. ## conclusion in this study, we proposed a system to distinguish between human-generated and ai-generated texts. our approach considered both semantic and syntactic aspects. for the semantic analysis, we focused on smaller text segments instead of the entire document, as we believed that ai models could produce similarly coherent long texts as humans. the results confirmed our assumption. our syntactic analysis, which employed a basic model to categorize texts based on their grammatical patterns using upos tags, revealed no significant differences in upos tag distribution between ai-generated and human-written texts. however, the analysis of word order identified distinct patterns in the semantic approach. this finding suggests that relying solely on upos tags for differentiation may be insufficient. in conclusion, our proposed system demonstrated superior performance compared to the official baseline, achieving a 3.9% improvement in the multilingual subtask. these results emphasize the significance of considering texts in smaller segments rather than analyzing them as a whole. moreover, our discoveries suggest that focusing solely on grammar, as indicated by their upos tags, may not sufficiently distinguish between ai-generated and human-written texts. therefore, for future research efforts, it might be beneficial to utilize graph neural networks (gnns) to examine the grammatical connections among words. this involves representing word embeddings as nodes and their grammatical relationships as edges based on their constituency parsing or dependency parsing trees.