CIC - NLP at G en AI Detection Task 1: Advancing Multilingual Machine-Generated Text Detection (2025)

Abstract: Machine-written texts are gradually becoming indistinguishable from human-generated texts, leading to the need to use sophisticated methods to detect them. Team CIC-NLP presents work in the Gen-AI Content Detection Task 1 at COLING 2025 Workshop: the focus of our work is on Subtask B of Task 1, which is the classification of text written by machines and human authors, with particular attention paid to identifying multilingual binary classification problem. Usng mBERT, we addressed the binary classification task using the dataset provided by the GenAI Detection Task team. mBERT acchieved a macro-average F1-score of 0.72 as well as an accuracy score of 0.73.

## introduction several researchers have worked on various binary classification tasks using ml models and llms in nlp, focusing on different areas such as hate speech detection @xcite @xcite , sentiment analysis @xcite , fake news detection @xcite , and hope speech identification @xcite . these efforts aim to discern the nuanced aspects of human communication. some of these classification tasks have been conducted on non-english and multilingual texts @xcite @xcite . with the advancements in large language models (llms), machine-generated content across various platforms, including news outlets, social media, educational, and academic publications @xcite has reached an outstanding quality. recent models like chatgpt, @xcite , llama 2 @xcite , and jais @xcite generated remarkable coherence in responding to diverse user queries. this rapid advancement has raised concerns about the potential misuse of machine-generated text in different fields such as journalism, education, and academia @xcite , in addition to the influence on operations @xcite , disinformation @xcite , spam, or unethical authorship @xcite . moreover, it poses challenges to maintain information integrity and ensure the accuracy of shared information. consequently, the ability to effectively distinguish between humangenerated and machine-generated content has become crucial for detecting possible instances of misuse @xcite @xcite . while significant progress has been made in detecting machinegenerated text in english, we still need to improve it in multilingual settings. in response to this gap, coling workshop organizers launched gen-ai content detection task 1: this shared gen-ai content detection task 1 introduces a new binary multilingual mgt detection challenge to accelerate research in this area and improve cross-lingual detection capabilities @xcite @xcite . being a shared task, it brings together researchers and practitioners interested in detecting machine-generated content reliably in many languages, reflecting the collaborative spirit and multidisciplinary innovation of shared tasks. at the broader level, the gen-ai content detection task 1 also highlights the importance of machinegenerated text (mgt) detection. also, it addresses the problem of keeping content authentic, fighting misinformation, and driving ethical use cases of ai in the multilingual realm. as cic-nlp team, we used mbert to detect and classify mgt as distinguished from human-generated text (hwt), the method used and results obtained are extensively highlighted in other sections of this report. ## literature review over the last few years there has been a great focus in the use of language models which in turn has created the need for keen classification of authentic and fake texts; this was historically stated mostly as a binary problem. the genai detection task 1 includes distinguishing between text written by a human and text written by a computer. there are two key approaches that have been broadly applied to text classification : classification with supervised methods @xcite @xcite @xcite and unsupervised (zero-shot) methods @xcite . supervised methods normally do better in terms of accuracy but are more likely to overfit, particularly when new language structures are used @xcite . on the other hand, unsupervised methods offer flexibility due to the absence of label information, however, they might call for impractical white-box access to the generating model. huge advancements in llms are currently driven by various platforms such as chatgpt powered by gpt-3.5, gpt-4 (openai, 2023), opt @xcite , llama @xcite , palm @xcite , lamda (y arcas, 2022), and bloom, and emergent models like vicuna @xcite and alpaca @xcite . these models containing millions to billions of parameters are trained on huge amounts of data, have shown extraordinary results across multiple fields including finance, customer support, and the educational sector. some of their most impressive features include their ability to write text that references human-generated text so closely that most people will initially find it hard to distinguish between the two. also, it is possible to note that their multilingual skills enable them to generate clear and high-quality text in more than fifty languages @xcite , thus making them more and more appropriate in the global business environment, but at the same time posing even higher problems to mgt detection. to the best of our knowledge, several benchmarks have been proposed to assess multilingual mgt detection models in different languages @xcite . for example, the human chat-gpt comparison corpus (hc3) @xcite compares chatgpt-generated text and human-written text, with authentications of english and chinese languages using logistic regression models and roberta-based classifiers built from features of giant language model test room (gltr). others have replicated such approaches by testing other detectors including roberta, xlm-r @xcite , logistic regression based on features from nela and other stylometric classifiers @xcite . multitude has also been introduced by researchers within the news domain for 11 languages that offers a strong test bed for multilingual detection baselines @xcite . to detect mgt, researchers released benchmark environment @xcite to compare machine-generated text detection across multiple languages using monolingual and multilingual bert models, which is consistent. as a result of comparison, it was found that multilingual-specific models tend to perform better than others. @xcite discussed challenges in multilingual nlp tasks and strategies for model adaptation across languages. while their work on sentiment analysis is almost exclusively concerned with model adaptation, their observations about the problem of improving machinegenerated sentences are relevant for our work. also, according to current literature, transformer models, such as lora-roberta and xlm-roberta, are found to be more accurate compared to classical machine learning techniques in multilingual mgt detection tasks, see for example @xcite . to summarize, researchers have been able to refine their methods of distinguishing human writing from computer scripts by integrating statistical analysis with other language models. the further development of these approaches proves that there are still challenges to differentiating between the advanced results produced by llms and works created by humans. prior work has mainly considered the classification of synthetic text in few languages, certain llms, or certain domains like news @xcite . our work extends this scope to multiple languages and include a range of diverse and popular llms across different domains. to sum up, the previous methods and works provided useful information regarding the efficiency of various approaches to identifying ai vs. human written text, but more works required. ## methodology we deployed multiple nlp techniques for data preprocessing, detection, and sorting to assess the performance of our approach to the binary multilingual machine-generated text detection task in the context of transformer-based models. next, we loaded and preprocessed a broad multilingual dataset to normalize input formats and then applied language detection to guarantee certain types of processing on specific languages. we tokenized text into language-appropriate segments, translated text between language pairs, and sorted operations adjusted to that language's unique characteristics in this gen-ai content detection task 1. using the mbert language model, which we pretrained and fine-tuned on the provided training datasets, we enhanced the model output by carefully approaching various linguistic constructs. focusing on efficient management of code-mixed and pure multilingual data, our methodology determined the tokenization method by polyglot such that each input is associated with a particular language. ## results on the test set, the model predicted the classes with an accuracy of 0.7348 and a macro-average f1-score of 0.7265, which indicates balanced test performance across the languages. our results also demonstrate that the model is capable of handling multiple languages without much performance degradation. we present figure 2 showing the confusion matrix for better analysis of the model predictions as it revealed the model strength towards accurately predicting mgt with accuracy of 0.88 and the model got weak results by confusing some model micro_f1 macro_f1 mbert 0.734 0.726 ## conclusion this paper shows how a multilingual transformerbased model detects machine-generated text in various languages. our results confirm the model's adaptability and scalability and evidence to its promising performance in high-resource languages and its potential for improvement in low-resource scenarios. we show that with appropriate data preprocessing, machine-generated text detection can be successfully extended to multilingual applications using fine-tuning and balanced datasets. this work will be continued to improve the performance for low-resource languages and deploy the model to handle more complex linguistic features such as code-switching and mixed scripts.