CIC - NLP at G en AI Detection Task 1: Leveraging D istil BERT for Detecting Machine-Generated Text in E nglish (2025)

Abstract: As machine-generated texts (MGT) become increasingly similar to human writing, these dis- tinctions are harder to identify. In this paper, we as the CIC-NLP team present our submission to the Gen-AI Content Detection Workshop at COLING 2025 for Task 1 Subtask A, which involves distinguishing between text generated by LLMs and text authored by humans, with an emphasis on detecting English-only MGT. We applied the DistilBERT model to this binary classification task using the dataset provided by the organizers. Fine-tuning the model effectively differentiated between the classes, resulting in a micro-average F1-score of 0.70 on the evaluation test set. We provide a detailed explanation of the fine-tuning parameters and steps involved in our analysis.

## introduction the rapid advancement of large language models (llms) has revolutionized nlp across all fields @xcite @xcite @xcite @xcite . machine-generated text (mgt) refers to text produced using ai algorithms with little or no human intervention. on the other hand, human-written text (hwt) relies on natural language and contextual understanding in its approach to construction. ai models are able to generate new content by being trained on large text datasets. however, it can be challenging for ai models to replicate the attributes of hwt accurately. in this work, we leverage the intricacies between mgt and hwt to develop a model that can successfully distinguish between the two. ai models can generate grammatically correct and contextually relevant text, but they often lack true cohesion between the sentences they produce @xcite . hwt conveys personal experience, emotions, and cultural understanding in its content, while mgt can only mimic surface-level emotional cues without grasping the underlying sentiment or context, leading to misrepresentation. detecting mgt means identifying and carefully analyzing misleading or inappropriate content @xcite . this detection has also become crucial for addressing ethical issues arising from using ai @xcite . llms often reflect biases and limitations inherent in the data on which they were trained @xcite . mgt may lean toward certain linguistic patterns, expressions, or topics that are common in the training dataset, which can differ from the more varied and dynamic expressions found in hwt. these biases can also help distinguish between mgt and hwt. to address some of these highlighted issues, the organizers of coling workshop 2025 introduced a large novel dataset @xcite , @xcite , @xcite . the uniqueness of this dataset is the fact that it broadly covers different text types from several human and ai sources, thus making it a valuable dataset to test several proposed models in exploring the attributes of mgt and hwt. ## related work early efforts in detecting mgt focused on identifying telltale signs of machine output, such as lack of coherence, repetitive patterns, or unnatural phrasing @xcite . however, as generative models continue to improve significantly in terms of fluency, the challenge of detecting mgt has become increasingly difficult. traditional rule-based approaches and ml classifiers have been largely insufficient to keep pace with the increasingly sophisticated text classification capabilities of deep learning (dl) models @xcite . recent approaches have thus turned to dl, particularly transformer-based models, which offer state-of-the-art performance in various nlp tasks (hoang; @xcite @xcite . the introduction of transformer models marked a significant breakthrough in nlp. research by various authors has shown that transformer-based models are effective in performing a range of nlp tasks @xcite . distilbert has recently been explored for various text classification tasks, including sentiment analysis @xcite @xcite ) and text summarization @xcite . its efficiency and robustness make it a promising candidate for distinguishing between mgt and hwt. the key advantage of using dis-tilbert for this task lies in its ability to effectively capture semantic and syntactic patterns in text, which are often subtle yet critical for differentiating between the two types of text. the authors in @xcite explored the challenges and feasibility of detecting ai-generated text across various domains and llms. they highlighted the growing ability of llms to generate human-like text, emphasizing the need for effective detection methods to mitigate risks such as misinformation and plagiarism. they argued that previous research had been limited to specific domains or language models, while real-world detectors needed to handle diverse inputs without prior knowledge of their source. to address this gap, they created a testbed containing a variety of humanwritten and machine-generated texts from multiple llms. their results revealed significant challenges in distinguishing between human-authored and machine-generated texts, especially with outof-distribution data. the authors also investigated linguistic patterns that could aid in differentiation, using tools like stanza to analyze named entities, part-of-speech tags, and other features. their analysis showed that including texts from diverse domains and llms reduced the linguistic differences between human and machine-generated texts, making detection more difficult. the approach described in @xcite focused on the participation of the turing-testers team in the autextification shared task at iberlef 2023, which aimed at automated text identification. the task consisted of two subtasks: the first involved distinguishing between human-written and machine-generated texts, while the second focused on attributing texts to specific large language models (llms). the authors addressed subtask 1 in both english and spanish, testing a combination of traditional machine learning and deep learning methods, along with integrating additional metadata related to readability, complexity, sentiment, emotion, and toxicity of the texts. the experiments demonstrated the effectiveness of fine-tuning a multilingual bert uncased model for detecting ai-generated texts. however, the inclusion of additional features, such as metadata, led to a decrease in performance, even compared to traditional machine learning models, suggesting that bert's architecture alone was sufficient for classifying text. the authors concluded that, while their experiments provided valuable insights into the effectiveness of different models for detecting ai-generated text, further research was needed. ## results in this section, we present the results of the test dataset provided by the organizers of the coling 2025 conference as predicted by the pre-trained distilbert model on the english test supplied through google drive. the prediction of the model was uploaded to the codabench platform featured by the organizers of the conference for submission, and the micro-and macro-average f1 scores were computed accordingly. the results proved the model's ability to distinguish between humanwritten text and ai-generated text. the details of the results are presented in ## conclusions the application of distilbert, a compact and efficient transformer model for the detection of aigenerated text illustrates the advancement in the potential of transformer-based models to distinguish nuances in language style and origin. this task addresses specific challenges concerning the size of the dataset and the fact that llm ai-generated text often mimics human-like patterns, thereby requiring the model to capture subtle stylistic and structural differences. our approach leverages dis-tilbert's efficient architecture and tokenizer capabilities to effectively identify these distinctions with a micro-average f1-score of 0.70. this result highlights distilbert's ability to generalize and perform well on the intricate task of text detection and classification, showing that smaller models can yield robust outcomes even in sophisticated nlp tasks. the performance obtained in our experiments emphasizes both the viability of distilbert in this domain and the promise of continued exploration with other transformer-based models. future work will focus on expanding the dataset to cover a broader array of ai-generated text from various models and fine-tuning distilbert and similar models to achieve even more refined detection capabilities.