L3i++ at G en AI Detection Task 1: Can Label-Supervised LL a MA Detect Machine-Generated Text? (2025)

Abstract: The widespread use of large language models (LLMs) influences different social media and educational contexts through the overwhelming generated text with a certain degree of coherence. To mitigate their potential misuse, this paper explores the feasibility of finetuning LLaMA with label supervision (named LS-LLaMA) in unidirectional and bidirectional settings, to discriminate the texts generated by machines and humans in monolingual and multilingual corpora. Our findings show that unidirectional LS-LLaMA outperformed the sequence language models as the benchmark by a large margin. Our code is publicly available at https://github.com/honghanhh/llama-as-a-judge.

## introduction the blooming of large language models (llms) has led to a significant step forward in producing different machine-generated content across diverse channels and platforms (e.g., news, social media, question-answering forums, educational, and even academic contexts). the generated texts become increasingly fluent and coherent with the advent of recent models (e.g., gpt-4o, claude 3.5). however, this also resulted in concerns regarding their potential misuse, such as spreading misinformation and causing disruptions in the education system. consequently, there is a need to develop automatic systems to identify machine-generated text to mitigate its potential misuse. inspired by the work of @xcite , we investigate the feasibility of training a binary sequence classifier that can reliably differentiate between text generated by humans and text that appears human-like but is generated by machines but leverage the performance with the integration of a llama-as-a-judge in three different settings on the larger monolingual and multilingual corpora from @xcite . the main contribution of this paper is as follows: ## related work the success of llms in various downstream nlp tasks @xcite leads to the overuse and abuse of the information generated by llms. however, it is essential to acknowledge that the outputs generated by llms are not always accurate, giving rise to the issue of hallucination @xcite . researchers have developed several automatic detection methods @xcite that can identify the mg texts from the hw texts, which initially can be divided into two categories, i.e., metric-based and model-based methods. ## data we evaluate the feasibility of our approach with english foot_0 and multilingual 2 corpora from @xcite . both corpora are the continuation and improvement of @xcite with additional training and testing data generated from novel llms and including new languages. ## methodology this section tackles the problem by formulating it as supervised sequence classification tasks. we then introduce our proposed architecture and present how we fine-tune them before indicating how we assessed their performance. ## discussion unidirectional vs. bidirectional context. the unidirectional ls-llama's focus on sequential learning, coherence recognition, and specialized training objectives makes it particularly well-suited for the task of mg text detection. in contrast, the bidirectional ls-llama, while powerful in capturing overall context, may struggle with the specific sequential dependencies that are critical for effectively distinguishing hw from mg texts. this fundamental difference in architecture and training approach likely contributes to the observed performance advantage of unidirectional ls-llama. masking strategy. the "masked" aspect refers to how models are trained to predict missing parts of the input. in unidirectional masked ls-llama, the focus is often on learning to predict the next token or fill in gaps based on prior context. this can enhance their ability to understand coherent patterns typical in hw texts, which explains the higher performance of masked ls-llama compared to unmasked settings, which can potentially suffer from data leaks. ## error analysis we conduct several analyses to investigate how different factors would affect the detection performance of our best classifier, namely ls-llama. figure 2 illustrates the confusion matrices for the english and multilingual test sets. these matrices reveal a notable tendency for higher error rates in detecting mg content. this observation suggests the model may be calibrated to prioritize detecting mg (label 1) instead of hw texts (label 0). we elaborated our analysis regarding the error rate by text length and textual analysis of misclassification. the results suggest that error rates may not consistently increase or decrease with longer or shorter texts; instead, they vary based on data. however, there is a tendency for the classifier to have higher errors when the length of the text is from 10,000 to 20,000 words (see figure 3 ). ## conclusions in conclusion, we conducted a comparative study of label supervision llama, so-called ls-llama to highlight the potential and feasibility of finetuning an llm to discriminate between hw and mg texts. three different settings have been applied, including unidirectional masked, unidirectional unmasked, and bidirectional. our findings suggest that unidirectional masked ls-llama outperformed two other settings and the benchmarks for both monolingual and multilingual sets.