Nota AI at G en AI Detection Task 1: Unseen Language-Aware Detection System for Multilingual Machine-Generated Text (2025)

Abstract: Recently, large language models (LLMs) have demonstrated unprecedented capabilities in language generation, yet they still often produce incorrect information. Therefore, determining whether a text was generated by an LLM has become one of the factors that must be considered when evaluating its reliability. In this paper, we discuss methods to determine whether texts written in various languages were authored by humans or generated by LLMs. We have discovered that the classification accuracy significantly decreases for texts written in languages not observed during the training process, and we aim to address this issue. We propose a method to improve performance for unseen languages by using token-level predictive distributions extracted from various LLMs and text embeddings from a multilingual pre-trained langauge model. With the proposed method, we achieved third place out of 25 teams in Subtask B (binary multilingual machine-generated text detection) of Shared Task 1, with an F1 macro score of 0.7532.

## introduction recently proposed large language models (llms) have demonstrated the ability to generate natural language with a level of fluency akin to that of humans, but they can still produce content that includes incorrect information @xcite . due to this fluency, people may not realize that the generated text contains inaccuracies, making it easier for false information to spread as if it were true. this can lead to various negative consequences. as a result, detecting text generated by llms has become increasingly important. in particular, with numerous language models now supporting multilingual text generation, identifying llm-generated text across different languages has also become a significant research topic. in this paper, we discuss methods to determine whether texts written in various languages are authored by humans or generated by llms. more specifically, we describe the system we developed for subtask b of shared task 1 (i.e., binary multilingual machine-generated text (mgt) detection) @xcite at the coling 2025 workshop on detecting ai generated content (daigenc). the goal of this task is to develop a high-performance binary classification system. to create the dataset used in this task, a variety of llms, ranging from closed models such as gpt-4 to open-source models such as the llama series @xcite , were utilized. one of the main challenges in this shared task is that a significant number of samples in the evaluation set are written in languages that the models did not observe during the training phase. it is known that when pretrained language models (plms) foot_1 , which have been pre-trained on multilingual raw corpora, are fine-tuned with largescale natural language understanding task datasets in specific languages, these models can effectively perform those tasks even on samples written in languages not observed during training @xcite . this is referred to as zero-shot cross-lingual transfer learning @xcite . in this work, we conducted a preliminary study to examine the effectiveness of zero-shot cross-lingual transfer learning in multilingual mgt detection. we trained the multilingual e5-large @xcite model on two different datasets. one dataset consisted of the entire training data provided for subtask b of shared task 1, and the other was composed only of samples written in english from that dataset. evaluation was conducted on 27,045 samples from the development set, excluding english samples. the model trained on the entire dataset showed a very high f1 score (0.9806), as there were no samples written in unseen languages. however, when the model was trained only on english samples, although there was some evidence of zero-shot cross-lingual transfer learning, a significant drop in performance was observed (f1 score: 0.7965). in this work, samples written in seen languages are inferred using a multilingual plm trained through a standard supervised fine-tuning approach. to more accurately distinguish the source of texts written in seen languages, we investigate various multilingual plms. however, for texts written in unseen languages, inference is performed differently from the traditional approach. to determine whether samples written in unseen languages are mgts, we explore features that can be commonly used for the multilingual mgt detection task, regardless of the language. in monolingual mgt detection, llms are known to assign high probability value to each generated token of the mgt @xcite . however, it is unclear whether this will be useful in multilingual mgt detection. this is because many unseen languages are likely low-resource languages with insufficient training data, meaning llms may not have learned many tokens for these languages. therefore, it is uncertain whether llms will assign high probabilities to all tokens in texts written in such unseen languages. in this study, we examine the effectiveness of a model that uses token-level predictive distributions extracted from various llms as features for multilingual mgt detection. the previous study @xcite used only the llama-2 models, but we utilized various models to reflect the characteristics of different llms. to address the issue of differing tokenization results across models, we also propose a novel network architecture. additionally, we found that using meaning representations extracted from a multilingual plm that had not been fine-tuned further improved performance. previously, such meaning representations were not utilized together. the experimental results showed that tokenlevel predictive distributions extracted from various llms and embeddings from a multilingual plm are useful in multilingual mgt detection. the system we proposed achieved third place out of 25 teams in the shared task 1, with an f1 macro score of 0.7532. ## conclusion in this work, we proposed a system for determining whether samples written in unseen languages are mgts or not, and our approach achieved third place in subtask b of shared task 1. however, while we obtained a relatively high f1 score compared to the baseline, it was not significantly higher. for future work, we should focus more on investigating features that can better distinguish samples in unseen languages.