MULTIT u DE : Large-Scale Multilingual Machine-Generated Text Detection Benchmark (2023)

Abstract: There is a lack of research into capabilities of recent LLMs to generate convincing text in languages other than English and into performance of detectors of machine-generated text in multilingual settings. This is also reflected in the available benchmarks which lack authentic texts in languages other than English and predominantly cover older generators. To fill this gap, we introduce MULTITuDE, a novel benchmarking dataset for multilingual machine-generated text detection comprising of 74,081 authentic and machine-generated texts in 11 languages (ar, ca, cs, de, en, es, nl, pt, ru, uk, and zh) generated by 8 multilingual LLMs. Using this benchmark, we compare the performance of zero-shot (statistical and black-box) and fine-tuned detectors. Considering the multilinguality, we evaluate 1) how these detectors generalize to unseen languages (linguistically similar as well as dissimilar) and unseen LLMs and 2) whether the detectors improve their performance when trained on multiple languages.

## introduction machine text generation has significantly progressed in the past few months thanks to a new generation of large language models (llms). first, it was the arrival of chatgpt and later gpt-4 that made available inexpensive generation of text in a range of languages to millions of people with chatgpt becoming the fastest growing consumer application in history. second, the introduction of llama @xcite opened new 1 the dataset is available at zenodo upon request for research purposes only: https://zenodo.org/records/ 10013755 . the source code is available at: https://github. com/kinit-sk/mgt-detection-benchmark . train test: en test: non-en difference en 0.9292 0.6903 ↓ 25.7% due to the potential for misuse of machinegenerated text (mgt) for influence operations @xcite , disinformation @xcite , spam or unethical authorship @xcite , there has been a substantial amount of research on the task of machine-generated text detection @xcite @xcite . although gpt-3 was already capable of generating text in languages other than english and despite the availability of multilingual bloom @xcite , most of the prior works on this task have been (until very recently) still focusing on gpt-2 with english-only support or newer models like @xcite , gpt-neox @xcite or gpt-j (wang and komatsuzaki, 2021) which were all trained on an english language only datasets. recently, first mgt datasets in languages other than english -autextification for spanish @xcite and ruatd for russian @xcite -were made public for the detection task. there is also a dataset containing 5 languages @xcite , but this was obtained by the use of machine translation with human corrections, which renders it less useful for mgt detection benchmarking due to the potential noise. thus, a dataset comprising authentic texts in multiple languages from a single domain (i.e., a text form, such as a news article or social media post) is still missing, hampering the comprehensive evaluation of detection methods in multilingual setting. at the same time, prior works have already shown that detectors fine-tuned on data in english fail to generalize to other languages (e.g. to german in case of @xcite showing a drop from 0.946 auc roc to 0.537), which is also confirmed by our results (see in this paper, we aim to address shortcomings of prior works and focus on the multilingual aspect of mgt detection task (a binary classification of a text to be human-written or machine-generated). our main contributions are: datasets. since transformer-based generative language models became ubiquitous in 2018, researchers have released datasets to address the new problem of machine-generated text detection. the most popular datasets are gpt-2 foot_0 @xcite , 2019) and grover 3 (zellers et al., 2019) generated texts. however, as more language models (lm) got deployed, the need for more datasets from different lms arose. therefore uchendu et al. (2020) released the first multi-generator (8 lms) dataset in the news domain and uchendu et al. ( 2021 ) released the first benchmark dataset (19 lms) and environment for machine-generated text detection. next, researchers released datasets for different domains -academic papers @xcite , news, reddit posts & recipes @xcite , amazon reviews (adelani et al., 2020), multi-modal (i.e., images & texts) news articles @xcite , tweets @xcite , covid-19 articles @xcite , deepfake text in-the-wild @xcite , gamification of mgt detection @xcite , essays @xcite , promptgeneration @xcite , and multi-generator (27 llms) multi-domain (i.e., news, story, question, argument, scientific, etc.) data @xcite . however, the vast majority of these datasets are in english. a few researchers released non-english datasets in russian @xcite , chinese @xcite , iberian languages @xcite , and m4 which contains russian, chinese, urdu, indonesian, & arabic languages @xcite . in light of the increasing number of multilingual llms, we generate the largest multilingual dataset for machine-generated text detection containing 11 languages. ## benchmark dataset as suitable multilingual human-machine pair dataset containing authentic non-english texts and machine texts generated by sota text-generation models is not available, we have put together a new dataset, called multitude (benchmark dataset for multilingual machine text detection). its human segment comprises texts in 11 languages (authentic news articles) from the massivesumm dataset (varab and schluter, 2021) (see figure 2 ). titles of the selected articles have been used in the prompts for 8 language models to generate corresponding machine texts. the titles were split into train and test portion of the dataset, ensuring machine and human texts generated for the same title to be in the same split. the train split is used to fine-tune detectors in monolingual (a single training language) as well as multilingual (multiple training languages) manner; the test split of the dataset is used for evaluation of the detectors' performances. details regarding text generation and pre-processing of both, human and generated, texts can be found in appendix b. language selection. we have deliberately selected major languages from three different language families as our training languages -english (germanic), spanish (romance), and russian (slavic). to see whether linguistic similarity influences the transferability of the detection, we have selected two genealogically related test languages for all of them -dutch and german for english, czech and ukrainian for russian, portuguese and catalan for spanish. on top of that, we have also generated test data for linguistically completely unrelated arabic (semitic) and mandarin chinese (sino-tibetan). most of the languages use latin script. russian is the only training language that uses a different script -cyrillic. we have deliberately selected czech (latin) and ukrainian (cyrillic) as its slavic neighbours to see how the script affects the results. arabic and chinese use their own scripts. overall, our selection of languages is still biased towards indo-european languages and latin script. the selected languages are just representatives to see the effect and answer our research questions, while avoiding waste of resources (including more languages than necessary). using the published source code, the study can easily be extended to other languages. ## detection methods for the purpose of this benchmark, the mgt detection methods are divided into three categories. the first category includes the black-box detectorszero-shot methods available either through web interface or api, providing only small amount or no information about the underlying model or method used for detection, typically provided as commercial paid services. the second category includes statistical detectors -zero-shot methods, relying on distributional differences between generated and human-written text. the third category includes the fine-tuned detectors -language-model-based methods, which require fine-tuning of the models for the mgt detection task. see appendix c for a complete list and more details on detection methods used in the benchmark. ## experiments and results we evaluate various aspects of multilingual capabilities of the existing sota mgt detection methods. mainly, we focus on detection capabilities in english and non-english languages. however, we also analyze cross-lingual relations and generalization capabilities of detectors fine-tuned on a specific language and specific mgt generator data. firstly, in there are 315 fine-tuned detection methods, which is infeasible to show in a single table. for clarity, the table contains all evaluated black-box and statistical methods, but includes only the bestperforming version of each base model of finetuned methods (i.e., only one fine-tuned version of xlm-roberta is provided with information about language and generator llm data used for training). performance evaluation of all versions can be found in the associated github repository. the results are ordered according to the achieved macro average f1-score (since the test data are highly imbalanced in terms of machine vs human classes). this metric is used in all experiments if not stated otherwise. in the table, we also show other standard performance metrics, such as weighted average of f1-score, precision, recall, accuracy, and fpr (false positive rate) with fnr (false negative rate). these metrics are calculated based on a default classification threshold of 0.5. such a threshold can be calibrated based on various aspects (such as minimizing fpr or maximizing recall). therefore, we also show auc roc (area under the curve of receiver operating characteristic), which is a threshold-independent metric calculated based on prediction probabilities rather than the predictions themselves. unfortunately, due to missing prediction probabilities, it is available only for fine-tuned methods in our results. it must be noted that even when using optimal thresholds maximizing true positive rate and minimizing false positive rate, the key conclusions reported in this paper hold. we use the mentioned default threshold also when reporting results in the rest of this paper. based on the results, we can clearly see that fine-tuned methods outperform the others, when utilizing training data from all llms and all train languages (with two exceptions when fine-tuning on a single language performed better). we can ## discussion multilingual fine-tuned detectors perform the best. fine-tuning multilingual lms is the best approach based on our results, outperforming both english lms and statistical methods. they have better ability to generalize to other unseen languages unmatched by the other methods. still, their performance on the multitude dataset is far from perfect and the best model (mdeberta-v3-base) achieved macro f1 score of 0.85. as such, they can not be used to reliably detect mgts. linguistic similarity matters. our results show that the linguistic similarity between the languages influence how well they generalize to each other and how much the performance for the languages correlate. the typology of the languages, but also the script they use are important. multilingual mgt detectors should be trained and tested with a diverse set of languages to ensure the inclusivity of their performance. yet, the practical development is often hindered by the fact that different lms (used both as generators and detectors) support different sets of languages to different extent, making it hard to create one-model-fits-all solutions. english is an inappropriate default. as mentioned previously, the performance on the english language is an outlier and the models do not generalize to other languages that well from this language. english is often used as the de facto default language for many nlp use cases, including using it as a source language for crosslingual learning. this should be reconsidered for multilingual mgt detection. ## conclusion in the paper, we provide the first comprehensive benchmark of black-box, statistical and fine-tuned machine-generated text detection methods in multilingual settings using our novel multitude dataset, covering 11 languages and 8 sota llms. our results show that most currently available black-box methods do not work in multilingual settings and that the statistical approaches lag behind the fine-tuned ones. we also show that fine-tuning models in a multilingual manner (i.e., train data in multiple languages) results in better performance of detectors for unseen languages compared to monolingual fine-tuning. the generalization is strongly affected by language script as well as language family branches of the train and test languages. also, english seems a particularly inappropriate choice of a training language if one aims for generalization of machine-generated text detection to non-english languages. this further emphasizes the importance of creating multilingual benchmarks for machinegenerated text detection such as multitude. as a future work, we plan to extend it with a more diverse set of languages (in terms of scripts and language families) and with texts from other domains, especially social media.