KI n IT at S em E val-2024 Task 8: Fine-tuned LLM s for Multilingual Machine-Generated Text Detection (2024)

Abstract: SemEval-2024 Task 8 is focused on multigenerator, multidomain, and multilingual black-box machine-generated text detection. Such a detection is important for preventing a potential misuse of large language models (LLMs), the newest of which are very capable in generating multilingual human-like texts. We have coped with this task in multiple ways, utilizing language identification and parameter-efficient fine-tuning of smaller LLMs for text classification. We have further used the per-language classification-threshold calibration to uniquely combine fine-tuned models predictions with statistical detection metrics to improve generalization of the system detection performance. Our submitted method achieved competitive results, ranking at the fourth place, just under 1 percentage point behind the winner.

## introduction recent large language models (llms) are able to generate high-quality texts that are not easily detectable by human readers. a problem arises when such generated texts are misused for academic exams @xcite , plagiarism @xcite , disinformation spreading @xcite , etc. therefore, it is crucial to develop automated means to detect machine-generated texts. semeval-2024 task 8 @xcite consists of three subtasks: a) binary human-written vs. machine-generated text classification, b) multiway machine-generated text classification, and c) human-machine mixed text detection. in our work, we have focused on subtask a, especially its multilingual track. it covered 8 known languages for training (arabic, bulgarian, chinese, english, german, indonesian, russian, urdu), multiple domains (e.g., wikipedia, news, abstracts), and multiple text generators (e.g., @xcite . during our participation in the shared task, we have explored various alternatives. our best submitted solution (illustrated in figure 1 ) combines two fine-tuned llms (green-colored) with statistical detection (orange-colored) using a two-step majority voting (purple-colored) based ensemble method. such a system achieved fourth place in the final leaderboard, with a performance of 95% in accuracy, within 1 percentage point range behind the winning system. we have published the source code for easier replication purposes foot_0 . we have used the statistical detection methods implemented in the recently published imgtb framework foot_1 @xcite ), which will be extended to also support all the fine-tunning options that we have used in this work. our key contributions include: an interesting observation regarding our rmistral system is that although our per-language threshold calibration method would not bring the best accuracy on the test set (0.93), the threshold fixed to 1.0 (only predictions with a probability of 1, i.e. 100% confident, are marked as machine-generated) would win the competition (accuracy of 0.97). nevertheless, we have noticed such a threshold performance only after the deadline and we considered the model being over-fitted (we would not submitted the results) which turned-out to be false. ## background and methodology for the machine-generated text detection task, three main groups of methods are nowadays used @xcite . the first one is a stylometric detection, which uses linguistic features (e.g., n-grams) to differentiate between human and machine writing styles @xcite . the second group is a statistical detection, which uses statistical distribution based on a pre-trained language model (e.g., gpt2) to calculate various metrics (e.g., entropy) that can be used even without training (i.e., better generalization) to differentiate machine and human written texts @xcite . the last group is a fine-tuned detection, which further trains an already pre-trained language model for the detection task @xcite . in the semeval2024 task 8 @xcite , we have focused on the multilingual track of subtask a, which aimed at a binary classification to differentiate between human-written and machinegenerated texts. the provided dataset (not allowing additional training data) contained the predefined splits of train, dev, and test sets. the train and dev sets officially contained 8 languages (3 languages in the dev set only), while unknown number of languages is contained in the test set. due to a multilingual nature of the data and our previous experience in multilingual machinegenerated text detection @xcite , we wanted to try-out something new in this shared task. our initial idea was to experiment with a per-language "mixture-of-experts", which would consist of multiple models, fine-tuned in a monolingual way per each official language in the train and dev sets. since it was expected that surprise languages will be present in the test set, we would have used an additional multilingually fine-tuned model for other languages. however, we have started the experiments only few weeks before the deadline, which gave us little time to cope with the problems such as over-fitting and hyper-parameter optimisation (shown as severe towards the deadline). therefore, while training these per-language models, we also started to fine-tune the falcon-7b model @xcite for the machinegenerated text detection task, inspired by the winning system @xcite of the recent alta 2023 shared task (although english monolingual). since falcon-7b is pre-trained on two languages only (english and french), we did not want to use it in a standalone way due to uncertain cross-lingual capability. therefore, we have similarly fine-tuned the mistral-7b model @xcite , which is similarly sized generative model outperforming even some 13b parameters models in common benchmarks. we have not previously experimented with such a "big hammer" for the task; therefore, it was an interesting new experience for us. we have further combined these llms with statistical detectors to ensure better generalization of the system, which is described in the following sections. each prediction uses a separate classificationdecision threshold, which is applied on prediction probabilities and statistical metrics. these thresholds are calibrated in a per-language way, meaning that separate thresholds are used for each language officially present in the train and dev sets, plus an additional threshold for unknown languages (i.e., not officially present in the train and dev sets). the thresholds are calibrated based on the machine-system description * llm2s3 the system described in this paper. it is an ensemble using two-step majority voting for predictions, consisting of 2 llms (falcon-7b and mistral-7b) fine-tuned using the train set only, 3 zero-shot statistical methods (entropy, rank, binoculars) using falcon-7b and falcon-7b-instruct for calculation of the metrics, utilizing language identification and per-language threshold calibration. ## experimental setup for the experimental purpose, we have used the defaults splits of the provided dataset, namely the train and dev sets in the pre-deadline experiments, and the gold labels of the test set for the postdeadline evaluation of the pre-deadline system alternatives. the main system described in this paper uses only the train set in the training process; however, uses both the train and dev sets for the classification threshold calibration. some of the system alternatives used both the train and dev sets in the training process, as described in as the key evaluation metric in the shared task is accuracy, we have also used this metric for the preliminary system evaluation and selection of the alternative for submission. since classification task is sensitive to the used classification threshold, we have also used auc roc (area under curve of the receiver operating characteristic) as a threshold independent metric, providing better information about the classification capability. for the fine-tuning process, we have used the official baseline script foot_3 , modified to export machineclass prediction probabilities in addition to the predictions. since, it was not clear which version of the xlm-roberta model was marked as a baseline in the multilingual track (with the known accuracy of 0.72), we have trained both the base (xlm-r-b) and large (xlm-r-l) versions. in addition, we have also included mdeberta-v3-base (mdeberta) model in our baselines, since it performed the best in our previous work @xcite . to perform per-language models fine-tuning, we have used the source field of the train and dev data to select data only for the specific language. other parameters of the fine-tuning process remained unchanged. the fasttext language identification is used for a prediction, which uses the machine-class probability of the corresponding language-specific model. the used qlora peft fine-tuning process used the binary cross entropy with logits for loss calculations and 4-bit quantization using bitsandbytes foot_4 . the lora configuration foot_5 used an alpha of 16, a dropout of 0.1, r of 64, and the task type of sequence classification. unlike the baseline finetuning, this version used half-precision training, gradient accumulation of 4 steps, and evaluation each 1,000 steps. other parameters were the same. due to time constraints, we have not done any hyper-parameter optimization; thus, further improvements of the system are very likely possible. ## results the experimental results are provided in due to high accuracy and high auc roc metrics using the dev set, we considered rfalcon and rmistral over-fitted; therefore, we decided not to submit our rllm2s3 system. this turned-out to be a mistake, since it performed slightly better than the submitted llm2s3 on the test set. on the other hand, our suspicion of over-fitting plmoe (due to the similar observations) turned-out to be valid, since it performed much worse using the test set. therefore, it seems that per-language monolingually fine-tuned (i.e., lower amount of samples than multilingually fine-tuned) models require optimization of hyper-parameters to prevent overfitting and to better generalize to unseen texts. as an ablation study, we also provide the results for individual components of our system alternatives. as the results show, the ensembling into more complex systems of llm2s3 and rllm2s3 helped generalization of the classification performance. individual methods would not outperform the submitted system.