Automatic Detection of Machine-Generated Text Using Pre-Trained Language Models (2023)

Abstract: In this paper, I provide a detailed description of my approach to tackling the ALTA 2023 shared task whose objective is to build an automatic detection system to distinguish between humanauthored text and text generated from Large Language Models. By leveraging several pretrained language models through model finetuning as well as the multi-model ensemble, the system managed to achieve second place on the test set leaderboard in the competition.

## introduction large language models (llms) have experienced a drastic advancement over the past few years and brought a revolution to the domain of natural language processing @xcite . through the expansion of model parameters and the intensive pre-training on a large corpus, recent llms such as @xcite and llama2 @xcite have shown their capability to understand the human language and generate high-quality text. however, the growing attention to llms and their increasing availability to the public nowadays has inevitably led to some concerns as these models can be used in an inappropriate manner to cause harm to society. this includes fake news generation @xcite , fake product reviews generation @xcite and plagiarism @xcite . therefore, this calls for the construction of a reliable machine-generated text detection system to regulate the use of llms so that we can make the most of them. to explore the effective ways that can achieve this objective, alta 2023 @xcite organised a shared task with the goal of constructing an automatic detection system to distinguish between the humanauthored text and text generated by the llms. the task is formed as a binary classification problem. my team handled this task through the utilisation of some representative pre-trained models to tackle the classification problem for machine-generated vs human-authored text given the fact that they have already exhibited their strength in various natural language processing tasks. the models i experimented with include the vanilla bert @xcite , roberta @xcite and de-bertav3 @xcite which represent the chain of improvement for the bert-based models. i also implemented an ensemble model via majority voting over the best models to further enhance the performance. the rest of the paper will provide a detailed explanation of the design of my system as well as the performance with respect to the task. ## dataset the dataset provided by the alta 2023 shared task @xcite consists of text pieces of humanauthored and machine-generated text across a wide range of domains. the machine-generated text inside the dataset originates from different types of llms. the statistics of the dataset are presented in ## methodology following the process explained in bert @xcite , the pre-trained language models discussed in section 2.2 are adopted to build binary classifiers by adding a single classification layer on top of the last hidden state of the first token (the special '[cls]' token added by these pre-trained language models) for each of them, which is the contextual representation of the full text. the model architecture is shown in figure 1a . the original text pieces are tokenised using the corresponding tokeniser for each model and the tokens are input into the classifier. the classifiers are then finetuned on the provided training set so that they can learn the language patterns inside the data. the resulting models will be applied to make predictions about the development and test set to gain insight into their performance. besides the employment of each single pretrained language model to perform classification and obtain the results, i've further performed the multi-model ensemble through majority voting over the prediction results from the 3 models that express the best performance. the process is demonstrated in figure 1b . the voting is conducted as a hard voting where for each instance of the text pieces inside the test set, the label that is assigned to the text by most of the classifiers will be selected as the final label. the logic behind this is to improve the robustness of the detection system by combining the results from multiple models. the performance of the resulting models is evaluated using the accuracy_score foot_7 from scikit-learn as specified by the alta 2023 shared task. ## conclusion in this paper, i've presented my automatic detection system for the alta 2023 shared task that classifies machine-generated and human-authored text. the capability of pre-trained language models in handling the task is demonstrated by fine-tuning them on the dataset and constructing the classifiers. the benefits that the multi-model ensemble brings to the performance of the detector are also indicated by the experiment results. as a result, the best system achieves second place in the alta 2023 shared task.