Overview of the 2023 ALTA Shared Task: Discriminate between Human-Written and Machine-Generated Text (2023)

Abstract: The ALTA shared tasks have been running annually since 2010. In 2023, the purpose of the task is to build automatic detection systems that can discriminate between human-written and synthetic text generated by Large Language Models (LLM). In this paper we present the task, the evaluation criteria, and the results of the systems participating in the shared task.

## introduction the generative abilities of recent large language models (llms) such as chatgpt have shown impressive abilities in generating content with quality close to those generated by humans. despite the possible advantages of llms, the concern about inappropriate utilization of these generated contents, accompanied by social and ethical issues, has been underscored in several preceding studies @xcite @xcite . some of those llms are designed with watermarks @xcite . however, there is also the possibility of deploying llms without watermarks. consequently, effectively distinguishing texts by vanilla language models from the human-written text pieces has become an emerging and challenging task. the goal of the 2023 alta shared task is to build automatic detection systems that can discriminate between human-written and text generated by llms. the text comes from a variety of sources and different llms. formally, this is a binary classification problem, as each candidate sentence can be generated either by human or a llm. the evaluation metric is accuracy. section 2 presents related work. section 3 details how the data have been gathered and labeled. section 4 presents the evaluation framework. section 5 describes a baseline that was made available to the participants. section 6 lists the details of the participating systems and their results. finally, section 7 concludes this paper. ## related work the preliminary work for identifying machinegenerated text involves feature-based approaches, such as utilizing linguistic patterns @xcite and cues @xcite , e.g., bag-of-words. more recent work @xcite proposes to use detectors based on pre-trained language models. e.g., @xcite use roberta as the basis of the detector. after a fine-tuning process, roberta has been proven its prowess as a detector across multiple domains @xcite @xcite . to align with our research goals, we depart from the conventional assumption that detailed knowledge of synthetic data origin is readily available, which includes specifics about generative models, decoding strategies, and domains. in reality, such information often remains elusive. it is worth noting several recent works on discriminating human-and machine-generated texts, e.g., openai gpt-2 detector (openai, 2023), gptzero @xcite , detect-gpt @xcite , dipper @xcite and g3-detector @xcite , which train their detectors on collected datasets with labeled human-written and machine-generated texts. later on, a training-free detector dna-gpt @xcite was proposed to discover n-gram patterns in the machine-generated text. although some progress has been made in the corresponding task, its efficacy and reliability largely depend on the task settings, such as the domains of the generative tasks, the structures and scale of the generative models, etc. @xcite propose an assessment framework using evasive soft prompts, and @xcite further introduce ai detectability index as an evaluation metric for machine-generated text detection. related shared tasks include clin33 foot_0 , autextification foot_1 (sarvazyan et al., 2023) , detecting generated scientific papers foot_2 (robodasha, 2022), and machine learning model attribution challenge foot_3 @xcite . ## data gathering the data for the 2023 alta shared task has been gathered from four generative benchmarks across multiple domains in the data. these comprise machine translation, and specifically the wmt (de-en) benchmark @xcite , summarization, with cnn-dailymail (cnndm) @xcite , and language pre-training, including wiki-data and the openwebtext benchmark @xcite . the human-written text are directly extracted from the ground-truth sentences in the above benchmarks. in contrast, the machine-generated text are produced by several widely-used generative models, all of which are gpt-based models. specifically, these models contain gpt2-large, gpt3.5turbo, and gpt4. we have used gpt2 model files through the huggingface repository foot_4 , and then finetuned these models on the aforementioned datasets. for the gpt3.5-turbo and gpt4 models, we use prompt-based text generation through the openai api foot_5 . specifically, we use the following prompts for different generative benchmarks: translation: please translate the following german sentence into english. summarization: please summarize the following long paragraph with a short summary. language pre-training: please paraphrase the following sentence. the final data used in the 2023 alta shared task was selected by random sampling from the gathered data to ensure 50%-50% between human and machine-generated text ( ## evaluation framework the evaluation framework was implemented as a codalab competition foot_6 with three phases. in the development phase, labelled training and unlabelled development sets were made available. participant systems could submit their system output on the development set up to 100 times, and the evaluation results were made public to all participating systems via a leaderboard. in the test phase, an additional unlabelled test set was made available, and participating systems could make up to 3 submissions. the results of the test phase form a separate leaderboard and are used for the final ranking reported in this paper. a third unofficial submissions phase has no end date and is available to all participant systems so that they can make additional submissions on the test data. these submissions form a separate leaderboard and are not used for the final ranking. ## baseline we formulate the detection framework as a binary classification task. based on previous observations @xcite , roberta has proven successful in various detection tasks. therefore, to provide a starting point for participants, we provide the vanilla roberta-large @xcite ## participating systems and results a total of 9 teams submitted runs in the development phase, and 8 submitted in the test phase 10 . the alta shared tasks have two categories, a student category where student members are not allowed to have completed a phd degree and cannot be employed full time (with the exception of student supervisors), and an open category for those who are not eligible for the student category. however, this year (2023) only teams in the student category submitted in the test phase. tests of statistical significance foot_10 indicate that the difference between the first and the second team is statistically significant. all of the participating systems that submitted a system description to us reported to have used llms in different ways, often as part of ensemble approaches, sometimes in addition to other approaches. team od-21 @xcite used falcon-7b and label smoothing. they also used prompting techniques for samples with low confidence scores. team detectorbuilder @xcite used an ensemble with majority voting of bert, roberta, and debertav3. team aast nlp @xcite used an ensemble with majority voting of distill-bert, xlmroberta, and roberta. team samnlp (joy and aishi, 2023) used a feature-level ensemble of debertav3 and xlm-roberta, where these llms are jointly trained by concatenating their last layer and adding subsequent lineal layers. team vdetect @xcite experimented with various ensemble approaches using a varied range of models including several transformer models, rnns, and cnn, plus svm and naive bayes. team synthdetectives @xcite used an ensemble of albert, electra, roberta, and xlnet, where the predictions of these llms are fed to a linear regression classifier. ## conclusions the 2023 alta shared task focused on the discrimination between human-written text and machinegenerated text. all systems submitting runs to the test phase had accuracy results over 0.95, and the baseline based on roberta had an accuracy result of 0.9765. the top system submitted to the shared task had an accuracy of 0.9910, yet the difference with the second best system was statistically significant. we were pleased to observe such good performance by the participants. this indicates that the task of identifying machine-generated text can be easy when used as a shared task like the one presented here. this task may become more difficult in the future as technology evolves.