Stacking the Odds: Transformer-Based Ensemble for AI -Generated Text Detection (2023)

Abstract: This paper reports our submission under the team name ‘SynthDetectives’ to the ALTA 2023 Shared Task. We use a stacking ensemble of Transformers for the task of AI-generated text detection. Our approach is novel in terms of its choice of models in that we use accessible and lightweight models in the ensemble. We show that ensembling the models results in an improved accuracy in comparison with using them individually. Our approach achieves an accuracy score of 0.9555 on the official test data provided by the shared task organisers.

## introduction transformer @xcite is a sequenceto-sequence model that has enabled the training of large language models (llms). llms such as gpt enable text generation in response to userdefined prompts, allowing for wide applicability. as a result, they have proliferated into several aspects of society, both for good and for bad. text generated from llms, when used unethically, can have several detrimental implications: they can cause widespread fake news, dispense away with all notions of academic honesty and authorship, and threaten to replace human-generated information with ai-generated data at large. motivated by these existential concerns, many models have been developed to distinguish aigenerated content from human's. alta is participatory in tackling this issue by announcing the alta 2023 shared task, whose goal is to build 'automatic detection systems that can discriminate between human-authored and synthetic text generated by large language models (llms)' @xcite . the text comes from a variety of sources in terms of domains (e.g. medical, law), and source model (e.g. gpt-x, t5). technically, participating teams are required to build an automated system to solve a binary classification task, distinguishing between human and ai-generated text. models are evaluated based on robustness and accuracy. there is no requirement on the efficiency and run-time performance. our team participated in the said shared task. the code is available here foot_0 . we stack multiple transformer-based models in an ensemble and show that the ensemble performs better than the individual models. in this paper, we will discuss existing works in the domain, our analysis of the original training data, our proposed pipeline and architecture, our experimental results, and suggested future work. stylometric attribution detects neural text generator (ntg) using ensembles of classical machine learning (ml) models trained on stylometric features such as liwc (linguistic inquiry & word count), pos tags, n-grams, readability score, writeprints, empath. these models work best on a small dataset. however, as we increase the data size, they are outperformed by deep learning models rapidly. deep learning: glove-based attribution: glove @xcite is an unsupervised learning algorithm that aggregates global word-word co-occurrence statistics from text to build word representation. glove-based models use these embeddings with rnn and lstm, which was considered sota before bert @xcite . ## dataset three subsets of the dataset are presented: training, validation, and testing. the training set contains 18,000 entries, and the validation and testing each contains 2,000 entries. evaluation is based on the testing set which was not released until the testing phase of the competition. the training set contains three columns 'id', 'text', and 'label' (1 if humangenerated, 0 if ai-generated). the validation and testing set each contains two columns 'id', and 'text'. samples of the training set are shown in when analyzing the dataset, we find that the aigenerated and human-generated text is evenly split into 9000-9000 entries respectively in the training set. we also find that the average word count per text is relatively low. the mean length is in the 34-35 range in the three subsets, with a standard deviation in the 26.7-27.9 range, a maximum of 172-193, and a minimum of 1, making this a short sequence task. to find the main domains of the text, we remove all stop words from each set and find the frequency of n-gram phrases from the cleaned corpus, and pick the top-k elements from each set. we look at the n-gram range of (3, 4), with @xmath0. we find that overwhelmingly all the phrases are in the domain of law across the three sets. the following list is the union of the three sets with the above configuration: {'court of appeals', 'of the court', 'of the united', 'of the united states', 'opinion of the', 'the court of', 'the court of appeals', 'the district court', 'the opinion of', 'the united states'}. ## approach our approach uses a stacking ensemble of classifiers (as shown in figure 1 ) to perform our training, validation, and testing. a stacking ensemble of classifiers acts similarly to a weighted voting classifier. our choice of architecture is inspired by @xcite , which achieved high performance in the ruatd shared task 2022 on artificial text detection in russian @xcite . we train each weak classifier using the above dataset split, and then we concatenate the raw predictions on the training set together and feed them to the meta-learner. we use a simple logistic regressor as our meta-learner. our criteria for picking models are ease of use, short-sequence-task-based models, and variety in model architecture. we also choose only encoder-only models, since they are built for regression/ classification tasks, and we can conveniently extract the [cls] token from their last hidden state to perform logistic regression. as a result, we use albert, electra, roberta and xlnet as the transformer-based models. to optimise the training cycle, we tokenise the entire dataset (with the respective model's tokeniser), and pass them through their respective pre-trained model to obtain the [cls] token from the last hidden state. we consider this to be our dataset and do our splitting, training, and testing on this processed dataset. to train, we pass the [cls] token through a single fully connected layer, with the input dimension equivalent to the model's [cls]'s dimension, and the output dimension of 2, then we softmax the output. after fine-tuning the weak models, we perform inference on the training split and concatenate the predictions which are fed for the meta-learner to train. ## results figure 2 shows our smoothed training loss using the gaussian kernel (since the original training loss displays too wide short-cycle variation, obfuscating the overall trend), figure 3 shows our validation loss which follows a similar pattern. finally, the alta shared task organisers provided us with a shared task test set i.e., the official test set. we achieve an accuracy of 0.9555 with our stacking ensemble on the official test set. ## conclusion & future work in this paper, we describe our system for the alta shared task 2023. we show how an ensemble of transformer-based models can be combined using a logistic regression classifier to predict if a text was generated by ai. we achieved an accuracy of 0.9555 using a stacking ensemble of basic encoderonly transformer models. our work presents a novel approach to ensemble transformer-based models to approach the alta shared task. however, this work identifies several potential directions for future work. ensembling models usually benefit from a variety of learners specialised in different types of inputs. we only implemented an ensemble of transformer classifiers, but it would be beneficial to integrate other non-transformer-based weak learners as detailed in section 2. especially useful would be to integrate contrastive learning in our training procedure. in addition, it would also be useful to perform data augmentation which can help generalise the model. one suggested technique is 'text continuation', where given a human-generated text, we slice the first n words and have an llm finish the sentence. furthermore, the scope of the shared task does not imply the possibility of an adversarial attack. it has been shown that the roberta detector can be attacked easily through misspelling @xcite . it would also be helpful to build detectors that are resilient in this regard.