A Simple yet Efficient Ensemble Approach for AI -generated Text Detection (2023)

Abstract: Recent Large Language Models (LLMs) have demonstrated remarkable capabilities in generating text that closely resembles human writing across wide range of styles and genres. However, such capabilities are prone to potential abuse, such as fake news generation, spam email creation, and misuse in academic assignments. Hence, it is essential to build automated approaches capable of distinguishing between artificially generated text and human-authored text. In this paper, we propose a simple yet efficient solution to this problem by ensembling predictions from multiple constituent LLMs. Compared to previous state-of-the-art approaches, which are perplexity-based or uses ensembles with a large number of LLMs, our condensed ensembling approach uses only two constituent LLMs to achieve comparable performance. Experiments conducted on four benchmark datasets for generative text classification show performance improvements in the range of 0.5 to 100% compared to previous state-of-the-art approaches. We also study that the influence the training data from individual LLMs have on model performance. We found that substituting commercially-restrictive Generative Pre-trained Transformer (GPT) data with data generated from other open language models such as Falcon, Large Language Model Meta AI (LLaMA2), and Mosaic Pretrained Transformers (MPT) is a feasible alternative when developing generative text detectors. Furthermore, to demonstrate zero-shot generalization, we experimented with an English essays dataset, and results suggest that our ensembling approach can handle new data effectively.

## introduction the domain of natural language generation (nlg) is witnessing a remarkable transformation with the emergence of large language models (llms) such as generative pre-trained transformer (gpt-4) @xcite , large language model meta ai (llama-2) @xcite , pathways language model (palm) @xcite , bard foot_0 , and text-to-text transfer transformer (t5) @xcite . llms, characterized by their large parameter size, have shown state-of-the-art capabilities in generating text that closely mirrors the verbosity and style of human language. they have shown exceptional performance across a wide range of applications, such as story generation @xcite , artificial intelligence (ai)-assisted writing @xcite , medical question answering @xcite , conversational response generation @xcite , radiology report generation @xcite , and code auto-completion @xcite . moreover, their capacity to generalize across tasks without the need for explicit training (referred to as zero-shot learning) or conditioning on only a few examples (referred to as few-shot learning) have substantially reduced the need for extensive, task-specific training efforts. these capabilities have significantly lowered the barrier of integrating llms into various language generation applications. with the ability to generate coherent human-like text, the llms can also be misused for unethical purposes, such as fake news generation @xcite , phishing or spamming @xcite , and fabrication of product reviews @xcite . it has become increasingly crucial for both humans and automated systems to be able to detect and distinguish ai-generated text, particularly when this text is employed for disseminating misinformation or propaganda @xcite . to address these challenges, automatic detection of ai-generated text has recently become an active area of research. diverse modeling strategies, ranging from simple statistical techniques to cutting-edge transformer-based architectures, have been investigated to develop solutions capable of distinguishing ai-generated text from those written by humans. gehrmann et al. (2019) proposed straightforward statistical methods for identifying model-generated text that could be integrated into a visual tool to aid in their detection process. the authors assumed that ai systems produce text from a limited set of language patterns for which they have a high level of confidence. @xcite and @xcite explored entropy, n-gram frequency, and perplexity to distinguish between human-authored and ai-generated texts. advanced deep-learning frameworks, such as transformer-based models have also been explored to improve the precision and reliability of ai-generated text detection techniques. detectgpt @xcite focused on generating minor perturbations of a text passage using a generic pre-trained t5 model. it then compared the log probability of both the original text and the perturbed versions to determine if the text is authored by a human or generated by ai. @xcite proposed a coherence-based contrastive learning (coco) model where the input text is represented as a coherence graph to capture its entity consistency. robustly optimized bidirectional encoder representations from transformers (bert) approach (roberta) embeddings are extracted and concatenated with sentence level graphical representations. in order to improve the model's performance, it is trained using a combination of contrastive loss and cross-entropy loss. most recently, @xcite proposed an ensemble modeling approach for detecting aigenerated text where the probabilities from various constituent pre-trained llms are concatenated and passed as a feature vector to machine learning classifiers. the ensemble modeling approach resulted in improved predictions compared to what any individual classifier could achieve independently. although the primary purpose of ai-generated text detectors is to mitigate risks associated with harmful ai-generated content, erroneously classifying genuine, human-authored work as ai-generated can, conversely, lead to significant harm. recently, there has been growing apprehension regarding the accuracy and reliability of these generative ai text detectors @xcite @xcite . @xcite highlighted potential bias observed with several gpt detectors. the authors showed that a majority of existing detectors incorrectly classified english writing samples from non-native english speakers as ai-generated. surprisingly, altering language created by non-native speakers with prompts like "enhance it to sound more like that of a native speaker" resulted in a significant drop in misclassification. this highlights that a majority of detectors prioritize low perplexity as a primary criterion for identifying text as ai-generated. since the potential bias in detectors is tied to perplexity scores, the authors propose a more robust and equitable redesign of these detectors. in addition, they propose thorough evaluation of these detectors that takes other important metrics such as bias and fairness into consideration. in this paper, we extend the work of @xcite by proposing an architecture that is simpler in design, while maintaining model performance. we validate the effectiveness of our model by benchmarking it on various publicly available datasets, including the automated text identification (autextification) @xcite dataset. we also study how inclusion of data generated by various llms in the training corpus affects the model performance and generalizability. in order to examine if our approach suffers from similar drawbacks and biases as other perplexity-based approaches, we evaluate the zero-shot performance of our trained model on the aforementioned english essays dataset @xcite and report the corresponding results. in summary, our key contributions in this paper are: 1) we propose a simpler non-perplexity based ai-text detector model that extends prior work 2) we demonstrate the robustness of our approach across multiple benchmark datasets, including the one that examines potential biases in model predictions, 3) we analyze the influence that training data from individual llms have on model performance and 4) we find that excluding gpt data from training sets improves the accuracy of detecting human-authored samples. ## datasets in this section, we provide a brief description of various publicly available benchmark datasets for ai-generated text detection. we also describe a number of datasets that we crafted and used in our experiments (henceforth referred to as curated datasets). ## approach we used an ensemble modeling approach similar to the one proposed by @xcite , where each input is passed through five pre-trained models, namely: 1. decoding-enhanced bert with disentangled attention (deberta) large foot_7 @xcite , 2. cross-lingual language model roberta (xlm-roberta) with cross-lingual natural lan- @xcite , and 5. xlm-roberta nli 13 @xcite . in the training phase, these models are fine-tuned using the training data for each dataset shown in in addition to experimenting with ensembling five models proposed by @xcite , we also conducted experiments with various combinations of these models using the same architecture. we observed that an ensemble of only the roberta base openai detector and the xlm-roberta nli model, along with the voting classifier, delivers performance comparable to the ensemble architecture. henceforth, we refer to this 9 https://huggingface.co/vicgalle/xlm-roberta-large-xnlianli 10 https://huggingface.co/roberta-large 11 a finetuned sequence classifier based on roberta-base (125 million parameters) 12 and roberta-large (356 million parameters) ## experiments in this section, we present an evaluation of our aigenerated text detection experiments. results are presented for multiple models on both benchmark and curated datasets. results from a zero-shot evaluation on the ewessays dataset is also presented. traditional classification metrics, namely, accuracy (acc), macro f1 score (f macro ), precision (p rec), and recall (rec) are reported for each of the experiments. ## conclusion in this research, we proposed a simple yet effective short ensemble model for distinguishing between ai-generated and human written text. we investigated the robustness of our proposed model across various benchmark datasets and observed that our model performs better compared to several stateof-the-art baselines. in addition, we crafted a set of datasets using open llms and examined their impact on model performance. our study shows that fine-tuning models with text generated from open llms performs comparable or better when compared to models fine-tuned on gpt-generated text. furthermore, we investigated the zero-shot generalization capabilities of our model on the ewessays dataset. we observed that our model outperformed several text detection tools in correctly classifying english essays authored by non-native english writers. however, it is important to note that our model with the highest accuracy on ewessays achieved a score of 42.8%, emphasizing the need for ongoing efforts to enhance both generalization and robustness in our approach. we also acknowledge that our approach should further be evaluated on detecting text generated from more advanced llms (llms with more than 175b parameters such as gpt-4 (openai, 2023)).