An Ensemble Based Approach To Detecting LLM -Generated Texts (2023)

Abstract: Recent advancements in Large Language models (LLMs) have empowered them to achieve text generation capabilities on par with those of humans. These recent advances paired with the wide availability of those models have made Large Language models adaptable in many domains, from scientific writing to story generation along with many others. This recent rise has made it crucial to develop systems to discriminate between human-authored and synthetic text generated by Large Language models (LLMs). Our proposed system for the ALTA shared task, based on ensembling a number of language models, claimed first place on the development set with an accuracy of 99.35% and third place on the test set with an accuracy of 98.35%.

## introduction in the realm of human-computer interactions, the recent advancements in ai-generated texts are hallmarked by the introduction of large language models (llms), such as gpt4 (openai, 2023), gpt3 @xcite , t5 @xcite , llama @xcite and much more. this has resulted in ai's ability to generate text of high quality and fluency comparable to that of humans. these language models have had widespread integration and adaptations across many different fields including but not limited to, law, medicine and education. nonetheless, similar to any revolutionary technology, llms possess both positive and negative aspects for our society. apart from spreading misleading information, the potential misuse of llms could lead to numerous social and ethical challenges, such as academic misconduct @xcite and spread of misinformation @xcite . the recent growth in adaption of large language models in many domains and their unprecedented ability to generate high quality fluent text similar to that of humans have caught researchers' attention. this lead to the development of systems with the goal of being able to differentiate between human-generated texts and machine-generated ones. those systems vary according to their scope of operation, ranging from domain specific ones that detect deep fakes based on specific models to more generalized ones, yet there have been efforts to build a unified model able to operate on different domains and generalize to novel llms despite not being trained on their respective data. large language models are expected to fundamentally change many aspects of life and with the trend in the number of large language models introduced each year @xcite , the challenges of detecting text generated by large language models are expected to reach new heights in the upcoming years. the alta 2023 shared task @xcite focuses on this important topic, offering a dataset for evaluation and training. the dataset addresses several issues and supports the creation of a single, readily generalizable model. our proposed model uses an ensemble-based approach paired with fine-tuning a number of language models. the structure of this research paper will unfold as follows: the related work section will provide an overview of various solutions explored by different researchers in the context of this problem. subsequently, the data section will detail the properties of the provided dataset and any preprocessing steps undertaken. in the system description section, we will go through the architecture of our proposed model. the results section will then offer a detailed analysis of the outcomes generated by the proposed system, complemented by a comprehensive evaluation of the model's overall performance. finally, the summary section will synthesize the paper's content, briefly touch on potential future research directions, and consider possible improvements to the model. ## related work we will touch on the most recent developments in identifying data produced by large language models in the section that follows. because of the widespread use of llms and their possible drawbacks, academics have been particularly interested in this area in recent years. many researchers have proposed systems that use both deep learning techniques and traditional machine learning models. one interesting approach was when @xcite built a logistic regression based detector which made use of tf-idf unigram and bigram features. the model was trained on gpt-2 outputs and webtext samples and yielded an accuracy up to 97% at 124 million parameters and up to 93% at 1.5 billion parameters. @xcite experimented with a number of conventional machine learning approaches, mainly support vector machines, random forests and logistic regression. in the realm of deep learning, many models were proposed to tackle the problem of aigenerated text, yet many of the proposed systems either focused on specific domains, or they were model specific @xcite . one interesting system was proposed by @xcite which consisted of training 3 detection models; a language model based on longformer @xcite , fasttext @xcite and gltr @xcite ) and testing the model on multiple settings to ensure its success ranging from domain-specific & model-specific to unseen domains and unseen models. many studies have also shown that text written by llm is more objective and less emotional than human-generated text @xcite . another factor has to do with the fact that llms have a condition called hallucinations, which results from the generation of material that is nonsensical or inconsistent @xcite . something that makes it possible to apply fact-verification procedures. a different strategy is known as "white box detection," where the detector can monitor any unauthorized or suspicious behavior by inserting hidden watermarks into its outputs and having complete access to the target language model @xcite . ## data the dataset used is the dataset provided in the alta 2023 shared task. below is an illustration of the dataset distribution. the dataset is derived from a number of sources, including several llms dataset train dev test texts 18000 2000 2000 ## system description in the subsequent section, we will outline our experimentation on the dataset, highlighting the key stages involved in the development of the previously mentioned system. ## results our ensembled models performed the best of the suggested systems, placing first on the test set and third on the development set, suggesting some progress on ensembling multiple learners. due to computational power constraints, we initially conducted our experiments using roberta-base. however, after experimenting with roberta-large, we discovered that when three roberta-large models were ensembled, they outperformed our top ranking system on the test set. ## discussion the results of these experiments showed that an ensembling-based approach is worth further exploring in the pursuit of a generalized model for classi-fying synthetic text data generated by llms. some potential further improvements include adding larger models to the ensemble, such as xlm-roberta-xl. other improvements include supplementing the development data set with more training data such as the one used in @xcite . another approach could be to further tune the hyperparameters of the individual members of the ensemble, which could lead to marginal improvements in the overall performance of the ensemble. overall, the system has promising implications and, with more research, could prove very fruitful in combating the spread of fake data in the modern world. addressing this problem is a very pressing matter as this spread of fake synthetic text data could spread far and wide and have catastrophic effects on the journalism industry,the education industry, along with several other industries. ## summary the presented system, utilizing an ensemble approach through hard voting, was thoroughly described in this study. the conducted experiments were comprehensively addressed. incorporating pretrained language models, along with ensembling, effectively addresses the challenge of identifying text generated by extensive language models, though there remains room for enhancement. our forthcoming research will concentrate on evaluating our model in analogous settings, utilizing data generated by recently developed large language models across diverse domains to assess its performance. another compelling avenue for future investigation involves conducting additional experiments with larger language models, particularly emphasizing the adaptation of roberta-large to our specific problem. this aspect warrants further exploration in subsequent research endeavors.