An Ensemble Method Based on the Combination of Transformers with Convolutional Neural Networks to Detect Artificially Generated Text (2023)

Abstract: Thanks to the state-of-the-art Large Language Models (LLMs), language generation has reached outstanding levels. These models are capable of generating high quality content, thus making it a challenging task to detect generated text from human-written content. Despite the advantages provided by Natural Language Generation, the inability to distinguish automatically generated text can raise ethical concerns in terms of authenticity. Consequently, it is important to design and develop methodologies to detect artificial content. In our work, we present some classification models constructed by ensembling transformer models such as Sci-BERT, DeBERTa and XLNet, with Convolutional Neural Networks (CNNs). Our experiments demonstrate that the considered ensemble architectures surpass the performance of the individual transformer models for classification. Furthermore, the proposed SciBERT-CNN ensemble model produced an F1-score of 98.36% on the ALTA shared task 2023 data.

## introduction nowadays, people have access to state-of-the-art llms which help them simplify some of their daily activities. one of the most notable breakthroughs in recent years is the evolution of openai's gpt models which are capable of generating text that looks as if they are written by a human. especially, the latest models such as chatgpt and gpt4 (ope-nai, 2023) have won global attention for providing solutions to any kind of question or concern that humans possess. moreover, these models produce outputs that appear to be written by a human. thus there is a potential risk in determining the authenticity of textual content that mankind refers to. especially, in a domain such as academia, leveraging generation models in composing articles might raise an ethical concern. for example in icml 2023, they have included a note under the "ethics" section prohibiting the use of text gen-erated by chatgpt and other llms, unless "presented as part of the paper's experiential analysis." foot_0 . accordingly, it is essential to have mechanisms for detecting artificially composed text from human written text. currently, a substantial amount of research has focused on the detection of automatically generated text. recent research ( @xcite , @xcite and @xcite ) mostly consider detection as a binary classification task and leverage sota classification models to distinguish machine-generated text from original text. besides, some employ statistical detection tools such as gltr @xcite or latest deep learning based tools such as gpt2 output detector foot_1 , detectgpt @xcite or gptzero foot_2 . moreover, several researchers @xcite , @xcite ) have published corpora composed of machine-generated content, which can be utilized by future research on detection. our work is based on the participation of our team in the alta shared task 2023 @xcite the objective of the task is to build automatic detection systems that can discriminate between human-authored and synthetic text generated by large language models (llms). their corpus is composed of artificial contents that belong to a variety of domains (such as law, medical) and are generated by models such as t5 @xcite and gpt-x. this paper is organized as follows. we provide the corpus and task description in section 2. in section 3, we describe our methodology and section 4, deliver the experimental setup and the official results. section 5 concludes this paper. ## methodology given that the shared task frames detection as a binary classification challenge, we utilized a range of classification models to address this objective. in the subsequent subsections, in-depth explanations are provided pertaining to the examined statistical, recurrent and transformer models, and the corresponding ensemble architectures. ## experiments and results the text underwent preliminary processing, involving the elimination of stopwords and stemming, before being supplied to either statistical or neural network architectures. the processed data was then transformed into numerical vectors using bag of words (bow) or tf-idf encoding techniques, which were subsequently utilized as inputs for the statistical models. all of the employed statistical models, as well as their corresponding ensemble methods, were imported from the scikit-learn library. for constructing lstm and cnn models, the relevant layers were imported from tensorflow's keras module. training these recurrent models, including those combined with cnn ensembles, involved running 10 epochs. the lstm and bi-lstm architectures were trained using batch sizes of 64 and 128, respectively. concerning transformer architectures and their associated ensembles, pre-trained models from hugging face @xcite were imported and subsequently fine-tuned through the utilization of simple transformers foot_3 . the bert tokenizer was consistently employed across all models. the fine-tuning process involved 3 epochs, a batch size of 16, and a maximum sequence length of 128. leveraging the t4 gpu hardware accelerator, the average training time for models was approximately 30 minutes. for standalone models, the input consisted of unprocessed text, while ensembles underwent pre-processing involving punctuation removal and conversion to lowercase. as represented in figure 1 , the cnn block of the ensembles was composed of three convolutional layers. the dataset was split in 80:20 ratio for training and testing. to assess the classification performance of the models under consideration, the f1 score was employed. this score, being a balanced combination of precision and recall, offers a comprehensive evaluation. each model underwent a total of five experimental iterations, and the resultant average f1 scores are presented in in general, the ensemble architectures have exhibited superior performance compared to their corresponding original models. our best-performing solution is the combination of deberta large with cnn, achieving an f1 score of 98.36%. considering that baseline models such as naïve bayes and tf.idf weighting obtain scores close to 90%, it is clear that the dataset is not well balanced. in fact, looking at the multinomial naïve bayes and the log probabilities differences for all features, we observed a thematic bias. specifically, the top most probable words in the negative category (human-generated) are law-oriented: "plaintiff", "defendant", and "judgment". on the other hand, llm-generated text contains words like "round", "league", "players", etc. therefore, it is not clear whether these results are generalizable to the general task of detecting artificial text. ## conclusion in this work, we have explored the application of different sota classification models on the detection of automatically generated text from human written text. moreover, we have created various ensemble methods with the aforementioned models and examined their performance on the detection model f1 statistical models nb + bow 89.04 pa + bow 84.07 svm + bow 87.51 nb + tf-idf 89.02 nb + tf-idf 91.00 nb + tf-idf 91.42 ensembles of statistical models voting (nb + pa + svm) + bow 90.29 stacking (nb + pa + svm) + bow 88.23 bagging (nb + pa + svm) + bow 91.56 boosting (nb + pa + svm) + bow 90.28 recurrent models lstm 49.08 bi-lstm 90.58 ensembles of rnns lstm + cnn 49.08 bi-lstm + cnn 90.02 transformer models bert base 90.81 scibert 94.89 deberta large 96.67 xlnet large 93.62 ensembles of bert models bert base + scibert 97.80 bert base + deberta large 97.47 ensembles of transformers with cnn bert base + cnn 97.42 scibert + cnn 97.56 deberta large + cnn 98.36 xlnet base + cnn 97.44 our results on the test data showed that generally the ensemble architectures outperform the considered original models. however, an analysis of the dataset raises some doubts about the generalizability of these results as it looks like the data are thematically biased. therefore, these results should be considered only within the scope of the alta 2023 shared task. as future work, we plan to examine the applicability of our ensemble architectures in detecting artificially generated text in multilingual corpora. another potential research direction involves assessing the effectiveness of knowledge-based approaches for detecting artificial text.