CUNLP at S em E val-2024 Task 8: Classify Human and AI Generated Text (2024)

Abstract: This task is a sub-part of SemEval-2024 competition which aims to classify AI vs Human Generated Text. In this paper we have experimented on an approach to automatically classify an artificially generated text and a human written text. With the advent of generative models like GPT-3.5 and GPT-4 it has become increasingly necessary to classify between the two texts due to various applications like detecting plagiarism and in tasks like fake news detection that can heavily impact real world problems, for instance stock manipulation through AI generated news articles. To achieve this, we start by using some basic models like Logistic Regression and move our way up to more complex models like transformers and GPTs for classification. This is a binary classification task where the label 1 represents AI generated text and 0 represents human generated text. The dataset was given in JSON style format which was converted to comma separated file (CSV) for better processing using the pandas library in Python as CSV files provides more readability than JSON format files. Approaches like Bagging Classifier and Voting classifier were also used.

## introduction we perform subtask a of the task 8 @xcite from the international workshop on semantic evaluation: semeval 2024 † which stated -multidomain, multimodal and multilingual machine-generated text detection. in this subtask we perform monolingual (english in this case) classification for ai generated vs human written texts. this binary classification task has utmost utility in real world scenarios like -content moderation on social media platforms, fake news detection that can impact organizations financially and people emotionally, detecting spam messages in email or communication channels like slack. to perform this task, we use a series of techniques including manual feature engineering for supervised learning techniques like logistic regression and bagging classifier as well as more complex techniques like neural networks and attention mechanism with transformers. we used supervised learning as well like k-nearest neighbours. the best approach found was a combination of transformers @xcite with hand engineered features like coherence @xcite of a text, complexity, length and emoji count. the accuracy and performance of these experiments are discussed in the later sections. in our experiments we found that some features were very influential like length of a text, vocabulary used in the text and coherence of a text. other features like complexity of the text had less weightage and were thus, not used in all experiments. even though transformers gave us the best accuracy we also used some other approaches that were competitive as well. we also had some limitations in the usage of computing resources where one of our approaches that combines tf-idf vector along with transformers uses over 50 gb of ram that exceeds the amount of any available computing resource available to us. ## background dataset -the dataset that was used was provided by semeval that is an extension of the m4 dataset @xcite . which had approximately 133551 data points in the training set and the dev set contained 5000 samples. the dataset contained texts from various sources including wikipedia, reddit, wikihow, and peerread for english texts. the ai generated text was curated from generative models like chatgpt, cohere, dolly v-2 and bloomz. after analysing the data, we found that the dev set data only included the data points from bloomz and there were none of bloomz model's generated texts in the training set. this was meant to test the real-life situation where a new generative model can come into picture when our model would not have seen that generative model's pattern. an exploratory data analysis of the text, gave the following interesting observations: we also explored some aspects of sentence structure like coherence, complexity and length of the sentences. we used these features along with the tf-idf vectors as an input to the logistic regression model, bagging classifier and k-nearest neighbours achieving a max training set accuracy score of 0.91 and 0.61 on dev set using bagging classifier. we also used a voting classifier which performed better, achieving an accuracy score of 0.68 on the dev set using the above-mentioned models. later we used transformers (bert) with combination of the above-mentioned models in the voting classifier. there were two more approaches where we tried topic modelling and feature repetition which yielded better results. one thing to note here is that when a certain text is generated by ai it contains some sort of template or pattern around it. so, to use that we tried unsupervised learning to make possible clusters of the texts, to identify which class of template the text might belong to. this approach included the use of the k-means clustering method, which reported a dev set accuracy of 0.57 heather et al. @xcite mentions the use of simple machine learning techniques with great accuracy. ahmed et al. @xcite compared different methodologies and tools and how each of them perform on unseen data. in any of the literature tf-idf was not used along with any other features, and we experimented by including these features in our approach along with topic modelling setup that was novel. ## system overview text classification even though an already accomplished task becomes challenging even for state-of-the-art models like transformers. in this task the adaptability of gpt makes it even more challenging to differentiate between the two types of texts. also, as ai progresses to understand human emotions @xcite and behaviour it is expected from the model to generate texts i.e. convey its thought in a more human centric manner. we aim to tackle the same starting with the standard machine learning algorithms and then moving on to much more complex models like attention based transformer models, example -bert @xcite , roberta @xcite among others. we describe below in detail the specification used along with each approach and mention its accuracy and experimental setup. for this task, we have used tf-idf vectorization technique. along with that we also analysed text structures and engineered 3 main features related to the task at hand. these were complexity of the sentence, coherence of a text and length of text (tokenization). these features were used by the algorithms described below and are described in the next section in detail. ## experimental setup various experiments were performed on the given dataset. the train-test split for all the experiments was kept the same to the ratio of 80:20. this split comes from the training data itself and the dev set was kept unseen from the model during the training phase. the best results on the dev set after hyperparameter tuning are logged in the results section of the paper. in this section we discuss the following: ## results we observed that the model combined with the attention mechanism of transformers with tf-idf vectors provides is with the best results. however, it should be noted that the dimensionality of the vectors has been significantly reduced due to its computational complexity and thus is bound to affect the accuracy. the results mentioned in the below table ( ## conclusion this binary classification task of predicting the mode of text generation is non-trivial in the aspect that as the generative models are largely trained on human generated text, they have learned to write more like humans and thus this becomes a challenging task. however, using proper means and computational methods, it is possible to segregate them using conventional feature extraction techniques combined with self-attention mechanism of transformers as seen in the experiments. we aim to use the topic modelling approach combined with tf-idf and transformers further in the future that might yield promising results.