S em E val Task 8: A Comparison of Traditional and Neural Models for Detecting Machine Authored Text (2024)

Abstract: Since Large Language Models have reached a stage where it is becoming more and more difficult to distinguish between human and machine written text, there is an increasing need for automated systems to distinguish between them. As part of SemEval Task 8, Subtask A: Binary Human-Written vs. Machine-Generated Text Classification, we explore a variety of machine learning classifiers, from traditional statistical methods, such as Naïve Bayes and Decision Trees, to fine-tuned transformer models, suchas RoBERTa and ALBERT. Our findings show that using a fine-tuned RoBERTa model with optimizedhyperparameters yields the best accuracy. However, the improvement does not translate to the test set because of the differences in distribution in the development and test sets.

## introduction large language models (llms) are becoming more and more accessible, which has resulted in an increase in machine-generated content across a wide variety of domains, including education, technology, and science. with this increase in machine generated texts from llms, and with the increase in the quality of llm created texts, concerns regarding but not limited to fake product review generation @xcite spam/phishing @xcite and fake news generation @xcite @xcite have arisen. @xcite demonstrated that humans can only detect such misuses of llms at chance level, which demonstrates the clear need for automated systems to detect machine generated content. in this paper, we describe the iucl submission to semeval task 8 @xcite ; we focused mostly on comparing traditional and neural models. our best system ranked 70th out of 137 submissions. ## related work in terms of impressionistic differences between human generated text and llm generated text, it has been observed that llms tend to be more focused (i.e. less diversion from the subject at hand), more objective, and highly formal. human texts, on the other hand, are overall more emotional, subjective, and less formal. in terms of linguistic difference, humans use fewer nouns and conjunctions, while employing more punctuation and adverbs. dependency relations are also shown to be shorter. lastly, human texts have higher type/token ratios in texts of the same length @xcite current llm models include gpt-2 @xcite , gpt-3 @xcite , ctrl @xcite and chatgpt. we will first begin by discussing statistical approaches to detecting machine-generated content, then using llm technology itself to do so. @xcite use a bag-of-words approach with tf-idf feature vectors (both unigrams and bigrams) and a logistic regression model to differentiate between human-written web pages and text generated web pages from gpt2. they examine a different number of parameters of the llm (117m, 345m, 762m and 1,542m) as well as different sampling methods (k-sampling, p-sampling and pure sampling). this is because an assumption that many researchers take is that language models sample from the head to generate natural looking text e.g. max sampling @xcite and kmax sampling @xcite . their findings are that the larger the llm, the harder to detect how machine-like the generated text is and k samples are easier to detect than pure samples, probably due to the fact that k samples over-produce common words, which is easy to detect using statistical methods. @xcite use bert and a group of statistical features: the probability of each word, ab-solute rank of each word, and entropy of the distribution, and create a tool for users to see specifically what features are more likely to be machine generated over human generated. they clearly show that the model gpt-2 oversamples certain words; it is worth pointing out, however, that as llms grow more sophisticated, such methods may not work as well. @xcite use finetuning on roberta and find that it can detect text generated from gpt-2 with an accuracy of 95%. the roberta detector has also been used in detecting fake news articles from several llms @xcite , amazon product reviews @xcite , and biomedical texts @xcite . ## data we used the m4 dataset @xcite provided by the semeval-2024 task 8: multigenerator, multidomain, and multilingual black-box machine-generated text detection. we used the english data provided for subtask a, the monolingual (english) binary classification task. the dataset for this subtask consists of 119,757 samples of human-written and machine generated text. there are an additional 5,000 samples as a development set. the test set consists of 34,272 samples. about 53% of the samples in the training set are machine generated while the rest are human written. the machine generated text was produced by a range of models: chatgpt and davinci by openai, dolly by databricks, cohere. the sources from which the human texts are taken are reddit, wikihow, arxiv, wikipedia and peerread. in contrast, the development set consists of an equal ratio of human and machine generated samples. the machine generated samples are entirely from the bloomz model. the human sources are also equally distributed between wikihow, wikipedia, reddit, arxiv and peerread. in the test set, 52.5% of the texts are machine generated with gpt4, cohere, chatgpt (gpt3.5), bloomz, dolly, and davinci as sources. note that this means optimizing a system on development data is difficult since the test data are much closer to the training data than the development data. further details about the data and the task are available at the overview of the shared task @xcite . we present a comparison of a range of classifiers (see below). for those experiments, we use the development set of 5,000 samples for benchmarking and finetuning the model performance. ## conclusion and future work in this project, we have investigated the performance of various machine learning models. we found that our best performing model is a base xlm-roberta model that is fine-tuned on the dataset. using the smaller albert or the large xlm-roberta models resulted in decreases in accuracy. however, we also see that finetuning is very sensitive to underlying data characteristics, since the gains we saw on the development set did not translate to equivalent gains on the test set. there is a significant scope for improvement in the performance of the models by working on further text preprocessing and feature engineering. future work includes using ensemble methods that combines the finetuned models along with a model using ratio features.