Simple models are all you need: Ensembling stylometric, part-of-speech, and information-theoretic models for the ALTA 2024 Shared Task (2024)

Abstract: The ALTA 2024 shared task concerned automated detection of AI-generated text. Large language models (LLM) were used to generate hybrid documents, where individual sentences were authored by either humans or a state-of-the-art LLM. Rather than rely on similarly computationally expensive tools like transformer-based methods, we decided to approach this task using only an ensemble of lightweight “traditional” methods that could be trained on a standard desktop machine. Our approach used models based on word counts, stylometric features, readability metrics, part-of-speech tagging, and an information-theoretic entropy estimator to predict authorship. These models, combined with a simple weighting scheme, performed well on a held-out test set, achieving an accuracy of 0.855 and a kappa score of 0.695. Our results show that relatively simple, interpretable models can perform effectively at tasks like authorship prediction, even on short texts, which is important for democratisation of AI as well as future applications in edge computing.

## introduction detecting human-versus ai-generated content is important, for multiple reasons, including misinformation detection @xcite , academic integrity @xcite , even healthcare records @xcite . increasingly, documents are likely to be hybrid-written, with portions of text being aigenerated, and potentially edited or augmented by humans. this introduces the challenge of authorship attribution of short texts such as individual sentences within a longer document, which can confound traditional approaches @xcite . the alta 2024 shared task is squarely focussed on this challenge, presenting a sentencelevel authorship attribution task between humanand ai-generated sentences, where those sentences belong to a longer, hybrid-written document. existing state-of-the-art approaches to this type of task are larger transformer-based, with models like seqxgpt @xcite and segmentationbased approaches @xcite showing strong performance. however, for many of the application domains above, there will likely be a desire to use "traditional" models for reasons of explainability and trustworthiness. also, a current trend in machine learning is towards the use of lower-dimensional models, for reasons of speed, accessibility of data, explainability and ability to run "at the edge" such as on mobile devices. motivated by this, and because we wanted to build on the existing large academic literature on authorship attribution, we opted to use "traditional" models such as those coming from stylometry, linguistics, and information theory. in order to experiment with a number of methods, we developed an ensemble approach comprising five such models. this ensemble model performed reasonably well on the held-out test set, with an accuracy of 0.855 and kappa score of 0.695. we hope our results demonstrate that relatively simple, interpretable models can perform well at distinguishing ai-generated from human-generated text, and that these models can still have relevance in a variety of application domains requiring explainable models. ## data the full details of the shared task description can be found in @xcite . the task consisted of a training phase (phase-1) where models could be trained on a training set and tuned/tested on a development set via multiple submissions, and then a testing phase (phase-2) where a final model was assessed on an unseen held-out test dataset. our training dataset comprised 212794 data points. features included the id (article id), 'domain' (the domain the article belongs to, such as news, academic, etc.), the sentence to make predictions on, and the true label of the sentence. the training dataset was class-imbalanced, with around two-thirds of its data points belonging to the 'machine' class and one-third to the 'human' class. ## methods our approach uses an ensemble of five separate models, the predictions of which are combined together to make an overall prediction. ## results a similar trend is observed in plots between other principal components as can be seen in fig -ure 2 where we see that the machine data points seem to be concentrated in certain regions whereas the human data points expand out a bit more than the other class. this might suggest that the human style of writing can have more variability compared to that of ai. ## discussion our system was relatively simple, and therefore unlikely to ever achieve the highest scores in this shared task. nonetheless, we think it performed very well, and demonstrates that simple models based on traditional methods can still be effective at distinguishing between human-and ai-generated text. how long this remains the case as generative large language models increase in sophistication remains an open question, however. our approach had a number of limitations, which area left as future work. firstly, we didn't consider the article structure, instead treating each individual sentence independently. this was partly in the interests of time, and because some methods used were less amenable to incorporating hierarchical structure than others. hierarchical document structure could be incorporated in some methods, for example the naive bayes model @xcite . we also did not always consider the domain of the document in the classification, for example in the information-theoretic model. this could be incorporated by splitting the documents in h a based on domain, which might lead to an improvement in classification performance. finally, we could consider each model's prediction confidence as part of the ensembling method. in the methods deployed here we only used the binary outcome predictions from each model as inputs to the ensembling method. however, incorporating a measure of the confidence of each model as inputs into the ensembling procedure is a more principled approach and has potential to improve the predictions, particularly in borderline cases where there might be disagreement between models. this would be straightforward to do for e.g., naive bayes which produces probabilities as predictions, but would require the development of some heuristics for other methods, e.g., potentially using the difference in cross-parsed-entropy rates as a measure of prediction confidence for the information-theoretic model.