G roningen Team A at S em E val-2024 Task 8: Human/Machine Authorship Attribution Using a Combination of Probabilistic and Linguistic Features (2024)

Abstract: Our approach primarily centers on feature-based systems, where a diverse array of features pertinent to the text’s linguistic attributes is extracted. Alongside those, we incorporate token-level probabilistic features which are fed into a Bidirectional Long Short-Term Memory (BiLSTM) model. Both resulting feature arrays are concatenated and fed into our final prediction model. Our method under-performed compared to the baseline, despite the fact that previous attempts by others have successfully used linguistic features for the purpose of discerning machine-generated text. We conclude that our examined subset of linguistically motivated features alongside probabilistic features was not able to contribute almost any performance at all to a hybrid classifier of human and machine texts.

## introduction large language models capable of generating human-like text have become quite ubiquitous very quickly. there are now many such models which are commonly used to generate text across different domains and in different languages. with their increasing availability and capabilities, it has subsequently become necessary to find ways to distinguish machine-generated text from that which is produced by humans. humans alone are not able to detect machine-generated text consistently, not even experts in this task @xcite , and current commercial solutions fall short @xcite . it is natural then that this problem has seen wide 1 https://github.com/rug-1-at-semeval24-task8/ code discussion and participation over several domains and languages, including the creation of datasets and proposal of different feature sets and model types @xcite to name a few), but it is still far from being solved. this leads us to the semeval-2024 shared task 8 that this paper is concerned with "multidomain, multimodal and multilingual machine-generated text detection" @xcite . this task is about distinguishing human-written text from machine-generated text in multiple different domains, modalities and across different languages. the languages included in the task are: arabic, bulgarian, chinese, english, indonesian, russian and urdu. the domains are varied and range from wikipedia pages to arxiv research papers to reddit posts. ## related work due to the similarities in the architecture and training of different text-generation machines, generated text may possess universal characteristics that distinguish it from text written by humans. @xcite set up a series of human evaluation and linguistics analyses to understand the characteristic features and patterns, where a study by @xcite looked at the differences in human vs ai-generated text. the studies found that humans tend to have much more diverse and expressive vocabulary, and often tend to diverge from the topic more than chatgpt does @xcite . this idea is supported by @xcite , who performed a probabilistic analysis of the vocabulary in human-and machine-generated texts and found that generation models tend to have a relatively limited and pre-dictable vocabulary. some work focuses on stylometric features, as these may be productive in discerning the original author of a text @xcite . @xcite show that models based on such feature sets perform strongly within the domain, but the choice of training dataset may have a notable effect on performance. feature-based detectors work fairly well for simple binary classifications in a single domain, but tend to fall short when attempting more complex problems which consist of additional styles and sources of texts @xcite , where shorter texts can have a negative impact on performance @xcite . conversely, language models may prove to be the optimal tool for detecting machine-generated text. recent attempts mostly use (ro)bert(a)-based models @xcite that are pre-trained for language understanding, and fine-tune them using datasets of human-and machine-generated text @xcite @xcite . these models are then able to detect authorship with varying levels of success. much of the focus in this area has been on developing useful datasets for fine-tuning and finding optimal models and methods of fine-tuning. an lstm, as introduced in @xcite , is a version of a rnn (recurrent neural network) that utilizes long term short memory to deal with issues present in regular rnns caused by larger gap lengths, which can be especially relevant in nlp tasks such as ours. lstms have been used with success to perform authorship attribution @xcite which suggests they may be useful in distinguishing human and machine authors as well. the semeval-2024 shared task 8 revolved around distinguishing human-written texts and machinegenerated texts. it was divided into multiple subtasks. the goal of subtask a was to perform binary classification on a given text to determine whether it is human-written or machine-generated. the monolingual track of this subtask only included text in english, whereas the multilingual track included text in english, russian, chinese, arabic, urdu, indonesian and bulgarian. subtask b focused on multi-way machine-generated text, where the goal of the task was to determine whether a given text is written by a human or generated by a machine, and if generated by a machine -which specific language model was it that generated the text? for all subtasks, we used the datasets provided by the task organizers. these datasets are an extension of the m4 dataset from @xcite . the datasets include texts from multiple domains, such as reddit discussions, wikipedia pages and arxiv papers to name a few, as well as multiple languages as stated above. in addition, the dataset for subtask b contains machine-generated texts from multiple models. for more information about the shared task, see @xcite . ## system overview the basic components of our design consist of both document-level and token-level features. document-level features (detailed in section 4.1) are extracted directly from the text, and the output features of document-level features are concatenated for further use in an mlp for classification. token-level features, i.e., the measure of predictability, are the probability of the input text according to a large language model, are fed into a bilstm network which converts sequences into a fixed-length representation by concatenating both directions; the details of token-level features are outlined in 4.2. document-level and token-level features are concatenated and then passed to an mlp for classification. this design remains consistent for both subtask a and subtask b, differing only in the dimensionality of the mlp output representation, which requires adjustments to the number of output classes. ## experimental setup we initially divided the training data into two subsets: training and development. we utilized the development set for testing purposes during the model development phase. however, following the availability of the test data, we incorporated the entire training set for training and the development set for validation. we then evaluated the model's performance on the raw documents in the publicly available test set. for preprocessing, we employed the xlm-roberta tokenizer for features related to perplexity and predictability. additionally, we utilized the stanza @xcite tokenization pipeline for features such as stylometric analysis, entity coherence, information redundancy, and partof-speech (pos) tagging. our model was implemented using pytorch. we employed accuracy as the official evaluation measure. ## conclusion our overall conclusion is that our examined subset of linguistically motivated features alongside probabilistic features was not able to contribute almost any performance at all to a classifier of human and machine texts. while some features did differ in value between human and machine texts, these differences did not translate into a learning advantage for a hybrid model. our findings underscore the nuanced challenges inherent in developing robust detection mechanisms for machine-generated text, emphasizing the need for further exploration and refinement of feature engineering strategies to effectively address this evolving domain.