T ue CICL at S em E val-2024 Task 8: Resource-efficient approaches for machine-generated text detection (2024)

Abstract: Recent developments in the field of NLP have brought large language models (LLMs) to the forefront of both public and research attention. As the use of language generation technologies becomes more widespread, the problem arises of determining whether a given text is machine generated or not. Task 8 at SemEval 2024 consists of a shared task with this exact objective. Our approach aims at developing models and strategies that strike a good balance between performance and model size. We show that it is possible to compete with large transformer-based solutions with smaller systems.

## introduction recent developments in the field of nlp have brought large language models (llms) to the forefront of both public and research attention. with the introduction of @xcite , made accessible to the public through chatgpt, the gates were open to the generation of high-quality text through ai. this leads to a sort of arms race both in integrating ai into customer-facing products, as well as in developing the models themselves, leading, among others, to facebook's llama @xcite and the open source model mistral @xcite . many fields have seen a dramatic increase in the use of llms, including arts, education, software development, and many more. initial public reaction to the popularisation of llm-powered chat interfaces already highlighted its potential problems for plagiarism detection in scientific, educational and other contexts @xcite . in a landmark development, the 5-month-long strike of the writer's guild of america 2 resulted in an agree-1 https://github.com/cicl-iscl/tuecicl_ semeval2024 2 https://www.vox.com/culture/2023/9/24/ 23888673/wga-strike-end-sag-aftra-contract ment which included safeguards for writers against the use of artificial intelligence. 3 the widespread use of language generation technologies is only expected to grow. however, this development is accompanied by a surging need to automate the process of flagging machine-generated text. crothers et al., 2023 offers a comprehensive survey of machine-generated text detection strategies. statistical methods, with global text vector representations, were among the early strategies adopted to tackle the issue. these feature vectors include, for example, tf-idf, frequency features investigating word or n-gram distributions, readability-related features such as the gunning-fog index, or linguistic features such as pos-tag distributions and coreference resolution relationships within a text. the subsequently introduced neural approaches show better performance, even when paired with the aforementioned text representation strategies. most prominently, transformer fine-tuning has established itself as more or less of a standard, with roberta @xcite in particular being the most strongly represented model. zero-shot approaches, optionally coupled with fine-tuning, have also seen experimentation, but have been observed to generalise poorly across domains. task 8 at semeval-2024 @xcite ) is a shared task built around the idea of detecting machine-generated texts across a variety of domains and setups. it is structured across three subtasks (subtask a, subtask b, subtask c), our team decided to only tackle subtasks a and c. subtask a is a binary classification task, with the objective of determining whether a text is human-or machine-generated. the subtask has a monolingual (english) and a multilingual track -our team only submitted for the monolingual track. subtask c is a boundary detection task: here, texts have a human segment followed by a machine-generated segment. the objective of the subtask is to correctly predict the boundary index. our approach for both tasks was to try to obtain competitive solutions with low resource cost. for this reason we deployed two different model classes: lstm-based models @xcite and multilayer perceptrons (mlp) @xcite and relied on various strategies of representing the input texts, either at the token or at the text level. we experimented with character-level approaches, systems relying on pretrained word2vec @xcite embeddings, and linguistically motivated features at the text level through spacy @xcite and the textdescriptives @xcite package. ## conclusion and discussion our objective for task 8 at semeval-2024 was to compete with large-scale solutions with models that could run on commonplace systems like midrange laptops. for this purpose, we discarded llmbased solutions that are prevalent in the related work in the field, opting instead for lstms and mlps whose size can more easily be controlled. while our team did manage to keep model size under control (none of the proposed solutions require more than 1 gb of memory), the systems we proposed performed less than ideally in the task itself. on both subtasks we participated in, our best models failed to match the transformer baseline in the test set, despite positive results in the development set. despite the final results, we believe our approach to be valid. our development processes likely ended up producing models that were overly tuned to the development set. with more time, it would be possible to produce more refined and generalizable solutions. trying different training strategies, like contrastive learning, or different architectures, such as mixture-of-experts systems, might be a good direction to follow in future work. to most problems that arise in the field of nlp, researchers and companies increasingly respond with huge models that require dedicated servers to run. but for many users, keeping their data safely on their own machines is a priority, thus discarding many of the contemporary llm-based services. system designers should aim to strike a compromise between size and performance, and prioritise users being able to own their workflows when possible. like our attempt did for this shared task, we believe researches should consider these objectives when proposing new solutions.