Team U nibuc - NLP at S em E val-2024 Task 8: Transformer and Hybrid Deep Learning Based Models for Machine-Generated Text Detection (2024)

Abstract: This paper describes the approach of the UniBuc - NLP team in tackling the SemEval 2024 Task 8: Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection. We explored transformer-based and hybrid deep learning architectures. For subtask B, our transformer-based model achieved a strong second-place out of 77 teams with an accuracy of 86.95%, demonstrating the architecture’s suitability for this task. However, our models showed overfitting in subtask A which could potentially be fixed with less fine-tunning and increasing maximum sequence length. For subtask C (token-level classification), our hybrid model overfit during training, hindering its ability to detect transitions between human and machine-generated text.

## introduction task 8 from semeval 2024 competition @xcite aims to tackle the complex challenge of distinguishing between human and ai generated text. doing so is crucial for maintaining the integrity and authenticity of information as it helps prevent the spread of misinformation and ensures that content sources are accountable. by developing tools for this task, which work in a multilingual setting, and releasing them open source we can combat non-ethical uses of ai such as propaganda, misinformation, deepfakes, social manipulation and others. the systems developed for subtasks a and b are based on transformer models with different layers selection and merging strategies, followed by a set of fully connected layers. the training is split in two phases: a) freezing phase, where the transformer weights are not updated, only the fully connected layers are updated with a specific learning * equal contributors rate; b) fine-tuning phase, where the selected layers of the transformer and the fully connected layers are updated with a different (usually smaller) learning rate. for the subtask c, a different architecture was used, combining character level features, extracted with a cnn model, with word embeddings and fed into a bidirectional lstm followed by a set of fully connected layers. the same training strategy with different learning rates was used. our error analysis revealed that overfitting remains a primary challenge, despite our initial precautions. we learned that for future fine-tuning of transformer models, we should dedicate a lot more time to prevent overfitting. we made our models open source in a github repository. ## background the competition had 3 tasks explained below (figure 1 ). subtask a had 2 sub-tracks: monolingual (english only) and multilingual. we participated in all 3 tasks with the best result being second place on subtask b ( ## system overview in this paper, we focused our research on two different system architectures: transformer based models (3.1) and hybrid deep learning models (3.2). both architectures use a block of fully connected layers (figure 2 ) with the base structure being initiated with a linear layer, succeeded by normalization, a tanh activation function, followed by a dropout layer (0.5). finally, it concludes with a linear layer with an output size of 1 for subtask a and 6 for subtask b . ## conclusions and future work in conclusion, our architecture and training methods produced good results for subtask b (securing the second place). however, our models demonstrated signs of overfitting for subtask a. we could not find a proper explanation for why the model architecture work better on subtask b and is overfitting that much on the other task. our future endeavors will explore several avenues: