Multi-Loss Fusion: Angular and Contrastive Integration for Machine-Generated Text Detection (2024)

Abstract: Modern natural language generation (NLG) systems have led to the development of synthetic human-like open-ended texts, posing concerns as to who the original author of a text is. To address such concerns, we introduce DeB-Ang: the utilisation of a custom DeBERTa model with angular loss and contrastive loss functions for effective class separation in neural text classification tasks. We expand the application of this model on binary machine-generated text detection and multi-class neural authorship attribution. We demonstrate improved performance on many benchmark datasets whereby the accuracy for machine-generated text detection was increased by as much as 38.04% across all datasets.

## introduction there has been considerable activity in the field of detecting machine-generated text. driven by the significant growth and the increasing prevalence of large language models (llms) and natural language generation (nlg) models. this has led to the production of high-quality human-like texts that have brought about useful applications in many domains such as machine translation, text summarisation and data generation @xcite @xcite . irrespective of the many useful applications, the deployment of nlg models has concurrently given rise to serious concerns such as plagiarism, and spreading misinformation and hate speech @xcite @xcite . therefore, the need to discriminate between human and machinegenerated text becomes paramount, especially in light of the growing sophistication and rapid updates of these models. given the diverse applications of nlg models, authorship attribution (aa) methods have been increasingly employed to detect the original author of synthetic data generated by machines @xcite @xcite . the main concern with traditional aa methods is that, typically, they are feature-based systems and consist of largely document-specific features. therefore, the application of this traditional model is often author, dataset and model-specific @xcite . previous research addressed the need for generalisable detection systems to identify machine-generated text @xcite @xcite . research involving the use of llms in authorship attribution has demonstrated that the simple fine-tuning of pre-trained language models can surpass the accuracy of traditional methods significantly @xcite @xcite . in particular, we introduce deb-ang, a pretrained deberta model with a specialised angular loss and contrastive loss integration. additionally, we demonstrate improved classification when applying deb-ang to several well-known machinegenerated text and authorship attribution datasets. contrastive learning is an unsupervised representation learning technique, aiming to learn a representation of data such that similar instances are close in the representation space whereas dissimilar instances are far apart @xcite . loss functions are crucial in contrastive learning as they quantify the similarity and dissimilarity between pairs, guiding the model to learn meaningful representations for class discrimination @xcite @xcite . however, recent studies suggest that various loss functions, including cross-entropy loss, contrastive loss and triplet loss, fail to consider the intrinsic angular distribution exhibited by the low-level and highlevel feature representations @xcite , which contributes to our choice of using angular loss in deb-ang. angular loss is a scale-invariant loss function designed to improve the learning sim-ilarity metrics by considering the angle between vectors @xcite . in summary, the contributions of this work are four-fold: for evaluating text classification models. these datasets were constructed by leveraging state-of-the-art language models, including gemma-7b ( team et al., 2024), gpt4-turbo (openai, 2023) and flan-t5-large (chung et al., 2022). 4. we conduct linguistic error analysis of incorrectly and correctly classified examples. ## conclusion and future work in this research, we have created a custom de-berta model integrating contrastive and angular loss. to our knowledge, this is one of the first attempts at this integration and we have demonstrated the success of the proposed deb-ang model on several datasets. we investigated more fine-grained machine-generated text detection by classifying model variations and developers. we were able to outperform prior approaches in machine-generated text detection with a minimum improvement of 0.23 % and a maximum improvement of 38.04% across all datasets. we were able to classify model variations with accuracy scores ranging from 0.64% to 5.68%, and to identify developers with an accuracy improvement of 3.20%. for authorship attribution, we were able to improve classification with a maximum accuracy of 17.48% on the extensive turingbench dataset which is characterised by high topical dissimilarity. future work will involve identifying texts in which multiple nlg models or humans have been used to intentionally mask the writing style of a text. additionally, a more extensive examination of linguistic features of synthetic data across generations of llms can provide insights into language evolution these models.