SC a LAR at S em E val-2024 Task 8: Unmasking the machine : Exploring the power of R o BERT a Ensemble for Detecting Machine Generated Text (2024)

Abstract: SemEval SubtaskB, a shared task that is concerned with the detection of text generated by one out of the 5 different models - davinci, bloomz, chatGPT, cohere and dolly. This is an important task considering the boom of generative models in the current day scenario and their ability to draft mails, formal documents, write and qualify exams and many more which keep evolving every passing day. The purpose of classifying text as generated by which pre-trained model helps in analyzing how each of the training data has affected the ability of the model in performing a certain given task. In the proposed approach, data augmentation was done in order to handle lengthier sentences and also labelling them with the same parent label. Upon the augmented data three RoBERTa models were trained on different segments of data which were then ensembled using a voting classifier based on their R2 score to achieve a higher accuracy than the individual models itself. The proposed model achieved an overall validation accuracy of 97.05% and testing accuracy of 76.25%. and our standing was 18th position on the leaderboard.

## introduction in the current day scenario, ai has noticed a major boom due the emergence of large language models (llms) in the field of natural language processing (nlp). these llms are capable of generating text with any given context that they have been trained on making them versatile to a lot of applications. llms have also showcased their unrivaled ability to code basic to complex programs. many large language models (llms) depend heavily on the data used for their training. consequently, they may occasionally provide inaccurate information, especially in contexts where precision is crucial, such as sensitive or professional advice. hence ai-generated text classification has become increasingly important due to the surge in the use of language models for content creation. accurately identifying the source of a text, whether human-written or generated by a specific language model, is crucial for various applications, such as combating misinformation and plagiarism detection. subtask b -multi-way machine-generated text classification shared task aims to not only detect text generated by these language models, but also specifically distinguish between outputs generated by different models. this in a real life scenario helps in determining the transparency and vulnerability of a model to attacks and reasoning as to why particular models perform in certain ways. different contributions of the paper is as follows :-• data augmentation: data augmentation is a crucial task of increasing the volume of available data with specific manipulations which also helps build a more robust model able to tackle edge case scenarios. we propose a novel approach to handle long texts. we initially set a specific threshold to split them into smaller segments while preserving label information, ensuring efficient model training. we observe how effective and relevant language models are in tackling natural language processing tasks such as the current shared task when compared to other neural network based lstm or other sequence models. our final submission had a test accuracy of 76.25% and our standing was 18th position in the leader-board. 1 1135 2 background our work improves model generalizability in largescale tasks by utilising insights from @xcite and building on recent studies. previously methods proposed by authors in @xcite collected 500 scientific articles from 10 domains including biology, chemistry, it and others and used chatgpt to paraphrase texts for each article. the authors extracted certain features such as perplexity, semantic document and six others to use classifiers on these extracted features. the authors used three classifiers: xgboost, random forest, and multi-layer perceptron, to train and test models for detecting human-generated and ai-generated texts, as well as human-generated and ai-rephrased texts. they performed a 5-fold cross-validation and evaluated the models using accuracy and f1score which majorly motivated our approach. another work @xcite used similar techniques to the previous one while using school topics as their dataset. in their work @xcite the authors use ensemble neural model that generates probabilities from different pre-trained llms which are used as features to a tml classifier following it. author in @xcite presents his work on text classification ensemble learning method based on multi-angle perturbation heterogeneous base classifiers and validates the effectiveness of the algorithm through experiments. in a similar work @xcite the authors propose a new meta-learning ensemble method that fuses baseline deep learning models using 2-tiers of meta-classifiers. furthermore, our method for comprehending model decision-making in short text classification-particularly in identifying ai-generated content-is influenced by methods from works on short text classification. authors in their work @xcite use a sliding window to align the sentences with the labels and preserve the edge characteristics of the long text. another work in the same field @xcite ) categorizes text data augmentations into symbolic and neural methods. symbolic methods use rules or discrete data structures to form new examples, while neural methods use auxiliary neural networks to sample new data. our research aims to advance the development of robust and adaptable machine learning models customised to particular tasks through this synthesis of diverse viewpoints. these viewpoints are then combined back again with the help of em-sembling ensuring no loss of data. ## methodology our methodology majorly focuses on exploiting pre-trained language models such as the roberta model @xcite and enhancing its performance through a much simpler traditional approach of ensemble learning. we worked on the m4 based dataset with our methodology @xcite the ensemble model shows better performance compared to all 3 roberta-base models which were trained on different segments of augmented data. it reduces over-fitting and increases interpretability for any given task. ## system overview as mentioned, we first perform data augmentation. before we get into the details of our experimental setup, we want to elaborate on different measures we took in order to augment our data. data augmentation for training data was performed by carefully splitting the validation data while noting that there is no major imbalance in the class distribution. this was followed by training three different roberta models on different combinations of training and validation dataset. we had 3 validation data splits namely val1, val2 and val3. for model1 we used val2 and val3 in training and val1 for validating the model1 and so on. ## experimental results as a part of our experimental setup we used p100 gpu which is available through kaggle. further we used the robertaforsequenceclassification available through transformers library along with robertatokenizerfast for the modelling aspect. the learning rate used was a fixed one and we found it optimal at 1e -5 along with crossentropyloss. adamw optimizer was used with weight decay coefficient of 1e -2 and β 1 = 0.9 and β 2 = 0.999. batch size of 20 was used for training and validation. our proposed methodology beat the baseline model which was a roberta model with an average accuracy of 0.75. our experimental results with respect to each of the roberta model is displayed along with the improvement in performance with the use of r2-score based weighted voting classifier. in testing phase our model gave an accuracy of 76.25% which shows clear signs of over-fitting compared to 97.05% in validation accuracy. ## conclusion in the proposed methodology, we beat the baseline roberta model and further enhance the performance of the model using r2-score based voting classifier. the model has performed well on the training data when compared to testing data which shows slight signs of over-fitting. in the light of ensemble learning for pre-trained language models we see that the models are very sensitive to overfitting hence should be used with caution. techniques like early stopping and using data augmen-4 1138 tation. further on embeddings from llms can be used to tackle this task more effectively.