Weighted Layer Averaging R o BERT a for Black-Box Machine-Generated Text Detection (2024)

Abstract: We propose a novel approach for machine-generated text detection using a RoBERTa model with weighted layer averaging and AdaLoRA for parameter-efficient fine-tuning. Our method incorporates information from all model layers, capturing diverse linguistic cues beyond those accessible from the final layer alone. To mitigate potential overfitting and improve generalizability, we leverage AdaLoRA, which injects trainable low-rank matrices into each Transformer layer, significantly reducing the number of trainable parameters. Furthermore, we employ data mixing to ensure our model encounters text from various domains and generators during training, enhancing its ability to generalize to unseen data. This work highlights the potential of combining layer-wise information with parameter-efficient fine-tuning and data mixing for effective machine-generated text detection.

## introduction language modeling is a foundational task in nlp, and encompasses learning of all the features that make up language. different levels of linguistic information are stored in language models' (lm) hidden states. this may include syntax, morphological features, phrasing, and so on. @xcite our aim is to leverage this encoded information to help us discern machine-generated text. the advent of large language models (llms) has transformed the digital landscape, and this has also led to the proliferation of machine-generated text in spaces spanning from legal proceedings, to articles, to school submissions. with this, there has been a consequential rise in the need to be able to distinguish between machine-and humangenerated text across domains. just as important is the need to be able to identify the generators for text that has been flagged as being generated by machines. in this paper, we describe our methodology and attempts to create a system that can perform the task effectively. ## system overview we have used roberta-base for all experiments in the scope of this paper. the baseline set by the task organizers is reported to have been from a finetuned roberta model. roberta has the same architecture as bert, but uses a byte-level bpe as a tokenizer and uses a different pretraining scheme and has become a sota model since its release @xcite . ## data data for the task was provided by the organizers @xcite . it is an extension of the m4 dataset @xcite . the name stands for multi-generator, multi-domain, and multi-lingual corpus for machine-generated text detection. as the name suggests, the dataset has been created with text from different generators spanning multiple domains. the data for subtask a and b follow the same format, consisting of source (such as wikipedia), model (such as dolly), label (such as human), and the text to be classified. the data for subtask c contains text with a combination of human-and machinegenerated text, and a label indicating the word index at which the split occurs. for our experiments, we resplit the training and dev datasets and split them uniformly across generators and domains in an 80-20 split'. our split of the dev set is bigger than the official dev set, to get a better estimate of our model's performance. ## experimental setup we use roberta's tokenizer and trained our models for subtask a (monolingual) (binary classification) and subtask b (multi-class classification) on the resplit train data and use the resplit evaluation data for early stopping. our hyperparameter configuration has been specified in appendix a. ## results our model while doing really well on our evaluation set, falls short on the test set scoring around 13 percentage points lower than the baseline for subtask a and around 1 percentage point lower than the baseline for subtask b. this could be attributed to the model not being as good in generalising to unseen domains and generators. we hypothesize more hyperparameter tuning, better aggregation of ## conclusion we have demonstrated that linguistic information encoded in the various layers of large language models such as roberta can be used to effectively demonstrate if a text is machine-generated or not, across different domains and generators.