Feature-Level Ensemble Learning for Robust Synthetic Text Detection with D e BERT a V 3 and XLM - R o BERT a (2023)

Abstract: As large language models, or LLMs, continue to advance in recent years, they require the development of a potent system to detect whether a text was created by a human or an LLM in order to prevent the unethical use of LLMs. To address this challenge, ALTA Shared Task 2023 introduced a task to build an automatic detection system that can discriminate between human-authored and synthetic text generated by LLMs. In this paper, we present our participation in this task where we proposed a feature-level ensemble of two transformer models namely DeBERTaV3 and XLM-RoBERTa to come up with a robust system. The given dataset consisted of textual data with two labels where the task was binary classification. Experimental results show that our proposed method achieved competitive performance among the participants. We believe this solution would make an impact and provide a feasible solution for detection of synthetic text detection.

## introduction in recent years, the remarkable advancements in large language models (llms) have showcased an unprecedented revolution in the field of natural language processing @xcite . these models e.g. gpt-x and t5, demonstrate the ability to generate text that closely resembles content created by humans. however, there is a risk of abuse that makes this a double-edged sword and raises moral questions. the spread of synthetic text produced by llms carries the risk of spreading false information @xcite , interfering in elections @xcite , and jeopardize the credibility of scientific knowledge @xcite . in this context, the alta shared task 2023 foot_0 introduced a task where researchers have to develop automated detection systems with the capacity to discriminate between human-written text and text generated by large language models (llms). the aim of this task is to mitigate the unethical application of llms and promote their conscientious and responsible utilization in various domains. the dataset provided by the alta shared task 2023 is used in building and assessing the automatic text detection systems. the task is fundamentally a binary classification problem. each text is labeled as either 0 (ai-generated) or 1 (humangenerated). the text samples are derived from diverse domains, including law and medicine, and span a spectrum of llms. the efficacy of the models will be assessed based on their accuracy and their resilience in detecting synthetic text. this text label i am asking you this because the fans of the band are completely devoted. they experience the days leading up to the concert very intensely. 0 a reston man has been charged with abduction after police say he dragged a woman from the sidewalk and tried to remove her clothes. text label id language found rektor på gammel hellerup gymnasium, jørgen rasmussen, ønsker ikke at udtale sig om sagen. 1 36 german en réalité, la superteam qui semble se profiler est composée de : 1 5628 french e le parole "programma di aggiustamento strutturale", "ristrutturazione" e "default" si possono benissimo tradurre in genocidio sociale. 1 15541 italian ## dataset description the dataset statistics are summarized in each text in the dataset is tagged with either a 0 (for ai-generated) or a 1 (for human-generated). the training dataset is evenly balanced, ensuring an equal number of both categories. in our analysis, we found that text length in the training set typically range from 10 to 50 and never exceed 200 which can be seen in figure 1 . also, we came across that some words of the texts are in languages other than english, e.g. german, afrikaans, romanian, french, etc., in about 124 rows of the train dataset. some examples are shown in ## conclusion in conclusion, the growing capabilities of large language models (llms) have brought both opportunities and challenges in the field of natural language processing. the rise of synthetic text generated by llms has raised ethical concerns, including the spread of misinformation and potential misuse in various domains. to address this, the alta shared task 2023 was introduced. in this paper, we presented our approach to this task, where we focused on building a feature-level ensemble model using two state-of-the-art transformer models. we conducted a comprehensive analysis of the dataset, which revealed the need to handle multilingual content. our approach leveraged feature-level ensemble learning, utilizing the strengths of both models, and included data augmentation to enhance performance. while we secured the 4th position in the original contest, the inclusion of validation data improved our model's accuracy, bringing it to the 2nd position when submitted after the contest's conclusion. notably, the top-performing teams in the competition exhibited marginal differences in accuracy, emphasizing the high level of competitiveness in the task. we believe that our proposed method provides a promising solution for the detection of synthetic text, contributing to the responsible and conscientious use of llms in various applications. as llms continue to evolve, robust detection systems like the one presented in this paper become increasingly important to address the ethical challenges associated with ai-generated text.