A Prompt in the Right Direction: Prompt Based Classification of Machine-Generated Text Detection (2023)

Abstract: The goal of ALTA 2023 Shared Task is to distinguish between human-authored text and synthetic text generated by Large Language Models (LLMs). Given the growing societal concerns surrounding LLMs, this task addresses the urgent need for robust text verification strategies. In this paper, we describe our method, a fine-tuned Falcon-7B model with incorporated label smoothing into the training process. We applied model prompting to samples with lower confidence scores to enhance prediction accuracy. Our model achieved a statistically significant accuracy of 0.991.

## introduction the rapid evolution of large language models (llms) has significantly facilitated the generation of complex, human-like text at scale @xcite . these llms have found applications in various domains, including ai-assisted writing @xcite , medical question answering @xcite @xcite , financial @xcite @xcite , and legal sectors @xcite . leading models like openai's gpt-3 @xcite ), meta's opt @xcite , and big science's bloom @xcite have the ability to produce content that closely mimics human-created text, making it challenging to distinguish between machine-generated and human-generated content. however, it's important to note that these models lack a genuine understanding of the content they generate. this limitation can lead to intended negative consequences when this machine-generated content is used in downstream applications. for instance, llms have been used to carry out academic fraud @xcite @xcite , disseminate fabricated news stories @xcite @xcite , and manipulate public opinion @xcite @xcite . given the widespread use of llms by the general public @xcite and the rapid global dissemination of information, there is a growing risk of disinformation affecting both individuals and organisations. to address these issues, it is crucial to differentiate between content authored by llms and humans. this distinction is essential for ensuring that machine-generated content is used appropriately in various applications while maintaining oversight. understanding the specific llm responsible for generating content can help users be aware of potential biases and limitations associated with that model. this interest has led to active research in the area of automatic detection of ai-generated text. recent work, such as detectgpt @xcite , focuses on techniques for identifying ai-generated content by perturbing text samples and comparing log probabilities. other approaches involve using llms such as deberta @xcite or ensemble methods @xcite for multi-class ai detection tasks, illustrating the evolving nature of this research domain. in this paper, we present our participation in the alta 2023 shared task @xcite , which centres on the automatic detection of synthetic text produced by llms. participants are challenged with the task of identifying synthetic text across a wide spectrum of sources, spanning different domains and llms, including prominent models like t5 @xcite and gpt-x @xcite . the primary assessment criterion is accuracy, and participants are encouraged to explore diverse methodologies and approaches to construct effective text detection systems. our approach involved the fine-tuning of a falcon-7b (institute, 2023) model, complemented by the integration of label smoothing during the training process. furthermore, we leveraged prompting techniques @xcite for samples exhibiting lower confidence scores, to guide our model, resulting in improved predictions and an overall enhanced system accuracy. our participation in this shared task yielded a successful outcome, as our method attained an overall accuracy of 0.991. this achievement underscores the effectiveness of our approach in discerning between human-authored and llm-generated text, making a substantial contribution to the ongoing endeavours aimed at addressing the challenges associated with synthetic text. ## related work text classification is a field that extensively investigates the extraction of features from unprocessed text data to predict text categories. this topic has witnessed substantial research efforts over recent decades, leading to the development of various models tailored for this purpose. traditional models like naive bayes, logistic regression, support vector machines, random forest, and k-nearest neighbors have been widely explored @xcite . machine learning boosting techniques, including extreme gradient boosting and adaptive boosting, have demonstrated their prowess in delivering high performance @xcite @xcite @xcite . deep learning models, such as convolutional neural networks and recurrent neural networks, have surpassed traditional methods in text classification tasks @xcite @xcite . in recent years, transformer-based language models have risen to prominence for natural language processing tasks due to their enhanced parallelization capabilities and self-attention mechanisms @xcite , compared to prior models like rnns @xcite . however, it's crucial to acknowledge that while transformer models excel in the domains for which they were trained, they can be less adaptable when dealing with out-of-domain or unseen samples. their profound understanding of specific contexts, stemming from vast pre-training data, makes them experts in those domains, yet can hinder their ability to generalise effectively @xcite @xcite . the focus on the knowledge they acquire during fine-tuning might result in a degree of "domain bias," making them less suitable for broader applications. to mitigate the limitations of domain-specificity in transformer models, a hybrid approach in text classification is increasingly gaining recognition @xcite . the concept of ensembling transformer models with traditional approaches, such as naive bayes, support vector machines, or ensemble learning, can harness the benefits of both worlds @xcite . the specialised domain knowledge acquired by transformer models can be combined with the interpretability, simplicity, and robustness offered by traditional techniques, ultimately leading to more versatile and adaptive text classification models. ## results the organisers of the alta 2023 shared task provided both a development and a test set for evaluation. while predictions were made on both sets, it's worth noting that the official rankings are determined based on the results from the test set. accuracy is the metric used to assess the model's performance. for this paper, we exclusively present the results of our test set predictions. the comprehensive leaderboard can be accessed on the alta codalab competition website foot_2 . our approach, under the team name od-21, as showcased in all scores presented in ## conclusion in this paper, we have presented our submission to the alta 2023 shared task, a binary classification challenge distinguishing generative ai content from human writing. our proposed approach, using a falcon-7b language model combined with label smoothing and model prompting, has demonstrated considerable promise. with a top-ranking accuracy score of 0.991, our system has showcased the effectiveness of these techniques in this specific task. looking forward, there is an opportunity for further research and refinement. future work should focus on extending our system's capabilities to tackle more challenging scenarios, including out-of-domain problems and multi-class authorship attribution tasks.