Deloitte at # SMM 4 H 2024: Can GPT -4 Detect COVID -19 Tweets Annotated by Itself? (2024)

Abstract: The advent of Large Language Models (LLMs) such as Generative Pre-trained Transformers (GPT-4) mark a transformative era in Natural Language Generation (NLG). These models demonstrate the ability to generate coherent text that closely resembles human-authored content. They are easily accessible and have become invaluable tools in handling various text-based tasks, such as data annotation, report generation, and question answering. In this paper, we investigate GPT-4’s ability to discern between data it has annotated and data annotated by humans, specifically within the context of tweets in the medical domain. Through experimental analysis, we observe GPT-4 outperform other state-of-the-art models. The dataset used in this study was provided by the SMM4H (Social Media Mining for Health Research and Applications) shared task. Our model achieved an accuracy of 0.51, securing a second rank in the shared task.

## introduction the field of natural language generation (nlg) is undergoing a significant transformation driven by the emergence of llms like @xcite , and many other large language models. these models are capable of generating text of human-level quality for a wide range of applications, such as data annotation @xcite , medical question answering @xcite , conversation response generation @xcite , and code auto-completion @xcite . notably, the ability of these models to learn without extensive training data (zero-shot learning) or with just a few examples (few-shot learning), simplifies their integration into various language generation applications. while the llms demonstrate the ability to understand the context and generate coherent humanlike responses, they do not have a true understanding of what they are producing @xcite . this could potentially lead to adverse consequences when used in downstream applications. for example, consider an application of a llm tasked with summarizing a medicinal drug datasheet inadvertently produces wrong dosage information. this generation of plausible but false content (referred as hallucination @xcite ), can unintentionally spread misinformation, false narratives, fake news, and spam. similarly, the use of llms in data annotation has sparked a debate within the research community due to potential issues like inherent biases and hallucinations associated with these models @xcite . motivated by these challenges, the automatic detection of ai-generated outputs has emerged as an active area of research. the detection of data annotations generated by llms closely resembles the process of identifying ai-generated text, which aims to distinguish between human-authored and machine-generated content @xcite . in this paper, we explore methodologies utilized for ai-generated text detection, with a particular emphasis on zeroshot detection techniques, and examine the synergies between these two areas. the ai-generated text detection methods predominantly involve the analysis of outputs from llms utilizing features such as entropy, log-probability scores, and perplexity @xcite @xcite to distinguish between human-written and machine-generated content. building upon this foundation, detectgpt @xcite introduced the concept of analyzing negative log probability curvature, identifying a distinct pattern in ai-generated text. subsequent advancements, such as dna-gpt @xcite , improved performance by analyzing the divergence of n-grams between the original text and llm-prompted versions. while zero-shot detection methods are effective, their success often rely on direct access to the internal mechanisms of the specific llm that generated the text. however, the underlying architecture and weights of many llms, including openai's gpt-4, are not publicly available. as a result, these techniques often depend on a substitute llm, presumed to have mechanisms similar to the proprietary model. the reliance on a proxy llm may limit the robustness and generalizability of zeroshot detection in various scenarios. ## gpt-4 as ai-annotated detector while gpt-4 has exhibited proficiency in various natural language processing (nlp) tasks, its potential for distinguishing between human-annotated and ai-annotated data remains largely untapped. in this study, we explore the effectiveness of gpt-4 in distinguishing between tweets it annotated and those annotated by humans in the medical domain. our experimental analysis indicates that gpt-4 outperform other state-of-the-art models in detecting ai-annotated data. ## results to assess the performance of our approach in the zero-shot setting, we curated three distinct evaluation sets -set 1, set 2, and set 3 as shown in we conducted a comparative evaluation of our gpt-4-based approach against multiple zero-shot detection models. the two most effective baselines methods we identified were: 1) binoculars @xcite , which compares the perplexity and cross-perplexity of two closely related language models to identify ai-generated text; and 2) fast-detectgpt @xcite , which utilizes conditional probability curvature to efficiently detect ai content, particularly from models like gpt. ## conclusion in this paper, we investigated the potential of gpt-4 to detect tweets annotated by itself in zero-shot setting. our experiments highlighted the complexity of this task, particularly for short texts in the medical domain, achieving an accuracy of 0.51. for future work, we aim to improve the model's performance by incorporating the reasoning behind its predictions. by using this reasoning as additional input, we aim to enhance the model's ability to differentiate between data annotations predicted by llm and those annotated by human experts.