B rain S torm @ i REL at # SMM 4 H 2024: Leveraging Translation and Topical Embeddings for Annotation Detection in Tweets (2024)

Abstract: The proliferation of LLMs in various NLP tasks has sparked debates regarding their reliability, particularly in annotation tasks where biases and hallucinations may arise. In this shared task, we address the challenge of distinguishing annotations made by LLMs from those made by human domain experts in the context of COVID-19 symptom detection from tweets in Latin American Spanish. This paper presents BrainStorm @ iREL’s approach to the #SMM4H 2024 Shared Task, leveraging the inherent topical information in tweets, we propose a novel approach to identify and classify annotations, aiming to enhance the trustworthiness of annotated data.

## introduction data annotation, essential for improving machine learning models, involves labeling raw data with relevant information. however, this process is often costly and time-consuming. in recent times, the field of natural language processing (nlp) has seen a transformative shift with the widespread adoption of large language models (llms) like @xcite ), gemini @xcite ) and bloom @xcite ). these advanced models have shown remarkable capabilities in automating data annotation @xcite ), aiding in a crucial yet labor-intensive step in machine learning workflows. however, despite their impressive performance, the integration of llms in annotation tasks has sparked a debate within the research community. proponents highlight their efficiency and consistency, while skeptics point to potential issues such as underlying biases and hallucinations. while many recent efforts have focused on distinguishing between human and machine-generated text @xcite , @xcite , @xcite ), detecting whether annotations are done by llms offers a novel perspective on ai detection. the advent of powerful llms, while driving innovation, poses risks of increased spread of untruthful news, fake reviews, and biased opinions, highlighting the need for a variety of detection technologies. this paper addresses task 7 of the smm4h-2024 @xcite ): the 9th social media mining for health research and applications workshop and shared tasks, focusing on the identification of data annotations made by llms versus those made by human domain experts. our objective is to develop methods for distinguishing between annotations made by llms and those by human experts in the context of covid-19 tweets in latin american spanish. this task is crucial for evaluating the generalizability and reliability of llms in real-world applications, particularly in health-related nlp tasks. ## methodology our approach to identifying whether a tweet was labeled as containing covid-19 symptoms by an llm or a human domain expert involves several key steps. we begin by preparing the dataset and leveraging both original and translated tweet texts to evaluate the performance of different models. additionally, we incorporate topical embeddings to enhance the distinction between human and llm annotations. ## results we evaluated the models on the test set using the accuracy score provided by the organizers on co-dalab. we observe that topical spanish (with bertopic) achieved a score of 0.50, indicating that the incorporation of topical embeddings did not improve the performance over the baseline in the original spanish tweets. topical english (translated tweets with bertopic) achieved a score of 0.51, showing a marginal improvement over the baseline, suggesting some potential in the use of translated tweets and topical embeddings. while the results indicate only slight improvements, they underscore the challenges inherent in distinguishing between human and llm annotations in this specific context. ## conclusion this study explored the feasibility of distinguishing between human and llm annotations in covid-19 symptom detection from tweets in latin american spanish. by leveraging both language-specific and domain-specific models, along with topical embeddings, we aimed to enhance the accuracy of annotation classification. our findings reveal that while topical embeddings and the use of translated tweets offer some promise, the improvements are marginal. the results suggest that more sophisticated techniques or additional features might be necessary to achieve significant enhancements in performance. the slight improvement observed with translated english tweets suggests that the method has potential when combined with domain-specific models like bertweet, pointing to the importance of further exploring multilingual and domain-adaptive approaches. there is a need to conduct detailed ablation studies to isolate the impact of various components, such as the translation process, topical embeddings, and different pre-trained models. an investigation into more advanced topic modeling techniques or the integration of other context-aware embeddings will also help. by addressing these areas, we can further enhance the reliability of distinguishing between human and machine annotations, ultimately contributing to more trustworthy nlp systems in critical domains like healthcare.