712for T ask7 at # SMM 4 H 2024 Task 7: Classifying S panish Tweets Annotated by Humans versus Machines with BETO Models (2024)

Abstract: The goal of Social Media Mining for Health (#SMM4H) 2024 Task 7 was to train a machine learning model that is able to distinguish between annotations made by humans and those made by a Large Language Model (LLM). The dataset consisted of tweets originating from #SMM4H 2023 Task 3, wherein the objective was to extract COVID-19 symptoms in Latin-American Spanish tweets. Due to the lack of additional annotated tweets for classification, we reframed the task using the available tweets and their corresponding human or machine annotator labels to explore differences between the two subsets of tweets. We conducted an exploratory data analysis and trained a BERT-based classifier to identify sampling biases between the two subsets. The exploratory data analysis found no significant differences between the samples and our best classifier achieved a precision of 0.52 and a recall of 0.51, indicating near-random performance. This confirms the lack of sampling biases between the two sets of tweets and is thus a valid dataset for a task designed to assess the authorship of annotations by humans versus machines.

## introduction tweeting has become a significant way for people to connect and communicate with each other individually and as a community. in 2021, 23% of adults in the united states reported to using twitter @xcite . this type of social media is used for a variety of reasons, including news, entertainment, communication between family and friends, information on brands, and for developing a professional network. for example, covid-19 spurred twitter users to share their personal experiences, emotions, and beliefs during a turbulent and confusing time of a global epidemic @xcite . given the prevalence of its use, tweet content can provide information for understanding public sentiment and identifying specific areas of concern. however, sifting through the volumes of tweets can be an arduous task. machine learning provides an automated approach for analyzing tweets. leveraging its tremendous processing speed, a machine learning model, such as bert and gpt-3, can be trained to identify specific patterns and annotate features of interest quickly compared to manual annotation by humans @xcite . the #smm4h 2024 shared task 7 builds over the dataset from task 3 of the 2023 social media mining for health. task 3 involved a dataset of 10,150 tweets describing covid-related symptoms @xcite . the dataset was annotated by human experts who were medical doctors and also native speakers of latin-american spanish. the 2023 challenge involved training a machine learning model that could identify and extract symptoms in the tweets that were either personally described or mentioned by a third party. the leading model achieved an f1 score of 0.94 for identifying the character offsets of covid-19 symptoms. the dataset provided for the #smm4h 2024 shared task 7 utilized some of the tweets from #smm4h 2023 that were annotated by either human or machine @xcite . the intended aim was to identify whether the tweet was annotated by a human or a machine. due to the lack of additional tweet annotation data for this classification task, we first conducted an exploratory data analysis to investigate if the two subset of data -those annotated by humans versus machines -were fundamentally different. then, we trained a bert-based classifier to explore differences in content between the two subsets of tweets. our primary research goal was to confirm the absence of sampling biases that could affect the training of a classifier designed to distinguish between human and machine annotations. ## exploratory data analysis the task 7 dataset included only the tweet text and an associated label indicating the author of the annotation, without any additional text annotations. while our initial understanding of the task was that the model should distinguish between annotations generated by humans versus machines, the absence of any generated annotations confirmed the overall purpose of the task. we framed our work to leverage the available data and examine the differences between tweets annotated by humans and those annotated by machines. we trained a binary tweet classifier to verify that there are no significant differences between the tweets in the two annotation groups. this aims to ensure that the tweets that were sampled for human annotation vs. machine annotation did not suffer from selection bias, and such bias did not affect the classifier trained to predict annotation authorship, as intended in the original task 7. we first approached the challenge by inspecting the provided dataset. there was a total of 4,603 tweets of which 2,288 were labelled "human" and 2,315 labeled "machine." the counts were sufficiently balanced. then, we performed an exploratory data analysis to dive deeper into the two subset features. as seen in fig. 1 , a comparison of the tweet lengths, number of mentions, and emojis did not show any significant differences between those labeled as humans or machines. this initial comparison suggests that both groups of tweets have similar textual features. word cloud analysis on the human and machineannotated tweets were also similar, indicating that the overall words and their frequencies were similar between both groups (fig. 2 ). finally, analyzing the bi-grams, which include pairs of consecutive words, also didn't show any significant differences figure 3: comparative analysis of tweet bi-grams by class 150 ## bert model selection and fine-tuning data pre-processing: after conducting the exploratory data analysis, we found that the number of emojis and mentions was similar between the two groups. therefore, we removed emojis and mentions from the tweets to enable the model to focus on the core textual content. we retained the stop words to preserve the context of the tweets, which is beneficial for training a large language model. ## results as shown in in the post-evaluation phase, we found that configuration 2 yielded better performance and consistency, with an average precision of 0.51 and a recall of 0.54, which was a slight increase compared to the initial submission. tuning other hyperparameters did not yield any significant improvement. ## discussion despite initial misunderstanding of the task, we realized that the actual aim of the shared task was to determine whether the dataset used in previously published work on generating machine-based annotations incurred any training bias. the dataset provided for training and validation did not contain any additional information aside from the tweet text and label. particularly, there was no information about the specific covid symptoms from the human or machine annotator that may have been informative in distinguishing between the human and machine annotations. based on our results, our models were unable to find any key distinguishing factors separating human-annotated and machine-annotated tweets. this strengthens the conclusions of the baseline system mentioned in the challenge, which achieved an 82% classification accuracy for a tweet annotation classifier. our best models, formulated as tweet authorship classifiers, only achieved a random-chance performance, indicating that the tweets labeled by machines were indistinguishable from those labeled by humans. this aligns with our exploratory data analysis, which also revealed that the features of the two subsets of tweets are very similar. ## conclusion our best fine-tuned beto model achieved an overall f1 score of 0.52, and was slightly better than our submitted run, which achieved an f1 score of 0.51. our submitted run achieved the highest performance on #smm4h 2024 task 7 to distinguish between tweets that were annotated by humans vs. machines. our findings from exploratory data analysis and training a classifier between the two subsets of tweets indicate no sampling biases and help confirm the applicability of this dataset in training a tweet annotation authorship classifier.