CHAAI @ SMM 4 H ’22: R o BERT a, GPT -2 and Sampling - An interesting concoction (2022)

Abstract: This paper describes the approaches to the SMM4H 2022 Shared Tasks that were taken by our team for tasks 1 and 6. Task 6 was the “Classification of tweets which indicate self-reported COVID-19 vaccination status (in English)”. The best test F1 score was 0.82 using a CT-BERT model, which exceeded the median test F1 score of 0.77, and was close to the 0.83 F1 score of the SMM4H baseline model. Task 1 was described as the “Classification, detection and normalization of Adverse Events (AE) mentions in tweets (in English)”. We undertook task 1a, and with a RoBERTa-base model achieved an F1 Score of 0.61 on test data, which exceeded the mean test F1 for the task of 0.56.

## introduction the shared tasks of the social media mining for health (smm4h) are focused on overcoming difficult challenges in utilizing natural language processing techniques for deriving health-related information from social media data @xcite . task 6 of the seventh smm4h (in 2022) was the "classification of tweets which indicate selfreported covid-19 vaccination status (in english)". our team, "champions of health and artificial intelligence" (chaai), found task 6 particularly interesting, as the challenge is similar to a key research area we are conducting in saefvic @xcite to leverage social media monitoring to improve vaccine safety surveillance. detecting self-reporting in healthrelated online posts helps to identify genuine descriptions of health events. it is as important to know who is speaking as it is to know what is being spoken about; as in detecting adverse events following immunization (aefi) @xcite @xcite ; experiences of covid-19 disease @xcite ; and drug effects @xcite . task 1 was the "classification, detection and normalization of adverse events (ae) mentions in tweets (in english)". the adverse events mentioned in the task description are adverse drug events (ade), which are unexpected side effects following consumption of medications. task 1a was for the classification of ade, task 1b was to identify the spans of the relevant text, and task 1c required matching the colloquially expressed reactions to meddra standard concept ids. whereas task 6 was akin to obtaining an online data stream of likely candidates for aefi detection, task 1a's challenge of precisely identifying an ade was akin to differentiating specific aefi in that data stream. in both tasks, we had to deal with massive imbalances between the under-represented positive class and the negative class, which matches realworld conditions. dealing with class imbalances was a major focus of our response to the tasks. ## task 6: classification of tweets which indicate self-reported covid-19 vaccination status identifying tweets of users reporting on their covid-19 vaccination status. ## conclusion for task 6, although our best model on validation data was an ensemble of a bertweet-large and two roberta-large models, surprisingly, the ct-bert model (model 4) was our best model on test data. its f1 score of 0.82 was close to the 0.83 f1 score of the baseline smm4h model, and significantly better than the median f1 score of 0.77 for the task. however, the ensemble was only marginally behind -to 3 decimals its f1 score was 0.814, compared to 0.819 for the ct-bert. for task 1a, our best result on the test dataset was from the roberta-base model. notably, thanks to its balance of precision and recall, its f1 score matched last year's winning f1 score of 0.61 @xcite . moreover, it exceeded this year's mean f1 score of 0.56. scores are presented in