BERT Goes Brrr: A Venture Towards the Lesser Error in Classifying Medical Self-Reporters on T witter (2021)

Abstract: This paper describes our team’s submission for the Social Media Mining for Health (SMM4H) 2021 shared task. We participated in three subtasks: Classifying adverse drug effect, COVID-19 self-report, and COVID-19 symptoms. Our system is based on BERT model pre-trained on the domain-specific text. In addition, we perform data cleaning and augmentation, as well as hyperparameter optimization and model ensemble to further boost the BERT performance. We achieved the first rank in both classifying adverse drug effects and COVID-19 self-report tasks.

## introduction over the years, social media has been used as a massive data source to monitor health-related issues @xcite @xcite , such as flu trends @xcite , adverse drug effects @xcite , or viral disease outbreak such as the covid-19 @xcite . in general, leveraging massive self-reported data is considered useful for supplementing the otherwise long and costly process of clinical trials in obtaining a more comprehensive picture of the issue in hand. nevertheless, analyzing text data from social media is challenging due to its noisy nature, which stems from the prevalence of linguistic errors and typos. in this work, we leverage bert @xcite to handle noisy text through domainspecific pre-training, data cleaning, augmentation, hyperparameter optimization, and model ensemble. with this training pipeline, we achieved the best performance in social media mining for health (smm4h) 2021 shared task @xcite for classifying adverse drug effect and covid-19 self-report from twitter text. ## bert goes brrr we participated in 3 classification tasks: task 1a to classify the adverse drug effect (ade), task 5 to classify covid-19 potential case, and task 6 to classify covid-19 symptoms. the distribution of the datasets are given in we used bert for all three tasks, implemented with the huggingface toolkit @xcite . for each task, we started off by fine-tuning the offthe-shelf bert-base @xcite , which resulted in a fairly good performance ( among the 3 tasks, we achieved the best score for task 1a and task 5. our standing for task 6 by the time of this paper submission is currently unknown. still, our performance on task 6 is above the median, as seen in ## improving bert in this section, we dissect each technique we introduce to our submission model. ## conclusion we describe our team submission for social media mining for health applications shared task 2021. our system achieved the best performance for classifying adverse effect mentions and self-reporting potential cases of covid-19 in english tweets. our system is based on bert model. we observe improvement over the off-the-shelf bertbase from using domain-specific bert, rigorous data cleaning, data augmentation, hyperparameter optimization, and model ensembling. among those techniques, we find that domain-specific bert, data cleaning, and model ensembling improve the performance on all tasks, whereas data augmentation and hyperparameter optimization are more situational. overall, we obtain 17% and 13% improvement on task 1a and task 5 respectively (