Fine-tuning Transformers for Identifying Self-Reporting Potential Cases and Symptoms of COVID -19 in Tweets (2021)

Abstract: We describe our straight-forward approach for Tasks 5 and 6 of 2021 Social Media Min- ing for Health Applications (SMM4H) shared tasks. Our system is based on fine-tuning Dis- tillBERT on each task, as well as first fine- tuning the model on the other task. In this paper, we additionally explore how much fine- tuning is necessary for accurately classifying tweets as containing self-reported COVID-19 symptoms (Task 5) or whether a tweet related to COVID-19 is self-reporting, non-personal reporting, or a literature/news mention of the virus (Task 6).

## introduction fine-tuning off-the-shelf transformer-based contextualized language models is a common baseline for contemporary natural language processing @xcite . when developing our system for task 6 of the 2021 social media mining for health applications (smm4h), we quickly discovered that fine-tuning distilbert @xcite , a smaller and distilled version of bert @xcite , outperformed training traditional, non-neural machine learning models. fine-tuning distilbert on the released training set resulted in a micro-f1 of 97.60 on the task 6 release development set. while this approach was not as successful for task 5 (binary-f1 of 51.49), in this paper, we explore how much fine-tuning is necessary for these tasks and whether there are benefits to first training the model on the other task since both are related to covid-19. 1 2 task description both task 5 and task 6 focused on classifying tweets related to covid-19 @xcite . task 5 required classifying tweets as describing self-reporting potential cases of covid-19 or not. 1 all code developed is publicly available at https:// github.com/mfleming99/smm4h_2021 . tweets were extracted via manually crafted regular expressions for potential self-reported mentions of covid-19 and then annotated by two people. 1, 148 tweets were labeled as containing a selfreporting potential cases and 6, 033 were labeled as "other." the other tweets that might discuss covid-19 but do not specifically reporting a user's or their household's potential cases were labeled as "other." foot_0 systems were ranked by f1-score for the "potential case" class. in task 6, systems must determine whether a tweet related to covid-19 is self-reporting, nonpersonal reporting, or a literature/news mention of the virus. 1, 421 released examples are labeled as self-reporting, 3, 567 as non-personal reports, and 4, 464 as literature/news mentions. systems were evaluated by micro-f1 score. ## method we fine-tuned distillbert using the implementation developed and released by huggingface transformer's library @xcite . we trained the model for 3 epochs, using a batch size of 64 examples, warm-up steps of 500 for the learning rate scheduler and a weight decay of 0.01. following @xcite recommendation to add minimal task hyper-parameters when finetuning pre-trained models, we used the remaining default hyper-parameters from the library's trainer class. all models were trained across 2 nvidia rtx 3090's. ## conclusion we discussed our straightforward approach of finetuning a distilbert model on tasks 5 and 6 of the 2021 social media mining for health applications shared tasks. while not attaining state-of-the-art, these results are competitive and demonstrate the benefit of leveraging large scale pre-trained contextualized language models. we additionally explored the benefits of first training the model on the corresponding task and determine when this can be beneficial. future work might consider jointly 5 these numbers differ from the official leaderboard during the evaluation as we discovered a bug related to loading our pre-trained models during the post-evaluation period. indicates the model was finetuned on the specific task and indicates the model was first fine-tuned on the other task. the first line reports the results trained on the combination of the corresponding train and development sets -7, 174 for task 5 and 9, 452 for task 6. the remaining lines are based on a ensemble of the 5 models trained on the corresponding number of examples using a majority vote.