OCHADAI at SMM 4 H -2021 Task 5: Classifying self-reporting tweets on potential cases of COVID -19 by ensembling pre-trained language models (2021)

Abstract: Since the outbreak of coronavirus at the end of 2019, there have been numerous studies on coro- navirus in the NLP arena. Meanwhile, Twitter has been a valuable source of news and a pub- lic medium for the conveyance of information and personal expression. This paper describes the system developed by the Ochadai team for the Social Media Mining for Health Appli- cations (SMM4H) 2021 Task 5, which aims to automatically distinguish English tweets that self-report potential cases of COVID-19 from those that do not. We proposed a model ensemble that leverages pre-trained represen- tations from COVID-Twitter-BERT (Müller et al., 2020), RoBERTa (Liu et al., 2019), and Twitter-RoBERTa (Glazkova et al., 2021). Our model obtained F1-scores of 76% on the test set in the evaluation phase, and 77.5% in the post-evaluation phase.

## introduction since the outbreak of coronavirus at the end of 2019, there have been numerous studies on coronavirus in the nlp arena. meanwhile, twitter has been a valuable source of news and a public medium for the conveyance of information and personal expression. this paper describes the system developed by the ochadai team for the social media mining for health applications (smm4h) 2021 task 5, which aims to automatically distinguish english tweets that self-report potential cases of covid-19 from those that do not. we proposed a model ensemble that leverages pre-trained representations from covid-twitter-bert @xcite , roberta @xcite , and twitter-roberta @xcite . our model obtained f1-scores of 76% on the test set in the evaluation phase, and 77.5% in the post-evaluation phase. ## system overview in this section, we overview the pre-processing steps, pre-trained language models and training prodedure used by our system. ## conclusion we presented the ochadai system submitted to the smm4h-2021 task 5. we proposed an ensemble model that leverages pre-trained representations from covid-twitter-bert @xcite , roberta @xcite , and twitter-roberta @xcite . our best performing model obtained an f1-score of 77.5%. in future efforts, we plan to further improve our model by exploring other pre-trained language models and ensemble techniques.