UACH - INAOE at SMM 4 H : a BERT based approach for classification of COVID -19 T witter posts (2021)

Abstract: This work describes the participation of the Universidad Autónoma de Chihuahua - Instituto Nacional de Astrofísica, Óptica y Electrónica team at the Social Media Mining for Health Applications (SMM4H) 2021 shared task. Our team participated in task 5 and 6, both focused on the automatic classification of Twitter posts related to COVID-19. Task 5 was oriented on solving a binary classification problem, trying to identify self-reporting tweets of potential cases of COVID-19. Task 6 objective was to classify tweets containing COVID-19 symptoms. For both tasks we used models based on bidirectional encoder representations from transformers (BERT). Our objective was to determine if a model pretrained on a corpus in the domain of interest can outperform one trained on a much larger general domain corpus. Our F1 results were encouraging, 0.77 and 0.95 for task 5 and 6 respectively, having achieved the highest score among all the participants in the latter.

## introduction the social media mining for health applications (smm4h) 2021 shared task aimed to address the challenges presented in natural language processing (nlp) applied to text obtained from social networks, specifically twitter, to gain medical insights @xcite . this year's smm4h proposed 8 different problems that involved classification and named entity recognition (ner) tasks. our team focused on tasks 5 and 6, both dealing with classification of covid-19 related tweets in different situations. we decided to approach this problem with transformer-based models @xcite , since they are considered state-of-theart in many nlp applications. also, we hypothesized that a model trained on domain specific texts could performed better than one trained in a much larger but general-domain corpus. to test our hypothesis, we implemented two models that share the same architecture but were trained on different data sets; on the one hand, the large uncased version of bert (bert-large) @xcite , which is a model pretrained on a very large corpus (wikipedia), and, on the other hand, ct-bert that is a model based on bert-large but pretrained on a smaller corpus of covid-19 related tweets @xcite . ## our approach for both tasks we implemented models based on the deep neural transformer architecture @xcite , since they have achieved state-ofthe-art (sota) results in several nlp tasks. we studied the impact of fine-tuned bert large and ct-bert pretrained models @xcite ) @xcite . for both models we employed the pytorch implementation available from the huggingface library @xcite . ## experimental setup in this section we describe the training process for the two models in both tasks. ## conclusions and future work transfer learning has shown to achieve above average results for various nlp tasks, where domain specific models can attain better results even if they were trained with less data than a general domain model. based on the results obtained, we conclude that a model trained with quality domain-specific data (ct-bert) can outperform a model trained with a much larger amount of general domain data (bert). in the experimental stage we also considered the biobert model, which was trained on a medical corpus @xcite , but it was not possible due to time constraints. thus, we envision a potential future work to further compare the reach of several domain-specific models in the prediction of social media posts that could embed covid-19 infection risk information. based on the error analysis, we plan to further improve our models' performance by considering wide & deep learning techniques @xcite , which help to enhance their generalization ability by adding other types of features through the wide branch.