Classification of Tweets Self-reporting Adverse Pregnancy Outcomes and Potential COVID -19 Cases Using R o BERT a Transformers (2021)

Abstract: This study describes our proposed model design for SMM4H 2021 shared tasks. We fine-tune the language model of RoBERTa transformers and their connecting classifier to complete the classification tasks of tweets for adverse pregnancy outcomes (Task 4) and potential COVID-19 cases (Task 5). The evaluation metric is F1-score of the positive class for both tasks. For Task 4, our best score of 0.93 exceeded the mean score of 0.925. For Task 5, our best of 0.75 exceeded the mean score of 0.745.

## introduction the social media mining for health application (smm4h) shared tasks involve natural language processing challenges using social media data for health research. we participated in the smm4h 2021 task 4 @xcite , focusing on automatically distinguishing tweets that selfreport a personal experience of an adverse pregnancy including miscarriage, stillbirth, preterm birth, low birthweight, and neonatal intensive care (annotated as "1") from those that do not (annotated as "0"). this task is a follow-up to smm4h 2020 task 5, which involves three classes of tweets that mention birth defects. we also participated in smm4h 2021 task 5. this new binary classification task involves automatically distinguishing tweets that self-report potential cases of covid-19 (annotated as "1") from those that do not (annotated as "0"). potential cases includes those tweets indicate the user or a member of the user's household was denied testing for, was symptomatic of, was directly exposed to presumptive or confirmed covid-19 cases, or had experiences that pose a higher risk of exposure to covid-19. other tweets related to covid-19 may discuss topics such as testing, symptom, traveling, or social distancing, but do not indicate someone may be infected. this paper describes the ncuee-nlp (national central university, dept. of electrical engineering, natural language processing lab) system for the smm4h 2021 task 4 and task 5. our solution explores how to use the roberta transformers @xcite with involved language models and classifier fine-tuning to predict tweet classes. the evaluation metrics of both tasks are f1-score for the positive class (i.e., tweets annotated as "1"). for task 4, our best score of 0.93 exceeded the median score of 0.925. for task 5, we achieved a best score of 0.75 exceeding the median score of 0.745. the rest of this paper is organized as follows. section 2 investigates the related studies. section 3 describes the ncuee-nlp system for the tweet classification tasks. section 4 presents the evaluation results and performance comparisons. conclusions are finally drawn in section 5. ## related work our participated smm4h 2021 task 4 is a follow-up to smm4h 2020 task 5, which focused on detecting tweets that mention birth defects. a hard-voting ensemble of nine biobert-based models was used to achieve a higher macroaveraging recall @xcite . the elmo word embeddings and data-specific resources were adopted to achieve a higher macro-averaging precision @xcite . ensemble bert flavors were studied to detect tweets that mention birth defects @xcite . two-views based cnn-bigru networks ## the ncuee-nlp system figure 1 shows our ncuee-nlp system architecture for the smm4h 2021 shared tasks. specially, our system is composed of two main parts: roberta transformers and fine-tuning. roberta (a robust optimized bert pretraining approach) @xcite is a replication study of bert pretraining @xcite that carefully measures the impact of key parameters and training data size. we observe that roberta transformers have usually performed well for many smm4h 2020 tweet classification tasks @xcite . hence, we explore the usage of roberta transformers and fine-tune the downstream tasks. for task 4, we use training, validation, and test datasets provided by task organizers to fine-tune the language model to improve the embedding representation. then, the tweets with class labels in the training dataset were used to fine-tune the classifier. ## evaluation the experimental datasets were mainly provided by task organizers @xcite . for task 4, we have a total of 5,514 tweets in the training set, including 2,484 positive tweets and 3,030 negative tweets. the validation set contains 973 tweets (438 positive and 535 negative). finally, there are a total of 10,000 tweets in the test set. for task 5, we have 6,465 tweets (1,026 positive and 5,439 negative) in the training set. the validation set contains 716 tweets (122 positive and 594 negative). finally, there are 10,000 tweets in the test set. we also have a total of 16,067 tweets from task 6 for fine-tuning the language model. all tweets were pre-processed to convert emojis into the corresponding codes defined by the unicode consortium. the pre-trained roberta-large model was downloaded from huggingface @xcite . the hyper-parameters used for both tasks are as follows: training batch size 64, learning rate 4e-5, and maximum sequence length 128. tables 1 and 2 respectively summarize the results for tasks 4 and 5. the evaluation metric is the f1-score of the positive class for both tasks. it's obvious that we have consistent results for both tasks, with a performance boost coming from finetuning the language model. our best results for both tasks slightly exceeded than the respective median scores of all submissions by 0.005. ## conclusions this study describes the ncuee-nlp system participating in smm4h 2021 task 4 for adverse pregnancy outcome and task 5 for potential covid-19 cases, including system design, implementation and evaluation. for task 4, our best f1-score of 0.93 exceeded the median score of 0.925. for task 5, our best f1-score of 0.73 exceeded the median score of 0.725.