Approaching SMM 4 H 2020 with Ensembles of BERT Flavours (2020)

Abstract: This paper describes our solutions submitted to the Social Media Mining for Health Applications (#SMM4H) Shared Task 2020. We participated in the following tasks: Task 1 aimed at classifying if a tweet reports medications or not, Task 2 (only for the English dataset) aimed at discriminating if a tweet mentions adverse effects or not, and Task 5 aimed at recognizing if a tweet mentions birth defects or not. Our work focused on studying different neural network architectures based on various flavors of bidirectional Transformers (i.e., BERT), in the context of the previously mentioned classification tasks. For Task 1, we achieved an F1-score (70.5%) above the mean performance of the best scores made by all teams, whereas for Task 2, we obtained an F1-score of 37%. Also, we achieved a micro-averaged F1-score of 62% for Task 5.

## introduction in recent years, researchers around the world came to realize the usefulness of social media data for extracting health information. the social media mining for health applications (#smm4h) shared task 2020 @xcite brings to the forefront the problem of extracting information from health social media posts. smm4h was also organized in the previous years and involved several tasks, such as automatic detection of tweets mentioning medication, of tweets describing medication intake or adverse reactions, of tweets mentioning vaccination behaviour, as well as tasks on extraction and normalization of adverse effects utterances @xcite . this year's competition had five tasks and our team focused on tasks 1, 2, and 5. task 1 was a binary classification one that involved distinguishing between tweets in which some medications or dietary supplements were mentioned (the positive class) and other tweets (the negative class). the challenge of this task, unlike the 2018 similar task, consisted in the highly imbalanced data sets, that is, the tweets had a distribution of the two classes similar with the one encountered in practice. therefore, the positive class counted for only 0.2% of the examples. classification of multilingual tweets that report adverse effects (task 2) was also a binary classification task that was divided in multiple subtasks, each one concerning a different language. our team submitted solutions only for the english subtask. for both tasks 1 and 2, the evaluation metric was the f1-score for the positive class. on the other hand, classification of tweets reporting a birth defect pregnancy outcome (task 5) was a multi-class classification task with the following classes: defect, possible defect, and non-defect. for this task, the evaluation metric was the micro-averaged f1-score of the first two classes. we started to approach the competition by studying methods for data preprocessing and for overcoming the class unbalancing issue. afterwards, we tested multiple language models for classification and we contributed by further pre-training the best language model on social media data. moreover, we increased the robustness of our approach by constructing an ensemble which managed to obtain 70.5% f1-score, whereas the average of the best scores for the task 1 was 66.28%. this paper is further structured as follows. section 2 presents previous works that helped us in developing our solutions. section 3 describes the proposed models. in section 4, we show and interpret the results of the systems. finally, section 5 summarizes the conclusions of our work. ## 2 related work as related tasks were also organized in previous years of smm4h, a significant amount of work has been done in developing solutions to the proposed problems. therefore, previous studies already established some directions for appropriate methods to preprocess social media data, for practices to address imbalanced data sets, or for language models that are more effective for the given tasks. thus, @xcite showed which data preprocessing steps are more likely to obtain better results on tweets. the challenge of learning from imbalanced data sets has been reviewed by @xcite . their paper analyzed some general solutions like over-sampling and under-sampling and also offered useful guidelines in applying these techniques. moreover, @xcite approached the problem by assigning different weights to the imbalanced classes and this method proved particularly useful in our experiments. @xcite used transfer learning approaches, showing that bidirectional encoder representations from transforms (bert) @xcite and universal language model fine-tuning (ulmfit) @xcite are able to handle classification tasks in the medical domain. also, gondane (2019 ) leveraged the focus on biomedical language of bert for biomedical text mining (biobert) @xcite . the issue of detecting medication mentions in social media data has been previously approached with impressive results. @xcite showed that ensemble classifiers can achieve performance close to humans in recognizing mentions of medications in tweets, on balanced data sets. @xcite obtained notable results on social media data mentioning drugs using a neural network based on a multi-head self-attention mechanism. ## results in the practice phase, we submitted predictions on the validation sets for tasks 1 and 2 -english. for the first task, biobert-classweights achieved an f1-score of 67.6% with a precision of 69.6% and a recall of 65.7%, while biobert-pretraintweets obtained an f1-score of 77.61% with a precision of 81.2% and a recall of 74.2%. for task 2 -english, biobert-pretraintweets achieved 53.87% f1-score on the validation set. in the evaluation phase, we submitted three solutions for the first task: one prediction from biobert-pretraintweets and one prediction from each ensemble described above. the prediction of the ensemble 2 scored above the mean scores for task 1. we also submitted one solution for each task 2 -english and task 5. tables 1 , 2 , and 3 show the reported scores for each submission, alongside with the averaged score of best submissions of all teams that participated. the precision and recall for the first two submissions on the task 1 were not reported by the organizers. ## conclusion in this paper, we experimented with ensembles of bidirectional transformers in the context of social media texts and we studied the value that pre-trained bert flavours, like biobert or clinicalbert, bring in solving classification tasks in the medical domain. we succeeded in obtaining a score above the average of the best scores using ensemble 2 on task 1 and we showed that bert-based classifiers can give acceptable results even with highly unbalanced data sets. we also showed that a bert-based language model, pre-trained for a rather colloquial language, improved the results on the given tasks of social media data. our results on task 1 show that, in cases where one of the classes is poorly represented, ensembles increase the prediction performance. further experiments should consider including an enlargement of the corpus of tweets used for pretraining biobert, so that the model will be more capable of representing this type of data. the next step would be to design a new bert flavour pretrained on social media texts. another future direction in addressing class imbalance might consist in using data augmentation in order to generate examples of the less represented class @xcite .