Automatic Detecting for Health-related T witter Data with B io BERT (2020)

Abstract: Social media used for health applications usually contains a large amount of data posted by users, which brings various challenges to NLP, such as spoken language, spelling errors, novel/creative phrases, etc. In this paper, we describe our system submitted to SMM4H 2020: Social Media Mining for Health Applications Shared Task which consists of five sub-tasks. We participate in subtask 1, subtask 2-English, and subtask 5. Our final submitted approach is an ensemble of various fine-tuned transformer-based models. We illustrate that these approaches perform well in imbalanced datasets (For example, the class ratio is 1:10 in subtask 2), but our model performance is not good in extremely imbalanced datasets (For example, the class ratio is 1:400 in subtask 1). Finally, in subtask 1, our result is lower than the average score, in subtask 2-English, our result is higher than the average score, and in subtask 5, our result achieves the highest score. The code is available online.

## introduction according to the united states centers for disease control and prevention (cdc), drugs administered for alleviating common sufferings are the fourth biggest cause of death and birth defects are the leading cause of infant mortality. these are the most important medical problems for human society. @xcite twitter, a popular micro-blogging service, has received much attention recently. it is an online network used by millions of people around the world to stay connected to their friends, family members, and coworkers through their computers and mobile telephones @xcite . on average, one in a thousand messages from public twitter data is health-related. these health-related twitter posts can help us analyze various human health-related phenomena. for example, there are limited methods for studying birth defects in infants, and this knowledge has been challenging @xcite . this situation provides a challenging opportunity due to the increasing number of related tweets on twitter. with this motivation, five shared tasks are conducted as part of the social media mining for health applications (smm4h) workshop 2020 hosted by the university of pennsylvania health language processing (hlp) lab. our team participated in subtasks 1, 2-english, and 5 of the workshop. some samples from the training set are given in sub-task sentence gold label sub-task 1 @username you can try vitamins for olivia! hudson has taken vitamins since the nicu &amp; it helps him gain weight 1 can someone get me some chippy chips please 0 sub-task 2 english today i'm an emotional mess. that's what happens when i think i could wean myself off of cymbalta 1 @username. prozac makes us all better people. #prozacnation 0 sub-task 5 @uesername, my baby was born with microcephaly in the us due to #cytomegalovirus not #zika. why no warning about that? #cdcchat" 1 "username @usrename in a way, mine are advantaged by the older having down's syndrome. i can recommend it!" 2 girl born without an ear can finally hear properly for the first time after surgeons rebuild it 3 we try a variety of approaches on these tasks, including classical machine learning methods, cnn models, rnn models, and transformer-based models. we find that the transformer-based neural models consistently outperform other methods. the framework of our implementation is shown in figure 1 . firstly, we combine the official training set and the validation set to get the new data set, which is split into the new training set and the validation set by using the stratified 5-fold cross-validation foot_1 . secondly, the test set is predicted by fine tuning the model. thirdly, we create pseudo-label to combine training set and input these data into model training and prediction. finally, we get the final result by hard voting. 2 related work normally, data in the health field is difficult to obtain. fortunately, the existence of social media such as twitter has alleviated this situation. however, it has caused problems that the data is difficult to handle. ## conclusion in this work, our method is an ensemble of various fine-tuned transformer-based models on these tasks. we obtain decent results for these tasks organized as a shared task in social media mining for health workshop -2020. our biggest regret in this work is that in extremely imbalanced data sets, we have not done too much processing on the data sets and can't achieve promising results. in the future, our work will focus on solving the problem of extremely imbalanced data. the code is available online. 3