Pre-trained Transformer-based Classification and Span Detection Models for Social Media Health Applications (2021)

Abstract: This paper describes our approach for six classification tasks (Tasks 1a, 3a, 3b, 4 and 5) and one span detection task (Task 1b) from the Social Media Mining for Health (SMM4H) 2021 shared tasks. We developed two separate systems for classification and span detection, both based on pre-trained Transformer-based models. In addition, we applied oversampling and classifier ensembling in the classification tasks. The results of our submissions are over the median scores in all tasks except for Task 1a. Furthermore, our model achieved first place in Task 4 and obtained a 7% higher F1-score than the median in Task 1b.

## introduction social media platforms such as twitter have been widely used to share experiences and health information such as adverse drug effects (ades), thus attracting an increasing number of researchers to conduct health-related research using this data. however, because social media data consists of user-generated content that is noisy and written in informal language, health language processing with social media data is still challenging. to promote the use of social media for health information extraction and analysis, the health language processing lab of the university of pennsylvania organized social media mining for health applications (smm4h) shared tasks. this year, the smm4h shared tasks included 8 subtasks @xcite . our team, the sarker lab at emory university, participated in six classification tasks (i.e., task 1a, 3a, 3b, 4, and 5) and one span detection task (i.e., task 1b) of the smm4h 2021 shared tasks. in recent years, transformer-based models such as bert @xcite and roberta @xcite , whose advantage is modeling of long-range context semantics, revolutionised the field of nlp and achieved state-of-the-art results in different nlp tasks. encouraged by those suc-cesses, we developed separate systems for classification and span detection both based on pre-trained transformer-based models. we experimented with different transformer-based model variants, and the model that achieved the best result on the validation set was selected as the final system. in addition, we performed undersampling and oversampling to address the problem of data imbalance and applied an ensemble technique in the classification tasks. the performances of our submissions are above the median f 1 -scores in all tasks except for task 1a. furthermore, our model achieved first place in task 4 and obtained a 7% higher f 1 -score than the median in task 1b. ## conclusion in this work, we developed two systems based on pre-trained transformer-based models for multiple health-related classification tasks and one span detection task for the smm4h 2021 shared tasks. we experimented with different transformer-based model variants as well as sampling strategies and applied an ensemble technique in the classification tasks. the results of our submissions are over the median f 1 -scores in all tasks except for task 1a. furthermore, our model achieved first place in task 4 and obtained a 7% higher f 1 -score than the median in task 1b. for future work, we will investigate methods to improve the generalizability of pre-trained transformer-based models to deal with various health-related expressions in social media data.