Transformer Models for Drug Adverse Effects Detection from Tweets (2020)

Abstract: In this paper we present the drug adverse effects detection system developed during our participation in the Social Media Mining for Health Applications Shared Task 2020. We experimented with transfer learning approach for English and Russian, BERT and RoBERTa architectures and several strategies for regression head composition. Our final submissions in both languages overcome average F1 by several percents margin.

## introduction in the world of pandemic threats it is important to pay a special attention to the process of drug discovery. for the pharmaceutical industry it was always important not only to develop a new drug, but also to detect its possible adverse effects (aes) even to smallest group populations as soon as possible. although early stages of a clinical trial reveal most of a drug aes, constant monitoring and feedback collection at the last pharmacovigilance stage is mandatory as it allows to identify rare or slowly evolving aes. with current development of social media networks and artificial intelligence methods it is tempting to mine this information from the publicly available user data such as twitter messages. if successful one can get an additional source of valuable information. the above stated problem got into the research focus of 2020 social media mining for health applications (smm4h) workshop @xcite . the organizers divided this problem into three consequent sub-tasks: 1) medical tweets detection; 2) classification of tweets that report aes and 3) extraction and normalization of aes from text. specifically the second task was to build a binary classification model able to distinguish tweets that report an ae of a medication from those that do not. we concentrate our efforts only on this task as it is the only one that offers multilingual version (english, french and russian). in the recent years, methods for natural language processing (nlp) developed rapidly. nowadays transformer-based neural architectures @xcite dominate the field. our goal for this study was to benchmark bert @xcite and roberta @xcite architectures and several strategies for regression head composition on this practical problem from medical domain with realworld data. ## data the organizers provided labeled data alone with the train / validation split. in our experiments we don't use the provided split, but join train and validation parts for each language and perform 5-fold cross-validation (cv) procedure @xcite . this allows us to estimate standard deviations for the models. ## our approach as the train data were very limited we resort to a transfer learning approach. that is, we take an nlp model pre-trained on a large corpus of texts (a backbone model) and fine-tune it for a specific task at hand. in this study as such backbone models for english we used bert-base and roberta-base from transformers library @xcite . for the russian bert model we fine-tuned rubert @xcite . there was no russian roberta model, so we trained one from scratch on the public data @xcite and made it available. foot_1 this model was trained with adam optimizer (starting learning rate 3 × 10 -6 ) on 16 gpu (tesla v100) and the batch size of 1024 (with gradient accumulation) for about 60k steps. originally bert and roberta architectures designed a special classification token (cls) to summarize the state of a whole input text sequence (into a single embedding) and allow to configure downstream tasks (thought classification or regression head). it is known that in nlp-related tasks max-pooling operation over the embeddings associated with sentence keyword selection and the mean-pooling is a way of encoding the general meaning of the sentence. so we experimented with this pooling strategies over the last encoder states. additionally the option with concatenation of embeddings (cls & max & mean) were explored. the resulting hidden state followed by additional fully connected layer of the same size and ends with regression output. based on cv metric maximization strategy the threshold were selected to make final binarization of predictions. ## results and conclusions the primary evaluation metric for the task was f 1 -measure toward the positive class as a trade-off between precision (p ) and recall (r) @xcite . the 5-fold cv results of our experiments are shown in first of all, for the english final submission we selected roberta model with (cls & max & mean) regression head. based on the test data it showed f 1 = 57% (@xmath0, @xmath1. that is better (by 11%) than average performing system f 1 = 46% (@xmath2, @xmath3. for the russian submission we selected bert model with mean regression head. in this case our system metrics is perfectly matched between cv and test, f 1 = 48% (@xmath4.1%, @xmath5.5%). which is also 5.3% improvement compared to average scores for this task f 1 = 42.7% (@xmath6.2%, @xmath7.3%). the results allow us to conclude that the drug adverse effects detection in raw text still remains challenging problem and requires more elaborate methods to reach the acceptable quality.