KFU NLP Team at SMM 4 H 2020 Tasks: Cross-lingual Transfer Learning with Pretrained Language Models for Drug Reactions (2020)

Abstract: This paper describes neural models developed for the Social Media Mining for Health (SMM4H) 2020 shared tasks. Specifically, we participated in two tasks. We investigate the use of a language representation model BERT pretrained on a large-scale corpus of 5 million health-related user reviews in English and Russian. The ensemble of neural networks for extraction and normalization of adverse drug reactions ranked first among 7 teams at the SMM4H 2020 Task 3 and obtained a relaxed F1 of 46%. The BERT-based multilingual model for classification of English and Russian tweets that report adverse reactions ranked second among 16 and 7 teams at two first subtasks of the SMM4H 2019 Task 2 and obtained a relaxed F1 of 58% on English tweets and 51% on Russian tweets.

## task 2: classification of multilingual tweets the goal of this task is to detect tweets that report an adverse effect of a medication. we present our results for russian and english subsets of tweets. the data for task 2 consists of 3 distinct sets of tweets posted in english, russian, and french. we present our results on two of them: russian and english with 20,544 and 6,090 tweets, respectively. the training sets are highly imbalanced as only 1,903 english and 533 russian tweets are labeled as positive examples. further, we refer to the union of two train sets as a bilingual train. we note that the dev sets are not used for training. ## task 3: extraction and normalization of adverse reactions this task's objective is to detect adr mentions and then map these entities to concepts in a controlled vocabulary. ## conclusion in this work, we have explored an application of domain-specific bert models pretrained on healthrelated user reviews in english and russian to the task of multilingual text classification and extraction and normalization of adverse drug reactions. experiments have shown that our bert models outperform general multilingual bert and biobert on two tasks, achieving the best results in task 3. there are several directions for future work arising from the results of our experiments. one potential direction is to investigate the impact of multilingual transfer learning from other corpora on the classification of russian health-related texts. another future direction is to verify the efficiency of english data for text classification in languages other than russian. for task 2, transfer from english to french remain to be explored.