KFU NLP Team at SMM 4 H 2021 Tasks: Cross-lingual and Cross-modal BERT -based Models for Adverse Drug Effects (2021)

Abstract: This paper describes neural models developed for the Social Media Mining for Health (SMM4H) 2021 Shared Task. We participated in two tasks on classification of tweets that mention an adverse drug effect (ADE) (Tasks 1a & 2) and two tasks on extraction of ADE concepts (Tasks 1b & 1c). For classification, we investigate the impact of joint use of BERTbased language models and drug embeddings obtained by chemical structure BERT-based encoder. The BERT-based multimodal models ranked first and second on classification of Russian (Task 2) and English tweets (Task 1a) with the F1 scores of 57% and 61%, respectively. For Task 1b and 1c, we utilized the previous year’s best solution based on the EnDR-BERT model with additional corpora. Our model achieved the best results in Task 1c, obtaining an F1 of 29%.

## introduction text classification, named entity recognition, and medical concept normalization in free-form texts are crucial steps in every text-mining pipeline. here we focus on discovering adverse drug effects (ade) concepts in twitter messages as part of the social media mining for health (smm4h) 2021 shared task @xcite . this work is based on the participation of our team in four subtasks of two tasks. task 1 consists of three subtasks, namely 1a, 1b, and 1c each of which corresponds to classification, extraction, and normalization of ades. for task 2, train, dev, and test sets include russian tweets annotated with a binary label indicating the presence or absence of ades. for the 1b task, named entity recognition aims to detect the mentions of ades. task 1c is designed as an end-to-end problem, intended to perform full evaluation of a system operating in real conditions: given a set of raw tweets, the system has to find the tweets that are mentioning ades, find the spans of the ades, and normalize them with respect to a given knowledge base (kb). these tasks are especially challenging due to specific characteristics of user-generated texts from social networks which are noisy, containing misspelled words, abbreviations, emojis, etc. the source code for our models is freely available foot_0 . the paper is organized as follows. we describe our experiments on the multilingual and multimodal classification of russian and english tweets for the presence or absence of adverse effects in section 2. in section 3, we describe our pipeline for named entity recognition (ner) and medical concept normalization (mcn). finally, we conclude this paper in section 4. ## tasks 1a & 2: multilingual classification of tweets the objective of tasks 1a & 2 is to identify whether a tweet in english (task 1a) or russian (task 2) mentions an adverse drug effect. ## tasks 1b & 1c: extraction and normalization of ades the 1b task's objective is to detect ade mentions. task 1c is designed as an end-to-end task. systems have a free-form tweet as input and should be able to produce a set of extracted medical concepts. for this task, we develop a pipeline that (i) first detect ade mentions and then (ii) link extracted ades to the concepts from the medical dictionary for regulatory activities (meddra) @xcite . following the best results in smm4h 2020 task 3 @xcite , we utilize a endr-bert model 5 with dictionary based features for the named entity recognition (ner) task. we adopted the dictionaries from @xcite . as in the best solution of the smm4h 2020 task 3, we adopted extra training data for the ner task, we used the csiro adverse drug event corpus (cadec) @xcite and cometa 5 https://huggingface.co/cimm-kzn/endr-bert corpus @xcite . for the normalization task, we applied two models: (i) a classifier @xcite , (ii) a novel neural model based on similarity distance of bert vectors of concepts @xcite . following @xcite , we utilize additional data for training. other corpora are filtered to match a vocabulary of the smm4h 2021 train set. we combined two models based on a threshold. for instance, given (i) prediction c bs from from bert-based similarity method with the distance equals to d and (ii) prediction c clf from the classification approach, the final prediction is set to c bs , if d is less than a threshold, and to c clf , otherwise. for more detailed description of ner and end-to-end entity linking model please refer to @xcite . ## conclusion in this work, we have explored an application of domain-specific bert models pretrained on healthrelated user reviews in english and russian to the task of multilingual and multimodal text classification, extraction, and normalization of adverse drug effects. our experiments show that multimodal architecture for classification of tweets outperforms other strong baselines and text classifiers. besides, our bert-based pipeline for extraction on med-dra concepts ranked 1st in task 1c. we foresee two directions for future work. first, future research will explore how different drug representation models and pretraining approaches affect classification performance. second, a potential direction is to verify the efficacy of multimodal classification for languages other than russian and english.