OFU @ SMM 4 H ’22: Mining Advent Drug Events Using Pretrained Language Models (2022)

Abstract: We describe in this paper our proposed systems for the Social Media Mining for Health 2022 shared task 1. In particular, we participated in the three sub-tasks, tasks that aim at extracting and processing Adverse Drug Events. We investigate different transformer-based pretrained models we fine-tuned on each task and proposed some improvement on the task of entity normalization.

## introduction social media platforms became a compulsory source of information thanks to its immediacy and accessibility. the recent covid-19 global crisis has demonstrated the necessity to increase the research effort toward processing health-related information with the aim of infoveillance, pharmacovigilance @xcite and fighting missinformation @xcite . indeed, the abundance of data provided by social media poses new challenges which encouraged the natural language processing (nlp) community to design and adapt suitable text processing approaches. the social media mining for health (smm4h) shared tasks @xcite follow this initiative and propose several nlp tasks for health information monitoring. in particular, smm4h task 1 proposes to tackle the problem of mining adverse drug events (ades) in tweets using the annotated dataset proposed in @xcite . we describe in this paper the systems that solve the three subtasks we participated in: 1) binary ades tweets classification. 2) ade span detection in tweets and 3) ade mention normalization. ## motivation active research effort have been spent on learning language representation models from large generalpurpose corpora. in particular, transformers-based models such as bert @xcite , xl-net @xcite and roberta @xcite greatly contributed to boosting the performance of many natural language processing (nlp) applications. although, such models have strong generalization abilities, they need an additional domain-adaptation training step in order to be employed in supervised settings for downstream tasks. for example, in the biomedical domain, pretrained language models (lms) such as clinicalbert @xcite , biobert @xcite have been proposed. while these domain-adapted models unarguably outperform general ones, the performance gain may vary depending on a number of factors, including the nature of the downstream task, the proximity of the task domain to the domain-adaptation training corpus and the task dataset size @xcite . furthermore, the characteristics of social media posts may adversely affect the general performance, as they are often in the form of short noisy texts with an informal writing style that involves abbreviations and misspelled words. in this work, we evaluate different pretrained language models on the three aforementioned tasks, investigating how well they transfer to different information extraction tasks in the context of social media and pharmacovigilance. ## task1a: tweet classification in task1a, the objective is to classify tweets as ade or noade. the training dataset is imbalanced and contains 1711 tweets reporting ade and 15674 noade examples. previous work such as @xcite , proposed a text processing pipeline and bidirectional lstm to classify ade tweets. @xcite proposed a lexicon-based approach that identified keywords used as markers of whether a tweet contains an ade. @xcite relied on a bert architecture for tweet classification, while @xcite additionally showed that ulmfit @xcite produces descent performance. following @xcite , we first preprocessed tweets using the tweet-preprocessor package 1 to remove re-tweets, hashtags, emojis and urls. we investigate several transformerbased pretrained language models @xcite which we fine-tuned on ade tweet classification. for each model, we take the pooled output ( [cls] token representation) followed by a linear layer and a softmax function for classification. for all tasks, we used the following models: bert: we used the "base-uncased" bert pretrained on general-domain text corpora. biobert @xcite : it is pretrained on biomedical domain corpora (pubmed abstracts + pmc full-text articles). we experimented with the biobert-large v1.1 (pubmed 1m), a version based on bert-large-cased with a custom 30k vocabulary. xlnet: with a different architecture, training objectives and without sequence length limit, xl-net performs autoregressive (ar) and permutationbased language modelling (plm) to encode bidirectional contexts. @xcite showed xlnet performance improvements over bert on several benchmark datasets. we used the xlnet-base pretrained model version. roberta @xcite : similar to bert, roberta is trained on larger general-domain corpora without the next sentence prediction objective. it has been shown that it outperforms bert on several downstream tasks. ## task1b: ade span detection in tweets given a tweet, the goal is to detect a span that refers to an expressed ade. similar to @xcite @xcite , we formulated the span detection task as a sequence labeling problem. we automatically annotated the dataset with labels following the bio tagging scheme. hence, the beginning of ade mention is tagged with b-ade, the inside tokens of an ade mention are tagged with i-ade, and the rest of tokens are tagged with o (outside ade mention). our model relies mainly on a sequence encoder (bert, roberta, biobert), followed by a conditional random field (crf) layer to predict output labels. we additionally implement a bilstm + crf as a strong baseline for sequence labeling. we used the same hyper-parameter settings as for the classification task, and computed the f1 score of the correct detected spans. ## task1c: ade mention normalization in task1c, we map ade mentions to their standard concept ids in the medical dictionary for regulatory activities (meddra) vocabulary. like @xcite , we used meddra dictionary from sider, the database of drugs and side effects @xcite . we followed a metric learning approach to address the entity normalization task, where mentions are encoded using a neural network, and entity labels are represented using pretrained continuous vector representations. during training, these representations are optimized with the objective of minimizing the embedding distance between mention representations and their normalized entity representation according to a given metric. in this work, span mention representations are obtained by encoding their texts using one of the previously used encoders (bert, biobert, xl-net, roberta), while entities are encoded using word2vec @xcite representations obtained after training word embeddings on pubmed and pmc texts @xcite . we therefore average the word embeddings of the words composing entity labels to build vector representations of all entities in the meddra dictionary. during inference, we computed the cosine similarity between the vector representations of each de-tected span mention and all the entity vector representations in the meddra dictionary. finally, we normalize mentions following two retrieval methods: 1) the nearest neighbor (nn) retrieval method which consists in selecting the closest entity representation in the embedding space, according to the cosine similarity. 2) the cross-domain similarity local scaling (csls) @xcite , which is a similarity measure that computes the cosine similarity normalized with the mean cosine similarity of each entity vector representation with its k entity neighbors (see @xcite for more details). ## conclusion in this work, we explored different transformerbased pretrained models in a transfer learning setting for several information extraction tasks. our approaches proposed to solve the three subtasks of smm4h shared task 1, by exploring the best transfer pretrained model. the results we obtained are different from our expectations since domainspecific pretrained models such as biobert did not outperform general-domain on domain-specific downstream tasks. we suggested that generaldomain pretrained models have better generalization abilities to extract ade on social media texts. further experiment and analysis are planned in order to draw more meaningful conclusions. our results on the test set are showed in