Autobots Ensemble: Identifying and Extracting Adverse Drug Reaction from Tweets Using Transformer Based Pipelines (2020)

Abstract: This paper details a system designed for Social Media Mining for Health Applications (SMM4H) Shared Task 2020. We specifically describe the systems designed to solve task 2: Automatic classification of multilingual tweets that report adverse effects, and task 3: Automatic extraction and normalization of adverse effects in English tweets. Fine tuning RoBERTa large for classifying English tweets enables us to achieve a F1 score of 56%, which is an increase of +10% compared to the average F1 score for all the submissions. Using BERT based NER and question answering, we are able to achieve a F1 score of 57.6% for extracting adverse reaction mentions from tweets, which is an increase of +1.2% compared to the average F1 score for all the submissions.

## introduction with the world adapting to the new normal, social media is proving to be a key resource for humans. with more people sharing their life experiences in social media platforms, pharmaceutical firms can benefit by leveraging the power of deep learning and natural language processing for digital pharmacovigilance. in this paper, we showcase our systems for task 2 & 3 of the social media mining for health applications shared task 2020 @xcite . inspired by the current research using transformer architectures, and the results that kfu nlp team @xcite had achieved at smm4h 2019 using bert @xcite , we experimented with a suite of different transformer architectures. transformers @xcite are solely based on attention mechanisms, which dispense recurrence and convolutions entirely, enabling parallel processing and state of the art models. @xcite in their research uncovered that bert was significantly under trained, and proposed roberta: a robustly optimized bert pretraining approach. we fine tuned roberta on the english training tweets to classify a tweet as containing adverse reaction mention or not. for the russian and french tweets classification tasks, we fine tuned rubert @xcite and camembert @xcite respectively. for extracting the adverse reaction mentions from a tweet, we devised an end to end pipeline by posing the task as a named entity recognition (ner) task as well as a question answering task. we fine tuned bert, scibert @xcite and biobert @xcite and created an ensemble ner, and fine tuned biobert qa for the question answering module. post adverse mention extraction, we normalised the mention to the meddra code using pre-trained fasttext @xcite embeddings and cosine similarity. the paper is organized as follows. in section 2 we describe the problem statements. section 3 describes the architectures and methods that were implemented for each of the tasks, and showcase our results in section 4. we discuss some of the challenging aspects of the problems in section 5, and finally conclude in section 6. 2 task and data description ## results strict and relaxed f1 scores are used for evaluating the models. under strict mode of evaluation, adr spans are considered correct only if both start and end indices matches with the indices in the gold standard annotations. under relaxed mode of evaluation, adr spans are considered correct only if spans in predicted annotations overlapped with the gold standard annotations. in our system, this leads to significant differences between the two f1 scores. our roberta based english tweet classifier for task 2 outperforms most systems, and achieves a test f1 score of 0.56. with the multi staged adverse reaction extraction pipeline, we are able to achieve a relaxed f1 score of 0.576, which is above the average f1 of all the other submitted systems. unfortunately, due to highly imbalanced training data, our french tweet classifier is not optimally trained, and fails to correctly identify any french tweets containing adverse reaction mentions. below we summarize the performance of our systems on all the different tasks. ## discussion as represented in ## conclusion in this work, we have experimented with different transformer models for classifying adr tweets as well as extracting adr terms. we have leveraged several transformer based pre-trained models like roberta, biobert etc. also, we have devised a multi step pipeline for extracting the adr terms from a tweet. we can immediately think of two future improvements, firstly for the non english tasks in task 2 we can develop a translation model to translate the tweets into english before classifying using the task 2 english tweet classifier. secondly, bettering the ner and meddra mapping, we want to incorporate a model that will be jointly trained to perform multiple tasks. for example, given a text, the model should be able to extract the adr extracts as well as classify the tweet as adr or non-adr, as well as map it to the correct meddra code. also, we will add a relationship extraction task, where we will identify the relation between the drug and adr. we hypothesize that such a model should outperform a standard model as it will incorporate features and information sharing across tasks. for example, the ner would make a less false positive classification for non-adr tweets.