Transformer-based Multi-Task Learning for Adverse Effect Mention Analysis in Tweets (2021)

Abstract: This paper presents our contribution to the Social Media Mining for Health Applications Shared Task 2021. We addressed all the three subtasks of Task 1: Subtask A (classification of tweets containing adverse effects), Subtask B (extraction of text spans containing adverse effects) and Subtask C (adverse effects resolution). We explored various pre-trained transformer-based language models and we focused on a multi-task training architecture. For the first subtask, we also applied adversarial augmentation techniques and we formed model ensembles in order to improve the robustness of the prediction. Our system ranked first at Subtask B with 0.51 F1 score, 0.514 precision and 0.514 recall. For Subtask A we obtained 0.44 F1 score, 0.49 precision and 0.39 recall and for Subtask C we obtained 0.16 F1 score with 0.16 precision and 0.17 recall.

## introduction information extraction from social media is widely studied nowadays, as platforms like facebook, twitter, instagram, or reddit become the main place for people to share their opinions and experiences. concurrently, a wide range of applications with completely different topics arises with the current advances in natural language processing (nlp), as the volume of posted information has become impossible to be manually analysed. for example, the social media mining for health (smm4h) applications shared task @xcite is focused on health applications and introduces a dataset of annotated tweets with the aim to analyse adverse drug ef-fects (ade) mentioned by users. this year's edition @xcite proposed eight different tasks, out of which we focused on the first task entitled classification, extraction and normalization of adverse effect mentions in english tweets. this task was further divided into three subtasks, as follows. subtask 1a was a binary classification task of tweets, focused on identifying whether the message contains ade or not. subtask 1b was a named entity recognition task on top of subtask 1a, centered on extracting the span of text containing the ade of the medication. subtask 1c was a named entity resolution task on top of both previous subtasks, aimed at predicting the normalized concept of the extracted adverse effect from the preferred terms included in the medical dictionary for regulatory activities (meddra) 1 . all three subtasks are addressed simultaneously using a multi-task learning (mtl) architecture @xcite that leverages acquired knowledge from one subtask to another. furthermore, we approached the challenge of unbalanced classes in the first subtask by considering class weights and by augmenting the training data set. the paper is structured as follows. the second section describes previous work that inspired our solution, while the third section presents our employed method. the fourth section presents the results of our work, followed by discussions and the final section that summarizes our findings and presents future research paths. ## results four configurations were compared in terms of performance. the first configuration (mtl) is a baseline relying on the previously described mtl architecture. weighted binary cross-entropy loss and feedback from the extractor to the normalizer (line 12 from algorithm 2) are enabled, but the normalizer uses only the ade span, without the entire tweet (line 10 from algorithm 2). the second configuration (mtl + boostin-gensemble) starts from mtl, but instead of the simple classifier model, it uses an ensemble of three models trained in a boosting manner. the first classifier (classifier1) is identical to the classifier from the first configuration. the second classifier (clas-sifier2) was trained on a modified training set in which the miss-classifies examples from classifier1 are over-sampled by a factor of three, whereas the correctly classified examples are down-sampled by the same factor. the third classifier, classifier3, is also trained on a modified training set in which examples with different results from classifier1 and classifier2 are over-sampled by a factor of three, while the rest of the examples are down-sampled by the same factor. the third configuration, denoted mtl + en-hancedensemble, further tries to improve the performance of the second configuration by adding two more classifiers to the ensemble, classifier4 and classifier5, trained now on the augmented training set while considering equivalent over-and down-sampling approaches. the fourth configuration, namely mtl + en-hancednormalizer, is similar to the first configuration, but with the normalizer is trained on both the ade span and the entire tweet. ## discussions another limitation of our method is highlighted in the second example. the meddra term of slurred speech is a rather rare label, not even present in the training set. even though our system correctly extracts the span containing the adverse effect, it is unable to correctly predict the ptid. the false positive example of "drunk" labeled as drunk like effect shows that our model finds it hard to discern appearances from facts. a similar bias can be observed in the third example, where the model fails to extract the spans "sleep" and "stomach is a cement mixer" most likely because it learned that interrogations ask about adverse effects rather than offer information about them. the fourth example denotes subtle errors, like grasping the difference between the meddra terms of sleepiness and somnolence, which are likely to be mislabeled even by humans. annotated sample model prediction meddra meddra ...trazodone, it takes the light right outta your eyes... ...trazodone, it takes the light right outta your eyes ... sleepiness one of the things i hate most about quetiapine is when i take it for the first few hours i slur my words, so people assume i'm merely drunk. slurred speech one of the things i hate most about quetiapine is when i take it for the first few hours i slur my words, so people assume i'm merely drunk . fluid retention, drunk-like effect ciprofloxacin: how do you expect to sleep when your stomach is a cement mixer ? sleeplessness, stomach perforation ciprofloxacin: how do you expect to sleep when your stomach is a cement mixer? just woke up. since i started on the higher dose of quetiapine i'm sleeping even more ...; i feel knackered when i wake . sleepiness, groggy on awakening just woke up. since i started on the higher dose of quetiapine i'm sleeping even more ...; i feel knackered when i wake. ## conclusions and future work we introduced a transformer-based multi-task learning architecture employed for task 1 from the social media mining for health applications shared task 2021. task 1 was concerned with the classification of tweets incorporating adverse effects of medication and, for the positive tweets, with the extraction and normalization of the adverse effects. we started from a pretrained domainspecific bert language model (i.e., biobert) which was further finetuned in a multi-task setting. a hard parameter sharing mtl model was trained on the three subtasks of smm4h task 1. furthermore, class weights and data augmentation were considered to overcome the problem of the unbalanced dataset from subtask 1a. our model achieved the highest score for subtask 1b (i.e., adverse effect span detection) with an f 1 -score of 51%, arguing that mtl can enhance adverse effect extraction from social media posts. in terms of future work, adversarial training @xcite will be considered to improve the robustness of our approach.