C omp L x@ SMM 4 H ’22: In-domain pretrained language models for detection of adverse drug reaction mentions in E nglish tweets (2022)

Abstract: The paper describes the system that team CompLx developed for sub-task 1a of the Social Media Mining for Health 2022 (#SMM4H) Shared Task. We finetune a RoBERTa model, a pretrained, transformer-based language model, on a provided dataset to classify English tweets for mentions of Adverse Drug Reactions (ADRs), i.e. negative side effects related to medication intake. With only a simple finetuning, our approach achieves competitive results, significantly outperforming the average score across submitted systems. We make the model checkpoints and code publicly available. We also create a web application to provide a user-friendly, readily accessible interface for anyone interested in exploring the model’s capabilities.

## introduction the shared task @xcite of the 2022 social media mining for health applications (#smm4h) workshop proposed ten sub-tasks in the domain of social media mining for health monitoring and surveillance. from the perspective of natural language processing (nlp), these tasks present a considerable challenge since the nature of social media posts requires dealing with both a significant level of language variation (informal and colloquial expressions, ambiguity, multilingual posts) and data sparsity, as well as a widespread presence of noise such as misspellings of clinical concepts and syntactic errors. in the 2022 instantiation of the #smm4h shared task, our team participated in: (i) sub-task 1a, the classification of english tweets containing mentions of adverse drug reactions (adrs) @xcite , (ii) sub-task 3, the classification of english tweets (3a) and webmd reviews (3b) contain-ing mentions of changes in medication treatments, and (iii) sub-task 8, the classification of english tweets self-reporting chronic stress. in this paper we primarily describe our approach for task 1a, as that constituted the major focus of our efforts. to address these challenges, we finetune a variant of a roberta @xcite model, a transformer-based @xcite language model pretrained on approximately 128 million tweets @xcite on each sub-task's provided dataset. without any domain adaptation efforts (apart from standard finetuning on the downstream task) or hyperparameter optimizations, the model outperforms the average of all submissions for sub-task 1a by a 9% absolute difference in f1score. in the following sections, we introduce the subtasks' datasets, describe the model architecture and training setup, report our results, and conclude with a discussion of related research and potential avenues for future work. ## datasets in section [1] we provided a brief summary of each sub-task in which we participated. for each of them, participants were given access to a labeled training and validation set, as well as an unlabeled evaluation set that was used to determined the final performance of the submitted systems. ## experiments and results in this section, we give a brief description of the system we used to conduct our experiments, share our results and provide a brief discussion. ## conclusion and future directions we demonstrated that a roberta model @xcite pretrained on approximately 128 million tweets performs very competitively when finetuned on english tweet classification for adrs. using only a standard finetuning approach, our model obtained competitive results, outperforming the average of all submissions for sub-task 1 by a 9% absolute difference in f1-score. this constitutes yet another testament of the fact that large pre-trained language models have rightfully become the default approach in virtually all nlp tasks. with respect to potential future work, there is a large collection of available options. text classification and, more generally, binary classification is one of the oldest and most widely researched topics in nlp. most approaches aiming to improve performance of classification models can be broadly categorized into three groups, depending on the segment of the machine learning workflow that they are targeting. data augmentation methods typically target the initial part of the workflow, the data, aiming to increase the quantity, quality and diversity of the training dataset to ensure that model performance is robust to small syntactic or semantic perturbations in the inputs. transformations acting directly on strings, such random token insertions or deletions, synonym/antonym replacements and related techniques @xcite have shown significant performance improvements, especially in lowresource scenarios much like the one in this shared task. a second approach, evidently a natural extension of the previous technique, would be to target vector encodings of the tokens and/or documents that are produced by the various layers of the neural networks. we can distinguish two different approaches here: (i) improve the language model backbone during the pretraining phase, or (ii) improve the weights of the language model backbone during finetuning. the research community has devoted intensive efforts in the former approach, as can be observed by the ever-increasing list of transformerbased pretrained language models @xcite @xcite @xcite , inter multi alia) released. model size, in terms of total number of trainable parameters, has been consistently shown to correlate strongly with downstream performance, so opting for a larger pretrained model would be a reasonable first steps towards more transferable vector representations (and hence improved performance) in the downstream task. the latter approach would include domain adaptation techniques, such as continued self-supervised pretraining followed by supervised finetuning, which has been shown @xcite to consistently lead to superior results relative to direct finetuning. finally, one could aim to improve performance by modifying aspects of the objective function. @xcite , in an extensive series of experiments, show that the established practice of using a cross-entropy loss for classification is not well-founded and show through a variety of diverse experiments that a square loss can, in many cases, significantly improve performance.