Exploring a Unified S equence- T o- S equence T ransformer for Medical Product Safety Monitoring in Social Media (2021)

Abstract: Adverse Events (AE) are harmful events resulting from the use of medical products. Although social media may be crucial for early AE detection, the sheer scale of this data makes it logistically intractable to analyze using human agents, with NLP representing the only low-cost and scalable alternative. In this paper, we frame AE Detection and Extraction as a sequence-to-sequence problem using the T5 model architecture and achieve strong performance improvements over the baselines on several English benchmarks (F1 = 0.71, 12.7% relative improvement for AE Detection; Strict F1 = 0.713, 12.4% relative improvement for AE Extraction). Motivated by the strong commonalities between AE tasks, the class imbalance in AE benchmarks, and the linguistic and structural variety typical of social media texts, we propose a new strategy for multi-task training that accounts, at the same time, for task and dataset characteristics. Our approach increases model robustness, leading to further performance gains. Finally, our framework shows some language transfer capabilities, obtaining higher performance than Multilingual BERT in zero-shot learning on French data.

## introduction before market release, drugs are regularly tested for safety and effectiveness in clinical trials. however, since no clinical trial is large enough to find all potential adverse events (aes) on a wide and diverse range of population, pharmacovigilance continuously monitors the market to timely intervene, in case unexpected aes are discovered. ## related work early efforts in automated pharmacovigilance have targeted electronic health records (ehr) to detect evidence of aes @xcite . however, not all aes lead to clinical visitations: many users prefer to discuss their experiences with drugs on the internet, and this fact led to a growing interest in the automatic detection of adverse events from social media platforms. some of the early machine learning systems for ae detection from social media data used a combination of various classifiers along with word embeddings as features @xcite @xcite . after the introduction of challenges such as social media mining for health applications (smm4h) @xcite and cadec @xcite , most works focused on neural networks @xcite . with the development of attention mechanism @xcite , transformerbased language models such as bert @xcite and its biomedical (e.g., biobert @xcite , clinicalbert @xcite and pubmedbert @xcite ) and non-biomedical variants (e.g., spanbert @xcite ) obtained state-of-the-art performance in ae detection @xcite @xcite . models like bert and its variants can be described as encoder-only: in order to carry out a specific task, a decoder has to be followed by taskspecific trainable network, most often in the form of a linear layer. recent developments in nlp led to the introduction of models such as t5 @xcite , which is an encoder-decoder transformer architecture. in a series of studies, t5 and its variants have shown performance gain on various datasets and applications @xcite , despite being smaller in terms of parameters. the prefix training approach adopted by t5 allows users to fine-tune on various tasks concurrently, creating a single model that can incrementally learn while being capable of performing different tasks simultaneously. to our knowledge, the present contribution is the first to frame ae detection and extraction as generative problems. ## experimental settings general figures for all the datasets are reported in ## conclusions in order to address several typical challenges of the healthcare domain (small, imbalanced and highly variable datasets, cross-lingual data), we proposed to treat ae detection and ae/drug/dosage ex-traction tasks as sequence-to-sequence problems, adapting the t5 architecture and improving over all the baselines in both the detection and the extraction tasks. to maximize the benefit of multi-task and multi-dataset learning, we introduced a new training strategy that extends @xcite , showing that our approach accounts for multiple and diverse datasets and leads to consistent improvements over the original t5 proposal. finally, the model also shows some language transfer abilities in the zero shot setting, leaving the door open for future experiments to extend our training framework towards multilinguality @xcite .