How Far Can We Go with Just Out-of-the-box BERT Models? (2020)

Abstract: Social media have been seen as a promising data source for pharmacovigilance. Howev-er, methods for automatic extraction of Adverse Drug Reactions from social media plat-forms such as Twitter still need further development before they can be included reliably in routine pharmacovigilance practices. As the Bidirectional Encoder Representations from Transformer (BERT) models have shown great performance in many major NLP tasks recently, we decided to test its performance on the SMM4H Shared Tasks 1 to 3, by submitting results of pretrained and fine-tuned BERT models without more added knowledge than the one carried in the training datasets and additional datasets. Our three submissions all ended up above average over all teams’ submissions: 0.766 F1 for task 1 (15% above the average of 0.665), 0.47 F1 for task 2 (2% above the average of 0.46) and 0.380 F1 score for task 3 (30% above the average of 0.292). Used in many of the high-ranking submission in the 2019 edition of the SMM4H Shared Task, BERT contin-ues to be state-of-the-art in ADR extraction for Twitter data.

## introduction any medicinal product that has an effect, has the potential of causing harmful effects, in some individuals under some circumstances @xcite . defined as 'a response to a drug that is noxious and unintended and occurs at doses normally used in man for the prophylaxis, diagnosis or therapy of disease, or for modification of physiological function' (who meeting on international drug monitoring: the role of national centres, 1972), adverse drug reactions (adrs) are a public health concern and have been identified as the 5th leading cause of deaths within the european union, with an estimated rate of 197,000 deaths per year and a cost of 79 billion euros within eu per year @xcite . monitoring the safety of medicinal products over time, as they enter the market and get used in much more heterogenous populations than the clinical trials in which their safety was primarily assessed, is therefore essential for public health and is the goal of pharmacovigilance. traditionally, pharmacovigilance has relied on spontaneous reporting systems such as faers in the us and vigibase, the who database if individual case safety reports, gathering reports of suspected adrs from more than 130 national spontaneous reporting systems, including faers @xcite . although these systems are particularly effective for detecting rare and serious adrs, they suffer from limitations, notably under-reporting @xcite . the appearance of social media platforms such as twitter has provided pharmacovigilance with new data sources which could potentially complement spontaneous reports by the breadth of coverage of the populations @xcite . twitter alone was boasting 321 million active users as of february 2019 and thus could partially address the under-reporting problem spontaneous report systems suffer from. nevertheless, it is yet unclear how social media could be integrated meaningfully into pharmacovigilance activities. the 2015-launched research consortium web-radr @xcite has concluded that, under the current performance of methods for detection and extractions of adrs, twitter has limited value for pharmacovigilance @xcite . although, the methods they have employed are not today's state-of-the-art nlp methods @xcite , it shows there is a need for developing performant algorithms for automatic detection, extraction and characterization of adrs and their associated medicinal products. to stimulate the development of such methods, the social media mining for health (smm4h) has launched several shared tasks over the years, with focus on specific aspects of the adr extraction problem. this year, we participated in task 1 (binary classification for tweets containing a drug name or a dietary supplement), task 2 (binary classification for tweets containing an adr mention) and task 3 (ner for adr mentions and normalization to meddra® the medical dictionary for regulatory activities terminology is the international medical terminology developed under the auspices of the international council for harmonisation of technical requirements for pharmaceuticals for human use (ich)). as the bidirectional encoder representations from transformer (bert) @xcite was proven to be powerful in the 2019 smm4h shared task @xcite , we decided to apply pre-trained bert models everywhere, with some fine-tuning to the tasks data, to see how this simple approach, requiring no domain expertise, would lead us. in this document, we report on the results of this experiment.