BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection (2021)

Abstract: Pretrained transformer-based models, such as BERT and its variants, have become a common choice to obtain state-of-the-art performances in NLP tasks. In the identification of Adverse Drug Events (ADE) from social media texts, for example, BERT architectures rank first in the leaderboard. However, a systematic comparison between these models has not yet been done. In this paper, we aim at shedding light on the differences between their performance analyzing the results of 12 models, tested on two standard benchmarks. SpanBERT and PubMedBERT emerged as the best models in our evaluation: this result clearly shows that span-based pretraining gives a decisive advantage in the precise recognition of ADEs, and that in-domain language pretraining is particularly useful when the transformer model is trained just on biomedical text from scratch.

## introduction the identification of adverse drug events (ades) from text recently attracted a lot of attention in the nlp community. on the one hand, it represents a challenge even for the most advanced nlp technologies, since mentions of ades can be found in different varieties of online text and present unconventional linguistic features (they may involve specialized language, or consist of discontinuous spans of tokens etc.) @xcite . on the other hand, the task has an industrial application of primary importance in the field of digital pharmacovigilance @xcite . this raising interest is attested, for example, by the acl workshop series on social media health mining (smm4h), in which shared tasks on ade detection have been regularly organized since 2016 @xcite @xcite . with the recent introduction of transformers architectures and their impressive achievements in nlp @xcite , it is not surprising that these tools have become a common choice for the researchers working in the area. the contribution of this paper is a comparison between different transformers on ade detection, in order to understand which one is the most appropriate for tackling the task. shared tasks are not the best scenario for addressing this question, since the wide range of differences in the architectures (which could include, for example, ensembles of transformers and other types of networks) does not allow a comparison on the same grounds. in our view, two key questions deserve a particular attention in this evaluation. first, whether there is an advantage in using a model with some form of in-domain language pretraining, given the wide availability of transformers for the biomedical domain @xcite . second, whether a model trained to predict coherent spans of text instead of single words can achieve a better performance @xcite , since our goal is to identify the groups of tokens corresponding to ades as precisely as possible. two models that we introduce for the first time in this task, spanbert and pubmedbert, achieved the top performance. the former takes advantage of a span-based pretraining objective, while the latter shows that in-domain language data are better used for training the model from scratch, without any general-domain pretraining. ## evaluation the results for the two datasets are shown in for both models, the addition of crf generally determines a slight improvement on cadec, while it is detrimental on smm4h. on smm4h, the f1-scores of biobert, scibert and bio-clinicalbert consistently improve over the standard bert, but they are outperformed by its crfaugmented version, while on cadec they perform closely to the standard model. the results suggest that in-domain knowledge is consistently useful only when training is done on in-domain text from scratch, instead of using general domain text first. scibert is also trained from scratch, but on a corpus that is less specific for the biomedical domain than the pubmedbert one @xcite . the models are also being compared with tm-rleiden: we can notice that both versions of spanbert and pubmedbert outperform it on cadec (the differences are also statistically significant for the mcnemar test at @xmath0.001), while only the basic versions of the same models retain an advantage on it on smm4h (also in this case, the difference is significant at @xmath1.001). ## conclusions we presented a comparison between 12 transformers-based models, with the goal of "prescribing" the best option to the researchers working in the field. we also wanted to test whether the span-based objective of spanbert and in-domain language pretraining were useful for the task. we can positively answer to the first question, since spanbert turned out to be the best performing model on both datasets. as for the in-domain models, pubmedbert came as a close second after spanbert, suggesting that pretraining from scratch with no general domain data is the best strategy, at least for this task. we have been the first, to our knowledge, to test these two models in a systematic comparison on ade detection, and they delivered promising results for future research. for the next step, a possible direction would be to combine the strengths of their respective representations: the accurate modeling of text spans on the one side, and deep biomedical knowledge on the other one.