Stress Test Evaluation of Biomedical Word Embeddings (2021)

Abstract: The success of pretrained word embeddings has motivated their use in the biomedical domain, with contextualized embeddings yielding remarkable results in several biomedical NLP tasks. However, there is a lack of research on quantifying their behavior under severe “stress” scenarios. In this work, we systematically evaluate three language models with adversarial examples – automatically constructed tests that allow us to examine how robust the models are. We propose two types of stress scenarios focused on the biomedical named entity recognition (NER) task, one inspired by spelling errors and another based on the use of synonyms for medical terms. Our experiments with three benchmarks show that the performance of the original models decreases considerably, in addition to revealing their weaknesses and strengths. Finally, we show that adversarial training causes the models to improve their robustness and even to exceed the original performance in some cases.

## introduction biomedical nlp (bionlp) is the field concerned with developing nlp tools and methods for the life sciences domain. some applications of these techniques include e.g., discovery of gene-disease interactions @xcite , development of new drugs @xcite , or automatic screening of biomedical documents @xcite . with the exponential growth of digital biomedical literature, the importance of bionlp has become especially relevant as a tool to extract relevant knowledge for making decisions in clinical settings as well as in public health. in order to encourage the development of this area, public datasets and challenges have been shared with the community to solve these tasks, such as biosses @xcite , hoc @xcite , chemprot @xcite and bc5cdr @xcite , among others. at the same time, neural language models have shown significant progress since the introduction of models such as w2v @xcite , and more recent models like elmo @xcite and bert @xcite . these models, trained over large corpora (med-line and pubmed in the biomedical domain) have obtained remarkable results in most nlp tasks, including bionlp benchmarks @xcite . however, they have not been systematically evaluated under severe stress conditions to test their robustness to specific linguistic phenomena. for this reason, the objective of this paper is to evaluate three well-known neural language models under stress conditions. as a case study, we evaluate ner benchmarks since it a key bionlp information extraction task. our stress test evaluation is inspired by the work of @xcite , which proposes the use of adversarial evaluation for natural language inference by adding distractions in sentences, and evaluating models on this test set. we propose an adversarial evaluation black-box methodology, which does not require access to the inner workings of the models in order to generate adversarial examples @xcite . specifically, we make perturbations to the input data, also known as edit adversaries, that could cause the models to fall into erroneous predictions. additionally, we train the models with the proposed adversarial examples, which is a methodology used in previous works @xcite to strengthen the neural language models during the training process. we hope that our work will motivate the development and use of adversarial examples to evaluate models and obtain more robust biomedical embeddings. ## methodology we follow a black-box attack methodology @xcite , which consists of making alterations in the input data to cause erroneous predictions in the models. the following subsections describe each of the adversarial sets, and their construction foot_0 . we show examples of the stress tests in noise adversaries these adversaries test the robustness of models to spelling errors. inspired by @xcite , we constructed adversarial examples that try to emulate spelling errors made by human beings. we used spacy models @xcite to retrieve the medical words of each corpus and add noise to them. we used two types of alterations: i) keyboard typo noise (k) involves replacing a random character in each relevant word with an adjacent character on qwerty english keyboards. this methodology could be adapted to keyboards with other designs or languages. ii) swap noise (w) consists of selecting a random pair of consecutive characters in each relevant word and then swapping them. ## experiments in this section we report the results of our experiments. note that all percentage drops or increases are expressed relative to the original score, not as percentage points. ## conclusions in this work, we have investigated whether large scale biomedical word (w2v) and contextualized word embeddings (bert and elmo) are robust with respect to black-box adversarial attacks in the biomedical ner task. our experimental results show different sensitivities of the models to misspellings and synonyms. among the main findings, we show that bert-based models are generally better prepared for adversarial attacks, but they are still fragile, leaving room for future improvement in the field. elmo-based models show lower robustness in most cases but consistently outperformed bert in some specific scenarios. w2v proves to be more brittle but shows similar patterns in terms of relative performance drops. we also demonstrate that by training with adversaries, we can considerably decrease the drop in performance and even improve the models' original performance when trained with synonyms, as they act as a form of regularization and augmentation of data.