Fair Evaluation in Concept Normalization: a Large-scale Comparative Analysis for BERT -based Models (2020)

Abstract: Linking of biomedical entity mentions to various terminologies of chemicals, diseases, genes, adverse drug reactions is a challenging task, often requiring non-syntactic interpretation. A large number of biomedical corpora and state-of-the-art models have been introduced in the past five years. However, there are no general guidelines regarding the evaluation of models on these corpora in single- and cross-terminology settings. In this work, we perform a comparative evaluation of various benchmarks and study the efficiency of state-of-the-art neural architectures based on Bidirectional Encoder Representations from Transformers (BERT) for linking of three entity types across three domains: research abstracts, drug labels, and user-generated texts on drug therapy in English. We have made the source code and results available at https://github.com/insilicomedicine/Fair-Evaluation-BERT .

## introduction aggregating knowledge about entities across different domains and corpora is critical for many information extraction (ie) applications. in biomedical research and healthcare, the entity linking problem is known as medical concept normalization (mcn). medical concepts may have different types (e.g., drugs, diseases, or genes/proteins) and may be retrieved from different single-typed ontologies. effective mapping of the same concepts across different ontologies (the mcn task) is the holy grail of modern medical nlp. most mcn methods meanwhile are evaluated on test sets of widely differing sizes and domains and a narrow subsample of concepts from specific terminology. moreover, the reported results of neural networks vary substantially on different corpora, with, for example, accuracy ranging at least from 91% to 96% on research abstracts @xcite and accuracy from 77% to 89% on social media texts @xcite . owing to their superior semantic learning capabilities, bert @xcite and other neural architectures have been widely used in recent state-of-the-art (sota) models for the mcn task on research abstracts and social media texts @xcite @xcite @xcite @xcite . these studies mostly share the same limitations regarding their evaluation strategy: models are usually trained and evaluated on entities of the same type from a single domain. often, concept unique identifiers (cuis) used in training are included in the test set. a recurring problem, which arises with supervised models, is how to reuse trained models for a different purpose; this requires coding to a specific terminology. in this work, we take the task a step further from existing research by exploring current benchmarks and cross-terminology transfer between entity mentions in research abstracts, drug labels, and user-generated texts. we perform an extensive evaluation of five biomedical corpora manually annotated with concepts regarding diseases, chemicals, human genes, and adverse drug reactions (adrs). we utilize two models: (i) a baseline that ranks concepts for a given mention by comparing biomedical bert vectors @xcite with the euclidean distance; (ii) a supervised sota model biosyn @xcite . the work reported here aims to advance sota models in biomedical concept normalization of entity mentions with a variety of entity types and differences in surface characteristics of mentions. in this work, we seek to answer the following research questions: rq1: do test sets of current benchmarks lead to an overestimation of performance? rq2: how do surface characteristics of entity mentions affect the performance of the bert-based baseline? rq3: does a model trained on one corpus work for the linking of entity mentions of another type or domain in the zero-shot setting? we use the following publicly available benchmarks with official train/dev/test splits. descriptive statistics of these datasets are shown in ## models for concept normalization we utilize two bert-based models: (i) a baseline method based on the ranking of biobert representations, (ii) a sota model named biosyn @xcite . we use biobert base v1.1 for both models that was pre-trained on pubmed abstracts (4.5b words in total) for 1m steps. biobert ranking this is a baseline model that used the biobert model for encoding mention and concept representations. each entity mention or concept name is passed first through biobert (we use the average over all outputs of bert) and then through a mean pooling layer to yield a fixed-sized vector. the inference task is then reduced to finding the closest concept name representation to entity mention representation in a common embedding space. we use the euclidean distance as the distance metric. the nearest k concept names are chosen as top-k concepts for entities. biosyn biosyn @xcite is a recent sota model that utilizes the synonym marginalization technique and iterative candidate retrieval. the model uses two similarity functions based on sparse and dense representations, respectively. the sparse representation encodes the morphological information of given strings via tf-idf, the dense representation encodes the semantic information gathered from biobert. biosyn achieves sota results on ncbi, bc5cdr, tac sets over previous works @xcite @xcite . we have used the publicly available code provided by the authors at https://github.com/dmis-lab/biosyn and reproduced the results successfully. we follow the default parameters of biosyn as shown in @xcite : the number of top candidates @xmath0, the mini-batch size is 16, the learning rate is 1e-5, the dense ratio for candidate retrieval is 0.5. we have trained biosyn for 20 epochs for all datasets. ## single-and cross-terminology evaluation we train biosyn on the train/dev set of each corpus with a source dictionary, evaluating it on the respective test set (in-domain performance). for cross-domain evaluation, we assess models trained on source data on the test sets of all other corpora (i.e., the target). specifically, both biosyn and biobert ranking models retrieve the nearest concept name in a target dictionary for a given mention representation at inference time. we note that cross-terminology evaluation provides a challenging setup for developing supervised models, especially for linking to concepts not encountered during training (zero-shot concepts). we evaluate this task in information retrieval (ir) scenario, where the goal is to find within a dictionary of concept names and their identifiers the top-k concepts for every entity mention in the texts. let acc@k be 1 if the correct cui is retrieved at rank k, otherwise 0. for composite entities, we define acc@k as 1 if every prediction for a single mention is correct. in particular, we use the top-1 accuracy as an evaluation metric, following previous works @xcite @xcite @xcite . to answer rq1, we compare the results of models on official and refined test sets in according to to answer rq3, we compare performance differences in tables 2 and 3 . the models trained on ncbi, cdr disease, bc2gn, and tac data perform on par with the model trained on the cdr chemical train set (approx. 74% acc@1), while the model trained on cdr chemical showed a 6% drop on these subsets. ## conclusion we have presented the first comparative evaluation of medical concept normalization (mcn) datasets, studying the ncbi disease, bc5cdr disease & chemical, bc2gn gene, tac 2017 adr, and smm4h 2017 adr corpora. we perform an extensive evaluation of two bert-based models on six datasets in two setups: with official train/test splits and with the proposed test sets that represent refined samples of entity mentions. our evaluation shows great divergence in performance between these two test sets, finding an average accuracy difference of 15% for the state-of-the-art model biosyn. we also performed a quantitative evaluation of biosyn in the cross-terminology mcn task where models were trained and evaluated on entity mentions of various types with concepts from different terminologies. knowledge transfer can be effective between diseases, chemicals, and genes with an average drop of 2.53% accuracy in the performance on ncbi, bc5cdr, and bc2gn sets. for tac and smm4h sets with adrs from drug labels and social media, biosyn models trained on four other corpora show a substantial decrease in performance (-10.2% and -33.1% accuracy, respectively) compared to in-domain trained models. to our surprise, these models still outperformed the straightforward ranking baseline on biobert representations. we believe that refined datasets with cross-terminology evaluation can serve as a step toward reliable and large-scale evaluation of biomedical ie models. we foresee three directions for future work. first, a promising research direction is the multilingual evaluation of mcn models. second, in some cases current models choose a broader concept that is in a parent-child relationship with the correct concept; here future research may focus on the encoding of the concept hierarchy. third, the use of local and global contexts of entity mentions remains to be explored.