M ed D istant19: Towards an Accurate Benchmark for Broad-Coverage Biomedical Relation Extraction (2022)

Abstract: Relation extraction in the biomedical domain is challenging due to the lack of labeled data and high annotation costs, needing domain experts. Distant supervision is commonly used to tackle the scarcity of annotated data by automatically pairing knowledge graph relationships with raw texts. Such a pipeline is prone to noise and has added challenges to scale for covering a large number of biomedical concepts. We investigated existing broad-coverage distantly supervised biomedical relation extraction benchmarks and found a significant overlap between training and test relationships ranging from 26% to 86%. Furthermore, we noticed several inconsistencies in the data construction process of these benchmarks, and where there is no train-test leakage, the focus is on interactions between narrower entity types. This work presents a more accurate benchmark MedDistant19 for broad-coverage distantly supervised biomedical relation extraction that addresses these shortcomings and is obtained by aligning the MEDLINE abstracts with the widely used SNOMED Clinical Terms knowledge base. Lacking thorough evaluation with domain-specific language models, we also conduct experiments validating general domain relation extraction findings to biomedical relation extraction.

## introduction extracting structured knowledge from unstructured text is important for knowledge discovery and management. biomedical literature and clinical narratives offer rich interactions between entities mentioned in the text @xcite , which can be helpful for applications such as bio-molecular information extraction, pharmacogenomics, and identifying drug-drug interactions (ddis), among others @xcite . ## related work relation extraction (re) is an important task in biomedical applications. traditionally, supervised methods require large-scale annotated corpora, which is impractical to scale for broad-coverage biomedical relation extraction @xcite . distant supervision (ds) allows for the automated collection of noisy training examples by aligning a given knowledge base (kb) with a collection of text sources @xcite . ds was used in recent works @xcite with pre-trained language models using multi-instance learning (mil) by creating bags of instances @xcite for corpuslevel triple extraction. 1 in biomedical domain, 1 re is used to refer to two different tasks: sentence-level detection of relational instances and corpus-level triples extraction, a kind of knowledge graph completion or link prediction task @xcite . @xcite first proposed the use of the unified medical language system (umls) metathesaurus @xcite as a kb with pubmed @xcite ) medline abstracts as text collection. for broad-coverage tasks, @xcite implemented a knowledge-based attention mechanism @xcite for mutual learning with knowledge graph completion and entity type classification. @xcite introduced a largescale biorel benchmark focusing on drug-disease and gene-cancer interactions and showed significant performance using a comprehensive selection of baselines. recent works focused on using domain-specific pre-trained language models for distantly supervised biomedical relation extraction (bio-dsre). @xcite extended relation enriched sentence-level bert @xcite to handle bag-level mil and demonstrated that preserving the direction of the kb relationships can denoise the training signal. they also outlined the steps to create a broad-coverage benchmark from umls. following this, hogan et al. (2021) introduced the concept of abstractified mil (amil), by including different argument pairs belonging to the same semantic type pair in one bag, boosting performance on rare triples. for domain-specific bio-dsre, @xcite introduced the bere framework for latent tree learning and self-attention to use the semantic and syntactic information in the sentence for mil. they also introduced a drug-target interactions (dti) bio-dsre benchmark, suitable for drug repositioning, drawn from drugbank we found no such overlap for dti and tbga, where the datasets used in @xcite are not publicly available. noting these shortcomings, we introduce a new and accurate benchmark meddistant19 for broad-coverage bio-dsre. our benchmark utilizes clinically relevant snomed ct knowledge model and data original filtered auc f1 auc f1 amin et al. (2020a) 68.4 64.9 50.8 53.1 hogan et al. (2021) † 82.6 77.6 11.8 19.8 graph @xcite , extracted from the umls, that offers a careful selection of the concept types and is suitable for large-scale biomedical relation extraction. in supervised re, chemprot @xcite and @xcite focus on multi-class interactions between chemicalprotein and drug-drug respectively. eu-adr @xcite and gad @xcite focus on binary relations between genes and diseases, while cdr @xcite focuses on binary relations between chemicals and diseases. ## constructing the meddistant19 benchmark documents we used pubmed medline abstracts published up to 2019 foot_0 as our text source, containing 32,151,899 abstracts. following hogan et al. (2021), we used scispacy foot_1 @xcite for sentence tokenization, resulting in 150,173,169 unique sentences. we further introduce the use of scispacy for linking entity mentions to their umls cuis and filtering disabled concepts from umls, which resulted in entitylinked mentions at the sentence-level. named entity recognition (ner) and normalization were two primary sources of errors in biomedical re, as shown in @xcite . while scispacy is reasonably performant among other options for biomedical entity linking, it remains quite noisy in practice; e.g., @xcite showed that scispacy had only about a 50% accuracy on extracting concepts in benchmark datasets. despite this being a limitation, using scispacy is better than relying on string matching alone @xcite @xcite . knowledge base we use umls2019ab foot_2 as our primary knowledge source and apply a set of rules, resulting in a distilled and carefully reduced version of umls2019ab. the umls metathesaurus @xcite covers concepts from 222 source vocabularies, thus being the most extensive ontology of biomedical concepts. however, covering all ontologies can be challenging, given the interchangeable nature of the concepts. for example, programmed cell death 1 ligand 1 is an alias of concept c1540292 in the hugo gene nomenclature committee ontology @xcite , and it is an alias of concept c3272500 in the national cancer institute thesaurus. this makes entity linking more challenging since a surface form can be linked to multiple entity identifiers and easier to have overlaps between training and test sets since the same fact may appear in both with different entity identifiers. furthermore, benchmark corpora for biomedical ner @xcite and re @xcite focuses on specific entity types (e.g. diseases, chemicals, proteins), and are usually normalized to a single ontology @xcite . following this trend, we also focus on a single vocabulary for bio-dsre. we use snomed ct, the most widely used clinical terminology worldwide for documentation and reporting in healthcare @xcite . since umls classifies each entity in a type taxonomy of semantic types (sty) and semantic groups (sg) (fig. 2 ), this allows for narrowing the concepts of interest. following @xcite , we first consider 8 semantic groups in snomed ct: anatomy (anat), chemicals & drugs (chem), concepts & ideas (conc), devices (devi), disorders (diso), phenomena (phen), physiology (phys), and procedures (proc). we then remove conc and phen as they are far too general to be informative for bio-dsre. for a complete list of semantic types covered in meddistant19, see table a.4. similarly, each relation is categorized into a type and has a reciprocal relation in umls (table a.3), which can result in train-test leakage @xcite . these steps follow @xcite , with the difference that we only consider relations of type has relationship other than synonymous, narrower, or broader (ro); this is consistent with prior works in bio-dsre. we also exclude uninformative relations, same_as, possibly_equivalent_to, as-sociated_with, temporally_related_to, and ignore inverse relations as generally is the case in re. in addition, @xcite ensures that the validation and test set do not contain any new entities, making it a transductive learning setting where we assume all test entities are known beforehand. however, we are expected to extract relations between unseen entities in real-world applications of biomedical re. to support this setup, we derive meddistant19 using an inductive kg split method proposed by @xcite (see appendix a in their paper). ## experiments meddistant19 is released in a format that is compatible with the widely adopted re framework opennre @xcite . 5 to report our results, we use the corpus-level area under the precision-recall (pr) curve (auc), micro-f1, macro-f1, and precision-at-k (p@k) with @xmath0, 200, 300, 1k, 2k}, and the sentencelevel precision, recall, and f1. due to the imbalanced nature of relational instances, following @xcite , we report macro-f1 values, and following hogan et al. (2021), we report sentencelevel re results on relationships, including frequent and rare triples. ## discussion in the biomedical domain, health experts are often concerned with a particular type of interaction, for example, drug-target and gene-disease. however, the number of ontologies is constantly growing (222 in umls2019ab), thus a growing need for a more general purpose relation extraction benchmark. broad-coverage benchmarks exist for biomedical entity linking, such as medmentions @xcite , but they still lack many important concepts involved in relational learning. the research community has come up with several re benchmarks (see further, the train-test overlap highlights the need to systematically assess the proposed benchmarks for inconsistencies that can overestimate the model performance. similar assessments have shown up in qa generalization where train-test overlap inflates the model performance @xcite . related to @xcite exposed shallow heuristics while @xcite showed that neural re models could retain triples, primarily due to type hints. meddistant19 partially addresses these issues by an inductive setup that can offer insights into the generalization trend in biomedical re using unseen entities. ## conclusion in this work, we highlighted a need for an accurate broad-coverage benchmark for bio-dsre. we bridged this gap by utilizing snomed ct for constructing the benchmark and laying out the best practices. we thoroughly evaluated the benchmark with baselines and state-of-the-art, showing there is room to conduct further research.