Integration of Heterogeneous Knowledge Sources for Biomedical Text Processing (2022)

Abstract: Recently, research into bringing outside knowledge sources into current neural NLP models has been increasing. Most approaches that leverage external knowledge sources require laborious and non-trivial designs, as well as tailoring the system through intensive ablation of different knowledge sources, an effort that discourages users to use quality ontological resources. In this paper, we show that multiple large heterogeneous KSs can be easily integrated using a decoupled approach, allowing for an automatic ablation of irrelevant KSs, while keeping the overall parameter space tractable. We experiment with BERT and pre-trained graph embeddings, and show that they interoperate well without performance degradation, even when some do not contribute to the task.

## introduction integration of external knowledge sources (kss) is seen as a daunting task by the community. most kss like ontologies are large and complex. thus, a majority of the current efforts focus on leveraging a single task relevant ks using hand-tailored architectures @xcite @xcite . during the design process, knowledge sources are often selected through an ablation study, which is laborious and makes the result task-dependent. thus for every new task the set of relevant knowledge sources has to be identified with a similar study. it is possible to ignore the tailoring step and use all available kss, trusting that the training process will properly weigh the heterogeneous kss given the internal dynamics of the model. this ideal case requires sufficient training data, but most tasks (like many biomedicial tasks) have only small or moderate-sized training data, a common problem for large, monolithic machine learning systems @xcite . in those systems all kss are always contributing their expertise, which can result in decreased rather than improved performance. we explore here a way to integrate several large, heterogeneous kss with partly overlapping, partly divergent, and possibly even contradictory expertise in such a way that they interoperate well without adaptation, with no resulting performance decrease as well as low parameter implications. we use an integration of six kss as our experiment system and test it over seven different shared task datasets to assess its robustness. we visualize and inspect the contribution of each ks and analyze the parameter space in detail. the question is how to integrate multiple heterogeneous kss so that the same system can be used for multiple, unrelated tasks without manual adaptation and without large overhead. we argue that a system with decoupled modules is suitable for this purpose. decoupled modules can be activated conditioned on the input, allowing the system to ignore an irrelevant ks and thus preventing performance loss with fewer parameter updates at each training step @xcite . in this paradigm, instead of hand-picking kss, an internal and automatic ablation is performed at each step for all kss, making it easy to use the same system for different tasks, with the least detrimental effects. our kss consist of the pre-trained language model bert, as well as six structured knowledge repositories designed for human useage: wordnet, dbpedia, conceptnet, mesh, go, and umls. for all but conceptnet we found open source graph embeddings, and we embed con-ceptnet using rdf2vec (see section 2). the recently proposed multi-input rim framework @xcite comes close to our ideas and we use it here for decoupled integration of our kss. @xcite showed successful decoupled integration of simple kss like gazetteer lists that were task appropriate but did not report on experiments with large, structured kss. we test the same system on 7 different biomedical shared task datasets and show that our heterogeneous kss interoperate well and achieve synergy, despite their overlap in coverage. our results improve on two baselines contributed by the knowledge-enhanced models biobert @xcite and kb-bert @xcite . the system is competitive with state of the art systems (see ## heterogeneous knowledge sources specialized ontological resources contain quality curated information and are often very large and complex. a graph-based knowledge representation is symbolic and discrete, making it hard to use in a machine learning framework, as most machine learning models prefer conducting computations on continuous data. the past few years have seen several techniques to embed graph structures into vector spaces. inspired by distributional word representations @xcite , where each word is embedded in a low dimensional space, graph embedding models embed a graph into a vector space. in graph embedding models, entities (nodes) and relations (edges) are represented by vectors or matrices @xcite . inspection of graph embedding models shows that they can capture a fair amount of ontological information. for instance @xcite show that related concepts are often close to each other in the vector space. we use the following ontologicial resources encoded using a pre-trained graph embedding: wordnet is a lexical databese that defines word senses by their relations to other senses @xcite . the most important relation in wordnet is synonymy that is used to group synonymous senses into synsets. dbpedia @xcite extracts knowledge from wikipedia info boxes, providing a large number of facts, largely focused on named entities that have wikipedia articles. we use the pre-trained rdf2vec @xcite embeddings of wordnet and dbpedia, which are available from kgvec2go web-site. 1 conceptnet is a large multi-lingual graph of general knowledge @xcite . concept-net uses closed class of 36 relations. to embed conceptnet a set of graph embeddings is obtained in-house, using rdf2vec. mesh or medical subject headings is a hierarchical vocabulary, produced by the us national library of medicine (nlm) @xcite . it is used for indexing, cataloging, and searching of biomedical and health-related information in pubmed. foot_1 mesh is also embedded using a pretrained graph embedding called mesh2vec @xcite . go or gene ontology @xcite is a controlled vocabulary that describes geneand protein-related terms. we use the pre-trained go2vec embeddings @xcite for encoding the gene ontology. umls or unified medical language system @xcite is a rich and large semantic network of biomedical vocabularies developed by nlm. umls comprises 127 semantic types and 54 semantic relations. currently umls encompases 222 biomedical vocabularies including mesh, go, drugbank, etc. for umls, we use the embeddings provided by @xcite . ## tasks we choose seven biomedically oriented datasets from different shared task competitions that range from simple classification tasks over multi-label classification and relation extraction to sequence labeling tasks. comparing results for the same system on such a variety of tasks and datasets (including ner on spanish!) allows us to be confident that the decoupled integration together with sparse activation in the mirim architecture successfully avoids interference of the kss and performance degradation. ## conclusion this paper presents extensive experiments on decoupled integration of heterogeneous kss such as language models and pre-trained graph embeddings. the same system with all kss was used for all tasks, without special calibrations, demon-strating reusability of extant knowledge sources. the tasks differed in terms of complexity as well as their knowledge requirements (specialized or general knowledge). the results show that for the tasks considered here, the kss interoperate well and they do not confound each other's performances. moreover, we showed that a system that leverages multiple ks does not necessarily show significant improvement, rather the sparse activity of modules is required to effectively improve performance. inspection of activation patterns shows that a decoupled system can ignore irrelevant/redundant kss, showing an automatic ablation behavior. we show that in terms of the number of trained parameters, a decoupled approach is efficient. the sparse activity significantly reduces the number of trained parameters. moreover, since the pretrained graph embeddings are not fine-tuned, the overall model does not have large parameter implications. we also stress the ease of reusing and replicating such a decoupled system, since the same pretrained embeddings will be used by different users. moreover, the pre-trained embeddings do not have to be stored on the same machine that the model is trained on. kgvec2go 6 , for instance, provides an api through which pre-trained embeddings are accessible. this ultimately results in lightweight models. in conclusion, a decoupled approach allows for robust and efficient integration of heterogeneous kss, allowing the user to leverage multiple knowledge sources, without any need for special calibration or tailoring.