UMLS - KGI - BERT : Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognition (2023)

Abstract: Pre-trained transformer language models (LMs) have in recent years become the dominant paradigm in applied NLP. These models have achieved state-of-the-art performance on tasks such as information extraction, question answering, sentiment analysis, document classification and many others. In the biomedical domain, significant progress has been made in adapting this paradigm to NLP tasks that require the integration of domain-specific knowledge as well as statistical modelling of language. In particular, research in this area has focused on the question of how best to construct LMs that take into account not only the patterns of token distribution in medical text, but also the wealth of structured information contained in terminology resources such as the UMLS. This work contributes a data-centric paradigm for enriching the language representations of biomedical transformer-encoder LMs by extracting text sequences from the UMLS.This allows for graph-based learning objectives to be combined with masked-language pre-training. Preliminary results from experiments in the extension of pre-trained LMs as well as training from scratch show that this framework improves downstream performance on multiple biomedical and clinical Named Entity Recognition (NER) tasks. All pre-trained models, data processing pipelines and evaluation scripts will be made publicly available.

## introduction in recent times, transformer language models @xcite have become the most popular and effective sequence modelling framework in almost all areas of applied natural language processing. unsupervised pre-training on large quantities of text allows transformers to capture rich semantic and syntactic patterns that can be transferred to many specialised language processing objectives. as such, transformer models that use the transfer learning paradigm whereby the model is trained in an unsupervised manner on a large text corpus and then fine-tuned on a downstream supervisedlearning task have achieved state-of-the-art results across a wide range of general and domain-specific applications. the proliferation of textual data in the biomedical domain (electronic health records (ehrs), clinical documents, pharmaceutical specifications, etc) has precipitated the broad adoption of deep learning & nlp techniques for information extraction and processing @xcite @xcite . moreover, it has been shown that language models are capable of encoding clinical knowledge to a certain extent @xcite . biomedical and clinical nlp, however, is widely recognised to present particular challenges that do not apply to the same extent in other domains, in particular the need to incorporate structured domain knowledge into text encodings @xcite . in order for neural language modelling to be reliable in a discipline as highly specialised as medicine, there is a more acute need for models to learn directly from domain-specific terminologies, as opposed to relying solely on corpus-based learning. thus, a significant amount of research effort in the medical nlp community has been directed towards the question of how best to inject information from knowledge graphs (kgs) into lms @xcite @xcite . however, a generalisable, widely-accepted approach to this technique that can be easily transferred across different problem settings, models and training corpora has yet to emerge. in addition, research into knowledge graph integration in nlp in the biomedical domain has tended to focus on english-language corpora; the utility and transferability of these techniques for other languages, for which less textual resources are available, as well as for multilingual models, remains therefore an under-explored area. this paper aims to contribute to the resolution of these issues by proposing a general framework for training bert encoders @xcite using the umls (unified medical language system, bodenreider ( 2004 )) alongside free-text corpora. the main contributions of this work are as follows: ## methodology in this work, we experiment with training bert language models with three knowledge graph reasoning tasks derived from the umls, in addition to the standard masked-language modelling objective: entity prediction, link prediction and triple classification. ## experiments for the evaluation of the approach described in the previous section, we restrict our attention in this paper to ner tasks. where possible, we use the datasets and training-evaluation-test splits that are publicly available via the huggingface datasets library 1 . ## conclusions and future work this paper introduces umls-kgi, a framework for training bert models using knowledge graphs requiring highly minimal adjustments to the standard language modelling paradigm. we show the potential of this method to increase the performance of bert models on various ner tasks. the results presented in this paper suggest that for clinical ner tasks, high-quality small-scale datasets derived from structured information, alongside alongside relatively small clinical text corpora, can be as effective as large-scale corpora for pre-training bert models. we make our models and dataprocessing pipelines freely available online. future work in this direction will involve the incorporation of more diverse graph-based reasoning tasks in the pre-training strategy with more finegrained representation of relation types, as well as intrinsic evaluation of the umls-kgi-bert language representations via embedding visualisation and interpretability studies.