A Knowledge-Graph-Based Intrinsic Test for Benchmarking Medical Concept Embeddings and Pretrained Language Models (2022)

Abstract: Using language models created from large data sources has improved the performance of several deep learning-based architectures, obtaining state-of-the-art results in several NLP extrinsic tasks. However, little research is related to creating intrinsic tests that allow us to compare the quality of different language models when obtaining contextualized embeddings. This gap increases even more when working on specific domains in languages other than English. This paper proposes a novel graph-based intrinsic test that allows us to measure the quality of different language models in clinical and biomedical domains in Spanish. Our results show that our intrinsic test performs better for clinical and biomedical language models than a general one. Also, it correlates with better outcomes for a NER task using a probing model over contextualized embeddings. We hope our work will help the clinical NLP research community to evaluate and compare new language models in other languages and find the most suitable models for solving downstream tasks.

## introduction in healthcare, text plays a role of enormous importance. one of the media that a medical practitioner can persist is the text in clinical records @xcite . text is one of the richest forms of information inside the electronic health record, so it is fundamental to develop tools to extract information from these text sources. to create these tools in this field, we must pay special attention to ensuring quality and reproducibility. analyzing unstructured texts written by humans is challenging since it is complex to formally understand and describe the rules governing human language, as it is ambiguous and constantly evolving. natural language processing (nlp) is an interdisciplinary field of artificial intelligence that seeks to develop algorithms capable of understanding, interpreting, and manipulating these unstructured texts @xcite . in the medical context, using nlp helps to address tasks such as extracting medical entities, disease coding, text classification, and relation extraction, among others. however, one of the steps before solving any of these tasks is to create robust numerical representations of the text so that the computer can handle this data. word embeddings are dense, semantically meaningful vector representations of a word. these models have proven to be a fundamental building block of neural network-based architectures @xcite . although these models have obtained excellent results for several nlp tasks, their main drawback is that they provide a singleword representation in a given document. this is not optimal since a word meaning may depend on the sentence in which it appears. this type of word embedding is known as static word embeddings. contextual representation models handle this issue by creating word representations based on sentence-level context. these representations are commonly retrieved from pretrained language models @xcite . classic examples of these models are elmo, bert, roberta, flair, albert, among others. however, contextualized word embeddings may not represent words as well as static ones, as results obtained in @xcite suggest. although contextualized word embeddings have these drawbacks, we can use these numeric representations of words to understand plm representations. specifically, we are interested in studying how domain-specific and general-domain plm represent clinical and biomedical concepts. in this study, we aim to create a simple and efficient test for measuring concept embeddings' quality and comparing clinical and biomedical plm performance using a relevant knowledge base and graph, the unified medical language system (umls). a knowledge graph is an extensive network of entities relevant to a specific domain. the network describes each entity's semantic types, properties, and relationships. knowledge graphs represent real-world entities and their relations in a graph, define possible classes, and allow to relate arbitrary entities with each other @xcite . the umls is a knowledge graph that combines many clinical and biomedical vocabularies and standards to enable interoperability between computer systems @xcite . the umls consists of multiple knowledge sources. one is the metathesaurus, a large, multi-purpose, and multilingual vocabulary database that contains information about biomedical and clinical-related concepts, their various names, and their relationships. another source is the semantic network, a consistent categorization of all concepts represented in the metathesaurus, providing a set of valuable relationships between these concepts. in this work, we used both knowledge sources. two testing frameworks have been developed to measure the quality of language representations. first, an extrinsic test framework that uses the language representations to construct a more complex architecture to solve a specific downstream task. second, an intrinsic test framework that measures the capacity of the language representation to resolve semantic questions regarding the language domain it represents @xcite @xcite . to construct intrinsic tests, we must compose questions based on a source of truth. this source can be expert knowledge, where we ask human experts to write each one of these questions manually, or we can use a knowledge base to compose these questions automatically. we used the umls knowledge graph to automatically derive a concept similarity intrinsic test using the length of the shortest path in the graph to compute a true similarity measure between concepts. this intrinsic test will be used as a metric to check how good language representations are, but also as a comparison measure of whether clinical and biomedical plm are better compared to general ones in downstream tasks such as named entity recognition (ner). regarding the clinical and biomedical domain in english, there are several models to obtain contextualized embeddings, such as bioelmo @xcite , clinical bert @xcite , scibert @xcite , biobert @xcite , among others. however, there remains a significant lack of language models in spanish. the only models available are scielo flair @xcite , clinical flair @xcite , and clinical and biomedical versions of roberta @xcite . although these studies have shown that incorporating domain-specific contextualized embeddings significantly improves the models' performance in several extrinsic tasks, comparing their performances with intrinsic tests is still necessary. since plm creates word-level contextual representations, it is necessary to define a method for combining these vectors to create sentence-level embeddings. for this purpose, a popular technique is the mean pooling of contextual word embeddings @xcite . however, this method may lead to poor results if the plm is not explicitly trained for similarity. another study has proposed transforming the distribution of sentence-level embeddings to generate isotropic and smooth representations @xcite . creating these sentence-level representations is fundamental for testing the intrinsic tests proposed in this research. common approaches to evaluate biomedical plm performance are benchmarks such as blue @xcite and blurb @xcite , which are built for the english language. there is no relevant benchmark in spanish, and every author selects some annotated datasets to evaluate plm performance on specific downstream tasks. although the amount of annotated datasets in spanish is growing, there is a lack of intrinsic tasks that can help to understand if a plm is improving, and this research tries to fill that gap. ## methods our proposed method creates a semantic similarity intrinsic test with medical concept pairs and their semantic distances. we extracted these concept pairs from the umls 1 term graph and computed their distances as the length of the shortest directed path of parent relationships between the concepts. we measured the correlation of the knowledge-graph-derived distance to the cosine similarity of the terms string descriptions on an embedding space projected using different language representations. finally, we compare these correlations with the performance on downstream tasks of each language representation. ## results we queried 20,000 pairs of random atoms to select umls concepts from the graph database. figure 2 shows the histogram of those pairs by path length. we can see that pair frequency increases as path length increase until seven parent relationships of distance. after that point, the frequency of pairs decreases until it reaches 14 relations of distance. we removed all path lengths containing less than 300 pairs of concepts to calculate the metrics ρ and δ. then, we plot a boxplot of cosine similarity by path length for every plm. figure 3 shows such a boxplot for a general-domain bert trained in spanish text @xcite 10 . this plot allows us to understand how cosine similarity distributes along path length. it is clear from the plot that average cosine similarity decreases as path length increases. however, the decline is near null or even negative from path length four onwards. moreover, the average cosine similarity is not going near zero. we hypothesize this pattern is because all concepts are related to clinical and biomedical domains and also due to the anisotropic behavior of sentence embeddings obtained from plm. as discussed in @xcite , contextualized embeddings obtained from plm tend to distribute not evenly in the embedding space but in a small portion of it. therefore, they still have a relatively high similarity when comparing dissimilar concepts. to compare several plm, we plot only average cosine similarity by path length for every language model, as shown in figure 4 . as we can see, average cosine similarity by path length varies for different base architectures and domains of plm. however, they all repeat the same decline pattern as path length increases. interestingly, when ρ metric is greater for a clinical model compared to a biomedical one, f1 scores for ner probing tasks are also greater, as we can see in the case of roberta architecture for can-temist and pharmaconer datasets. in the case of ct-ebm-sp and nubes, there are no such differences, but f1 scores for clinical and biomedical are almost the same. on the contrary, when ρ metric is greater for a biomedical model compared to a clinical one, then f1 scores present a similar behavior, as we can see in the case of flair architecture for cantemist, pharmaconer, and ct-ebm-sp datasets. and as same as the previous situation, f1 scores for another dataset (nubes) are almost the same. we do not observe this pattern for δ metric. this finding suggests that ρ metric could be applied as a useful intrinsic test for comparing plm within the same base architecture. however, it is important to note when comparing ρ metric for different base architectures, we do not find a clear relation with f1 scores. consequently, we present the ρ metric as an intrinsic test to measure improvements for plm within the same base architecture. ## conclusion and future work using domain-specific plms for downstream tasks has allowed reaching the state-of-the-art in several benchmarks. however, since these models are trained in large corpora, fine-tuning them or training from scratch is time-consuming. therefore, before using these models to solve downstream tasks, it is crucial to create intrinsic tests that validate whether a domain-specific plm yields better results than its base version. in this study, we build an intrinsic test for clinical and biomedical plm using contextualized em-beddings and the umls knowledge graph. we suggest that our intrinsic test can help compare domain-specific plm performance within its base architecture, which could be used to evaluate improvements when building plm. our experimental results show that this intrinsic test can capture improvements in clinical and biomedical plm over general ones. also, it correlates with better results in a ner probing task over four datasets in spanish. in future work, we can implement this study for other languages. additionally, we can compare our intrinsic test with other probing tasks such as pos-tagging or coreference or even other clinical downstream tasks such as patient mortality or unplanned readmission. on the other hand, since our experimental datasets contain nested entities, but for simplicity, they were ignored, we would like to explore the use of contextualized embeddings in models that can address them, such as those proposed in @xcite . finally, we can compare several experimental settings, such as multiple numbers of concept pairs.