Are we there yet? Exploring clinical domain knowledge of BERT models (2021)

Abstract: We explore whether state-of-the-art BERT models encode sufficient domain knowledge to correctly perform domain-specific inference. Although BERT implementations such as BioBERT are better at domain-based reasoning than those trained on general-domain corpora, there is still a wide margin compared to human performance on these tasks. To bridge this gap, we explore whether supplementing textual domain knowledge in the medical NLI task: a) by further language model pretraining on the medical domain corpora, b) by means of lexical match algorithms such as the BM25 algorithm, c) by supplementing lexical retrieval with dependency relations, or d) by using a trained retriever module, can push this performance closer to that of humans. We do not find any significant difference between knowledge supplemented classification as opposed to the baseline BERT models, however. This is contrary to the results for evidence retrieval on other tasks such as open domain question answering (QA). By examining the retrieval output, we show that the methods fail due to unreliable knowledge retrieval for complex domain-specific reasoning. We conclude that the task of unsupervised text retrieval to bridge the gap in existing information to facilitate inference is more complex than what the state-of-the-art methods can solve, and warrants extensive research in the future.

## introduction transformers-based neural architectures @xcite currently hold the state-of-the-art performance on several nlp tasks and domains. in the biomedical domain itself, there exist several versions of transformers-based bert models @xcite that have been shown to be successful. however, an analysis of the availability of medical knowledge to these models is incomplete. to facilitate better understanding, in our research, we analyze a sample of errors made by biobert (v1.1) model @xcite on a clinical language inference task @xcite . we find that the errors related to domain knowledgebased reasoning, such as the knowledge of treatments administered for certain diseases, are dominant (40%). to address this limitation, we analyze a broad range of state-of-the-art methods to integrate medical knowledge in bert models from textual medical corpora. these methods have previously been shown to excel at evidence retrieval in the generic domain. the goal of our study is to understand whether these methods can be successfully applied for knowledge integration in the more complex setup of finding missing medical information for supporting sentence-pair inference. we explore both implicit and explicit knowledge integration, where implicit refers to indirect access to this knowledge by further language model pretraining on medical corpora, and explicit knowledge integration refers to the setup where a relevant sentence from external corpora is appended to the premise to support inference. for explicit knowledge integration, as the baseline method, we make use of the traditional best match 25 (bm25) algorithm @xcite for finding the most relevant sentence in the medical corpora. as a modification of this method, we additionally incorporate syntactic knowledge in the retrieval step. we do so by restricting the retrieved sentence to the one that contains at least one dependency relation between premise and hypothesis medical entities. in the third setup, instead of using bm25 scores and dependency paths, we train an end-toend model to first find the most relevant text block from wikipedia for a given instance, and then append it to the instance for classification. in both knowledge integration setups, we do not see any significant performance difference due to access to additional knowledge. on inspecting the sentences retrieved by the bm25 and dependency relation-based methods, we find that these methods successfully shortlist sentences related to the topic, but it is difficult to then automatically rank the best candidate among the shortlisted options. this best candidate should fill the information gap between the sentence pairs to enable pairwise inference. we expect to overcome the ranking issue when we instead train an end-to-end model that learns to dynamically retrieve relevant supporting knowledge alongwith pairwise classification, as opposed to static heuristic-based retrieval. however, we find that although the blocks of text retrieved in the end-to-end setup provide medical context, they are often unrelated to the desired information and are insufficient for improving inference. although knowledge-integration methods are effective for evidence retrieval in open domain qa @xcite , where the task is to retrieve a passage that mentions the correct entities, they are insufficient for the more complex task of augmenting missing information for pairwise domain knowledge-based reasoning in an unsupervised setup. entity span-based supervision simplifies the problem statement in the first case, hence resulting in the documented success. however, the more realistic setup of retrieving the specific context that can fill the information gap between pairs of sentences without supervision is not yet solved. ## related work since the bert models were found to be effective for a wide range of nlp tasks @xcite , several efforts have been extended towards improving them by more efficient training strategies @xcite @xcite , training them for different domains @xcite @xcite @xcite and languages @xcite @xcite @xcite . within the clinical domain, different models include the biobert models pretrained on pubmed abstracts and pmc full-text articles @xcite , scibert trained on scientific text @xcite , clinicalbert models trained on patient notes from the mimic-iii corpus @xcite (sometimes as a continuation of the biobert models) @xcite , and bluebert models that also use pubmed abstracts and mimic-iii patient notes for training @xcite . these models hold promising performance for clinical language processing @xcite and have become a popular choice for several classification tasks that involve the medical data, spanning tasks such as literature search and question answering for assisting healthcare professionals @xcite @xcite , as well as patient outcome prediction such as diagnosis prediction @xcite . despite being a popular choice, little is known about the medical knowledge of these models and their limitations when in-depth domain knowledge is required for correctly solving a task. much prior research has explored augmentation of background knowledge in neural models to make them more effective for downstream tasks. most common approaches include adapting entity embeddings learned by models such as bert by providing additional knowledge from different ontologies that define relations between entities. this can be done either by using templates to convert the relations to text before finetuning embeddings @xcite @xcite , by combining relational information from knowledge graphs with text embeddings @xcite @xcite @xcite , or by jointly learning knowledge graph and textual embeddings @xcite . these ontologies are either generic like wordnet @xcite , concept-net @xcite , and wikidata (vrandečić and krötzsch, 2014), or more specific to a particular domain like the umls @xcite ). an advantage of using ontologies is that the semantics of entities gets encoded in the learned representations, thereby enhancing their effectiveness. however, they are expensive to construct and either are incomplete, or do not exist for specialized domains. methods that make use of textual corpora for background knowledge integration are therefore more easily transferable to other domains. @xcite have shown earlier that having explicit access to external information can often improve reasoning skills of the state-of-the-art models, which we investigate further. use of tf-idf @xcite and bm25 scores has been frequently explored for evidence retrieval from wikipedia for open domain qa @xcite @xcite ). an-other popular approach includes representation similarity-based evidence retrieval @xcite . recently, joint training of retriever for span identification and pretraining language models have also been analyzed by @xcite ; @xcite ; @xcite . although the methods extensively explore qa, this line of work has not been explored much for language inference, especially in specialized domains. existing studies for augmenting medical knowledge for clinical language inference are limited to the use of umls knowledge graph embeddings @xcite , interaction weighting between premise and hypothesis based on distance in the umls @xcite , augmenting clinical concept definitions during representation learning @xcite and adding domain knowledge by means of pretraining existing models further on different in-domain corpora and closely related tasks @xcite @xcite . the closest work to ours is the contemporary work by @xcite that shows improvements when knowledge from wikipedia is implicitly integrated by training bert masked language models to predict disease names and their aspects (such as symptoms, treatments) given the corresponding context. in our work, we instead explore whether we can augment domain knowledge by dynamically fetching relevant context in an unsupervised manner to improve medical language inference. ## medical language inference in medical language inference, given a pair of sentences, the goal is to describe a logical relation between them. we make use of the mednli dataset @xcite , where the premise is a sentence borrowed from patient notes in the mimic-iii dataset @xcite , and the hypothesis is written by medical experts such that the premise either entails or contradicts the hypothesis, or their relation cannot be @xcite . entailment refers to whether the meaning of the second sentence, also known as the 'hypothesis', is already contained in the first sentence called the 'premise'. we explore whether the biobert v(1.1) model encodes sufficient medical knowledge for this task. in the same manner as @xcite , we model this task as a sentence pair classification task, where the final pooled bert [cls] representations of the premise and the hypothesis are processed through a dense neural layer to classify the correct class. we then perform manual analysis on a subset of 50 incorrectly classified instances in the development set to understand the type of errors made by the model. we eliminate ambiguity in the cause of errors by using an adversarial evaluation, where we modify an instance according to a potential cause of error, and monitor whether the output changes accordingly. in this manner, we obtain the distribution of errors presented in ## medical knowledge augmentation 4.1 external medical corpora different versions of bert that exist for biomedical tasks are either trained on journal abstracts and articles, or on patient notes. these articles and notes are written by and for an audience with an advanced level of domain knowledge. fundamental domain-specific information, such as an understanding of domain terminology, commonly accepted clinical practices for specific medical conditions, human physiology and anatomy, etc. is often also required for clinical language understanding. we hypothesize that access to such fundamental domain knowledge during model training would complement training on more advanced information. to explore this, we create two corpora -one containing only the medical subset of wikipedia (wikimed), and one with contents of a popular medical textbook (medbook). the wikimed subset is parsed from the html sources of the medical wikipedia dataset used in the android app by the kiwix team foot_0 . the medical subset of wikipedia contains about 40 million tokens, and the medical textbook corpus contains nearly 3.6 million tokens. ## conclusions and future work on investigating the error categories of biobert (v1.1) models on the clinical language understanding task, we find that despite having a strong performance, the models still make several mistakes on examples that require medical domain knowledge. to this end, we explored multiple methods to improve access of these models to medical domain knowledge by implicit and explicit knowledge retrieval and augmentation. however, we see that these extensions do not show significant improvement on the test sets. we conclude that state-of-the-art solutions lead to unreliable knowledge augmentation for language inference, as is shown by a detailed analysis in our work. future research should concentrate efforts towards developing methods to augment fundamental domain knowledge from textual corpora to solve the problem of advanced knowledge-based reasoning in these domains.