Self-Supervised Intermediate Fine-Tuning of Biomedical Language Models for Interpreting Patient Case Descriptions (2022)

Abstract: Interpreting patient case descriptions has emerged as a challenging problem for biomedical NLP, where the aim is typically to predict diagnoses, to recommended treatments, or to answer questions about cases more generally. Previous work has found that biomedical language models often lack the knowledge that is needed for such tasks. In this paper, we aim to improve their performance through a self-supervised intermediate fine-tuning strategy based on PubMed abstracts. Our solution builds on the observation that many of these abstracts are case reports, and thus essentially patient case descriptions. As a general strategy, we propose to fine-tune biomedical language models on the task of predicting masked medical concepts from such abstracts. We find that the success of this strategy crucially depends on the selection of the medical concepts to be masked. By ensuring that these concepts are sufficiently salient, we can substantially boost the performance of biomedical language models, achieving state-of-the-art results on two benchmarks.

## introduction natural language processing (nlp) in the biomedical domain poses a number of particular challenges. for this reason, several language models (lms) that are specialised towards the biomedical domain have been proposed, including biobert @xcite , clinicalbert @xcite , scibert @xcite , and @xcite . recent work has focused on analysing the capabilities of such models @xcite @xcite and enhancing them further @xcite @xcite . broadly speaking, biomedical lms have proven successful in capturing the meaning of specialised terminology, but they have been far less successful in enabling medical reasoning, e.g. question: a 38-year-old woman comes to the emergency department because of progressive headache, blurry vision, and nausea for 1 day. four days ago, she was diagnosed with a right middle ear infection. she appears lethargic. her temperature is 39.1°c (102.3°f), and blood pressure is 148/95 mm hg. ophthalmologic examination shows bilateral swelling of the optic disc. the corneal reflex in the right eye is absent. sensation to touch is reduced on the upper right side of the face. serum studies show increased concentrations of fibrin degradation products. which of the following is the most likely diagnosis? for predicting a likely diagnosis from a given patient case description. currently, the main strategies for alleviating this latter issue have centered on incorporating structured knowledge, especially in the form of knowledge graphs. for example, @xcite proposed a method to integrate a large biomedical knowledge graph into a language model through the use of adapters @xcite , while @xcite used graph neural networks to jointly reason about language model outputs and knowledge graphs. we focus on the task of interpreting patient case descriptions. to illustrate this task, ideally, biomedical language models for interpreting patient case descriptions would be pretrained on a task that involves predicting diagnoses, or other salient aspects of these patient cases. unfortunately, beyond the training fragment of benchmarks such as medqa, such labelled data is not readily available. as an alternative, we propose to generate a pseudo-labelled dataset, based on the heuristic that whenever a case descriptions mentions a disease, it is likely (although by no means guaranteed) that this disease is a valid diagnosis, and similar for other medical concepts such as treatments. to get access to a large set of case descriptions, we rely on abstracts of published case reports. in particular, starting from a collection of pubmed abstracts, we first use a simple heuristic to identify those that are likely to correspond to case reports. given a case report that mentions some disease, we then fine-tune the language model on the task of predicting that disease. note that the target disease is masked, as the task would otherwise be trivial. the pre-training task is formulated as a binary classification problem, i.e. given a patient description and a disease, is that disease the correct diagnosis (or more precisely, is it the disease that was masked). this formulation has the advantage that the input format is similar to that of multiple-choice question answering (qa) and natural language inference (nli). beyond diseases, we also experiment with predicting masked treatments. similar to the usual masked language modelling objective, our pre-training task involves making predictions about masked text spans. however, due to the fact that we specifically mask diseases and treatments, we hypothesize that this will improve the model's ability to take the whole case description into account when making predictions. finally, note that we consider this to be an intermediate fine-tuning step. in other words, we start from a state-of-the-art biomedical language model, which is then fine-tuned on the proposed task, before finally being fine-tuned on a downstream task. we find that this intermediate fine-tuning leads to substantial improvements in downstream tasks, even when using a biomedical lm that was already pre-trained on pubmed. to some extent, this comes from the fact that we specifically fine-tune the model on case reports. however, this in itself is not sufficient. to achieve good results, we find that a careful selection of the target concepts is needed. for instance, strong results are obtained when only masking medical treatments. when masking diseases, the improvements over the baseline are sometimes smaller. this is surprising, given that most questions in the considered benchmarks are about diagnosing diseases. upon closer inspection, the under-performance of strategies that rely on masking diseases appears to be related to the fact that diseases can be mentioned for two common reasons: (i) because the patient has been diagnosed with that disease, which is the case that underpins the intuition behind our proposed approach, or (ii) because the disease is relevant to the medical history of the patient. in the latter case, only a small part of the abstract may be relevant to the disease, which hampers the extent to which the model learns to focus on the case description as a whole. to address this issue, we propose to split abstracts in which multiple diseases are mentioned. despite the simplicity of the overall approach, our fine-tuning strategies enable significant improvements over the current state-of-the-art in two benchmarks that are focused on patient case descriptions: medqa @xcite and diskne @xcite . 1 ## related work the standard paradigm in nlp at the moment is to fine-tune a pre-trained lm, such as bert @xcite , on task-specific training data. however, it has been observed that adding an intermediate step, where the lm is first fine-tuned on a different task, for which training data is more abundant, can be highly beneficial @xcite @xcite @xcite . several works have investigated the role of intermediate tasks, in particular with the aim of analyzing when and why results improve @xcite . for the biomedical domain, one strategy has been to rely on transfer learning from generaldomain tasks. for instance, soni and roberts (2020) use general-domain question answering for intermediate training, to improve a clinical question answering system. another strategy has been to rely on different, but related tasks, such as pre-training on natural language inference to develop a question answering system @xcite . furthermore, several authors have proposed techniques for infusing the knowledge from biomedical knowledge graphs into lms @xcite @xcite . more closely related to our approach, @xcite propose a strategy which relies on the structure of wikipedia to infuse knowledge about diseases. for instance, to teach the model about how diseases are treated, they rely on the fact that diseasecentric wikipedia articles tend to have a section called treatment. they then combine the content of that section with a generated question-style sentence mentioning the aspect considered (i.e. treatment in this case) and a masked disease. however, rather than infusing encylopedic knowledge, our aim is to teach lms to interpret patient case descriptions. another related approach was introduced by @xcite , who propose to fine-tune a biomedical language model by using a masked language modelling objective which is modified such that only biomedical concepts are masked. this approach has some similarities with our work, e.g. the idea of masking biomedical concepts as an intermediate fine-tuning task, but there are also some clear differences. first, we formulate our task as a binary classification problem, rather than masked language modelling. moreover, we specifically target diseases and treatments, and we only mask one concept at a time (although all occurrences of that concept are masked). finally, since we focus on paragraph-level understanding, we pay particular attention to how these input paragraphs can be selected. as we will see, each of these differences has a clear impact on the empirical results. ## proposed method we consider the problem of making predictions from patient case descriptions. for instance, given a description that lists symptoms and other information about the patient (e.g. gender, age, and medical history), we would like to infer the corresponding diagnosis or to recommend suitable treatments. we are specifically interested in the potential of using freely available case reports from the medical literature to improve the ability of standard biomedical lms to make such predictions. in section 3.1, we first explain our overall strategy. subsequently, in section 3.2 we describe the specific variants that we included in our analysis. ## experiments in this section, we empirically analyse the different variants of the intermediate fine-tuning strategy. evaluation datasets we mainly focus on two benchmarks that are specifically focused on interpreting and reasoning about patient cases. first, we use medqa @xcite , which is a multiplechoice question answering benchmark. the questions are taken from medical exams and are specifically asking about what can be inferred from a given patient case description. we use the english version of this dataset (usmle). results for this benchmark are reported in terms of accuracy (acc). second, we use diskne @xcite , which has been derived from mednli @xcite . therefore, to use diskne, a license and access to mednli is required. instances of this benchmark consist of a patient case description and a disease, and the aim is to predict whether that disease can be inferred as diagnosis. this repurposing from the original mednli is of particular relevance to our experiments, given that many instances in mednli can be solved simply with linguistic knowledge. diskne contains a separate training-test split for each disease, and for each split, we consider the task of ranking all test cases, according to our confidence that the given target disease is a valid diagnosis. the results are averaged across all diseases and are reported in terms of mean average precision (map). we use the medical-similar variant of the benchmark. in addition, we also consider the english version of headqa @xcite , as a more general healthcare-oriented qa dataset. this dataset contains a broad variety of healthcare questions, most of which do not involve patient descriptions. however, the questions are designed to require complex medical reasoning. as such, we use this benchmark to analyse whether our proposed approach may also benefit such settings. ## conclusions we have proposed a strategy for intermediate finetuning of biomedical language models, to improve their ability to interpret patient case descriptions. the core of our strategy is to exploit abstracts of case reports found in the literature, as a surrogate of patient case descriptions, and to rely on the heuristic that diseases and treatments that are mentioned in such abstracts are likely to correspond to diagnoses and recommendations, respectively. despite its conceptual simplicity and without the cost of manual annotation, this approach was found to lead to clear performance gains, setting a new state-ofthe-art in medqa and diskne, while also benefitting more diverse datasets such as headqa.