Improving Neural Models for Radiology Report Retrieval with Lexicon-based Automated Annotation (2022)

Abstract: Many clinical informatics tasks that are based on electronic health records (EHR) need relevant patient cohorts to be selected based on findings, symptoms and diseases. Frequently, these conditions are described in radiology reports which can be retrieved using information retrieval (IR) methods. The latest of these techniques utilize neural IR models such as BERT trained on clinical text. However, these methods still lack semantic understanding of the underlying clinical conditions as well as ruled out findings, resulting in poor precision during retrieval. In this paper we combine clinical finding detection with supervised query match learning. Specifically, we use lexicon-driven concept detection to detect relevant findings in sentences. These findings are used as queries to train a Sentence-BERT (SBERT) model using triplet loss on matched and unmatched query-sentence pairs. We show that the proposed supervised training task remarkably improves the retrieval performance of SBERT. The trained model generalizes well to unseen queries and reports from different collections.

## introduction electronic health record (ehr) retrieval is important for clinicians, staff and researchers. the tools for performing clinically relevant searches could aid in many use cases such as clinical decision support (syeda-mahmood, 2010), auditing, revenue cycle management, and cohort selection for clinical studies. frequently, these searches involve retrieval of patients based on clinical findings that are often captured in unstructured textual reports such as radiology reports, encounter notes, etc. unlike structured query-based lookup of ehr, retrieval of unstructured (free-text) ehrs is much more challenging, requiring a semantic understanding of the underlying clinical conditions present or absent. conventional exact or approximate termbased retrieval methods such as bm25 @xcite often perform poorly in response to ad-hoc queries @xcite , as these methods lack the ability of semantic understanding of the clinical as well as language context. with the emergence of deep learning encoding models, new retrieval methods have emerged with studies showing bert-based neural methods outperforming bm25 on multiple retrieval benchmarks @xcite @xcite @xcite . the bert-based retrieval methods can be classified into two categories: the cross-attention (or interaction-based) models @xcite @xcite and the embedding-based (or representation-based) models @xcite . while the bert-style cross-attention models are very successful, they cannot be directly applied to large-scale retrieval problems because computing the similarity score for every possible query-document pair during inference can be prohibitively expensive. therefore, they were often used as a re-ranker after a initial candidate retrieval round using bm25. the embedding-based methods can pre-encode the documents, and only the queries need to be encoded upon retrieval. retrieval can be achieved via approximate nearestneighbor search in the embedding space very efficiently @xcite . in this study, we focus on the embedding-based retrieval bert models. specifically, we adopted the sentence-level retrieval setting, as studies suggested that the "best" sentence in a document provides a good proxy for document relevance @xcite . different pre-training tasks were used to train the bert-based models for retrieval. the pretraining tasks range from masked language modelling (mlm) over unlabeled free-text to supervised training on labeled datasets such as sts @xcite , ms marco @xcite or trec microblog track @xcite . how-ever, mlm is not tailored for the purpose of information retrieval (ir), and labeled datasets are usually small and not easily accessible. recently, pre-trained models on biomedical corpora such as bioclinicalbert @xcite and biobert @xcite ) can obtain embeddings with medical-domain-specific knowledge, but they were still trained with mlm. early studies @xcite showed that most clinical queries are actually short queries (e.g. a disease or a syndrome). we found that the existing bert models pre-trained with mlm performed poorly on short queries as well as negative queries (i.e. queries asking for lack of a finding). ideally, if a retrieval system could be trained by matched and unmatched query-sentence pairs, in both positive and negated instances, we can expect a higher precision and recall in retrieval. however, manually labeling a large dataset is impractical, particularly for the medical domain where the number of clinical findings is very large. training neural ir models using weak supervision has been previously investigated @xcite . these methods use unsupervised methods (e.g. bm25) or article headings to provide pseudo labels. however, these pseudo labels usually are imprecise and do not consider negative queries. moreover, the article headings are not always available. motivated by these challenges, we present a hybrid approach where we combine automated clinical finding detection with supervised querysentence pair learning. specifically, we use an automatic lexicon-driven concept detection method to detect relevant positive or negative chest x-ray (cxr) findings in sentences. these findings paired with the sentences containing them serve as weakly labeled training data for sentence-bert (sbert) @xcite . the resulting approach avoids manual annotation and can be scaled for training on a large number of query-sentence pairs. we show that the proposed training task remarkably improves the retrieval performance of sbert on datasets with automatic annotations and human annotations. ## discussion in this paper we demonstrated that the proposed supervised pre-training tasks with automated annotation can greatly improve the ir performance of sbert on short and negative queries. the proposed labeled data generation method can also be used to train the cross-attention bert models for further improvement when computation speed is not the bottleneck. we focused on short queries in this study, and bm25 still performs well on positive queries. the embedding-based bert models are expected to show more advantages over bm25 on complicated queries that require semantic understanding. having the comprehensive negation and finding labels for each sentence also allows us to assemble more complicated queries that include more than one finding, such as "a and b" or "a without c" where a, b and c represent three different findings. these more challenging tasks can be explored in the future work. the label generation tool can also be extended to training ir models in domains other than medical domain, such as finance, law, or retail, provided with the corresponding lexicons. in this study we did not evaluate retrieval on the report-level because we have the sentence-level annotations, which enable fine-grained evaluation. the report-level evaluation can be included in the future work. ## conclusion in this work we proposed to generate querysentence pairs automatically using a cxr lexicon for training embedding-based bert models on the ehr retrieval problem. we showed that the fine-tuned sbert obtained a substantial performance gain over the other pre-trained models. the trained model can also generalize well to unseen queries and data from another source. the proposed method can be especially helpful in training and evaluating neural ir models in domains with limited human-labeled data.