Interactive Span Recommendation for Biomedical Text (2023)

Abstract: Motivated by the scarcity of high-quality labeled biomedical text, as well as the success of data programming, we introduce KRISS-Search. By leveraging the Unified Medical Language Systems (UMLS) ontology, KRISS-Search addresses an interactive few-shot span recommendation task that we propose. We first introduce unsupervised KRISS-Search and show that our method outperforms existing methods in identifying spans that are semantically similar to a given span of interest, with >50% AUPRC improvement relative to PubMedBERT. We then introduce supervised KRISS-Search, which leverages human interaction to improve the notion of similarity used by unsupervised KRISS-Search. Through simulated human feedback, we demonstrate an enhanced F1 score of 0.68 in classifying spans as semantically similar or different in the low-label setting, outperforming PubMedBERT by 2 F1 points. Finally, supervised KRISS-Search demonstrates competitive or superior performance compared to PubMedBERT in few-shot biomedical named entity recognition (NER) across five benchmark datasets, with an average improvement of 5.6 F1 points. We envision KRISS-Search increasing the efficiency of programmatic data labeling and also providing broader utility as an interactive biomedical search engine.

## introduction one of the major challenges in developing machine learning models for biomedical text analysis is the scarcity of high-quality labeled data. manual annotation of biomedical text is a time-consuming process that demands specialized expertise, leading researchers to investigate alternative methods such as weak supervision @xcite @xcite and active learning @xcite to address this bottleneck. programmatic data labeling @xcite , a form of weak supervision in which domain experts develop heuristics (labeling functions) to provide noisy labels for large datasets, has been shown to be effective in leveraging domain expertise. however, developing diverse and high-quality labeling functions can be challenging, as it requires knowledge of the programmatic rule specification. some techniques have been proposed to suggest labeling functions to users @xcite @xcite , but they still rely on users' understanding of rule specifications to evaluate or modify the labeling functions. to address this challenge, we introduce an interactive span recommendation task. our key idea is to train a single model and adapt it to human feedback, enabling it to understand and treat similarity at various levels of granularity. this approach eliminates the need to train multiple models for different notions of similarity. conventional entity linking is one such notion of similarity, where a user may want to identify all mentions of a specific concept, such as "hypertension disease" from the unified medical language system (umls). however, a user may want the flexibility to iden-tify not only mentions of "hypertension disease" but also those of "hypertension treatments" and "hypertension comorbidities" simultaneously. this task extends beyond entity linking and can be more broadly described as interactive span recommendation. to tackle the interactive span recommendation task, we propose kriss-search, a method that enables domain experts to develop span recommendation models for searching unlabeled corpora. a crucial aspect of model performance in kriss-search is the choice of embedding space. the umls ontology offers a comprehensive set of biomedical concepts organized hierarchically. we adapt the umls-based self-supervised training technique of krissbert to generate the embedding space used by our method. we evaluate two versions of kriss-search. unsupervised kriss-search takes a single userselected query span from a biomedical corpus as input and returns semantically similar spans. however, in some cases, this single measure of similarity may not adequately overcome the inherent ambiguity in identifying spans based on one example. to address this limitation, we introduce supervised kriss-search, which employs active learning to incorporate human feedback and refine the concept of similarity used in the unsupervised version. in the context of programmatic data labeling, we envision unsupervised kriss-search recommending terms for users to incorporate into labeling functions and supervised kriss-search directly generating noisy labels, providing a more flexible alternative to labeling functions. our main contributions can be summarized as follows: ## methods in this paper, we compare various training strategies for the bert-base @xcite (100 million parameters) architecture in order to address our proposed task. while the training strategies discussed in this paper are specific to the bertbase architecture, they can also be applied to larger models. the methods we evaluate can be characterized as "contextual," "in-domain," "contrastive," and "interactive." "contextual" methods use the surrounding context to make recommendations, while "in-domain" methods are trained on data specifically related to the biomedical domain. "contrastive" methods utilize semantic similarity and dissimilarity during the training process. "interactive" methods involve human participation to guide model training. the four training strategies we compare are bert, pubmedbert, unsupervised kriss-search, and supervised kriss-search. each strategy implements an additional descriptor in the order they were listed, with supervised kriss-search implementing all four. to highlight the distinctions between bert, pub-medbert, unsupervised kriss-search, and supervised kriss-search, consider the following example. in the sentence "the patient received a pt assay," the query span "pt" refers to the concept "prothrombin time assay". bert, which is not specifically tailored to the biomedical domain or designed to employ contrastive or interactive techniques, may surface a false positive "platinum," which shares the same abbreviation "pt" but is not relevant to the biomedical domain. similarly, pub-medbert, which is trained on biomedical data but does not utilize contrastive learning, may generate a false positive "physical therapy," which is in the biomedical domain but semantically dissimilar to the query span. in contrast, both unsupervised and supervised kriss-search utilize contrastive learning, which makes them more likely to recommend semantically similar spans, such as "prothrombin time assay", as this similarity is explicitly incor-porated into the training process. now consider another example: "decrease in right lung mass compared to prior imaging". here, the user is interested in the query span "decrease in right lung mass", which represents a relationship between "decrease" and "right lung mass". in this scenario, the concept of similarity is complex and may require the interactive feature of supervised kriss-search to surface similar spans. ## conclusion we demonstrate that unsupervised kriss-search outperforms existing embedding methods for biomedical interactive span recommendation. supervised kriss-search utilizes humans-in-theloop to achieve high levels of performance on both granular and course grain span recommendation. future work will investigate whether kriss-search does indeed address the initial motivation -aiding programmatic data labeling as part of an interactive biomedical nlp system. nonetheless, we envision kriss-search being broadly useful as a general purpose interactive biomedical search engine.