Can We Edit LLM s for Long-Tail Biomedical Knowledge? (2025)

Abstract: Knowledge editing has emerged as an effective approach for updating large language models (LLMs) by modifying their internal knowledge. However, their application to the biomedical domain faces unique challenges due to the long-tailed distribution of biomedical knowledge, where rare and infrequent information is prevalent. In this paper, we conduct the first comprehensive study to investigate the effectiveness of knowledge editing methods for editing long-tail biomedical knowledge. Our results indicate that, while existing editing methods can enhance LLMs’ performance on long-tail biomedical knowledge, their performance on long-tail knowledge remains inferior to that on high-frequency popular knowledge, even after editing. Our further analysis reveals that long-tail biomedical knowledge contains a significant amount of one-to-many knowledge, where one subject and relation link to multiple objects. This high prevalence of one-to-many knowledge limits the effectiveness of knowledge editing in improving LLMs’ understanding of long-tail biomedical knowledge, highlighting the need for tailored strategies to bridge this performance gap.

## introduction recently, knowledge editing @xcite has emerged as a promising approach to efficiently update large language models (llms) by injecting new knowledge into their internal knowledge @xcite . these methods have shown remarkable performance in enhancing llms' performance across several general-domain tasks, such as question answering (qa) @xcite , knowledge injection @xcite , and reasoning @xcite . what is the therapeutic procedure of type 1 diabetes insulin therapy what is the therapeutic procedure of co-occur evans syndrome co-occur popular knowledge long-tail knowledge enzyme therapy enzyme therapy knowledge editing 1024 docs llm edited-llm 6 docs ... while knowledge editing methods have proven effective in general-domain tasks, their application to the biomedical domain presents unique challenges @xcite . specifically, realworld biomedical data often exhibit a long-tailed distribution, with a small amount of popular knowledge and a large amount of long-tail knowledge that appears rarely or only once @xcite . for example, the common disease "type 1 diabetes" is mentioned in over 106,138 papers in pubmed @xcite , while a rare disease like "evans syndrome" appears in only about 23 papers @xcite . recent studies indicate that the low frequency of knowledge in the pre-training corpus can hinder llms' understanding of this knowledge @xcite . figure 1 illustrates an example where llms struggle with lowfrequency biomedical knowledge. this is particularly problematic as llms are increasingly being used by healthcare professionals, including doctors, to assist in diagnosis and treatment recommendations @xcite . as llms become more integrated into clinical practice, their ability to accurately handle rare but critical biomedical knowledge becomes essential. this raises a critical question for knowledge editing in the biomedical domain: can knowledge editing methods effectively edit large language models to incorporate long-tail biomedical knowledge? in this work, we present the first comprehensive study to investigate the effectiveness of knowledge editing for long-tail biomedical knowledge. we focus on biomedical knowledge represented as knowledge triples and leverage knowledge probing @xcite to evaluate whether llms have effectively acquired this knowledge. specifically, knowledge probing is a technique that queries llms to assess their internal factual knowledge @xcite . as illustrated in figure 1 , we probe llms with questions generated from biomedical knowledge triples to determine whether they can correctly recall the target knowledge. by comparing the knowledge probing results of llms before and after editing, we can evaluate how effectively knowledge editing enhances llms' ability to handle long-tail biomedical knowledge. our key findings are: ## background and definitions this section defines long-tail biomedical knowledge and briefly introduces the knowledge probing and editing techniques used in our experiments. ## identifying long-tail biomedical knowledge due to the lack of biomedical datasets specifically designed to evaluate long-tail knowledge, we develop a pipeline to extract such knowledge. in this section, we outline the procedures for extracting long-tail biomedical knowledge, with further details provided in appendix a and figure 7 . we focus on biomedical knowledge represented as knowledge triples and extract these triples from snomed ct @xcite , which is a comprehensive biomedical knowledge graph comprising over 1.4 million clinical triples (benson and grieve, 2021), and widely used to evaluate llms' understanding of biomedical knowledge @xcite . following previous work @xcite , we adopt the cooccurrence numberi.e., how often a triple's subject and object appear in the same documentas a proxy for knowledge popularity. to identify the long-tail knowledge within these triples, we use an entity linking pipeline to compute the cooccurrence number of each triple in the pubmed corpus foot_0 , which is a widely used biomedical corpus for pre-training. in the entity linking pipeline, we use pubtator @xcite to annotate entities in the pubmed corpus and then use sap-bert @xcite to link knowledge triple entities to pubmed entities. subsequently, we compute the co-occurrence number for each triple. long-tail knowledge is defined as triples with a co-occurrence number less than 10 @xcite . as a result, we obtained 59,705, 14,087, and 28,375 triples for the training, validation, and test sets, respectively, stratified by varying levels of co-occurrence. the statistics of the dataset are presented in to evaluate llms ability to understand these triples, we generate question-answer pairs following @xcite . for each triple, we construct a question using the subject and relation, with the object serving as the answer. for example, for the triple ⟨diabetes, treated_by, insulin⟩, the corresponding qa pair is: what is diabetes treated by? answer: insulin. the template for constructing questions is provided in in this section, we investigate the effectiveness of knowledge editing methods in enhancing llms' ability to handle long-tail biomedical knowledge. since some editing methods, e.g., mend @xcite and ike @xcite , require training data, we follow the data splitting strategy proposed by @xcite to divide our clikt dataset into training, validation, and test sets (see ## discussion while our work highlights the challenges llms face in capturing and editing biomedical one-tomany knowledge, we acknowledge that addressing these limitations requires further exploration. we outline several promising directions that may improve performance: (1) retrieval-augmented generation: incorporating external biomedical knowledge by retrieving relevant documents or triples could help llms better handle long-tail biomedical knowledge. this approach has shown promise in open-domain qa @xcite @xcite and may be adapted for biomedical editing with domain-specific retrieval modules; (2) structure-aware finetuning: instead of treating each triple independently, future work could explore fine-tuning strategies that explicitly model the structure of one-to-many knowledge. for example, training objectives can be designed to encourage the model to recognise that multiple objects may be valid for a given subject-relation pair. ## conclusion in this paper, we investigate the effectiveness of knowledge editing methods for addressing the challenges of long-tail biomedical knowledge in llms. our results show that while existing techniques enhance performance on long-tail knowledge, they still fall short compared to their performance on high-frequency knowledge. this disparity is largely due to the prevalence of one-tomany knowledge structures in the biomedical domain, which complicate models ability to accurately represent and edit such information. our results highlight the need for advanced editing techniques specifically designed for long-tail knowledge. these techniques should prioritise strategies for effectively handling the intricacies of one-tomany knowledge scenarios, which are particularly common in the biomedical domain and remain a significant obstacle for current methods.