Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLM s via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains (2025)

Abstract: Large Language Models (LLMs) have demonstrated an impressive level of general knowledge. However, they often struggle in highly specialized and sensitive domains such as drug discovery and rare disease research due to the lack of expert knowledge, which is often costly to obtain. In this paper, we propose a novel framework (PU-ADKA) designed to efficiently enhance domain-specific LLMs by actively engaging domain experts within a fixed budget. Unlike traditional fine-tuning approaches, PU-ADKA proactively identifies and queries the most appropriate expert from a team, taking into account each expert’s availability, competency, knowledge boundaries, and consultation cost. We train PU-ADKA using simulations on PubMed publication data and validate it through domain expert interactions, showing promising improvements in LLM domain knowledge acquisition. Furthermore, our experiments with a real-world drug development team validate that PU-ADKA can significantly enhance LLM performance in specialized domains while adhering to strict budget constraints. In addition to outlining our methodological innovations and experimental results, we release a new benchmark dataset, CKAD, for cost-effective LLM domain knowledge acquisition to foster further research in this challenging area.

## introduction recent advancements in large language models (llms) have led to impressive performance gains across a wide range of tasks @xcite @xcite . however, these gains are not uniformly observed across all domains. in highly specialized and cost-sensitive fields, such as drug discovery and rare disease exploration, the acquisition of domain knowledge remains a challenge. traditional approaches like reinforcement learning from human feedback (rlhf) @xcite have demonstrated value in general settings, yet they struggle in contexts where expert knowledge is extremely expensive and sparse. this scenario is particularly pronounced in domains where domain expertise is fragmented among professionals with diverse competencies and availability constraints @xcite . consequently, there is a pressing need for novel approaches that can efficiently integrate domain expert feedback into llms while operating under tight budgetary and expert availability restrictions. to respond to this demand, we propose positive unlabeled active domain knowledge acquisition (pu-adka), which is designed to selectively engage with domain experts and acquire targeted feedback that can significantly enhance the performance of llms in specialized fields. unlike conventional fine-tuning methods that passively incorporate affordable human feedback @xcite , pu-adka actively queries the most appropriate expert from a team given each member's computational profile. the model can elaborately consider factors such as the candidate expert's knowledge boundary, cost of consultation, and expert availability, thereby optimizing the knowledge acquisition process within a fixed budget (e.g., total @xmath07, while consulting mary, a different expert, about cat-t for $4, ensuring cost-effective expert engagement. this dynamic querying mechanism allows the llm to refine its domain knowledge efficiently, making it particularly useful in critical domains like drug discovery and rare disease research, where expert knowledge is both sparse and expensive. our contributions in this paper are threefold and can be summarized as follows: ## conclusion and future work we propose pu-adka, a cost-aware active learning framework that enhances llms by selectively engaging domain experts based on their expertise, availability, and annotation cost. pu-adka improves budget efficiency and llm performance, as validated through simulations and real-world biomedical tasks. the release of the ckad dataset supports further research on domain-specific llm tuning. in future work, we plan to explore alternative backbone models and extend pu-adka to specialized domains beyond biomedical domain.