Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answering (2025)

Abstract: Large Language Models (LLMs) perform well in general QA but often struggle in domain-specific scenarios. Retrieval-Augmented Generation (RAG) introduces external knowledge but suffers from hallucinations and latency due to noisy retrievals. Continued pretraining internalizes domain knowledge but is costly and lacks cross-domain flexibility. We attribute this challenge to the long-tail distribution of domain knowledge, which leaves partial yet useful internal knowledge underutilized. We further argue that knowledge acquisition should be progressive, mirroring human learning: first understanding concepts, then applying them to complex reasoning. To address this, we propose Selct2Know (S2K), a cost-effective framework that internalizes domain knowledge through an internal-external knowledge self-selection strategy and selective supervised fine-tuning. We also introduce a structured reasoning data generation pipeline and integrate GRPO to enhance reasoning ability. Experiments on medical, legal, and financial QA benchmarks show that S2K consistently outperforms existing methods and matches domain-pretrained LLMs with significantly lower cost.

## introduction with the rapid advancement of large language models (llms), their effectiveness in general question answering has been widely validated @xcite @xcite . however, llms still exhibit noticeable performance gaps in domainspecific qa tasks @xcite . to address these challenges, a variety of approaches have been explored to improve domain-specific qa (dsqa) performance. a common solution is the use of retrieval-augmented generation (rag) @xcite ; @xcite @xcite , where an retriever is used to access external knowledge from a domain corpus. while rag helps incorporate up-to-date information, it introduces extra latency and computation due to redundant retrievals. additionally, distribution mismatches may lead the retriever to return irrelevant or conflicting information, increasing the risk of hallucinations @xcite @xcite @xcite . another line of research focuses on enhancing domain adaptation through continued pretraining @xcite @xcite @xcite . these methods can achieve strong performance, but they are extremely resource-intensive and often lack transferability to other domains. (biomistral @xcite requires training on a corpus of three billion tokens.) we argue that the fundamental reason behind llms' poor performance in dsqa lies in the longtail distribution of domain knowledge in pretraining data. as illustrated in figure 1 , llms have already internalized parts of domain knowledge during pretraining. while this knowledge is often incomplete, it can complement or even correct external domain inputs, making external-only methods suboptimal. furthermore, we believe knowledge acquisition should follow a human-inspired staged progression-first achieving conceptual comprehension, then advancing to complex reasoning. building on this insight, we propose a low-cost post-training framework, select2know (s2k), for domain-specific question answering, which integrates both internal parametric knowledge and external domain knowledge. specifically, we first introduce a token-level internal-external knowledge self-selection strategy to construct fusion training data. we then propose selective supervised fine-tuning (selective sft) to guide the model toward focusing on domain knowledge it has not yet mastered. in addition, we design a structured data generation pipeline to efficiently produce high-quality reasoning data, and incorporate group relative policy optimization (grpo) @xcite to enhance the model's ability to apply learned knowledge to real-world reasoning tasks. our main contributions are as follows: ## problem definition we aim to design a general pipeline that enables llms to efficiently generalize to domain-specific qa tasks with minimal cost. to closely reflect real-world scenarios, we make the following assumptions: (1) no existing qa training datasets are available in the target domain. our goal is to develop a pipeline p such that the resulting domain-adapted model m @xmath0, d) achieves strong performance on the domain qa task t qa . formally, we aim for perf(m d , t qa ) ≫ perf(m 0 , t qa ), where perf(•) denotes the evaluation performance on domain qa tasks. ## methods we introduce s2k, a low-cost post-training framework for adapting general llms to domain-specific qa. as illustrated in figure 2 , s2k first extracts question-style meta knowledge from raw domain corpora (section 3.1.1). we then design a tokenlevel self-selection mechanism to fuse internal and external knowledge (section 3.1.2), complemented by selective sft, which guides the model to focus on unfamiliar domain knowledge (section 3.2). we further introduce structured reasoning data generation pipeline (section 3.1.3), and incorporate grpo to enhance the model's reasoning ability for complex real-world scenarios (section 3.3). ## experiments we organize our experiments as follows: section 4.1 details the experimental setup. section 4.2 provides a quantitative comparison between our method and other question-answering enhancement paradigms. section 4.3 analyzes the sensitivity of key hyperparameters, revealing underlying mechanisms of our method. section 4.4 presents ablation study to examine the contribution of each module. finally, section 4.5 provides case studies illustrating the use of internal knowledge in practice. ## conclusion to address challenges in dsqa, we propose s2k, an efficient framework designed to enhance llm performance in long-tail domains. in vertical domains where no readily available qa datasets exist, s2k enables effective transfer and generalization of qa capabilities using only raw corpora. experiments across multiple representative vertical domains demonstrate its effectiveness. in addition to strong accuracy, s2k achieves comparable performance to domain-pretrained models at significantly lower cost.