Smart-Searcher: Incentivizing the Dynamic Knowledge Acquisition of LLM s via Reinforcement Learning (2025)

Abstract: Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the model’s internal knowledge.In this paper, we introduce Smart-Searcher, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. Smart-Searcher employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model’s internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning.Our experiments demonstrate that Smart-Searcher outperforms previous RAG and reasoning methods and achieves efficient retrieval.The code is available at https://github.com/RUCAIBox/R1-Searcher-plus .

## introduction large language models (llms) @xcite have demonstrated remarkable reasoning capabilities by only leveraging the information encoded in their parameters. however, their reliance on static, internal knowledge leads to notable limitations. at the simultaneously, this reliance easily leads to hallucinations @xcite , so llms may struggle with open-ended tasks @xcite . therefore, it is crucial to enable llms to access external information when * equal contributions. to address this issue, extensive research has focused on augmenting llms with external information sources (i.e., rag @xcite ). early approaches emphasize specific prompting strategies to guide llms @xcite and subsequent studies investigate to distill this capability into smaller llms through supervised fine-tuning (sft) @xcite . however, recent findings suggest that sft-based distillation can cause models to memorize solution paths, limiting their generalization to novel scenarios @xcite . further proposals include a test-time scaling method @xcite , notably employing the monte carlo tree search (mcts) framework @xcite to enhance solutionfinding by expanding the search space during inference, but this approach incurs significant inference overhead, reducing its practicality for widespread use. recent studies employ end-to-end outcomebased reinforcement learning (rl) to train models, enabling them to autonomously explore external retrieval environments during inference @xcite . this approach fosters the development of self-directed retrieval capabilities in llms as they reason. however, such models often become overly reliant on external search engines after training, neglecting the utilization of their internal knowledge. in practice, when humans attempt to solve factual questions, they first recall their internal knowledge, and only turn to search engines when they recognize a lack of information. at the same time, after obtaining the external searched information, humans would memorize this knowledge for future use. for llms, extensive pretraining on large-scale data has already endowed them with substantial internal knowledge @xcite . therefore, it is essential to equip models with the ability to dynamically switch between internal and external knowledge sources as needed. furthermore, models should be encouraged to effectively memorize useful information encountered during training @xcite , progressively enriching their internal knowledge and continuously evolving toward greater intelligence. in this paper, we present smart-searcher, a novel framework designed to teach llms to adaptively leverage both internal and external knowledge. this is a further exploration of our previously proposed r1-searcher. we adopt a two-stage training strategy: sft cold-start and rl for dynamic knowledge acquisition. in the first phase, we employ reject sampling to collect data that meets the format requirements and perform a cold start with sft. in the second stage, we further train the model using outcome-based rl to guide the model in dynamically acquiring knowledge, which is to encourage reliance on internal knowledge when confident, and invoke external search mechanisms when uncertain, based on a carefully designed reward design. additionally, we further introduce a memory mechanism, enabling the model to retain knowledge encountered during training by converting and memorizing retrieved content. this mechanism continuously enriches its internal knowledge, empowering it to effectively balance internal reasoning and external retrieval through autonomous exploration and timely memorization. to verify the effectiveness of smart-searcher, we conduct extensive experiments based on qwen-2.5-7b-instruct. notably, our method surpasses the strong baseline by up to 4.3% and reduces the retrieval count by 42.9% compared to vanilla rlbased approaches. our principal contributions are as follows: ## preliminary to enhance the performance of llms in opendomain multi-hop question answering tasks @xcite , in this work, we focus on enabling the model to autonomously decide when to use its internal knowledge or to invoke an external retriever to answer the given questions with the llm selfimproving paradigm, which can improve both reasoning effectiveness and efficiency. to this end, we introduce three special tokens to format the llm reasoning process, i.e., <internal>, <external>, @xmath0. concretely, during the reasoning process, the llm with parameters θ determines whether the current step requires external knowledge to help perform reasoning. if so, it triggers @xmath1, which is sent to a retriever to retrieve the top-k relevant documents d @xmath2,k } k @xmath3. these retrieved documents are incorporated into the reasoning path with another special @xmath4. otherwise, the model directly generates the related internal knowledge enclosed @xmath5. after several reasoning steps, the llm obtains the final answer and stops the reasoning process. since our approach is orthogonal to the rl algorithm, we conduct the experiments based on a widely used rl algorithm, i.e., rein-force++ @xcite , which is a stable rl algorithm without the critic model. to better accommodate the retrieval scenario, we mask the retrieved documents during the loss calculation process, as they serve as environmental observations rather than model-generated content. formally, for each question q, we first samples a group of outputs next, we incorporate the kl regularization into the reward scores r ϕ (q, o i,≤t ) , and then normalize the advantage scores: we utilize â to denote the set of all advantages in the global batch that contains âi,t . after obtaining the advantage scores, we set the mask value m (i, t) as 0 if this token belongs to an external document, otherwise we set m (i, t) = 1. finally, we employ the masks m (i, t) in the objective function to remove the influence of retrieved documents: pi,@xmath6,t âi,t , clip (p i,t , 1ε, 1 + ε) âi,t , ## methodology in this part, we introduce the smart-searcher framework, which aims to teach llm to adaptively utilize internal and external knowedge through two critical stages, i.e., sft cold-start (section 4.1) and rl for dynamic knowledge acquisition (section 4.2). concretely, in the first stage, we utilize the curated data to perform sft on the model, to standardize its responses in a specific format and enable it to leverage external retrievers and internal knowledge adaptively. in the second stage, we employ rl on llm that encourages the model to explore more effective actions and behaviours, and further incorporate the internal knowledge utilization encouragement and external knowledge memorization in the training process, to guide the model to dynamically acquire knowledge and continuously enrich its internal knowledge, which can lead to higher reasoning efficiency. ## further analysis 6.1 ablation study to validate the effectiveness of our proposed smart-searcher framework, we conduct a comprehensive ablation analysis of its key design elements. we design five distinct variants: ( 1 2 . as observed, all ablated variants exhibit a decline in performance compared to our full method, underscoring the integral contribution of each component. specifically, w/o stage-1 leads to a degra-dation in performance along with an increase in retrieval count. meanwhile, the performance of w/o stage-2 drops significantly, primarily because simple sft causes the model to over-rely on its internal knowledge. this highlights the necessity of our two-stage pipeline. futhermore, during rl training, w/o r group during rl also leads to a reduction in performance. this demonstrates the positive impact of group reward in successfully guiding the model to be more selective with external searches and to rely more on its internalized knowledge. similarly, w/o l m results in lower scores and a slight increase in retrieval count indicating that the memory mechanism for external knowledge can effectively internalize retrieved content as intrinsic knowledge of the model. ## conclusion in this paper, we introduced smart-searcher, a novel framework that enables large language models to dynamically integrate and alternate between internal knowledge and external retrieval. this is a two-stage training strategy consisting of an sft cold-start phase and rl for dynamic knowledge acquisition. the rl stage incorporates a reward mechanism to encourage internal knowledge utilization, and a memory module to convert retrieved information into internal knowledge. through this design, smart-searcher empowers llms to perform efficient retrieval-augmented reasoning while continuously enriching their internal knowledge via self-exploration and memory. experimental results on multi-hop tasks demonstrate that smart-searcher outperforms existing rag methods.