G en P o E : Generative Passage-level Mixture of Experts for Knowledge Enhancement of LLM s (2025)

Abstract: Typically, parametric adaptation methods such as domain-adaptive pretraining (DAP) and retrieval-augmented generation (RAG) have been considered effective approaches for adapting large language models (LLMs) to new knowledge or domains. To unify positive effects of parametric adaptation and RAG, this paper proposes \textbf{GenPoE} , i.e., “generative’’ passage-level mixture of experts (MoEs) for enhancing knowledge of LLMs. The key component is its novel \textit{MoE-generating hypernetwork} which takes in-context retrieved passages and generates their “expert’’ parameters, where these generated parameters are then integrated into LLMs by forming expert networks. With its use of “generated’’ parameters, GenPoE does not require a separate parameter training or finetuning stage, which is often costly. By parameterizing passages into expert networks, GenPoE likely exhibits robustness even when the retrieved passages are irrelevant. Experiment results in two open-domain question answering (QA) tasks present that GenPoE shows improved performances over other passage-level knowledge editing, and its combination of RAG produces superior performances over RAG. Our data and code will be available at https://github.com/Liu-Xuebing/GenPoE.

## introduction large language models (llms) have achieved remarkable success across a wide range of natural language processing tasks, especially in knowledgeintensive applications due to their vast pretrained knowledge @xcite . however, the dynamic nature of real-world knowledge necessitates continuous and efficient updates to these models. existing methods for enhancing knowledge of llms are broadly categorized to two approachesparametric adaptation and rag @xcite ; 1) parametric adaptation -such as domain-adaptive pretraining (dap) @xcite or parametric knowledge editing @xcite foot_0 -modifies or updates the parameters of large language models (llms). however, it often requires substantial computational and memory cost due to llm's huge parameters and the training time, and may suffer from the catastrophic forgetting, which can degrade previously learned knowledge and task performance. 2) rag retrieves relevant passages and uses them as additional in-context prompts before generating responses. however, rag likely suffers from performance degradation due to noisy or irrelevant retrieved content @xcite , and its lack of parameter-level integration may limit the model's ability to accumulate and evolve knowledge over time. given these characteristics, we assume that parametric adaptation and rag serve as complementary solutions, each addressing the limitations of the other. rag is less likely to suffer from computational overhead and catastrophic forgetting, while parametric adaptation tends to avoid the influence of noisy or irrelevant retrieved content due to its access to internalized parametric knowledge. to unify these complementary effects, this paper proposes genpoe, i.e., "generative" passage-level moes for enhancing knowledge of llms, by internalizing passages into moe parameters based on our novel moe-generating hypernetwork. more specifically, the moe-generating hypernetwork is trained via meta-learning framework: given a question and its corresponding gold passage, it learns to produce passage-level moe parameters such that the resulting moe-equipped llms maximize the autoregressive likelihood of generating the correct answer to the question. during inference, as an alternative to rag, genpoe employs the moe-generating hypernetwork through a passage injection stage to incorporate retrieved passages. once the top-k passages are retrieved, the hypernetwork dynamically constructs k passage-level moes, which are then integrated with the original llm parameters, as illustrated in figure 1 . experimental results on standard open-domain question answering (qa) datasets-including natural questions (nq) @xcite , and triviaqa (tqa) @xcite -demonstrate that genpoe achieves a stronger passage injection effect by outperforming baseline parametric knowledge editing methods. when combined with rag, genpoe gives rise to a more powerful rag variant that surpasses the performance of existing advanced rag approaches, confirming the complementary relationship between rag and parametric adaptation. our main contributions are summarized as follows: ## methodology figure 2 provides an overview of genpoe, highlighting its two key stages: (1) the meta-learning stage for training the hypernetwork (section 3.2), and (2) the passage injection via moes during inference (section 3.3.1). in the following sections, we first introduce our moe framework that integrates passage-level experts. ## conclusion in this work, we propose genpoe, a hypernetworkdriven mixture-of-experts framework for passagelevel knowledge enhancement, designed to address the limitations of large language models in effectively leveraging external prompts. given a query, relevant passages are dynamically encoded into expert modules via a hypernetwork, allowing knowledge to be injected into the model on-thefly. this design ensures that the generated experts are ephemeral and lightweight, leading to minimal computational overhead and negligible memory growth. such properties make genpoe not only effective in enhancing factual knowledge utilization but also highly suitable for real-world deployment scenarios, where scalability and efficiency are critical. extensive experiments on nq and tqa demonstrate that genpoe successfully incorporates knowledge while avoiding the degradation typically associated with general parameter-updatebased methods. the framework achieves strong editing effectiveness without sacrificing inference efficiency, validating its practicality for knowledgeintensive applications. for future work, we plan to explore more efficient and expressive hypernetwork and mixtureof-experts architectures, as well as stable continual learning techniques to further improve adaptability and long-term scalability. moreover, we aim to extend the framework to more challenging multi-hop reasoning tasks, where multiple passages must be integrated and jointly reasoned over. this would push genpoe beyond single-hop factual injection toward a more general paradigm for dynamic knowledge integration in large language models.