KBA lign: Efficient Self Adaptation on Specific Textual Knowledge Bases (2025)

Abstract: Although retrieval-augmented generation (RAG) remains essential for knowledge-based question answering (KBQA), current paradigms face critical challenges under specific domains. Existing methods struggle with targeted adaptation on small-scale KBs: vanilla unsupervised training exhibits poor effectiveness, while fine-tuning incurs prohibitive costs of external signals. We present KBAlign, a self-supervised framework that enhances RAG systems through efficient model adaptation. Our key insight is to leverage the model’s intrinsic capabilities for knowledge alignment through two innovative mechanisms: multi-grained self-annotation that captures global knowledge for data construction, and iterative tuning that accelerates convergence through self verification. This framework enables cost-effective model adaptation to specific textual KBs, without human supervision or external model assistance. Experiments demonstrate that KBAlign can achieve 90% of the performance gain obtained through GPT-4-supervised adaptation, while relying entirely on self-annotation of much smaller models. KBAlign significantly improves downstream QA accuracy across multiple domains with tiny costs, particularly benefiting scenarios requiring deep knowledge integration from specialized corpora. We release our experimental data, models, and process analyses to the community for further exploration(https://anonymous.4open.science/r/KBAlign-D160).

## introduction large language models (llms) have demonstrated their general capabilities across a wide range of downstream tasks @xcite , and the factual reliability of the models could be enhanced with common techniques such as retrieval augmented generation (rag) @xcite ). when applied in specific domains, however, the adaptation of models to knowledge base (kb) materials remains a crucial strategy to further improve performance @xcite . intuitively, adapted models align with the knowledge scope of the given kbs, and can directly memorize some domain information or better utilize the retriever through query rewriting. for example, both general qa model and the retriever may regard "llm" as an ai term, while the legal aligned model can generate disambiguated context in which "llm" stands for master of laws. existing adaptation methods usually construct domain models with large-scale training data @xcite , while there are quite specific needs in real-world scenarios corresponding to small-scale textual kbs, such as providing customized services based on user-specific document repositories, or plug-and-play integration of modules. in these cases, simple lm training on raw data may degrade performance, necessitating alternative approaches to align with specific domains @xcite . targeted fine-tuning, on the other hand, typically involves the incorporation of external knowledge signals @xcite to transform data into more structured tasks. when faced with constraints such as confidentiality, convenience, and limited computational resources, involving human annotation or relying on online large models becomes unpractictible. therefore, a low-cost adaptation to small-scale kbs that does not rely on external supervisions is urgently needed. drawing an analogy to the human learning process, rag is similar to open-book tests in which students could query kb materials. if they conduct self-study in advance, quickly grasping the fundamental content of the books themselves, the effectiveness and efficiency in tests can be improved. based on this idea, we purpose kbalign, a highly efficient self-adaptation approach tailored to specific textual kbs comprising self-improvement learning. generally, we align the model with the small domain in a highly efficient and completely self-supervised manner. as shown in fig. 1 , for self annotation, we organize the original kb materials in multiple grains and conduct self annotation to get instruction-response pairs that can cover various downstream task scenarios. for iterative tuning, we require the model to check its own responses and help modify common mistakes in the current stage for a faster convergence. meanwhile, we conduct targeted inference in which strategies such as query expansion and confidence verification are adopted to refine the response. experiments on fact qa, long-form qa, and professional field test have shown the effectiveness of our method across different backbone models. with a low cost, kbaligned models generally master the domain knowledge, memorize part of detailed information in kb, and also obtain some capability of better utilizing retrieved context. side experiments including ablation studies and performance curve analyses identify the most efficient self-annotated data amount and optimal training volumes, offering valuable guidance for effectively applying kbalign in practical scenarios. our main contributions are as follows: (1) we propose kbalign, a novel method for autonomous llm adaptation tailored to textual kbs. it helps llms perform kb adaptation relying entirely on self annotations; (2) we provide empirical insights into efficient self-adaptation to kbs, offering practical parameters and settings for deploying kbalign; knowledge enhancement. for some specific downstream requirements, there often exist highquality knowledge materials (e.g., domain kbs, personal documents or records), of which the data amount is not enough for model tuning, and knowledge enhancement methods can help improve the performance. there are two mainstream solutions. the first one is to rely on the strong in-context learning capability of llms @xcite , and adopt rag @xcite to enhance the model. apart from textual materials such as internet passages, it is proven that integrating special kbs and tools is also a good approach to improve the model performance with specific knowledge @xcite @xcite . to provide more useful information in context, strategies including designing better queries for retrieval are proposed @xcite . the second solution is to augment training data based on knowledge materials. llms can help synthesize data in more styles @xcite or convert the original data into formats more suitable for training @xcite . our method is special, emphasizing the self annotation instead of introducing new llms into the system. self improvement. there are some works exploring the self-improve capability of llms, most of which focus on the automatic generation and selection of reasoning steps for existing answers, being helpful in tuning @xcite and inference @xcite . self-play fine-tuning in an iterative manner (chen et al., 2024) also unlocks the full potential of golden data. even without the ground-truth answers, intern consistency of llms can be adopted as an important supervision signal that can achieve improvement @xcite . nevertheless, challenge remains for human-like self improvement, such as how to self correct the reasoning process @xcite . we observe the human learning process and design corresponding self-improvement methods. ## discussion in this paper, we introduced kbalign, a highly efficient self-adaptive method tailored for specific kbs. during the tuning stage, inspired by human learning strategies such as summarizatio and self-reflection, we propose a combined long-and short-dependency annotation method, as well as an iterative tuning approach. these techniques enable low-cost targeted training data augmentation and efficient adaptation without requiring external supervision. in the inference stage, we enhance the model's performance on kbqa tasks using query expansion and sampling-based self-verify strategies. our approach demonstrates significant improvements across various datasets spanning different domains and formats. additionally, detailed analysis provides empirical guidance regarding the best data amount required. in future work, we aim to focus on adaptive performance enhancement in more complex scenarios, such as utilizing new tools. additionally, we will explore the integration and collaboration of multiple models adapted to different subdomains.