U ni HGKR : Unified Instruction-aware Heterogeneous Knowledge Retrievers (2025)

Abstract: Existing information retrieval (IR) models often assume a homogeneous structure for knowledge sources and user queries, limiting their applicability in real-world settings where retrieval is inherently heterogeneous and diverse. In this paper, we introduce UniHGKR, a unified instruction-aware heterogeneous knowledge retriever that (1) builds a unified retrieval space for heterogeneous knowledge and (2) follows diverse user instructions to retrieve knowledge in specified types. UniHGKR consists of three principal stages, including heterogeneous self-supervised pretraining, text-anchored embedding alignment, and instruction-aware retriever fine-tuning, enabling it to generalize across varied retrieval contexts. This framework is highly scalable, with a BERT-based version and a UniHGKR-7B version trained on large language models. Also, we introduce CompMix-IR, the first native heterogeneous knowledge retrieval benchmark. It includes two retrieval scenarios with various instructions, over 9,400 question answer (QA) pairs, and a corpus of 10 million entries, covering four different types of data. Extensive experiments show that UniHGKR consistently outperform state-of-the-art methods on CompMix-IR, achieving up to 6.36% and 54.23% relative improvements in two scenarios, respectively. Finally, by equipping our retriever for open-domain heterogeneous QA systems, we achieve a new state-of-the-art result on the popular ConvMix task, with an absolute improvement of up to 5.90 points.

## introduction retrieval-augmented generation @xcite @xcite ) has become a pivotal technique for improving the faithfulness of generative large language models (llms @xcite ). by leveraging retrievers to extract relevant knowledge from largescale knowledge corpus, rag effectively reduces the hallucinations often produced by llms @xcite . although existing information retrieval (ir) methods @xcite have demonstrated effectiveness in retrieving information from homogeneous knowledge corpus, where knowledge is stored in a single structure, such as tables @xcite or text @xcite , most of these systems fail to recognize diverse user retrieval intents and retrieve heterogeneous knowledge from multiple sources. in heterogeneous ir, knowledge comes from multiple structures, making retrieval much more complex. relying solely on homogeneous knowledge often results in partial or incomplete retrieval results, limiting the applicability of these systems to a wider range of downstream tasks @xcite . for example, a retriever specialized in table-based retrieval @xcite cannot be easily applied to downstream tasks such as question answering (qa) based on knowledge graphs @xcite . in this paper, we propose the unified hetero-geneous knowledge retriever (unihgkr), a novel framework designed to retrieve information from heterogeneous knowledge corpus by following user instructions, as depicted in figure 1 . the unihgkr framework consists of three training stages: (1) unified embedding self-supervised pre-training: this stage addresses the lack of structured data in the original pretraining of the language model, laying the foundation for the creation of a unified embedding space. (2) text-anchored heterogeneous embedding alignment: in this stage, natural language text that shares the same semantic content as heterogeneous data is collected, and their embeddings are aligned using contrastive learning. this process creates a unified embedding space that captures semantic information, independent of the format in which the knowledge is presented. (3) instruction-aware heterogeneous retriever fine-tuning: at this final stage, the retriever is fine-tuned on heterogeneous knowledge retrieval tasks. to enhance the model's capability to follow user instructions, we introduce two specialized contrastive losses, termed 'type-balanced loss' and 'type-preferred loss', which are designed to optimize retrieval performance according to user instructions. in addition, existing heterogeneous ir benchmarks have limited knowledge coverage @xcite . for example, studies like @xcite focus only on two types of knowledge: tables and text. to address this gap, we introduce compmix-ir, the first-ever benchmark for heterogeneous knowledge retrieval. compmix-ir has over 9,400 qa pairs and a corpus of 10 million entries spanning four distinct knowledge types: text, knowledge graphs (kg), tables , and infoboxes . derived from the open-domain qa dataset comp-mix @xcite , compmix-ir transforms this qa task into a standard ir task (as detailed in section 3). to better reflect real-world retrieval needs, we define two distinct scenarios in this benchmark: (1) retrieving relevant evidence across all knowledge types, and (2) retrieving evidence of a specific type, as specified by user instructions. both scenarios utilize the same evi-dence pool, requiring the retriever to adapt queryevidence similarity based on the instructions. this setup mirrors the complexities of real-world retrieval tasks, offering enhanced practical relevance and utility for diverse applications. experimental results demonstrate the effectiveness of our proposed unihgkr over the existing methods, with relative improvements of up to 6.36% and 54.23% in two different scenarios. in addition to the bert-based unihgkr-base model, we also extend our framework to an llm-based retriever and train the unihgkr-7b model to verify scalability. both models achieve state-of-the-art (sota) performance on compmix-ir respective to their parameter scales. furthermore, in the context of open-domain heterogeneous qa, systems equipped with unihgkr retriever set a new sota on the convmix task @xcite , with an absolute gain of up to 5.90 points, further validating its real-world applicability. ## compmix-ir benchmark in this section, we provide a detailed description of the construction of compmix-ir, the definition of retrieval scenarios, and their instruction schema. ## unihgkr in this section, we introduce our problem formulation and the unihgkr framework. our unihgkr-base model adopts a single sharedencoder architecture, with parameters initialized from the bert-base model @xcite . the [cls] token from the final hidden layer is trained to serve as the embedding, following @xcite . ## experimental methodology in our main experiments, we train and evaluate retrievers on the compmix-ir, following the train, dev, and test set divisions in compmix. ## evaluation results in this section, we focus on comparing and discussing the performance of unihgkr with baselines on heterogeneous retrieval tasks and the application of unihgkr models in the open-domain qa task. we also explore the robustness and zeroshot performance of unihgkr in appendix e. ## conclusion in this paper, we introduced unihgkr, an instruction-aware unified heterogeneous knowledge retriever. first, we constructed compmix-ir, the first heterogeneous information retrieval task dataset containing a corpus of over 10 million entries across four heterogeneous data types. then, we defined two different heterogeneous information retrieval scenarios to meet the diverse retrieval needs of real-world users. we designed the uni-hgkr framework with three training stages. our experiments showed that unihgkr achieved stateof-the-art performance on compmix-ir benchmarks, both with the 110m bert-based retriever and the 7b llm-based retriever. applying our unihgkr retrievers can significantly enhance the performance of heterogeneous qa systems, achieving new sota results on the convmix dataset.