KM atrix: A Flexible Heterogeneous Knowledge Enhancement Toolkit for Large Language Model (2024)

Abstract: Knowledge-Enhanced Large Language Models (K-LLMs) system enhances Large Language Models (LLMs) abilities using external knowledge. Existing K-LLMs toolkits mainly focus on free-textual knowledge, lacking support for heterogeneous knowledge like tables and knowledge graphs, and fall short in comprehensive datasets, models, and user-friendly experience. To address this gap, we introduce KMatrix: a flexible heterogeneous knowledge enhancement toolkit for LLMs including verbalizing-retrieval and parsing-query methods. Our modularity and control-logic flow diagram design flexibly supports the entire lifecycle of various complex K-LLMs systems, including training, evaluation, and deployment. To assist K-LLMs system research, a series of related knowledge, datasets, and models are integrated into our toolkit, along with performance analyses of K-LLMs systems enhanced by different types of knowledge. Using our toolkit, developers can rapidly build, evaluate, and deploy their own K-LLMs systems.

## introduction knowledge-enhanced large language models (k-llms) system uses external knowledge to enhance the capabilities of large language models (llms) @xcite ), which alleviates the issues of hallucination and weak reasoning abilities for knowledge-intensive natural language processing tasks @xcite , @xcite , @xcite ). recently, k-llms have become a popular research topic and extensive works have been conducted from various dimensions such as knowledge, models, and enhancement methods @xcite ). early k-llms works primarily focused on free-textual knowledge enhancement @xcite al. (2020), qu et al. (2020) ), which led to the emergence of the retrieval-augmented generation (rag) research branch. recent studies have explored methods for jointly enhancing llms with heterogeneous knowledge (like tables, knowledge graphs, etc) using unified retrieval @xcite , @xcite ) or selective query @xcite , @xcite ). meanwhile, increasing attention is being directed towards adaptive enhancement methods research @xcite , @xcite ), which autonomously control the interaction between generation and retrieval @xcite ) to achieve better performance. moreover, with the development of k-llms, there is a need for an easy-to-use toolkit to flexibly implement k-llms works and compare different approaches under the same conditions. in recent years, many k-llms related toolkits @xcite , @xcite , @xcite , @xcite ) have emerged, but they still have the following shortcomings: 1) lacking support for joint enhancement with heterogeneous knowledge sources. the existing representative k-llms toolkits @xcite , @xcite , @xcite ) predominantly focus on textual knowledge enhancement. 2) lacking systematic support for various adaptive enhancement methods. coze foot_1 and ralle @xcite ) enabled the construction of naive k-llms (retrieval and generation) by selecting components, but they lacked support for building complex adaptive k-llms. flashrag @xcite ) implemented adaptive enhancement by simply integrating code of some existing k-llms works, lacking systematic integration of adaptive enhancement methods from different dimensions, like retrieval timing determination and retrieval source selection. 3) not highly customizable or easily combinable, and lacking comprehensive support for training, evaluation, and deployment of k-llms systems. langchain (chase (2022)) and haystack (pietsch et al. (2019)) are two fundamental k-llms toolkits which lacked integration of existing representative k-llms works, and did not provide sufficient flexibility for customization. fastrag (izsak et al. (2023)) and flashrag (jin et al. ( 2024 )) utilized customizable component design and integrate extensive existing datasets, knowledge, and models. however, they define component relations using hard-coding methods, which is not easily combinable. comparison of existing representative k-llms toolkits is shown in to address the aforementioned shortcomings, we introduce kmatrix: a flexible heterogeneous knowledge enhancement toolkit for llms. our toolkit uses both verbalizing-retrieval @xcite ) and parsing-query @xcite ) methods to support unified enhancement of heterogeneous knowledge (like free-textual knowledge, tables, knowledge graphs, etc). and we systematically integrate adaptive enhancement methods from two aspects: retrieval timing judgment @xcite ) and knowledge source selection @xcite ). to achieve high customizability and easy combinability, we deploy modular component definition and control-logic flow diagram design to flexibly construct components and their relations. in summary, our main contributions are: ## kmatrix toolkit as shown in figure 1 , our toolkit contains seven stages to complete knowledge-enhanced generation task. knowledge access, knowledge preprocessing, and knowledge integration are respectively used for the access, preprocessing, and unified fusion of heterogeneous knowledge. knowledge retrieval retrieves knowledge from a unified textual knowledge base and query parsing generates query statements for a unified querier. adaptive enhancement autonomously controls the interaction between generation and retrieval/query. generation stage receives task inputs and generates outputs with knowledge enhancement. all stages are implemented based on our modular component definitions. meanwhile, we design a control-logic flow diagram to combine components. next, we will introduce the seven stages of kmatrix and present our modular design approach & toolkit usage. ## experimental settings in this section, we evaluate the performance of k-llms constructed by kmatrix to demonstrate the entire lifecycle capabilities of our toolkit. ## experimental results we report experimental results from two aspects: knowledge access performance evaluation: single vs. multi-knowledge bases enhancement evaluation: ## conclusions we introduce kmatrix tootkit to facilitate the construction of adaptive heterogeneous k-llms system, which enables one-click support for training, evaluation, and deployment procedures. meanwhile, we integrate a rich collection of representative k-llms knowledge, datasets, and methods, and provide performance analysis of heterogeneous knowledge enhancement, which can offer assistance for future works. overall, kmatrix is particularly useful for k-llms practitioners without extensive expertise, and we hope kmatrix will contribute to the development of k-llms.