MATTER : Memory-Augmented Transformer Using Heterogeneous Knowledge Sources (2024)

Abstract: Leveraging external knowledge is crucial for achieving high performance in knowledge-intensive tasks, such as question answering. The retrieve-and-read approach is widely adopted for integrating external knowledge into a language model. However, this approach suffers from increased computational cost and latency due to the long context length, which grows proportionally with the number of retrieved knowledge. Furthermore, existing retrieval-augmented models typically retrieve information from a single type of knowledge source, limiting their scalability to diverse knowledge sources with varying structures. In this work, we introduce an efficient memory-augmented transformer called MATTER, designed to retrieve relevant knowledge from multiple heterogeneous knowledge sources. Specifically, our model retrieves and reads from both unstructured sources (paragraphs) and semi-structured sources (QA pairs) in the form of fixed-length neural memories. We demonstrate that our model outperforms existing efficient retrieval-augmented models on popular QA benchmarks in terms of both accuracy and speed. Furthermore, MATTER achieves competitive results compared to conventional read-and-retrieve models while having 100x throughput during inference.

## introduction retrieval-augmented models have enhanced performance in various natural language processing tasks, such as open-domain question answering @xcite . these models leverage a two-step process: an initial retrieval phase to gather relevant information from a knowledge source (retrieve), followed by a reading or comprehension phase to generate responses based on the retrieved context and input (read). however, the strong performance of retrievalaugmented qa models is offset by a substantial drawback of high inference latency @xcite . @xcite demonstrate that even with relatively small reader models, such as t5-base, retrieveand-read qa models struggle to process more than 10 questions per second. recent studies attribute the problem to the increase in context length for a reader to condition on @xcite @xcite . for instance, the fusion-in-decoder model @xcite retrieves 100 documents, each comprising of 250 tokens, resulting in the reader model attending to 25,000 tokens during answer generation; this poses a significant bottleneck during inference. to address this limitation, @xcite ) transforms the retrieved texts as neural memories. a neural memory is an efficient way of storing knowledge with a fixed length latent representation @xcite . as a result, memory-augmented qa models generate an answer conditioned on retrieved neural memories, rather than retrieved raw text. this approach shortens the context length, enabling memory-augmented models to respond to several hundred questions per second @xcite . while this improves the throughput compared to conventional retrieval-augmented models, there is still room for improvement in performance. another critical drawback of existing retrievalaugmented models, both conventional and memorybased approaches, is their narrow focus on a single type of knowledge source, either qa pairs or wikipedia articles @xcite @xcite . external knowledge sources come in various formats, such as unstructured, semi-structured, and structured, each with its own merits and use cases. for instance, unstructured data, like a wikipedia paragraph, is easily accessible and often covers a broad range of topics. however, it may suffer from noise and a lack of precision. in contrast, semi-structured knowledge, such as a question-answer pair (qa), is concise and clear but can be challenging to gather. previous retrieve-and-read approaches have mainly utilized a single type of knowledge source, and this limited focus results in reduced scope and coverage of knowledge. a potential approach to overcome this limitation is by synchronizing the knowledge structure, such as converting a wikipedia paragraph into qa pairs using a qa generation pipeline, as demonstrated in @xcite . however, this format transformation process comes with significant computational cost and the potential risk of introducing noise or corrupting knowledge during the transformation. to address these limitations, we introduce mat-ter, a novel memory-augmented qa model designed to retrieve information from diverse knowledge sources. unlike existing retrieval models, matter retrieves from multiple heterogeneous knowledge sources. this allows our model to maintain a comprehensive and type-agnostic knowledge index, enabling retrieval and conditioning on a broader range of knowledge snippets in response to questions. moreover, our model cross-encodes a given question and retrieved neural memories, ensuring a comprehensive understanding of input and context. with this efficient cross-encoding capability and access to heterogeneous knowledge sources, our model significantly outperforms existing efficient qa models in both zero-shot and fine-tuned settings. furthermore, it achieves a remarkable 100x throughput improvement over raw text-augmented qa models like fid @xcite , while maintaining competitive performance. overall, our approach strikes a balance between speed and performance demonstrated over popular qa benchmarks and supported by in-depth analysis. ## related work question answering is a central task in the nlp community, and various approaches have emerged to address it, falling into three main categories. firstly, closed-book approaches like t5 @xcite and bart @xcite have gained prominence relying solely on the input question and parametric knowledge to generate an answer. these qa models remain suboptimal as they rely exclusively on parametric knowledge @xcite . the second category comprises retriever-only models, where a retriever fetches a relevant qa pair based on the input question, often with a reranker to enhance qa performance @xcite @xcite . this approach restricts the knowledge source to qa pairs and relies on strong overlap between the candidate question and qa pairs in the index. the third category is the retrieve-and-read pipeline models, which have become the standard for building qa models with strong performance @xcite @xcite . this method involves a retriever fetching pertinent knowledge, followed by a separate reader model that generates answers based on the acquired context. however, it's worth noting that while these retrieve-and-read models excel in performance and combat hallucination, they are notorious for their slow inference speed @xcite @xcite . to address this issue, various strategies have been proposed, such as @xcite and @xcite , that dynamically determine which retriever results to read, aiming to reduce computational overhead during inference. additionally, @xcite eliminates cross-attention from most decoder layers and incorporates multiquery attention. emat @xcite takes a novel approach by having the retriever fetch neural memories, significantly accelerating the answer generation process. ## analysis retrieval vs reader in model performance with the superior results achieved by our proposed model on various qa benchmarks, one natural question arises: are the gains in em score solely attributable to using an off-the-shelf retriever? in fact, this is not the case; our strong reader models bring such superior em scores. for instance, the emat retriever achieves an em score of 43.3 on the tqa dataset, while the em score with the full emat model is 44.4, a gain of 1.1 points through reading the retrieved memories. on the other hand, a significant improvement is observed with our reader model: our off-the-shelf retriever model achieves an em score of 40.0, and when coupled with a reader model that cross-encodes retrieved memories, we achieve a score of 56.0, marking an absolute em score improvement of 16.0 points. while our retriever is inferior to that of emat, the em score of the full retrieve-read model surpasses that of emat by a significant margin. this clearly demonstrates that the retriever is not the core contributor to the performance gains; rather, it is the proposed memory-augmented model that brings a substantial improvement. scalability to varying k in this section, we analyze the scalability of model performance with varying values of k. as shown in ## conclusion in this paper, we propose an efficient memoryaugmented question answering model with multiple heterogeneous knowledge sources. the proposed qa framework is able to retrieve from and condition on retrieved knowledge from multiple sources with varying formats and achieves remarkable performance in popular qa benchmarks, while having high throughput.