Knowledge Base Index Compression via Dimensionality and Precision Reduction (2022)

Abstract: Recently neural network based approaches to knowledge-intensive NLP tasks, such as question answering, started to rely heavily on the combination of neural retrievers and readers. Retrieval is typically performed over a large textual knowledge base (KB) which requires significant memory and compute resources, especially when scaled up. On HotpotQA we systematically investigate reducing the size of the KB index by means of dimensionality (sparse random projections, PCA, autoencoders) and numerical precision reduction. Our results show that PCA is an easy solution that requires very little data and is only slightly worse than autoencoders, which are less stable. All methods are sensitive to pre- and post-processing and data should always be centered and normalized both before and after dimension reduction. Finally, we show that it is possible to combine PCA with using 1bit per dimension. Overall we achieve (1) 100 \times compression with 75%, and (2) 24 \times compression with 92% original retrieval performance.

## introduction recent approaches to knowledge-intensive nlp tasks combine neural network based models with a retrieval component that leverages dense vector representations @xcite @xcite . the most straightforward example is question answering, where the retriever receives as input a question and returns relevant documents to be used by the reader (both encoder and decoder), which outputs the answer (chen, 2020). the same approach can also be applied in other contexts, such as fact-checking @xcite or knowledgable dialogue @xcite . moreover, this paradigm can also be applied to systems that utilize e.g. caching of contexts from the training corpus to provide better output, such as the k-nearest neighbours language model proposed by @xcite or the dynamic gating language model mechanism by @xcite . all these pipelines are generalized as retrieving an artefact from a knowledge base @xcite on which the reader is conditioned together with the query. crucially, all of the previous examples rely on the quality of the retrieval component and the knowledge base. the knowledge base is usually indexed by dense vector representations foot_0 and the retrieval component performs maximum similarity search, commonly using the inner product or the l foot_1 distance, to retrieve documents 2 from the knowledge base. only the index alone takes up a large amount of size of the knowledge base, making deployment and experimentation very difficult. the retrieval speed is also dependent on the dimensionality of the index vector. an example of a large knowledge base is the work of @xcite which performs retrieval over a database of 1.8 billion documents. this paper focuses on the issue of compressing the index through dimensionality and precision reduction and makes the following contributions: in section 3, we describe the problem scenario and the experimental setup. we discuss the results of different compression methods in section 4. we provide further analysis in section 5 and conclude with usage recommendations in section 6. the repository for this project is available open-source. 3 2 related work reducing index size. a thorough overview of the issue of dimensionality reduction in information retrieval in the context of dual encoders has been done by @xcite . though in-depth and grounded in formal arguments, their study is focused on the limits and properties of dimension reduction in general (even with sparse representations) and the effect of document length on performance. in contrast to their work, this paper aims to compare more methods and give practical advice with experimental evidence. a baseline for dimensionality reduction has been recently proposed by @xcite in which they perform the reduction while training the document (and query) encoder by adding a low dimensional linear projection layer as the final output layer. compared to our work, their approach is supervised. in the concurrent work of @xcite , pca is also used to reduce the size of the document index. compared to our work, they perform pca using the combination of all question and document vectors. we show in figures 4 and 6 that this is not needed and the pca transformation matrix can be estimated much more efficiently. moreover, we use different unsupervised compression approaches for comparison and perform additional analysis of our findings. an orthogonal approach to the issue of memory cost has been proposed by @xcite . instead of moving to another continuous vector representation, their proposed method maps original vectors to vectors of binary values which are trained using the signal from the downstream task. the pipeline, however, still relies on re-ranking using the uncompressed vectors. this method is different from ours and in section 4.4 we show that they can be combined. finally, @xcite investigate filtering and k-means pruning for the task of knn language modelling. this work also circumvents the issue of having to always perform an expensive retrieval of a large data store by determining whether the retrieval is actually needed for a given input. ## compression methods having established the retrieval performance of the uncompressed baseline, we now turn to methods for compressing the dense document index and the queries. note that we consider unsupervised methods on already trained index, for maximum ease of use and applicability. this is in contrast to supervised methods, which have access to the query-doc relevancy mapping, or to in-training dimension reduction (i.e. lower final layer dimension). ## discussion in this section we briefly discuss the main conclusions from our experiments and analysis in the form of recommendations for nlp practicioners. importance of pre-/post-processing. as our results show, for all methods (and models), centering and normalization should be done before and after dimension reduction, as it boosts the performance of every model. ## summary in this work, we examined several simple unsupervised methods for dimensionality reduction for retrieval-based nlp tasks: random projections, pca, autoencoder and precision reduction and their combination. we also documented the data requirements of each method and their reliance on preand post-processing. future work. as shown in prior works, dimension reduction can take place also during training where the loss is more in-line with the retrieval goal. methods for dimension reduction after training, however, rely mostly on reconstruction loss, which is suboptimal. therefore more research for dimension reduction methods is needed, such as fast manifold or distance-based learning.