An Effective Post-training Embedding Binarization Approach for Fast Online Top-K Passage Matching (2022)

Abstract: With the rapid development of Natural Language Understanding for information retrieval, fine-tuned deep language models, e.g., BERT-based, perform remarkably effective in passage searching tasks. To lower the architecture complexity, the recent state-of-the-art model ColBERT employs Contextualized Late Interaction paradigm to independently learn fine-grained query-passage representations. Apart from the architecture simplification, embedding binarization, as another promising branch in model compression, further specializes in the reduction of memory and computation overheads. In this concise paper, we propose an effective post-training embedding binarization approach over ColBERT, achieving both architecture-level and embedding-level optimization for online inference. The empirical results demonstrate the efficaciousness of our proposed approach, empowering it to perform online query-passage matching acceleration.

## introduction the information retrieval community has witnessed an emerging slew of bert @xcite -based deep ranking models that achieves performance superiority in various retrieval benchmarks @xcite @xcite . despite their advantage in learning deeplycontextualized semantic representations, a major issue however is the heavy computational complexity. a recent model colbert @xcite detaches the query-passage contextual encoding in the proposed contextualized late interaction mechanism, achieving substantial progress in optimizing the runtime resource footprints. orthogonal to architecture simplification, embedding binarization, i.e., another model compression technique, has received growing attention across various applications @xcite @xcite . despite the promising advantages, it usually suffers from large performance degradation even with adequate training supports @xcite , in which the crux generally lies in: to tackle these issues, we propose an effective post-training binarization approach by introducing: ## preliminaries colbert @xcite . it comprises: (1) a query encoder f q , (b) a passage encoder f d , and (3) a query-passage score predictor. specifically, given a query q and a passage d, f q and f d encode them into a bag of fixed-size embeddings e q and e d as follows: ## bi-colbert methodology to tackle the aforementioned issue, we propose bi-colbert by introducing two effective and lightweight techniques: (1) semantic diffusion to hedge the information loss against embedding binarization, and (2) approximation of unit impulse function @xcite ) for more accurate gradient estimation. ## experimental evaluation we now evaluate our approach with the aim of answering the following research questions: • rq1. how does bi-colbert perform in the fine-grained top-k passage searching task? • rq2. is the proposed semantic diffusion technique effective to hedge the information loss? • rq3. how does the proposed gradient estimator compare to the previous counterparts? we implement our embedding binarization approach directly on pretrained colbert, denoted as colbert pretrain . to give a fair comparison, we use the same dataset (i.e., ms marco) and evaluation metric (i.e., mrr@10) with colbert. detailed experimental setups and baseline introduction are attached in appendix b. ## discussion for future work we summarize five promising future directions. 1. it is pragmatic to evaluate the adaptability of our approach to other bert-based models. 2. a promising direction could be using embedding binarization for other scenarios with efficiency demands @xcite @xcite @xcite . 3. colbert also employs faiss @xcite , a tool for large-scale vector-similarity search. thus, it is worth developing a similar index-based data structure specifically for retrieval in the discrete embedding space. 4. data augmentation, e.g., feature-based augmentation @xcite , is another effective technique to boost embedding informativeness before and after the binarization. 5. if the training resource is adequate, quantizationaware training @xcite resembles the standard fine-tuning and thus is promising to compensate for the performance degradation.