Pseudo-Relevance for Enhancing Document Representation (2022)

Abstract: This paper studies how to enhance the document representation for the bi-encoder approach in dense document retrieval. The bi-encoder, separately encoding a query and a document as a single vector, is favored for high efficiency in large-scale information retrieval, compared to more effective but complex architectures. To combine the strength of the two, the multi-vector representation of documents for bi-encoder, such as ColBERT preserving all token embeddings, has been widely adopted. Our contribution is to reduce the size of the multi-vector representation, without compromising the effectiveness, supervised by query logs. Our proposed solution decreases the latency and the memory footprint, up to 8- and 3-fold, validated on MSMARCO and real-world search query logs.

## introduction information retrieval (ir) aims to rank documents by their relevance scores to the given query, where neural ir models have achieved state-of-the-art performances by fine-tuning large language models. these models, encoding document d to query q into vectors for soft matching, can be categorized into cross-encoder jointly encoding q and d, and biencoder separately encoding the two into single vectors. our target scenario is commercial-scale ir, requiring online query latency to a web-scale corpus, where bi-encoders are generally preferred for lower latency. meanwhile, as a hybrid of the two, a single vector representation can be enriched by multivector representation, such as colbert @xcite preserving all token embeddings, for higher effectiveness. with multi-vector bi-encoder, once query and document vectors are indexed on memory, relevant documents can be efficiently identified, by a nearest neighbor (nn) search on the index @xcite . col-bert, by fully preserving all tokens in document representation, sacrifices on-memory efficiency, for enriching document representation. ## preliminaries and related work this section motivates the importance of queryawareness for document understanding, then compares and contrasts with existing approaches. ## proposed: query-aware selection at indexing time to enable query-aware selection at indexing time, we leverage existing relevant query logs for each d. however, in practice, most of the documents in a corpus do not have query logs. therefore, we train a pseudo-query extractor to predict relevant query terms from d, by designing supervision for the training based on the relevant q-d pairs from the query logs. at indexing time, the pseudo-query extractor selects relevant terms from both seen and unseen documents, by envisioning real, relevant queries. our extractor and retriever are built upon col-bert ( §3.1) which models term-level relevance between q-d. by leveraging soft matches between terms in q/d, we obtain supervisions on p(w|r) for training pseudo-query extractor ( §3.2). ## experiments in this section, we empirically validate the effectiveness of our pqa-colbert, with respect to ranking performance on passage ranking task. we report results for a single run. ## conclusion we studied enhancing document representation of bi-encoder, by balancing the benefit of the single-vector (of memory-efficient indexing) and the multi-vector (of preserving all relevant query terms) representation. specifically, inspired by relevance model, we supervise a pseudo-query extractor by using query logs of relevant querydocument pairs in the training dataset, to extract query-relevant terms at indexing time. our proposed model is validated on msmarco and re-alq benchmarks, for both passage and document retrieval tasks. our model achieves comparable ranking performance to multi-vector retriever and even to cross-encoder ranker, with significantly better memory-efficiency for indexing and lower latency for inference. because of the extremely large number of documents in a corpus, relevance annotations are given only to a few documents, while the others are often assumed to be non-relevant. for example, the user clicks as a relevance proxy often suffer from the ranking bias (also known as position bias), where clicks are often only examined on the topranked (or top-positioned) documents by a system @xcite . for msmarco, relevance annotations are biased to the documents that have exact matching terms to queries @xcite . such annotation biases have been a bottleneck for both training @xcite and evaluation @xcite . to alleviate such biases, one can design the annotation process in an interactive way between the annotators and a target retriever, such that the pool of documents presented to the annotators dynamically change as the system evolves @xcite , which we leave as future work.