Multi-View Document Representation Learning for Open-Domain Dense Retrieval (2022)

Abstract: Dense retrieval has achieved impressive advances in first-stage retrieval from a large-scale document collection, which is built on bi-encoder architecture to produce single vector representation of query and document. However, a document can usually answer multiple potential queries from different views. So the single vector representation of a document is hard to match with multi-view queries, and faces a semantic mismatch problem. This paper proposes a multi-view document representation learning framework, aiming to produce multi-view embeddings to represent documents and enforce them to align with different queries. First, we propose a simple yet effective method of generating multiple embeddings through viewers. Second, to prevent multi-view embeddings from collapsing to the same one, we further propose a global-local loss with annealed temperature to encourage the multiple viewers to better align with different potential queries. Experiments show our method outperforms recent works and achieves state-of-the-art results.

## introduction over the past few years, with the advancements in pre-trained language models @xcite , dense retrieval has become an important and effective approach in open-domain text retrieval @xcite @xcite . a typical dense retriever usually adopts a bi-encoder @xcite architecture to encode input query and document into a single low-dimensional vector (usually cls token), and computes the relevance scores between their representations. in real-world applications, the embedding vectors of all the documents are pre figure 1 : the illustration of our multi-view document representation learning framework. the triangles and circles mean document and query vectors separately. usually, one document can be asked to different potential queries from multiple views. our method comes from this observation and generates multi-view representations for documents to better align with different potential queries. however, being limited to the single vector representation, bi-encoder faces the upper boundary of representation capacity according to theoretical analysis in @xcite . in the real example from squad dev dataset, we also find that a single vector representation can't match well to multi-view queries, as shown in figure .1. the document corresponds to four different questions from different views, and each of them matches to different sentences and answers. in the traditional bi-encoder, the document is represented to a single vector while it should be recalled by multiple diverse queries, which limits the capacity of the bi-encode. as for the multi-vector models, cross-encoder architectures perform better by computing deeply-contextualized representations of querydocument pairs, but are computationally expensive and impractical for first-stage large-scale retrieval @xcite . some recent studies try to borrow from cross-encoder and extend bi-encoder by employing more delicate structures, which allow the multiple vector representations and dense interaction between query and document embeddings. including late interaction @xcite and attention-based aggregator @xcite . however, most of them contain softmax or sum operators that can't be decomposed into max over inner products, and so fast ann retrieval can't be directly applied. based on these observations, we propose multi-view document representations learning framework, mvr in short. mvr originates from our observation that a document commonly has several semantic units, and can answer multiple potential queries which contain individual semantic content. it is just like given a specified document, different askers raise different questions from diverse views. therefore, we propose a simple yet effective method to generate multi-view representations through viewers, optimized by a global-local loss with annealed temperature to improve the representation space. prior work has found [cls] token tends to aggregate the overall meaning of the whole input segment @xcite , which is inconsistent with our goal of generating multi-view embeddings. so we first modify the bi-encoder architecture, abandon [cls] token and add multiple @xcite tokens to the document input. the representation of the viewers in the last layer is then used as the multi-view representations. to encourage the multiple viewers to better align with different potential queries, we propose a global-local loss equipped with an annealed temperature. in the previous work, the contrastive loss between positive and negative samples is widely applied @xcite . apart from global contrastive loss, we formulate a local uniformity loss between multi-view document embeddings, to better keep the uniformity among multiple viewers and prevent them from collapsing into the same one. in addition, we adopt an annealed temperature which gradually sharpens the distribution of viewers, to help multiple viewers better match to different potential queries, which is also validated in our experiment. the contributions of this paper are as follows: ## conclusions in this paper, we propose a novel multi-view representation learning framework. specifically, we present a simple yet effective method to generate multi-view document representations through multiple viewers. to optimize the training of multiple viewers, we propose a global-local loss with annealed temperature to enable multiple viewers to better align with different semantic views. we conduct experiments on three open-domain retrieval datasets, and achieve state-of-the-art retrieval performance. our further analysis proves the effectiveness of different components in our method.