Q uadruplet BERT : An Efficient Model For Embedding-Based Large-Scale Retrieval (2021)

Abstract: The embedding-based large-scale query-document retrieval problem is a hot topic in the information retrieval (IR) field. Considering that pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, we present a QuadrupletBERT model for effective and efficient retrieval in this paper. Unlike most existing BERT-style retrieval models, which only focus on the ranking phase in retrieval systems, our model makes considerable improvements to the retrieval phase and leverages the distances between simple negative and hard negative instances to obtaining better embeddings. Experimental results demonstrate that our QuadrupletBERT achieves state-of-the-art results in embedding-based large-scale retrieval tasks.

## introduction large-scale retrieval systems such as search engines have been a vital tool to help people access the massive amount of online information. various techniques have been developed to improve retrieval quality in the last decades. due to the difficulty of computing search intent from the query text and accurately representing the semantic meaning of document requirements, most previous studies are based on classic term-weighting methods such as @xcite or tf-idf @xcite or simple context-free word embedding @xcite that perform well for the cases that keyword matching can address. however, these models only accept sparse handcrafted features and cannot capture complex semantic features. considering that pre-trained language models like bert @xcite and roberta @xcite have achieved great success in a wide variety of nlp tasks, more and more researchers propose bert-style models to solve large-scale retrieval problems. some previous work has confirmed the effectiveness of bert for enhancing retrieval systems. for example, yilmaz et al. (2019) apply a bertstyle model to document retrieval via integration with the open-source anserini information retrieval toolkit to demonstrate end-to-end search over large document collections. yang et al. (2019) build a bert-based reader to identify answers from a large corpus of wikipedia articles in an end-to-end fashion. padaki et al. ( 2020 ) use query expansion to generate better queries for bert-based ranker in retrieval. mass and roitman (2020) describe a weakly-supervised method for training bert-style models for ad hoc document retrieval. in bert, the prediction function f (query, doc) is a pre-trained deep bidirectional transformer model @xcite . while the above bert-style models are very successful, this approach cannot be directly applied to large-scale retrieval problems because predicting f for every possible document can be prohibitively expensive. thus, the methods mentioned above first use a less powerful but more efficient retrieval algorithm (retriever) such as an inverted index to reduce the solution space and then use the bert-style model to re-rank the retrieved documents. as shown in figure 1 , we refer to all such bert-style retrieval models as ranker. unlike these ranker which have recently seen significant advances, constructing a bert-style retriever is a new topic in the large-scale retrieval field, on which few studies have thus far focused. for example, @xcite present a modification of the pre-trained bert network that uses siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine similarity. @xcite build a two-tower transformer model with more pre-training data, which can significantly outperform the widely used bm-25 algorithm. @xcite distill knowledge from bert into a two-tower architecture network for efficient retrieval. as shown in figure 2 (a ) and (b), the existing bert-style retriever mentioned above simply builds a two-or three-tower network structure to compute distances between positive and negative instances, which ignores the fact there are not only simple negative instances in the dataset: some instances are seemingly positive but actually negative, which we call hard negative instances. as we all know, the retriever should have high recall; otherwise, many positive instances will not even be considered in the ranking phase. however, due to hard negative instances being literally related, treating them as equal to simple negative instances may harm the embedding of positive instances and lead the model to identify positive instances as negative ones mistakenly. the key to solving the problem mentioned above is incorporating the distances between hard negative and simple negative instances into the training step. our intuition is that hard negative instances are negative compared to positive instances but should be considered positive compared to simple negative instances. therefore, we explore a new way to incorporate distances between hard negative and simple negative instances into the training process and build a four-tower bert-style model named quadrupletbert. we have evaluated our model on two retrieval question-answering (reqa) benchmarks. experimental results show that our model registers huge improvements over existing bert-style retriever models and achieves state-of-the-art results. our main contributions are as follows: large-scale retrieval problems can be defined as: given a query, return the most relevant documents from a large corpus, where the corpus' size can be hundreds of thousands or more. the embeddingbased retrieval model jointly embeds queries and documents in the same embedding space and uses an inner product or cosine distance to measure the similarity between queries and documents. since embeddings of all candidate documents can be precomputed and indexed, the inference can be made efficiently with approximate nearest neighbor search algorithms in the embedding space (shrivastava and li, 2014; @xcite . let the query embedding model be φ(•), and the document embedding model be ψ(•) the distance function can be defined as: in this paper, we are interested in parameterizing the encoders φ and ψ as a four-tower bert which incorporates the distances between hard negative and simple negative instances into the training step. ## quadrupletbert as shown in figure 2 (c), the core of our model is a four-tower sentence-level bert relevance encoder. each tower of our retrieval model follows the architecture and hyper-parameters of the 12 layers bert model 1 . note that for all bert baselines, we all pre-train them on the specific downstream datasets by masked lm and next sentence prediction tasks @xcite . the embedding dimension is 768. the sequence length for the encoder is set to be 64. for all towers, taking the average of the encoding layer's hidden state on the time axis as the final embedding. ## hyper-parameter finetuning as a key hyper-parameter of our quadrupletbert model, m denotes the margin enforced between positive and hard negative and negative instances. we further investigated the influence of m on our model. with the squad and natural questions datasets, we train models with m is set to 0, 0.1, 0.2, 1, 1.5, and 2, respectively. the experimental results are shown in ## related work we have covered research on embedding based large-scale retrieval in section 1, related work that inspires our technical design is mainly introduced in the following: @xcite present a modification of the pre-trained bert network that uses multi-tower network structures to derive semantically meaningful sentence embeddings that can be compared using cosine similarity. huang et al. (2020) present a multi-tower word embedding retrieval method successfully applied in the facebook online search. besides, they mentioned that shuffling hard negative and simple negative instances in the training sets may help model learning, which inspired us to further investigate the effectiveness of hard negative instances. ## conclusion we have presented our four-tower quadruplet-bert model and demonstrated its usage and effect on large-scale retrieval. unlike many widely-used bert-style ranker models of large-scale retrieval tasks, our model focus on the retrieval phase. the multi-tower architecture making it extremely easy to be applied in retrieval systems. moreover, incorporating distances between hard negative and simple negative instances into the training process shows significant superiority in improving retriever model performance. we hope our work can inspire more sophisticated techniques of leveraging bert-style models in large-scale retrieval. leveraging hard negative instances for other natural language processing tasks such as text generation and information extraction is also worth investigating.