Less is More: Pretrain a Strong S iamese Encoder for Dense Text Retrieval Using a Weak Decoder (2021)

Abstract: Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space. Autoencoder-based language models are appealing in dense retrieval as they train the encoder to output high-quality embedding that can reconstruct the input texts. However, in this paper, we provide theoretical analyses and show empirically that an autoencoder language model with a low reconstruction loss may not provide good sequence representations because the decoder may take shortcuts by exploiting language patterns. To address this, we propose a new self-learning method that pre-trains the autoencoder using a weak decoder, with restricted capacity and attention flexibility to push the encoder to provide better text representations. Our experiments on web search, news recommendation, and open domain question answering show that our pre-trained model significantly boosts the effectiveness and few-shot ability of dense retrieval models. Our code is available at https://github.com/microsoft/SEED-Encoder/ .

## introduction recently, dense retrieval (dr) has progressed to more important roles in many language systems, for example, web search @xcite , question answering @xcite , and news recommendation @xcite . in the firststage retrieval of these scenarios, dr models generally employ a siamese/dual-encoder architecture in practice. the encoder model first separately encodes the user side (query, browsing history, or question) and the corpus side (document or passages) as individual embeddings in a learned representation space @xcite , where retrieval with simple similarity metrics are conducted effectively @xcite . a popular choice of text encoders in dr is the transformer network pre-trained by language modeling (e.g., bert) @xcite . it is unexpected that, unlike in other language tasks where pre-trained models simply excel, directly fine-tuning bert in dr often underperforms unsupervised sparse retrieval, e.g., bm25. some complicated procedures are almost necessary to effectively fine-tune pre-trained transformers in dense retrieval @xcite @xcite . one observation is that the pre-trained language models are not effective at encoding the semantics of the entire text sequence in one embedding, especially in dense retrieval where text sequences are mostly longer than 128 tokens @xcite . in some other modalities, autoencoders have been widely used to obtain high-quality data representations @xcite . they pair a decoder on top of the encoder, trains the decoder to reconstruct the data solely from the encoder's encodings, thus enforce an information bottleneck on the data encodings for better representation quality. recently, autoencoders have been brought in language pre-training. @xcite stacks a gpt-2 decoder on top of the bert encoder and trains the autoencoder via a conditional language modeling task. their learned encoder, optimus, provides better text encodings for glue and language generation tasks, but, as shown in our empirical study, does not provide better encodings for dense retrieval. this phenomenon inspires us to investigate why the standard setup of autoencoders in language modeling falls short in dense retrieval. we first notice that in the auto-regressive decoder, the model takes not only the cls encoding but also the previous tokens as input. our mathematical analysis shows that the decoder can exploit natural language patterns using its access to previous tokens and bypass the dependency on the encoder, especially when the sequence is long and the decoder is strong, e.g., gpt-2. as a result, the autoencoder achieving a low reconstruction loss value does not necessarily provide better text sequence encodings. our analyses lead to a quite simple solution: we present a new autoencoder pre-training strategy, which pairs the bert-style encoder with a weak decoder by restricting its parameter capacity and attention flexibility. this way, our seed-encoder, "strong text encoder by training with weak decoder", creates an information bottleneck in the autoencoder and forces the encoder to provide better text representations. in our experiments on three real-world applications, we confirm that seed-encoder produces better pre-trained checkpoints that seed dense retrieval models with higher accuracy and better few-shot ability. dense retrieval with text encoders. dense-retrieval systems often use the siamese/dual encoder architecture, where two sequences are encoded by the transformer separately, and their similarity is calculated upon their sequence embeddings. @xcite is among the first to study how to use bert in a siamese architecture and found that the cls representation does not perform as well as expected. recent research @xcite demonstrated that applying pre-trained models in dense text retrieval is not as straightforward. @xcite use bm25 to find negative samples to better fine-tune pre-trained models for dense retrieval. @xcite performs global noise constructive estimation and finds global negatives using the dr model for the dr model. ## method in this section, we first recap preliminaries in language pre-training and autoencoder. then we discuss the drawbacks of using strong decoders in autoencoder and address them with seed-encoder. ## experiments in this section, we present various experimental analyses to evaluate the seed-encoder on dense retrieval tasks. more results on other language tasks are in appendix a.2. ## conclusion in this paper we present seed-encoder, a selftraining framework dedicated to pre-training language models for dense text retrieval. we pre-train an auto-encoder that employs a weak decoder with restricted capacity and attention span following our mathematical derivation. the weak decoder helps seed-encoder capture more context information and generate better text representation. in our experiments on web search, news recommendation, and question answering, seed-encoder initialized dense retrieval models achieve state-of-the-art accuracy compared to several strong baselines. future work along this direction includes exploring more self-learning tasks and network architectures for sequence matching in dense retrieval scenarios.