Condenser: a Pre-training Architecture for Dense Retrieval (2021)

Abstract: Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs’ internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.

## introduction language model (lm) pre-training has been very effective in learning text encoders that can be finetuned for many downstream tasks @xcite . deep bidirectional transformer encoder @xcite lms like bert @xcite are the state-ofthe-art. recent works fine-tune the cls token to encode input text sequence into a single vector representation @xcite @xcite . the resulting model is referred to as dense encoder or bi-encoder. finetuning associates with vector similarities some practical semantics, e.g., textual similarity or relevance, and therefore the vectors can be used for efficient text comparison or retrieval by inner product. despite their efficiency, bi-encoders are hard to train. even with sufficient data, bi-encoders still require carefully designed sophisticated methods to train effectively @xcite @xcite . they can also take big performance hits in low data situations @xcite @xcite . another common use of deep lm is cross-encoder, pass compared text pair directly in and use attention overall tokens to do prediction. in contrast to biencoder, cross encoder trains easier and is effective in low data for similarity and ranking tasks @xcite . based on the same lm, however, bi-encoder and cross encoder have similar language understanding capabilities. to explain the difficulty in training bi-encoder not seen in cross-encoder, we look into the internal structure of pre-trained lm. we find lm like bert directly out of pre-training has a non-optimal attention structure. in particular, they were not trained to aggregate sophisticated information into a single dense representation. we term effort during fine-tuning to adjust the lm internal activation to channel its knowledge out for the target task, structural readiness. we argue bi-encoder fine-tuning is inefficient due to the lacking structural readiness. many updates are used to adjust model attention structure than learn good representation. based on our observations, we propose to address structural readiness during pre-training. we introduce a novel transformer pre-training architecture, condenser, which establishes structural readiness by doing lm pre-training actively condition on dense representation. unlike previous works that pre-train towards a particular task, condenser pre-trains towards the bi-encoder structure. our results show the importance of structural readiness. we experiment with sentence similarity tasks, and retrieval for question answering and web search. we find under low data setups, with identical test time architecture, condenser yields sizable improvement over standard lm and shows comparable performance to strong task-specific pretrained models. with large training data, we find condenser retriever optimize more easily, outperforming previous models trained with complicated techniques with a single round of negative mining. ## related work transformer bi-encoder lm pre-training followed by task fine-tuning has become one important paradigm in nlp @xcite . sota models adopt the transformer architecture @xcite @xcite . one challenge for applying deep transformer is their computation cost when used to retrieve text from large collections. motivated by this, @xcite propose sbert which trains biencoder from bert and uses vector product for efficient sentence similarity comparison. transformer bi-encoders were soon also adopted as dense retriever @xcite @xcite . dense retrieval dense retrieval compares encoded query vectors with corpus document vectors using inner product. while there are works on efficient cross-encoder @xcite , such models are still too costly for full corpus retrieval. by pre-encoding the corpus into mips @xcite index, retrieval can run online with millisecond-level latency. an alternative is the recently proposed contextualized sparse retrieval model @xcite . in comparison, dense retrieval is easier to use and backed by more matured software like faiss @xcite . pre-train bi-encoder @xcite are among the first to show the effectiveness of transformer bi-encoder for dense retrieval. they proposed to further pre-train bert with inverse cloze task (ict). ict uses pair of passage segment and full passage as pseudo training pair. @xcite find ict and other related tasks are "key ingredients" for strong bi-encoders. their results also show that models without pre-training fail to produce useful retrieval results under low data setups. @xcite propose to pre-train retriever and reader together for end-to-end qa system. the aforementioned methods are specialized task specific solutions for improving bi-encoder training based on contrastive loss. this paper provides an explanation for the learning issue and presents an architecture that establishes a universal solution using general language model pre-training. we also note that language model and contrastive pretraining are orthogonal ideas. in a follow-up work, we show further improved performance adding contrastive learning to condenser language model pretraining @xcite . effective dense retriever @xcite found carefully fine-tuning bert can produce better results than earlier pre-trained dense retrieval systems. to further improve the end performance of dense retrievers, later works look into better fine-tuning techniques. using a learned retriever to mine hard negatives and re-train another retriever with them was found helpful @xcite . ance @xcite actively mines hard negatives once after an interval during training to prevent diminishing gradients. it allocates extra resources to update and retrieve from the corpus retrieval index repetitively. @xcite proposed to jointly learn a pair of dense and sparse systems to mitigate the capacity issue with low dimension dense vectors. beyond fine-tuning, using more sophisticated knowledge distillation loss to learn bi-encoders based on soft labels has also been found useful @xcite . they first learn a teacher model and use its predictions at training time to optimize the dense retriever. these works all aim at producing better gradient updates during training, while condenser aims at better initializing the model. we will also show the combined improvement of condenser and hard negatives in experiments. another line of works question the capacity of single vector representation and propose to use multi-vector representation @xcite . capacity defines the performance upper bound and is one other issue than training (optimization), i.e. how to reach the upper bound. sentence representation we'd also like to make a distinction from works in universal sentence representation and encoder @xcite @xcite . they are featurebased methods rather than fine-tuning @xcite . in evaluation, they focus on using the learned embedding as universal features for a wide range of tasks @xcite . this paper considers task-specific fine-tuning of the entire model and focuses on the target task performance. ## method this section discusses the motivation behind condenser, its design, and its pre-training procedure. ## experiments in this section, we first describe details on how to pre-train condenser from bert. our fine-tuning experiments then look into the impacts of condenser under low and high data setup. to evaluate low data, we sample smaller training sets similar to @xcite , by sub-sampling the original train set. we keep dev/test sets unchanged across runs for direct comparison. we first validate our model with short sentence level tasks, then evaluate retrieval in open question answering and web search tasks following prior works @xcite . we will examine how swapping original bert with condenser improves performance, and how the improvements compare to various improved training techniques. ## attention analysis condenser is built upon the idea that typical pretrained lm lacks proper attention structure. we already see that we can fix the issue by pre-training with condenser in the last section. in this section, we provide a more in-depth attention analysis: we compare attention behaviors among pretrained/fine-tuned bert, ict, and condenser. we use an analytical method proposed by @xcite , characterizing the attention patterns of cls by measuring its attention entropy. a higher entropy indicates broader attention and a lower more focused attention. similar to @xcite , we show cls attention entropy at each layer, averaged over all heads, and averaged over 1k randomly picked wikipedia sections. in figure 2 , we plot attention from cls of various models. we see in figure 2a that bert has a drastic change in attention pattern between pretrained and fine-tuned models. this again confirmed our theory that typical transformer encoder lms are not ready to be fined-tuned into biencoder, but need to go through big internal structural changes. in comparison, we see in figures 2b , 2c that task-specific pre-trained ict and lm pretrained condenser only have small changes, retaining general attention structure. in other words, ict and condenser both established structural readiness, but in very different ways. both ict and condenser have broadening attention (increased entropy) in the later layers, potentially because the actual search task requires aggregating more highlevel concepts than pre-training. the results here again confirm our theory, that a ready-to-use structure can be easier to train; their structures only need small changes to work as an effective bi-encoder. ## conclusion fine-tuning from a pre-trained lm initializer like bert has become a very common practice in nlp. in this paper, we however question if models like bert are the most proper initializer for bi-encoder. we find typical pre-trained lm does not have an internal attention structure ready for bi-encoder. they cannot effectively condense information into a single vector dense representation. we propose a new architecture, condenser, which establishes readiness in structure with lm pre-training. we show condenser is effective for a variety of tasks, sentence similarity, question answering retrieval, and web search retrieval. with low data, condenser shows comparable performance to task-specific pre-trained models. it also provides a new pretraining perspective in learning effective retrievers than fine-tuning strategies. with sufficient training, condenser and direct fine-tuning can be a lightweight alternative to many sophisticated training techniques. positive results with condenser show that structural readiness is a fundamental property in easyto-train bi-encoders. our attention analysis reveals both condenser and task-specific pre-trained model establish structural readiness, suggesting task-specific objective may not be necessary. researchers can use this finding to guide the study of better lm for bi-encoder, for example, explore training condenser with other lm objectives. one big advantage of bert is that after cumbersome pre-training for once, fine-tuning is easy with this universal model initializer. this is however not true for bert bi-encoder, especially retriever, which needs careful and costly training. condenser extends this benefit of bert to bi-encoder. practitioners on a limited budget can replace bert with our pre-trained condenser as the initializer to get an instant performance boost. meanwhile, for those aiming at the best performance, training techniques and condenser can be combined. as we have demonstrated the combined effect of hard negatives and condenser, sophisticated but better techniques can be further incorporated to train condenser.