CELI : Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders. (2024)

Abstract: In text ranking, it is generally believed that the cross-encoders already gather sufficient token interaction information via the attention mechanism in the hidden layers. However, our results show that the cross-encoders can consistently benefit from additional token interaction in the similarity computation at the last layer. We introduce CELI (Cross-Encoder with Late Interaction), which incorporates a late interaction layer into the current cross-encoder models. This simple method brings 5% improvement on BEIR without compromising in-domain effectiveness or search latency. Extensive experiments show that this finding is consistent across different sizes of the cross-encoder models and the first-stage retrievers. Our findings suggest that boiling all information into the [CLS] token is a suboptimal use for cross-encoders, and advocate further studies to investigate its relevance score mechanism.

## introduction the two-stage retrieve-then-rerank pipeline has been the de facto design for many information retrieval systems. recently, both the retriever and the reranker systems have benefited from the advancement in pretrained language models @xcite . when using the pretrained models as crossencoders, the model views the query and document candidate together, allowing rich token interaction via the attention mechanism at all hidden layers. however, all the information then boils down to the representation of @xcite token at the final stage, during the computation of the relevance score between the query and the document. this raises the concern of whether the single token representation is sufficient to capture all salient information. pretrained language models have also been adopted in the retriever stage in various ways. @xcite and find that the [cls] token embedding could well capture query or document representations, whose similarity can be used to indicate the relevance level between the query and document. this line of methods @xcite , named as single-vector dense retrievers, while being effective for the in-domain scenarios, is found to be less robust on the out-ofdistribution (ood) datasets @xcite , possibly due to inadequate information at tokenlevel granularity. methods such as further pretraining or adding token-level interaction have been applied to improve the ood generalization, where multi-vector retrievers @xcite @xcite perform the best on both the in-domain and ood effectiveness among neural retrievers @xcite . this ability is usually credited to its design that computes the similarity score based on contextualized embeddings of all tokens, which provides richer token interactions compared to the single-vector dense retrievers. some of the previous works term the models as "mono{backbone}" when initialized from non-bert pretrained transformers, such as mono-electra @xcite . however, since the underlying model structure remains the same, we refer to them all as monobert while specifying the backbones where the models are initialized. ## experimental setup all cross-encoders are trained on ms marco @xcite , a dataset composed of queries from bing search log and a collection of passages sourced from the general web. it contains 8.8m passages, over 500k query-document pairs for training, and 6980 queries for inference. marco for 30k steps with a learning rate 1e -5 and a batch size 16. we use linear warmup on the first 3k steps, then linearly decay the learning rate on the rest of the steps. cross-encoders are trained on lce loss @xcite with 7 negative samples. 2 we experimented with three backbones, all available on hug-gingface @xcite : minilm (wang et al., 2020), foot_2 electra base , foot_3 and electra large @xcite ). 5 we use ms marco @xcite for the in-domain evaluation and 13 datasets from beir @xcite for ood evaluation, which covers 10 domains including wikipedia, finance, scientific, quora, and so on. at the inference stage, we always rerank top-1k results from the first-stage retrievers. on ms marco, we use tct-colbert @xcite as the retriever following @xcite . on beir, we use an extensive list of retrievers that covers the categories of sparse, single-and multi-vector dense retrievers. retrievers results are produced using pyserini @xcite , beir @xcite , or colbert (khattab and zaharia, 2020) repository. 6 following the datasets standard, we report mrr@10 on ms marco and ndcg@10 on beir. ## results and analysis in this section, we first demonstrate that such improvement is consistent over different model sizes and the first-stage retrievers, then analyze how the projected token dimension and the query length impact the improvement. ## conclusion in this work, we show that adding late interaction to existing cross-encoders brings visible improvement to its ood capacity without hurting in-domain effectiveness, even though the cross-encoder already processes the token interaction in earlier layers. extensive experiments on different model sizes and first-stage retrievers show that this improvement is consistent, and according to our analysis, the improvement is more prominent on longer queries. our findings suggest that boiling all information into the [cls] token is a suboptimal use for crossencoders, and further studies are required to better explore their capacities.