mGTE : Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval (2024)

Abstract: We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications.

## introduction text retrieval aims to find relevant passages or documents from a large corpus given a query @xcite . it is often implemented as a multi-stage process, consisting of two main components: a retriever and a reranker @xcite @xcite . the retriever identifies a set of candidate documents that are potentially relevant to the query based on the similarity between their sparse (lexical term weights) or/and dense representations from a text representation model (trm). while the reranker reorders these retrieved candidates to refine the results based on the relevance score generated by a more precise yet computationally demanding model that processes both the query and a candidate document together. recent advances in large language models (llms) and retrieval augmented generation @xcite random encoder 2k text encoder 8k text encoder 1k text embedder 8k trm 8k reranker contrastive fine-tuning contrastive pre-training mlm pre-training figure 1 : training pipeline. we first build an 8k longcontext multilingual encoder. then based on it, we train text representation and reranking models for retrieval. @xcite systems have led to an unprecedented surge in demand for versatile, plug-andplay trms and rerankers. these new applications heavily involve processing long and multilingual texts, which could not be addressed by conventional encoder-based models and urgently require upgraded ones. to this end, some resort to enhancing existing multilingual encoders, e.g., xlm-r @xcite , with extended context window up to 8192 @xcite . others turn to use multilingual llms which already have the required capabilities @xcite , but their models might be computationally expensive for self-hosted search services. in the english community, it has been proven that training long-context encoders from scratch is promising for text retrieval @xcite . in this work, we continue this journey, presenting systematic efforts in building the long-context multilingual text encoder, trm, and reranker. we suggest a holistic pipeline (figure 1 ) as well as several techniques in modeling and training for multilingual long-context retrieval. concretely, we first introduce a text encoder enhanced with rotary position embedding @xcite and unpadding @xcite , pre-trained by masked language modeling (mlm) @xcite and sparse vectors for efficient first-stage retrieval, as well as a cross-encoder reranker. we construct them via the contrastive learning objective @xcite with large-scale meticulously curated datasets, providing robust off-theshelf retrieval models. we conduct extensive experiments to verify our method. for the text encoder, we evaluate on two natural language understanding (nlu) benchmarks, i.e., xtreme-r @xcite and glue @xcite , and show that our encoder outperforms the same-sized previous stateof-the-art xlm-r. for the trm and reranker, we evaluate on multiple retrieval benchmarks with multilingual and long-context settings, e.g., mir-acl @xcite and mldr @xcite , where our models match the performance of state-of-the-art bge-m3 @xcite and achieve better long-context performance by a smaller size. we open-source our models and code to facilitate further research and applications. ## method 2.1 text encoder to construct powerful long-context multilingual text encoder models, we implement several enhancements to bert @xcite architecture and train it from scratch using the vocabulary of xlm-r foot_0 @xcite series. specifically, we replace the absolute positional embeddings with rope @xcite , and upgrade the feedforward network (ffn) to gated linear unit (glu) @xcite . to ensure compat-ibility with libraries like flashattention @xcite , we remove the dropout applied to attention scores. in addition, we pad the token embedding size to be a multiple of 64, which could speedup the model throughput @xcite . @xcite , we unpad the input batch to reduce redundant computations associated with padding tokens (figure 2 ). we use xformers @xcite to implement the variable length attention. it dispatch the attention forward and backward to different kernels foot_1 based on the numerical precision, attention head size and device type. we unpad the mlm labels as well to reduce the computation cost of predicting non-masked tokens. ## evaluation we separately evaluate our text encoder in §3.1, trm and reranker in §3.2 and §3.3. ## conclusion we present the holistic practice of building native 8192-context multilingual retrieval models. we first suggest a text encoder with rope and unpadding, which is pre-trained by a two-stage mlm curriculum for 8k context. evaluations on nlu benchmarks show that our encoder outperforms xlm-roberta in the same size. based on our encoder, we construct a hybrid trm and a crossencoder reranker by contrastive learning. the trm is pre-trained with reversed rope ntk scaling and fine-tuned to generate both matryoshka embeddings and sparse representations. results on monolingual and crosslingual retrieval benchmarks show that our trm and reranker are close to larger ones on regular datasets, and achieve better performance on long-context datasets. this means our models are more efficient for industrial applications. chinese pairs @xcite , multilingual pairs (cc-news foot_8 ), and crosslingual instruction and translation pairs @xcite . we filter the data by removing duplicates and low-quality pairs, resulting in a total of 2,938.8m pairs.