m A ggretriever: A Simple yet Effective Approach to Zero-Shot Multilingual Dense Retrieval (2023)

Abstract: Multilingual information retrieval (MLIR) is a crucial yet challenging task due to the need for human annotations in multiple languages, making training data creation labor-intensive. In this paper, we introduce mAggretriever, which effectively leverages semantic and lexical features from pre-trained multilingual transformers (e.g., mBERT and XLM-R) for dense retrieval. To enhance training and inference efficiency, we employ approximate masked-language modeling prediction for computing lexical features, reducing 70–85% GPU memory requirement for mAggretriever fine-tuning. Empirical results demonstrate that mAggretriever, fine-tuned solely on English training data, surpasses existing state-of-the-art multilingual dense retrieval models that undergo further training on large-scale MLIR training data. Our code is available at url.

## introduction fine-tuning a pre-trained transformer has proven to be highly effective in many tasks of natural language processing, including information retrieval (ir). despite its success, the recent state-of-theart dense retrieval (dr) models @xcite predominantly focus on english. this bias arises from the fact that constructing an effective dr model requires a substantial amount of annotated training data, which is predominantly available in english datasets @xcite . this makes it challenging for users of low-resource languages to benefit from the recent ir progress. to address this issue, researchers have explored leveraging pre-trained multilingual transformers, such as mbert @xcite and xlm-r @xcite , which exhibit promising language transferability even when fine-tuned on * work done during sheng-chieh's internship at vectara. english datasets alone @xcite . however, @xcite emphasize the importance of contrastive pre-training on multilingual corpora to achieve improved language transferability in ir. other approaches utilize multilingual parallel corpora or translation pairs for contrastive pretraining @xcite or fine-tuning @xcite . however, these solutions are viable only when significant computational resources or multilingual parallel data are available. recently, @xcite demonstrate that the existing dr models solely using [cls] (or averaged pooling) do not fully exploit the capability from pre-trained transformers for retrieval. @xcite propose aggretriever by combining the semantic and lexical features from the respective two components, @xcite and masked language modeling (mlm) prediction, which shows superior effectiveness on diverse english retrieval tasks. this inspires us to ask the question: can aggretriever benefit multilingual retrieval? 1 extending aggretriever to support multilingual retrieval poses challenges due to increased computation costs for extracting lexical features from the mlm component as the vocabulary size of the pre-trained model grows, making fine-tuning with limited resources challenging. in this work, we propose two simple approaches to approximate mlm prediction to extract lexical features from multilingual pre-trained transformers, making the training of maggretriever possible in one gpu within 40 gbs of memory. remarkably, maggretriever exhibits strong retrieval capability across multiple languages despite being fine-tuned on english data. the paper is structured as follows: we begin by providing background information on standard dr and aggretriever. we then present our proposed approaches to tackle the computational challenges involved in extending aggretriever to support multilingual retrieval. subsequently, we compare maggretriever with other state-of-the-art multilingual dr models and explore the potential for extending maggretriever to enable cross-lingual retrieval. aggretriever. in addition to using [cls] vectors to capture semantic textual features, @xcite further propose to capture lexical textual features from the pre-trained mlm prediction head by projecting each contextualized token embedding e q i into a high-dimensional vector in the wordpiece lexical space: ## our approach in this work, we extend aggretriever to multilingual retrieval, maggretriever. however, directly applying eq. ( 2 ) to multilingual pre-trained mlm heads over the whole vocabulary increases computation cost for both training and inference due to large vocabulary size |v wp | in mlm projector, w mlm @xcite . for example, the mbert and xlm-r have respective vocabulary size of 120k and 250k (vs bert's 35k) in the mlm projector. to address the issue, we propose two computationally efficient approaches to approximate mlm prediction in eq. ( 2 ). target-language prediction. one intuitive approach is to compute the probability distribution over the target-language token set of interest, denoted as v tg wp , by replacing w mlm and b mlm in eq. ( 2 ) with w tg mlm and b tg mlm , respectively. from the above equation, we only have to compute the matrix multiplication and softmax among |v tg wp | instead of |v wp | tokens. this approach assumes that only the tokens corresponding to the target language have responses when conducting mlm prediction; i.e., self prediction. since bert mlm pre-training task includes recovering the unmasked token itself, to further save computation cost, we may approximate mlm prediction by assigning a value of one to the token q i itself, and zero to the others: the operation removes the computationally expensive matrix multiplication and softmax operation in eq. ( 2 ). note that, combining eq. ( 3 ) and ( 5 ), the lexical representations, v q , from self prediction can be considered bag-of-word vectors with learned term weights. figure 1 illustrates whole-vocabulary, our proposed target-language and self mlm predictions. ## conclusion in this paper, we introduce maggretriever, an extension of aggretriever for multilingual retrieval, by combining lexical and semantic features in pretrained language models for dense retrieval. we propose target-language and self mlm predictions to enhance the efficiency of maggretriever. our study highlights the efficiency advantage of self mlm prediction in multilingual retrieval, while target-language mlm prediction offers flexibility for cross-lingual retrieval. importantly, maggretriever, solely fine-tuned on english data, demonstrates competitive multilingual retrieval capability compared to other state-of-the-art dense retrievers.