Cross-Lingual Training of Neural Models for Document Ranking (2020)

Abstract: We tackle the challenge of cross-lingual training of neural document ranking models for mono-lingual retrieval, specifically leveraging relevance judgments in English to improve search in non-English languages. Our work successfully applies multi-lingual BERT (mBERT) to document ranking and additionally compares against a number of alternatives: translating the training data, translating documents, multi-stage hybrids, and ensembles. Experiments on test collections in six different languages from diverse language families reveal many interesting findings: model-based relevance transfer using mBERT can significantly improve search quality in (non-English) mono-lingual retrieval, but other “low resource” approaches are competitive as well.

## introduction this work proposes techniques for leveraging relevance judgments in a source language (english) to train neural models for mono-lingual document retrieval in multiple target (non-english) languages, what we refer to as cross-lingual training. success in this task would make it easier to develop effective search engines in multiple (potentially lowresource) languages, without gathering expensive relevance judgments in each language. a blog post by google suggests that the company is exploring this approach to improving web search across a number of languages. 1 we are inspired by the work of @xcite , who explored the cross-lingual potential of multi-lingual bert as a zero-shot language transfer model for nlp tasks such as named-entity recognition and parsing. mono-lingual bert models @xcite have also proven effective in document retrieval @xcite @xcite . in particular, akkalyoncu @xcite demonstrated that bert models fine-tuned with passage-level relevance data can transfer across domains: surprisingly, fine-tuning on social media data is effective for relevance classification on newswire documents without any additional modifications. building on these results, we wondered if multi-lingual bert could enable cross-lingual training of neural document ranking models as well. the contribution of this work is to explore diverse methods to train neural document ranking models cross-lingually. while we are aware of two previous papers along these lines @xcite , this work explores a far broader range of techniques and adds more nuance to previous findings. beyond the basic approach proposed by these two papers, which we refer to as model-based transfer, we investigate additional approaches involving the translation of the training data, the translation of documents, hybrid models, as well as ensembles-which we broadly characterize into "high resource" and "low resource" settings. we show that various methods alone and in combination can yield robust increases in effectiveness across diverse languages with minimal resources, and that model-based cross-lingual transfer isn't the only way. ## approach this work adopts the standard formulation of document ranking: given a user query q, the task is to produce a ranking of documents from a collection that maximizes some ranking metric-in our case, average precision (ap). given source language relevance judgments (in english), our task is to train a mono-lingual document ranking model for a target (non-english) language; that is, the queries and the documents are both in, for example, bengali. ## experimental setup we experimented with six test collections (in chinese, arabic, french, hindi, bengali and spanish) from diverse language families (sino-tibetan, semitic, romance, and indo-aryan). dataset statistics are shown in for the token-based translation method, we used the muse bilingual dictionary @xcite and the aligned word embeddings from fast-text @xcite . for fine-tuning mbert, we followed the same experimental setup as @xcite . we used data from the microblog (mb) tracks from trec 2011-2014 @xcite or its translated counterparts, setting aside 75% of the total data for training and the rest for validation, which was used for selecting the best model parameters. we trained each model using cross-entropy loss with a batch size of 16; the adam optimizer was applied with an initial learning rate of 1 ˆ10 ´5. during fine-tuning, the embeddings were fixed. the model with the highest ap on the validation set was chosen. we ran all experiments on an nvidia tesla v100 16gb with pytorch version 1.3.0. each model was trained for up to 15 epochs, with an average running time of approximately two hours. for retrieval, we used the open-source anserini ir toolkit @xcite with minor modifications based on version 0.6.0 to swap in lucene analyzers for different languages. fortunately, lucene provides analyzers for all the languages in our test collections. the query was used to retrieve the top 1000 hits from the corpus using bm25 or bm25+rm3 query expansion; default anserini settings were used in both cases. reranking with mbert (see section 2.1) used the approach with higher ap (either bm25 or bm25+rm3); the top three sentences were considered in aggregating sentence-level evidence. we applied five-fold crossvalidation on all datasets and the parameters α, the w i 's, and β were obtained by grid search, choosing the parameters that yielded the highest ap. ## results our results are shown in finding #1: model-based transfer, model (2), improves upon the baseline, with significant gains (denoted by ĳ ) everywhere except for ap in arabic and p@20 in spanish. since mbert is widely available, mono-lingual retrieval improvements can be obtained "for free" with microblog relevance judgments in english. these results indicate that mbert effectively transfers relevance matching across languages. this finding confirms previous work (shi @xcite , but see additional discussion below. finding #2: comparing model-based transfer and the two approaches to translating training data, models (3) and ( 4 ), it is difficult to spot trends or reach definitive conclusions. model-based transfer does not consistently beat simply translating the training data. in terms of ap, google translate, model (3), outperforms model-based transfer for chinese and arabic; token-based translation, model (4), beats model-based transfer in hindi and achieves comparable scores in arabic and spanish. interestingly, it is not always the case that google translate ("high resource") is better than token-based translation ("low resource"); the latter achieves higher ap for hindi and bengali. a tukey's hsd test across models (2-4) showed no significant differences. these results suggest that model-based transfer is not the only effective approach, and that simply translating the training data is at least competitive; neither shi and @xcite nor macavaney et al. (2020) explored this obvious baseline. finding #3: results show that hybrid two stage training in the enñgt direction, model (6), can further improve over model-based transfer alone or translating training data with google translate alone, but the gains are not consistent; lower ap than either models (2) or (3) in chinese, bengali, and spanish. when compared to the baseline, model (6) yields significant improvement on chinese, arabic, and hindi (denoted by § ). in the opposite direction, gtñen, while the hybrid model ( 7 ) significantly outperforms the baseline in a few cases, it doesn't seem to be consistently more effective than either models (2) or (3). note that both hybrid approaches are "high resource" since they require google translate. finding #4: document translation, model ( 5 ), generally beats model transfer, but it requires substantial resources, such as large amounts of parallel text for training a translation system. because all our documents are in the newswire domain, the output of google translate is quite reasonable. since this approach avoids language mismatch between training and test, it can outperform the model-based transfer approach: these improvements are significant (denoted by ¶ ) for the spanish collection on both metrics, and for the arabic, bengali, and french collections on ap. finding #5: in general, ensembles outperform model transfer alone, with the "high resource" approaches beating the "low resource" approaches (as expected). comparing the interpolation and rrf methods, we see no consistent trends. a tukey's hsd test showed no significant differences between the four ensemble methods. ## discussion given the effectiveness of model transfer, we additionally investigated a research question focused on model (2): how much contextual information does mbert rely on besides term matching? inspired by the query-centric assumption @xcite ) that relevance information is localized in the contexts around query terms, we conducted the following experiments: for each query term, we only kept the texts around the matched tokens in each sentence within a window size, and used only those contexts for reranking. we tried window size 1 (only the matched query terms are kept), 3 (the matched query terms with their left and right tokens), 5, 7, 11, and "sentence" (the entire sentence is kept if at least one query token matched). if the segments are from the same sentence, they are concatenated to form a new "sentence". experimental results are shown in figure 1 for two representative collections. for comparison, we also repeat results of the baseline, either model (0) or (1), denoted bm25 in the figure, and the results of model (2), denoted full in the figure. we can see that as the window size increases, ap tends to rise as well. this seems intuitive, as context is needed for relevance matching. furthermore, results show that some words critical for determining relevance bm25 1 3 5 7 11 sent full segment size 0.295 0.300 0.305 0.2932 0.2973 0.2976 0.2998 0.3015 0.3036 0.3080 0.3081 ar bm25 1 3 5 7 11 sent full 0.290 0.295 0.300 0.305 0.310 0.2881 0.2933 0.2980 0.3039 0.3024 0.3022 0.3095 0.3101 bn are located quite far from the query terms; these are discarded when the window size is too small, leading to lower ap scores. however, if we only keep sentences that have at least one query term, the ranking effectiveness is already comparable to using all sentences (0.3080 vs. 0.3081 in arabic, 0.3095 vs. 0.3101 in bengali). this simple filter can decrease the inference time needed for ranking 60% to 80% depending on different characteristics of the collections. ## conclusion as a high-level summary, our experiments confirm that mbert can enable cross-lingual training of document ranking models. however, mbert's "multi-lingual capacity" for direct model-based transfer does not appear to be consistently better than other approaches of bridging language gaps. for example, simple approaches such as tokenbased translation of the training data also work well. however, model-based transfer requires only a single model, whereas the latter requires a model for each language. overall, our work contributes to a better understanding of how relevance judgments in high-resource languages can be leveraged to improve search in low(er)-resources languages. our code is available on github. 3