LAR e QA : Language-Agnostic Answer Retrieval from a Multilingual Pool (2020)

Abstract: We present LAReQA, a challenging new benchmark for language-agnostic answer retrieval from a multilingual candidate pool. Unlike previous cross-lingual tasks, LAReQA tests for “strong” cross-lingual alignment, requiring semantically related cross -language pairs to be closer in representation space than unrelated same -language pairs. This level of alignment is important for the practical task of cross-lingual information retrieval. Building on multilingual BERT (mBERT), we study different strategies for achieving strong alignment. We find that augmenting training data via machine translation is effective, and improves significantly over using mBERT out-of-the-box. Interestingly, model performance on zero-shot variants of our task that only target “weak” alignment is not predictive of performance on LAReQA. This finding underscores our claim that language-agnostic retrieval is a substantively new kind of cross-lingual evaluation, and suggests that measuring both weak and strong alignment will be important for improving cross-lingual systems going forward. We release our dataset and evaluation code at https://github.com/google-research-datasets/lareqa .

## introduction recent progress in self-supervised pretraining for language understanding has enabled training large multilingual models on 100+ languages at the same time, as in multilingual bert (mbert) and xlm-r @xcite . these models, despite being trained without any explicit objective of cross-lingual alignment, are surprisingly effective for cross-lingual transfer (pires et al., 2019; @xcite , suggesting that the models may have learned to "factor out" language and embed inputs into a language-agnostic space. at the same time, @xcite observe that mbert representations at all layers are highly accurate (>96%) at classifying language id, which the authors find "surprising given the model's zeroshot cross-lingual abilities". this raises an interesting question. to what degree are models like mbert and xlm-r language agnostic or easily adaptable to be so? have they effectively disentangled the language-specific signal from the underlying semantic content, with each occupying a separate subspace? can the learned representations be adapted by a lightweight alignment procedure to be truly language agnostic? to address these questions, we introduce a challenging new task, lareqa: language agnostic retrieval question answering, which requires models to retrieve relevant cross-lingual answers from a multilingual candidate pool. perform-ing well on this task demands a stronger degree of cross-lingual alignment than previous crosslingual evaluations like xnli @xcite . concretely, we propose to distinguish "weak" vs. "strong" alignment, defined as follows and illustrated in figures 1a-1b : weak alignment for any item in language l 1 , the nearest neighbor in language l 2 is the most semantically relevant item. (the specific notion of "relevance" may vary across tasks.) strong alignment for any item, all semantically relevant items are closer than all irrelevant items, regardless of their language. crucially, relevant items in different languages are closer than irrelevant items in the same language. 1 to our knowledge, lareqa is the first crosslingual benchmark to target strong alignment. 2 building on top of multilingual bert, we develop and test several baseline models on lareqa. we find that mbert already exhibits strong alignment between some language pairs, but that this can be improved significantly by leveraging machine translation to extend the set of training examples and encourage cross-lingual alignment. one observation that emerges from our experiments is that strong alignment comes at a cost. specifically, our baseline that reaches the best lareqa performance lags behind other baselines on the narrower task of retrieving relevant answers that match the question language. our main contributions are: (1) we propose a new framework for classifying different degrees of cross-lingual alignment. (2) we propose a challenging new benchmark to evaluate language bias in representations, setting stricter notions of crosslingual embedding space alignment. (3) we investigate the potential for multilingual bert to achieve "strong" cross-lingual alignment, including various fine-tuning techniques to improve alignment. (4) we publish our trained models and lareqa benchmarking code for others to reproduce. 3 1 stricter notions of cross-lingual alignment are possible, such as requiring that model representations remove any trace of the original text language, preventing language id from being reconstructed. we treat these as sub-types of "strong" alignment, but leave their investigation for future work. in this section, we present the practical task of answer retrieval from a multilingual candidate pool, and argue that this task goes beyond existing cross-lingual benchmarks in demanding models with strongly aligned multilingual representations. the task can be summarized as: given a question in one language and potential answers in many different languages, retrieve the best answer for the question, regardless of language. we begin by describing why this task is both useful and challenging. subsequently, we compare this task with existing cross-lingual tasks, and show how they differ in their ability to measure "language bias". ## lareqa having motivated the need for a cross-lingual benchmark that asks models to choose between languages, we now present a concrete case of such a cross-lingual evaluation, lareqa: language-agnostic retrieval question answering. ## baseline models we consider several neural baseline models for evaluation on lareqa. all our baselines are "dual encoder" models @xcite , encoding questions and candidate answers separately. unlike full cross-attention models, this architecture enables retrieval through approximate nearest neighbor search, and thereby scales well to large-scale retrieval problems. for more discussion of dual encoders for deep retrieval, see @xcite and @xcite . our baselines leverage multilingual bert @xcite , or "mbert", for cross-lingual pretraining. these baselines allow us to test (i) how well mbert already aligns languages in a language-agnostic space, and (ii) the degree to which it can be adapted to do so. we initialize each tower of a dual encoder model with pretrained mbert, sharing weights between the question and answer encoding towers 10 , as in figure 2 . to obtain final question and answer encodings, we normalize the bert cls token to unit l2 norm. the model score for a qa pair s(q, a) is the dot product of these encodings. we fine-tune the mbert towers for qa retrieval on squad training data using an "in-batch sam10 when feeding inputs to the answer encoding tower, we concatenate the answer sentence and its containing context paragraph ("answer context"), using bert's segment ids to distinguish between the two. pled softmax" loss @xcite , as this has been observed to converge quickly and perform well on retrieval tasks @xcite . the loss, given in equation 2 , encourages the score of a correct answer pair s(q i , a i ) to be higher than scores for incorrect answer pairings from the minibatch s(q i , a j ): we train four variants of our mbert model (110m parameters), using different fine-tuning datasets and batching procedures. each model is fine-tuned on 32 tpu-v3 cores, with a batch size of 2,048. the in-batch sampled softmax is calculated separately per core, over sub-batches of 64 qa pairs. we use the standard bert learning rate schedule, with an initial learning rate of 1e-5, which performed the best among {1e-6, 3e-6, 1e-5, 3e-5}. we train all baselines for 100,000 steps and observe no overfitting. our first baseline "en-en" adapts mbert to qa retrieval by fine-tuning on the 80,000 english qa pairs from the squad v1.1 train set, with the ranking loss from equation 2. this baseline tests how well mbert can perform language-agnostic retrieval when only fine-tuning on english data. our second baseline "x-x" extends the same squad train set by translating each example into the 11 xquad languages using an in-house trans- our fourth baseline "x-y" is similar to x-x, but allows a question and answer to be translated into different languages, giving 9,680,000 examples. this setup is the first to directly incentivize the model to treat answers from other languages as correct, which we expect to further reduce samelanguage bias. our final baseline "translate-test" is not a proper text embedding model, as it relies on an external translation system at test time. here, we simply translate any test data into english, and then score it with our en-en model. additionally, we compare the above baselines with universal sentence encoder multilingual qa @xcite , which specifically targets cross-lingual qa retrieval. however, this model only supports 8 of the 11 xquad languages, and we found it was not competitive with our mbert baselines, even when restricting evaluation to the supported languages. see appendix a for details. ## conclusion lareqa is a challenging new benchmark testing answer retrieval from a multilingual candidate pool. it goes further than previous cross-lingual benchmarks in requiring "strong" cross-lingual alignment, which is a step closer to truly languageagnostic representations. we believe there is significant headroom for models to improve on lareqa. our best initial baseline sidesteps the alignment problem by simply translating all test data to english. among embedding-based models, our strongest baseline ("x-y") actively removes language bias by augmenting training data to include machine-translated cross-lingual examples. however, to achieve strong alignment, this model sacrifices performance on both retrieval from a monolingual pool ( the heatmaps in figure 6 show for each question language (row), the frequency of different answer languages (column) among the top 100 retrieved candidates, for each of our baseline models on the xquad-r dataset. the strong diagonal in x-x indicates that when the question is in a given language, nearly all of the top 100 retrieved results are in the same language. overall, this measure of language bias is consistent with those discussed in section 5.2, with the models ranking x-@xmath0. interestingly, x-y performs almost perfectly on this "semantics-free" measurement of language bias. this is in contrast to the map performance of the same model in figure 4d , where the retrieval of correct answers is somewhat improved when the q and a languages match. taken together, we can say that x-y is nearly perfectly unbiased in which languages it retrieves on the whole, but is slightly biased as to which language pairs exhibit the strongest qa matching.