CL-ReLKT: Cross-lingual Language Knowledge Transfer for Multilingual Retrieval Question Answering
Abstract
Cross-Lingual Retrieval Question Answering (CL-ReQA) is concerned with retrieving answer documents or passages to a question written in a different language.
A common approach to CL-ReQA is to create a multilingual sentence embedding space such that questionanswer pairs across different languages are close to each other.
In this paper, we propose a novel CL-ReQA method utilizing the concept of language knowledge transfer and a new cross-lingual consistency training technique to create a multilingual embedding space for ReQA.
To assess the effectiveness of our work, we conducted comprehensive experiments on CL-ReQA and a downstream task, machine reading QA.
We compared our proposed method with the current state-of-the-art solutions across three public CL-ReQA corpora.
Our method outperforms competitors in 19 out of 21 settings of CL-ReQA.
When used with a downstream machine reading QA task, our method outperforms the best existing language-model-based method by 10% in F1 while being 10 times faster in sentence embedding computation.
The code and models are available at https://github.com/ mrpeerat/CL-ReLKT .
Introduction
Cross-lingual question answering allows a question posed in one language to be answered using materials written in a different language.
As exemplified in Figure 1, one may ask, "Who was the first king of Hongsawadee?" and have their answer retrieved from a collection of historical documents in Burmese or other languages.
To support the given example application, we require a retrieval system that can handle documents and questions in multiple languages at the same time.
That is, we want to map questions and answers from multiple languages into the same space for easy retrieval.
This functionality is also known as Cross-Lingual
Retrieval Question Answering (CL-ReQA).
[Figure]
1. Existing Methods
One prominent approach to CL-ReQA is multilingual sentence embedding , i.e., creating an embedding space that can handle questions and answers from different languages.
This approach can be further categorized into (i) LM-Based: finetuning a language model (LM), e.g., mBERT and XLM-R; (ii
USE-Based finetuning the Universal Sentence
Encoder (USE) for QA.
LM-based.
Devlin et al.
and Conneau et al.
(2020) proposed a pretrained large-scale language model (LM) with multiple languages (100+ languages) called mBERT and XLM-R, respectively.
Both solutions rely on finetuning the LM part to the target task.
Reimers and Gurevych (2020) showed an accuracy improvement from 11.6% to 88.6% after finetuning with a bilingual text mining task.
Finetuning LMs has been explored by many recent works, e.g., triplet loss with various supervised learning tasks (Reimers and
Gurevych, 2019), knowledge distillation (Reimers and Gurevych, 2020), dense network QA encoder (Karpukhin et al., 2020), and providing initial word embeddings for the translation task (Feng et al., 2020).
Nonetheless, finetuning these models requires a large number of training samples (more than 100,000 sentences in some cases (Reimers and
Gurevych, 2020; Zhang et al., 2021; Wang et al.,
2021)) to give the best performance in multilingual settings.
On the other hand, cross-lingual QA training corpora are usually smaller with only 1,000 to 1,500 questions per language.
We need a method that can operate with a limited amount of data.
(mUSE).
Based on the Universal Sentence Encoder (USE) architecture (Cer et al., 2018), Yang et al. (2020) proposed a training method utilizing a multilingual corpus with 16 different languages and multiple training objectives.
They call their pretrained network multilingual USE or mUSE.
Experimental results from Trijakwanich et al. (2021) show that mUSE provides superior performance over the LM-based methods.
However, this method performs poorly on languages outside the mUSE training corpus, i.e., unsupported languages .
This limitation hinders the adoption of mUSE on limited-resource languages.
1. Our Work
Proposed Method.
In this paper, our goal is to improve the robustness of multilingual sentence embedding that works with a wide range of languages, including those with a limited amount of training data.
Leveraging the generalizability of language knowledge transfer (LKT), we propose a Cross-Lingual Retrieval Language Knowledge Transfer (CL-ReLKT) framework.
Figure 1 illustrates how cross-lingual retrieval can be conducted through a multilingual embedding function h () .
Given a question-document pair ( q, d ) in any language, h ( d ) is closer to h ( q ) than any other documents using any similarity measure, e.g., cosine similarity.
[Figure]
Before
After
[Figure]
Learning Objective.
As shown in Figure 2, the proposed CL-ReLKT framework is designed to improve the embedding space by making crosslingual question-answer pairs closer to each other.
The crux of our proposed framework lies in the following two parts.
First, we formulate a LKT process to create a language-generalized student.
In particular, we leverage the fact that there is likely to be one language in a large multilingual corpus that dominates all others.
We use that language to help improve the embedding quality of other languages.
Second, we formulate a new loss function designed to improve the cross-lingual consistency between question-answer pairs in a multilingual environment.
We aim to improve the consistencies between the teacher (dominant language) and student (other languages) for the following teacherstudent output pairs: question-question, documentdocument, and document-question.
Experimental Studies.
To determine the effectiveness of our approach, we compared the proposed methods with the current best practices (discussed in Section 1.1) on the CL-ReQA task across three datasets in 15 languages.
Experimental results show that the CL-ReLKT framework outperformed all competitive methods on languages supported by mUSE in all cases.
The results on unsupported languages, i.e., languages outside of the mUSE training corpus, show that the CL-ReLKT framework improved the performance of the mUSE encoder significantly ( p < 0 . 05) in all cases.
Moreover, on a downstream task of machine reading
QA (MR-QA), our method obtained better F1 and exact match scores than those of the best existing
LM-based method in seven out of eight cases.
Last but not least, our method is also 10 times faster than the state-of-the-art LM-based competitor in sentence embedding computational cost.
Summary of Contributions.
• We propose a new language knowledge transfer method called Cross-Lingual Retrieval Language Knowledge Transfer (CL-ReLKT) to transfer knowledge from the dominant language to non-dominant languages and build a languagegeneralized encoder.
• We design a new loss function to enforce crosslingual consistency between dominant and nondominant language vector representations.
• To assess the performance and efficiency of the models, we conducted an extensive set of experimental studies involving 2 tasks, 15 languages, and 8 competitors.
Experimental results show the benefits of our proposed CL-ReLKT framework.
Moreover, we found that retrieving answers at the document level yields a significant improvement over the passage-level methods.
Background 2. Dominant Language
In a multilingual dataset, the distribution of languages tends to be imbalanced.
As shown in Figure 3, the number of sentences in English is approximately 50% of all sentences in the corpus used to construct mUSE (Yang et al., 2020).
[Figure]
# S e n t e n ce (
M illi on )
[Figure]
The number of training data in mUSE
Due to the stated language imbalance, the model performance in languages with a large amount of data tend to be substantially better than that in other languages (Arivazhagan et al., 2019; Wang et al., 2020).
This issue can be problematic when we want the model performance to be consistent across multiple languages.
For the case of mUSE, as shown in Figure 3, we can see that English is the dominant language in terms of training data available.
Hence, the Englishto-English retrieval performance tends to be better than all other language pairs.
To verify this performance gap, we conducted a CL-ReQA experimental study using questions in non-English and answer documents in English; mUSE was used to encode the questions and documents.
Experimental results show a significant performance improvement when the questions are translated into English instead of using the original non-English questions, i.e., translated questions from Russian to English improving the precision-at-1 from 43.3% to 52.8%.
For the full results, see Appendix A.3.
2. Language Knowledge Transfer (LKT)
There are many techniques to boost a model performance on low-resource languages using the structure obtained from rich-resource ones.
Transfer and multitask learning have been popular paradigms for leveraging rich-resource languages.
These methods usually rely on the shared-encoder strategy so that the language pattern learned in one language can be shared across all other languages using the same model (Lin et al., 2019; Nooralahzadeh et al., 2020;
Zoph et al., 2016; Schwenk and Douze, 2017; Neubig and Hu, 2018;
Yang et al., 2020; Feng et al., 2020).
These classes of techniques are commonly known as Language Knowledge Transfer (LKT) .
With a shared encoder, improvements on one language tend to benefit other languages as well.
Let us consider a scenario where we have a large number of question-answer pairs in English and a significantly smaller number of pairs in other languages, e.g., Russian, French, and German.
By letting other languages share the same encoder as we update the encoder weights while training with English data, we can also improve the general encoding performance of the model in other languages.
Proposed Method
In this section, we formulate our proposed methods by leveraging the two concepts discussed in the previous section, dominant language and language knowledge transfer .
In particular, we perform language knowledge transfer to transfer the knowledge from the dominant languages to other languages.
Our proposed method consists of two stages: teacher model preparation and CrossLingual Retrieval Language Knowledge Transfer (CL-ReLKT), which are described as follows.
3. Stage 1: Teacher Model Preparation
The purpose of this stage is to create a strong teacher for language knowledge transfer in the next stage.
For a base model, we use mUSE small for efficiency and performance reasons 2 .
There are several techniques that can be used to create the teacher model,mUSE teacher .
The results from our ablation study (Appendix A.5) suggest that triplet loss is a reasonable choice. As shown in
Equation 1, triplet loss L tp is a training objective that maximizes the cosine similarity cos ( · ) between anchor-positive pairs ( a, p ) and makes similarity between anchor-negative pairs ( a, n ) smaller than a given threshold α for all the training data M .
[Equation]
While the anchors a can be randomly sampled from the questions, we need the CL-ReQA model to help select positives p and negatives n . For negative sample categorization, we consider two options. First, we can directly use the original mUSE small model to categorize the negative samples according to the current embedding space (online fashion) (Kaya and Bilge, 2019). Second, we can apply the method proposed by Karpukhin et al.
(2020), which utilizes BM25 (Trotman et al., 2014) to produce textual similarity scores.
From the ablation study given in Appendix A.5.1, the results show that the first three epochs use the initial strategy for triplet mining (Kaya and Bilge, 2019) before proceeding to online mining (Kaya and Bilge, 2019) for five epochs yield the best performance.
3. Stage 2: CL-ReLKT
We now describe our method to improve the general CL-ReQA performance using the concept of language knowledge transfer (LKT).
As we mentioned in Section 2.2, LKT is a technique that the language pattern learned in one language can be shared across all other languages using the same model.
Applying the same concept to our problem, we can set the LKT process to improve the embedding consistency between the dominant language and other languages.
In particular, we setup the LKT environment as follows: (i) the teacher operates in the dominant language, i.e., English; (ii) the student operates in
non-dominant languages; (
iii)
the student tries to mimic the embedding outputs of the teacher. In what follows, we describe the teacher and student models, inputs, and the loss function for the training process.
[Figure]
(CL-ReLKT) compris
Teacher and Student Models.
As illustrated in Figure 4, the Cross-Lingual Retrieval Language Knowledge Transfer (CL-ReLKT) process consists of a teacher, student, and loss function.
Initially, the student’s parameters are initialized to the same values as those of the teacher trained in Stage 1.
During the training process, the teacher’s parameters are fixed; we only adjust the student’s parameters according to the loss function.
Inputs.
Let us now consider input questions and answer documents of the training process.
As illustrated in Figure 4, both teacher and student models accept the same document input d .
However, there are two different versions for each question,
English q en and non-English q ne .
The English question q en is a translation of the original one q ne .
This gives us a question pair ( q ne , q en ) for language knowledge transfer between different languages.
For simplicity, we use Google Neural Machine Translation (GNMT) to translate q ne into q en .
Note that if available, one may also use human-translated parallel questions.
The teacher model T () accepts q en as input, while the student model S () accepts q ne as input.
In other words, q en functions as the “reference” of the LKT process.
According to our assessment (Appendix A.5.3), English provides the best performance and hence is chosen as the dominant language for the training process.
Note that this finding also conforms with the data distribution shown in Figure 3.
Loss Function.
The goal of our CL-ReLKT loss function L CL-ReLKT is to let the student mimic the teacher’s knowledge from the dominant language to the student’s target language.
As shown in Figure 5, our loss function L CL-ReLKT has three consistency objectives, namely, question-question, documentdocument, and document-question.
We describe them as follows.
• Obj 1: Question-Question.
The first objective is to enforce the consistency between S () and T () when encoding the same question expressed in English q en and non-English q ne , respectively.
• Obj 2: Document-Document.
While adjusting the student S () for the first objective, we also want to keep its answer document encoding unchanged.
Hence, we want to maintain the consistency between T ( d ) and S ( d ) .
• Obj 3: Document-Question.
To accommodate the lookup process, the embedding space should also keep question-answer pairs consistent with each other.
As our third objective, we minimize
the discrepancy between the student’s question vector S ( q ne ) and the teacher’s document vector
T ( d ) .
[Figure]
We formulate the loss function L CL-ReLKT as a linear combination of these three consistency objectives.
Using the squared L2 norm as the discrepancy measure, we obtain the following loss function:
[Equation]
where M is the set of training samples used in a given batch, and β , λ , and ω are the weighting coefficients.
Discussion.
As stated earlier, the goal of the loss function is to transfer the teacher’s knowledge to the student operating in target languages.
Since the performance of the teacher’s dominant language is generalized, after the LKT process, other languages will have the same properties.
The experimental results show that the student can better handle unsupported languages and improve the performance of supported languages than the teacher model.
This improvement comes from the cross-lingual consistency objectives, Obj 1 and Obj 3, in the loss function, while Obj 2 maintains the monolingual consistency.
Moreover, L CL-ReLKT does not require the teacher and student models to be of the same architecture; it can be applied to any pre-trained models.
(For more information, see Appendix A.4)
Experimental Setup 4. Datasets
To evaluate the effectiveness of our method, we conduct our experiments on three well-known CLReQA corpora: XORQA, XQuAD, and MLQA.
All experiments were done by XX → EN where XX is the question language (15 languages), and EN is answer passages or documents.
XORQA (Asai et al., 2021a) is a benchmark dataset for multilingual open-retrieval question answering.
The dataset contains questions in a diverse set of seven non-English languages and answer documents in English.
We use the Gold Paragraph part of the corpus, which contains 12,895 documents and 8,949 question-answer pairs.
The authors, however, did not provide a public test dataset.
Thus, we divide the samples into train/dev/test (0.7/0.1/0.2).
XQuAD (Artetxe et al., 2020) is a dataset for evaluating cross-lingual question answering performance.
XQuAD comprises 48 documents and 13,090 question-answer pairs obtained from the development set of SQuAD v1.1 (Rajpurkar et al., 2016) with 11 languages.
Since XQuAD is too small for model training, we used it for testing only.
Translated questions from SQuAD v1.1 (training set) were used instead for training the models (the same setup as XQuAD (Artetxe et al., 2020)).
MLQA (Lewis et al., 2020) is also a dataset for evaluating cross-lingual question answering performance.
The dataset contains 15,806 documents and 33,706 question-answer pairs in seven different languages.
However, the authors did not provide any training dataset.
As a result, we combined the development and test datasets and divided them into train/dev/test (0.7/0.1/0.2).
4. Competitive Methods
We compare the performance of our method with two groups of competitive methods as follows:
LM-based .
As discussed in Section 1, one approach to CL-ReQA is to use an embedding space based on some language model.
In the experimental studies, we compare our methods to the following LM-based competitors.
• XLM-R-nli-stsb : A RoBERTa-based crosslingual model trained using the NLI and STS benchmark datasets (Reimers and Gurevych, 2019).
• mBERT-triplet : A BERT-based multilingual model finetuned with a QA dataset using triplet loss (Reimers and Gurevych, 2019).
• XLM-R ← SBERT :
A XLM-RoBERTa model trained by distilling from the sentence BERT model (Reimers and Gurevych, 2020).
• DPR : A dense-network solution using the multilingual BERT model to provide the cross-lingual QA capability (Karpukhin et al., 2020).
• CORA : An adaptation of DPR on multilingual Wikipedia QA data (Asai et al., 2021b).
• LaBSE : A multilingual sentence encoder using mBERT to provide initial word embedding vectors (Feng et al., 2020).
We retrained mBERT-triplet, XLM-R ← SBERT and DPR with the CL-ReQA training set following previous work (Reimers and Gurevych
; Asai et al., 2021b; Zhang et al., 2021).
Multilingual Universal Sentence Encoding (mUSE) .
As an alternative to LM, we can also construct a QA embedding space from a well-known multilingual sentence encoder, mUSE.
In particular, we consider the following mUSE variants.
• mUSE small : The mUSE small encoder was based on Convolution Neural Network (Kim, 2014).
• mUSE large : The mUSE large encoder was on the transformer architecture (Vaswani et al., 2017).
Note that although there exists a QA variant of mUSE, mUSE qa , we found that this QA variant does not provide any performance improvement over mUSE large .
As a result, we omit mUSE qa from our study.
Our proposed methods .
As previously discussed, we construct our proposed methods based on mUSE small .
The first method, mUSE teacher , is constructed from triplet loss where each triplet consists of a question, its corresponding answer document, and a non-answer document.
The second method, mUSE cl-relkt , is constructed from the process of Cross-Lingual Retrieval Language Knowledge Transfer (with all languages in the training datasets).
4. Hyperparameter and Evaluation Settings
Hyperparameter .
In these experiments, we use grid search on the following hyperparameters: learning rates, triplet loss margin ( α ), L CL-ReLKT ’s coefficients ( γ , β , ω ), and the number of negative samples for triplet loss.
The hyper-parameter configurations are given in Table 1.
[Table]
For the Cross-Lingual Retrieval Language Knowledge Transfer settings, we use a batch size of 8 with a total number of 10 epochs.
Since the student model receives the initial weights from the teacher, the CL-ReLKT’s loss value ranges between [10 − 3 , 10 − 5 ] and the loss value of Obj 3 is lower than those of other objectives.
To prevent the CL-ReLKT’s loss value from being too small, we multiply the value by γ and set the coefficient of Obj 2, λ , to 1.
In addition, we evaluate the precision score on the development set everysteps.
If the precision score does not improve, the learning rate is halved.
Evaluation .
We use precision at k where we set k to 1 (P@1) which is a common practice for the
CL-ReQA task (Ahmad et al., 2019; Yang et al., 2020; Guo et al., 2021) and cross-lingual retrieval tasks (Reimers and Gurevych, 2020; Feng et al., 2020).
We also provide precision at 5 and 10 results in Appendix A.2.
Furthermore, we used McNemar’s test as the significant statistical measurement ( p < 0 . 05) for all experiments.
Experimental Results 5. Passage- vs Document-bases on Machine Reading QA (MR-QA)
To determine the best answer retrieval unit for MRQA, we compare two scenarios.
(i) Passage-based :
Retrieving answers as passages; (ii) documentbased : Retrieving answers as documents.
For conciseness, we chose DPR, which is the stateof-the-art LM-based competitor, for comparison.
For testing, we chose XORQA, which is the newest
MR-QA benchmark.
For all test cases, we used the same machine reading comprehension model constructed from XLM-R.
In particular, we finetuned
XLM-R using the same training portion of XORQA described in Section 4.
[Table]
Results.
Table 2 displays the MR-QA scores as F and exact match (EM) and provides a comparison between the two MR-QA input options: passagebased and document-based.
In all cases, the document-based option improves over the passagebased one.
We can also see that our method signif-
[Table]
icantly outperformed DPR for both input options on average.
Discussion.
The document-based representation has advantages and drawbacks in comparison to the passage-based one.
In particular, by grouping passages associated with the same document together, the retrieval unit becomes larger, making it harder to miss an answer.
However, operating at the document level also means that we have to handle a larger input.
That is, for the three corpora used in the experimental studies, the model has to handle 128 tokens on average when the input is a passage, while the input size can be up to 1, tokens when the input is a document.
The results provide empirical evidence that the benefits outweigh the drawback.
We believe that as machine reading models improve over time, longer input passages will provide even better results.
For more information about the retriever’s performance, see
Appendix A.1.
5. CL-ReQA: Supported Languages
Previously, we have shown that document-based yield the best MR-QA results.
Thus, we will use this setting for the rest of the embeddings experiments.
In this experiment, we report the effectiveness of our proposed methods on mUSE’s supported languages where the answer retrieval unit is document-based.
We evaluated our methods against the competitors discussed in Section 4.2.
Results.
As shown in Table 3, our proposed models mUSE teacher and mUSE cl-relkt provide significant improvements from the base model, mUSE small .
Moreover, our models also outperformed the largest pre-trained variant of mUSE, mUSE large .
All of our proposed models also significantly performed better than the LM-based competitors.
The results also show that our consistency enhancement method, CL-ReLKT, were effective in all cases except AR for the MLQA dataset.
Discussion.
Experimental results verify that for languages supported by mUSE, our approach based on the language knowledge transfer concept (Section 2.2) can provide significant improvements over the teacher model.
However, the improvements were less significant than ours when the language knowledge transfer concept is applied to an LM to create mBERT-triplet and XLM-R-nli-stsb from mBERT and XLM-R, respectively.
Notice that methods based on mBERT performed poorly in Thai (TH).
We can also see that finetuning mBERT with triplet loss (mBERTtriplet) and multilingual dense retrieval (CORA) did not provide any improvements on Thai.
This is because Thai was not included in the construction process of mBERT (uncased-version), and the amount of the training data is insufficient to improve the model.
5. CL-ReQA: Unsupported Languages
Let us consider how well our proposed models performed when used with languages not supported by the base model, mUSE small , i.e., FI, RO, EL,
HI, and VI.
Similar to the study presented in the previous subsection, we used XORQA, XQuAD, and MLQA as our test corpora.
Results.
Table 4 presents the P@1 scores of our methods and the competitors.
We can see that the original mUSE models, mUSE small and mUSE large , did not perform well in these languages.
As expected, mUSE teacher had a tendency to provide some improvements over mUSE small .
This is because these languages were not included in the original training process, and the amount of data is insufficient to improve the performance of these languages.
In contrast, we obtained significant improvements through the CL-ReLKT methods, mUSE cl-relkt .
For five out of seven cases, mUSE cl-relkt were the best performer compared to other models.
Two LM-based methods, XLMR ← SBERT and LaBSE, were the best performer in HI with the test corpora of XQuAD and MLQA, respectively.
As an alternative to cross-lingual retrieval
[Table]
one can convert the problem to a monolingual retrieval one using a machine translation (MT) model.
We found that using an MT model (i [Table] e., GNMT,
MBART) with DPR following Asai et al.
(2021a) helps improve the performance of LM-based models.
However, the performance decreases in some languages, i [Table] e., in Finish (XORQA), DPR’s performance is dropped from 39.1 to 32.0 and dropped to 36.4 when GNMT and MBart were applied to the DPR, respectively.
For the full discussion and results see Appendix A.3.
[Table]
Discussion.
The performance gap between mUSE small and mUSE cl-relkt demonstrates the effectiveness of the proposed CL-ReLKT framework.
In particular, we can use CL-ReLKT to generalize a base sentence embedding model to handle languages that were not originally included in the training process.
Regarding the CL-ReLKT performance on Hindi (HI), one important observation is that Hindi is the only language in this study whose family is not represented in the original training data, which results in a lot of OOV tokens from unknown characters.
We provide more explanation in Appendix A.6.
5. Run-time Efficiency on Query Encoding
Let us now consider the efficiency of the methods.
Since this investigation focuses on the embedding methods, we consider only the sentence embedding computational time.
We used a DGX-1 machine using one Intel Xeon E5-2698 and one NVIDIA
Tesla V100 GPU to benchmark the models.
Results.
As shown in Table 5, the experimental result shows that mUSE small is the fastest.
The result shows that mUSE small took only ∼ 7.9 ms on average to encode one query at a time.
Since our method is based on the mUSE small architecture, we also obtain a similar run time.
For the
LM-based methods
, we found that LaBSE is the quickest one.
However, the method is still slower than mUSE teacher and mUSE cl-relkt by at least 80%.
We can also find that MT-assisted, i.e., GNMT and
MBart, are significantly slower than our methods.
For the MT-assisted results, GNMT used 258. [Table] for one query while MBart used 9,132 ms
[Table]
Discussion.
Since mUSE small is a much smaller model than BERT-base and XLM-R-base, it is advantageous to use our proposed methods when efficiency is a concern, e.g., edge deployment.
While
GNMT and MBart were effective in improving the performance of LM-based models (Appendix A.3), the additional machine translation cost renders the approach less desirable.
5. Ablation Studies on the Training Objective for LKT
This study compares our Cross-Lingual Retrieval
Language Knowledge Transfer method with other LKT techniques using the same baseline model.
To directly assess the effect of the LKT method, we use the original mUSE small instead of the mUSE teacher as the starting model in this study.
We compare four training objectives:
• We apply the training objective following Reimers and Gurevych (2020)’s work denoted mUSE mse where the training objective contains the first CL-ReLKT’s objective with an additional loss term that minimizes the difference between English and English embeddings from the previous iteration; • As mentioned in Section 5.3 the CL-ReLKT loss has three objectives.
mUSE qcl-relkt uses only the first objective; • mUSE qdcl-relkt uses the first and the second objective; and • Lastly, mUSE cl-relkt uses the full version of the CL-ReLKT loss function.
Results.
The experiment results are given in Table 6.
As expected, our training objective outperformed competitive training objectives.
The performance of mUSE cl-relkt outperformed other training objectives from six out of seven cases.
Especially
Reimers and Gurevych (2020)’s training objective, our method outperformed with significant results on six out of seven cases except for HI → EN in
MLQA.
[Table]
in unsupported languages.
Conclusion
In this paper, we propose a novel Cross-Lingual Retrieval Language Knowledge Transfer framework for CL-ReQA.
Our framework is designed to improve the general performance and enable the baseline model to handle unsupported languages by exploiting the concepts of Dominant Language and Language Knowledge Transfer .
Our method outperformed competitive methods in all cases of supported languages and five out of seven cases of unsupported languages.
Furthermore, we demonstrated that grouping passages associated with the same document together could benefit machine reading QA.
Responsible NLP Research Checklist
Did you discuss the limitations of your work?
The limitation of our work is out-of-domain problems.
We strongly advise against using our model with out-of-domain data.
Did you discuss any potential risks of your work?
There is a risk of retrieving incorrect documents causing the machine reading comprehension part to produce wrong answers. Did you discuss the license or terms for use and/or distribution of any artifacts? The XORQA dataset is under the MIT License, while
XQuAD and MLQA are under CC-BY-SA 4.0.
Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified?
For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)?
XORQA, XQuAD, and MLQA were created for retrieval QA assessments; we use them for this exact purpose.
Did you discuss the steps taken to check whether the data that was collected/used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it?
From our careful inspection, there is no offensive content or sensitive data included in the three datasets: XORQA,
XQuAD, and MLQA.
Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc.
or just a single run?
We reported the mean average score for all experiments, which is a common practice for reporting the performance.
If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy,
ROUGE, etc.)?
All experiments were done by
Tensorflow because mUSE is only available on
Tensorflow.
Making
monolingual sentence embeddings multilingual using knowledge distillation.
In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP) .
Holger Schwenk and Matthijs Douze. 2017.
Learning joint multilingual sentence representations with neural machine translation.
In Proceedings of the 2nd Workshop on Representation Learning for NLP,
Rep4NLP@ACL 2017, Vancouver, Canada, August 3, 2017 .
Nattapol Trijakwanich, Peerat Limkonchotiwat, Wannaphong Phatthiyaphaibun, Raheem Sarwar, Ekapol
Chuangsuwanich, and Sarana Nutanong. 2021.
Robust fragment-based framework for cross-lingual sentence retrieval.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language
Processing (EMNLP) .
Andrew Trotman, Antti Puurula, and Blake Burgess.
Improvements to bm25 and language models examined.
In Proceedings of the 2014 Australasian
Document Computing Symposium , ADCS ’14.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz
Kaiser, and Illia Polosukhin. 2017.
Attention is all you need.
In Advances in Neural Information Processing Systems .
Liang Wang, Wei Zhao, and Jingming Liu.
Aligning cross-lingual sentence representations with dual momentum contrast.
In EMNLP .
Association for
Computational Linguistics.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020.
Balancing training for multilingual neural machine translation.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics .
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo,
Jax Law, Noah Constant, Gustavo Hernandez Abrego,
Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope
A Appendix A. CL-ReQA: Passage- vs Document-bases
This study reports the effect of the passage-based and document-based input on retrieval performance.
The experiment was conducted in supported and unsupported languages of three datasets:
XORQA, XQuAD, and MLQA.
Results.
As shown in Table 7, the document-based input substantially outperform the passage-based one in all languages and all datasets.
The performance of mUSE cl-relkt on passage-based for supported and unsupported languages is lower than document-based significant on every unsupported language.
These results conform with those of the downstream task presented in Table 2.
[Table]
Discussion.
This experiment shows that the efficiency of document-based is more robust than passage-based significant.
Moreover, when we applied both inputs to a downstream task, MR-QA, the experiment results showed that changing the input from passage-based to document-based is more robust.
A. CL-ReQA: Precision at 5,
In this experiment, we study the effectiveness of our method on mUSE’s supported languages. We report the precision score at 5 and 10 on the XORQA dataset.
Results.
As shown in Table 8, our proposed model, mUSE cl-relkt , on P@5 and P@10 have the same narrative as P@1 (Table 3). That is, the cross-lingual retrieval language knowledge transfer model improved the performance from the teacher model, mUSE teacher , and it outperformed every LM-based model. Discussion. Experimental results verify that precision at other k’s values does not change any conclusion from our work.
The results show that our model outperformed other models in precision at 1, 5, and 10 settings.
[Table]
Our proposed methods [Table] A. CL-ReQA: Monolingual vs Cross-lingual Retrievals In this study
[Table]
we compare the CL-ReQA approach against the MT-assisted monolingual retrieval one.
For CL-ReQA methods, we chose DPR and mUSE cl-relkt .
For the MT-assisted methods, we used two translators, GNMT and MBart (Liu et al., 2020), to translate all questions into English which is the documents’ language.
These two translators were then applied to assist DPR in the same manner as the XORQA investigation (Asai et al., 2021a).
Results.
Table 9 shows that the two translators provide substantial improvements to DPR and mUSEbased.
The GNMT-assisted tended to perform better than the MBart-assisted in almost all cases.
As state in Section 3.2, our language knowledge transfer comprise of GNMT in the training data process.
Thus, the performance of our method with/without MT-assisted is similar.
Our method was the best performer in six out of nine cases.
Moreover, in seven out of nine cases, the performance of our method is decreased when MBart is applied.
Since the performance of MBart is lower than GNMT, the performance of mUSE cl-relkt +MBart is dropped significantly.
Discussion.
Machine translators can provide a quick solution to improve the performance of crosslingual retrieval.
In this way, the problem of crosslingual retrieval is converted into a monolingual one.
This approach can be useful when the language pair has a reliable translator, but there is insufficient QA data to create a cross-lingual retrieval
[Table]
model.
However, we consider the multilingual sentence embedding approach to be superior to the
MT-assisted one due to the following reasons: (i) the computational cost benefits of skipping the MT process; (ii) the MT-assisted approach can be negatively affected by a poor MT performance; (iii) the reliance of MT models can be a limitation for some language pairs.
See Appendix A.7 for a further analysis.
A. CL-ReLKT on Other Architectures
This study demonstrates our cross-lingual retrieval language knowledge transfer method on LM-based models.
For diversity, we select BERT-based (mBERT-triplet) and RoBERTa-based (XLM-R-nlistsb) because these models trained only on the dominant language, English, same as mUSE teacher .
Results.
As shown in Table 10, we applied the
CL-ReLKT framework on other architectures, i.e.,
BERT and RoBERTa, on supported languages of XORQA.
The experimental results show that
CL-ReLKT on LM-based models significantly improves XLM-R-nli-stsb.
Furthermore, when we use mBERT-triplet as the teacher instead of XLMR-nli-stsb.
The result shows a small improvement over the student instead of using XLM-R as the teacher model.
Discussion.
Our CL-ReLKT can be applied to any pre-trained model not limited to only in mUSE’s architecture.
However, to give the best results, let the student initialize the weight from the teacher.
In addition, the results from both models are not over mUSE’s performance.
[Table]
A. Ablation Studies
This study presents the effect of each design decision in the triplet loss and cross-lingual retrieval language knowledge transfer proposed.
Here, we investigate the following components: (i) training strategies; (ii) training data settings; (iii) reference languages for cross-lingual retrieval language knowledge transfer (CL-ReLKT); and (iv) distance functions for CL-ReLKT.
In each investigation, we use the best setting from the previous steps.
All experimental results were obtained from XX → EN retrieval on the XORQA test set across four languages, where XX can be one of these languages
Russian (RU), Korean (KO), Japanese (JA), and Finnish (FI).
A.5. Training strategies
As shown in Table 11, we compare training strategies with/without each of the following components: initialization, online updates, and other deep metric learning techniques.
As expected, the result shows that online negative sampling and initializing with BM25 helps improve the performance of triplet loss.
Furthermore, we also study the effect of replacing triplet loss with contrastive learning as presented in the current state-of-the-art work (Karpukhin et al., 2020).
We found that contrastive loss consistently provides a performance improvement over the original mUSE small model but still lags behind the triplet loss.
A.5. Training data settings
In terms of training data for the teacher training, there are two decisions we need to consider: (i) the answer representation unit: whether to use one passage or one document as the retrieval unit in the training process; (ii) the question language: whether to use the original questions (in multiple languages) or translate them all to English.
In the case of English, all English questions were trans- lated using GNMT.
As shown in Table 11, the passage and English combination provides the best performance.
A.5. Teacher’s language for CL-ReLKT
In this study, we explore the choice of language to function as the reference (teacher) in the CLReLKT process.
Intuitively, we want a language that is well-represented in the training corpora when constructing the original model.
Consequently, we compare English, Spanish, and German.
As expected, English, which is the dominant language in the training corpora of mUSE, provides the best performance.
These results also conform with the discussion on the dominant language provided in Section 2.1.
A.5. Distance functions for CL-ReLKT
The distance function is critical to the CL-ReLKT performance.
While there exists many distance functions we can apply to the LKT process, we consider two of the most widely used ones, cosine and squared L2.
As we can see, squared L provides the best performance.
A.5. Discussion
From the results, we conclude that the default settings of our proposed methods are (i) triplet loss as the training objective (ii) English question + passages as the answer representation unit (iii) English as the teacher language, and (iv) square L2 distance as the distance function for CL-ReLKT.
A. Language Analysis
In this section, we discuss Hindi, which our method performs worse than other competitive methods, and examines other languages that we have not shown in the tables, such as Telugu and Bengali.
As shown in Table 4, the mUSE cl-relkt ’s performance is the best in every language except Hindi.
This is due to the mUSE encoder’s tokenizer (sentencepiece), which cannot handle Hindi well compared to other unsupported languages (i.e., FI, RO,
EL, VI).
For instance, we measured the tokenizer’s out-of-vocabulary (OOV) rate of the mUSE’s tokenizer on Hindi (XQuAD) and found that the OOV rate of Hindi is ∼ 14.5%.
On the other hand, the
OOV rate on Greek on the same dataset is only 2.2% which is ∼ 12.3% lower than Hindi.
Since the language families of the languages used in mUSE are not Indo-Aryan (Hindi and Bengali) nor Dravid-
[Table]
ian (Telugu) 4 , the mUSE’s sentencepiece cannot handle Indo-Aryan and Dravidian language families well.
To make the mUSE encoder handle these language families better, we might need to retrain the mUSE sentencepiece tokenizer by using other tokenizers (i.e., sentencepiece in mBERT).
Since mBERT’s tokenizer is trained on more than 100 languages, mBERT and XLM-R perform better than the mUSE cl-relkt on Indo-Aryan languages.
Another solution would be to use the universal tokenizer (Gillick et al., 2016) which represents the input as bytes instead of characters.
A. Error Analysis: MT-assisted vs Ours
We provide some sample questions that our model and the MT-assisted model (DPR+GNMT) answer differently in Table 12.
We notice that our method does better than MT-assisted when the questions are very specific, such as the question relating to
Brothers Grimm.
In general, we find that our model generally performs better when the question contains names.
This is because machine translation can sometimes fail to translate names properly.
On the other hand, when the questions are not specific, it is a toss-up whether the prediction is correct for both models.
Rows 2 and 3 show examples of such vague questions.
Lastly, we found that our model performed particularly worse than MT-
[Table]
✓
[Table]
✓
The model in our method has to map questions from multiple languages and numbers close together.
This makes learning numerical concepts such as in example number four challenging.
We believe this is a good avenue for further research.