Ask Me Anything in Your Native Language
Abstract
Cross-lingual question answering is a thriving field in the modern world, helping people to search information on the web more efficiently.
One of the important scenarios is to give an answer even there is no answer in the language a person asks a question with.
We present a novel approach based on single encoder for query and passage for retrieval from multi-lingual collection, together with cross-lingual generative reader.
It achieves a new state of the art in both retrieval and end-to-end tasks on the
XOR TyDi dataset outperforming the previous results up to 10% on several languages.
We find that our approach can be generalized to more than 20 languages in zero-shot approach and outperform all previous models by 12%.
Introduction
Question answering (QA) is an important tool for information search on the Internet.
Since it is natural for a person to ask a question to get some information, the QA systems are designed to meet this requirement.
QA provides a wide range of tasks, from engineering to cornerstone scientific tasks.
Open-domain QA, in this direction, is an interesting example of a problem that connects both: multilingual knowledge sources form differing knowledge and supplement gaps in each specific language.
The requirement for modern language models to be cross-lingual is gradually becoming more and more important and is being incorporated into popular benchmarks such as XGLUE (Ruder et al., 2021) or XTREME (Liang et al., 2020), where the systems are evaluated not only by their performance metrics on single language tasks but the ones on many languages.
[Table]
lingualsearch index
Q : Milloin ja miten tumbleweed pääsi Amerikkaan?
in ﬁnnish When and how did tumbleweeds get to America?
A :
Se saapui pellavansiementen kanssa Etelä-Venäjältä 1870 It arrived from in the with ﬂaxseed 1870s southern Russia
Sentri encoder Солянка сорная [...] перекати-поле.
Распространен широко на юге России .
[...] в Северной Америке как интродуцент широко распространился.
Образует шаровидные кусты, которые осенью отрываются от Kali tragus
It ’s known simply as tumbleweed [...] in the 1870s , it appeared in South Dakota when ﬂaxseed from Russia turned out to be contaminated with Kali seeds [...]
==
In short, at the first step system retrieves information from factoid knowledge sources ( Wikipedia’s ) on diverse languages.
In the second step it fuses the retrieved information regardless of the language of each part, even in the absence of texts on query language.
Finally , it produces an answer on the query language that aggregates all (as it can) diverse pieces from other languages.
For this particular example, there is no answer 1 in
Finnish Wikipedia.
One of the reasons is that there aren’t many articles in Finnish Wikipedia because
Finnish is a low-resource language in general.
Moreover, the answer can be found in rich-resources languages such as English or Russian for example.
Generally, passage retriever is based on the so called dual-encoder, i.e. two independent modules of the same architecture for the question and context encoding.
However, in previous works (Qu et al., 2021; Gao and Callan, 2021) authors found out that on the one hand dual-encoder is not noiseresistant and on the other hand large batch training can improve stability of resulting embedding space.
For instance, in (Qu et al., 2021; Gao and Callan, 2021) authors used 512 × 8 and 512 × 4 respectively.
Ouguz et al.
(2021) present DPR-PAQ model combining large semi-supervised corpus for pre-training with the better and larger LM
RoBERT a large , DPR-PAQ achieves the state of the art results on Natural Questions dataset.
Nevertheless, large models and large batch training require abundance of GPU memory.
The mentioned models are represent dual-encoder scheme, where two independent encoders are used – one for the questions and one for the passages.
To reduce memory usage, we present a system including a single encoder used for both tasks, moreover, we show that the system using single encoder can learn an embedding space better suited for transfer learning and thus improve result in cross-lingual question answering in zero-shot scenario.
The system, we call Sentri , achieves a new state of the art on XOR TyDi QA cross-lingual dataset (Asai et al., 2021a) outperforming the previous approaches by 10% on retrieval task and 7% on end-to-end question answering task.
In addition, on MKQA multi-lingual dataset (Longpre et al., 2020), which contains translations of Natural Questions to different languages, in zero-shot scenario our system outperforms a strong baseline by 8%.
The overall contribution of this paper is two-fold:
(i) we present a system, including single encoder for questions and contexts, that achieves state-ofthe-art results in the retrieval and end-
to-end tasks of the XOR TyDi dataset, (ii) we provide an analysis of the system behaviour in zero-shot scenario on unseen languages proving the its transferability and lower resource consumption.
The rest of the paper is organized as follows.
Section 6 presents an overview of the recent studies on multilingual QA models.
Section 2 which details the dataset design choices, outlines the data preparation pipeline and data used for evaluation.
[Figure]
Section 3 presents the engineering choices and describes the resulting model and its training process.
We describe experimental setup and describe the achieved results in Section 4.
We provide additional results analysis in Section 5, and Sectionconcludes the paper.
Datasets
In this work, we use XOR TyDi and MKQA to evaluate our system onto and several datasets to (pre-)train it.
XOR TyDi (Asai et al., 2021a) is a multilingual open-retrieval QA dataset that enables crosslingual answer retrieval.
The dataset, based on questions from TyDi QA (Clark et al., 2020), articulates three new tasks that involve finding documents in different languages using multilingual and English resources.
It consists of questions written by information-seeking native speakers intypologically diverse languages: Arabian, Bengali,
Finnish, Japanese, Korean, Russian and Telugu.
Answer annotations are retrieved from multilingual document collections.
XOR-Retrieve is a crosslingual retrieval task where a question is written in the target language (e.g., Japanese), and a system is required to retrieve an English document that answers the question.
XOR-English Span is a cross-lingual retrieval task where a question is written in the target language (e.g., Japanese), and a system is required to output a short answer in
English.
XOR-Full is a cross-lingual retrieval task where a question is written in the target language (e.g., Japanese), and a system is required to output a short answer in the target language.
In our work, we concentrate on XOR-Retrieve and XOR-Full tasks and use XOR TyDi to train and evaluate our system.
2. Training Dataset Pre-processing
Since there are low-resource languages in the XOR
TyDi dataset, the task of preparing data is of priority importance.
To generate data in different languages, we used NQ and Trivia QA datasets described above.
Both NQ and Trivia QA are English language datasets.
To use them for training purposes in our setup we had to translate them to the languages of XOR TyDi.
The quality of machine translation still does not match the human translation quality in most of the languages and domains.
However, questions and answers in NQ and Trivia
QA datasets are short and easy to translate.
For each pair of question q and answer a from the datasets, we made translations from English to each language L
i using the M2M100 model described in (Fan et al., 2020), it is a state-of-the-art model for translation for many languages, including the ones we are interested in.
Thus we translated question-answer pairs and we got an aligned dataset.
The pairs are not enough, since the opendomain QA task is based on so-called support passages retrieved from some document collection.
To overcome this issue we mined the positive and hard negative sample passages for each language
L i .
We consider a paragraph from a document to be a positive sample if it is ranked high by a retriever model and includes the answer.
We use complicated morphology-aware answer detection technique which we describe in Appendix.
If a paragraph is highly ranked but contains no answer, we consider it as a hard negative sample.
The resulting statistics and analysis of the training dataset we also present in Appendix.
Information Retrieval
We took into account that most XOR TyDi languages have complex morphology and other linguistic features, which makes information retrieval less effective for the models using token comparison.
Thus we decided to normalize the morphology for at least the languages, which has publicly available stemmers, namely,
Arabic 1 , Bengali 2 , Korean 3 , and Russian 1 .
The
Telugu language has no publicly available stemmer, but there is a lemmatizer 4 , which we used.
For Korean we apply token splitting by the partof-speech tag, i.e. modifier POS as a Josa and Eomi are treated as separate tokens.
Unfortunately, we have not found accessible stemmers and/or lemmatizers for Japanese and Finnish languages.
We use this normalisation to improve positive passage mining for self-training procedure.
More details on normalisation could be found in Appendix.
Natural Questions (NQ) (Kwiatkowski et al., 2019) dataset is designed for end-to-end question answering.
The questions are mined from real
Google search queries and the answers are spans in
Wikipedia articles identified by annotators.
We use this dataset in two ways.
One way is for training and another one is for zero-shot evaluation.
The latter option is provided to us by MKQA dataset (Longpre et al., 2020).
It is a translation of 10 thousand question-answer pairs from NQ to 26 different languages, thus giving us an aligned dataset of 260 thousand question-answer pairs total.
The former option is described below.
We also use Trivia QA for pre-training of our model.
Joshi et al.
(2017) presented Trivia QA, a large-scale question-answering dataset that includes so-called evidence documents, allowing one to state a task of information retrieval.
Trivia QA includes 95 thousand question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average ending with 650 thousand total triples.
Method
The open domain question answering task heavily relies on retrieval from some (possibly more than one) document collections.
In the case of the crosslingual variant of this task, the usage of several (at least two - in English and in a target language) document collections is almost inevitable.
We evaluate our model in two cross-lingual setups: using English Wikipedia ( W eng ) to search for a relevant passage containing the answer to the question or using collection of multilingual reference passages from
Arabic, Russian, English, Finnish, Telugu, Bengali,
Japanese, Korean Wikipedia ( W multi ).
More formally, given a question q in language L i , a system retrieves the documents from W eng or W multi , and formulates an answer a .
Thus the system could be virtually split to retriever , which creates a list of relevant documents, and reader , which generates an answer using the most relevant documents.
The sample of the system output is presented on Fig.
3. Single Encoder Retriever
We follow common (Qu et al., 2021; Asai et al., 2021b; Ouguz et al., 2021) dual-encoder approach in data representation.
The system consists of question encoder E q ( · ) and passage encoder E p ( · ) which maps text to d -dimensional real-valued vectors.
Before run-time, E p ( · ) applied to all passages in knowledge source to create search index.
To find out relevant passages to certain question system operates a similarity function:
[Equation]
i.e. similarity between the question and the passage defined by the dot product of their vectors.
In this work we investigate case when
E q ( · ) =
E p ( · ) and call this approach as Single Encoder .
In addition to it, a model with E q ( · ) ̸ = E p ( · ) we call Bi-Encoder to avoid confusion.
The architecture that utilizes S ingle en coder approach for re tri eval ( Sentri ) shares one encoder for E q ( · ) and E p ( · ) contrary to bi-encoder which based on two separate models.
Since our model is used in a multi-lingual setting, the choice of multilingual models is natural for base model.
We use XLM-RoBERTa (Conneau et al., 2020) (large) in our experiments.
Training
Sentri is trained to give positive passages higher scores than negative passages.
More specifically, given a question q i in a language from
L together with its positive passage p + i and m negative passages { p − i,j } mj =1 sampled from W multi , we minimize the loss function:
L ( q i , p + i , { p − i,j } mj =1 ) =
[Equation]
where we aim to optimize the negative loglikelihood of the positive passage against a set of m negative passages.
For each question, we treat other passages in the training batch that do not answer this particular question as negative passages ( in-batch negative trick , Henderson et al. 2017; Karpukhin et al. 2020)
In particular, for batch size n each question can be further paired with m = n − 1 + n negatives (i.e., positive and hard negative passages of the rest questions) without sampling additional negatives.
Furthermore, in the case of multilingual data, it helps enforce the cross-lingual ability of the model because of an increasing number of cross-language pairs.
3. In-batch False Negatives Filtering
Although the above strategy can increase the number of negatives, some of them may turn out to be false negatives.
We analyze the batches generated for the training and found out that different questions in the same batch could have the same positive passages.
Since these positive passages are used for in-batch negative training that produces false negative pairs.
For English Wikipedia this overlap is significant but not so crucial like for lower resource Wikipedias.
For instance, for Natural Questions passages in 44% training triplets (questions, answer, passage) are used more than once in the dataset.
The sample of positive passage overlap for NQ is presented on Fig.
We use in-batch filtering allowing us to eliminate this overlap from generated batches and thus improve the overall system quality.
3. Self-Training
Several works (Qu et al., 2021; Izacard and Grave, 2020a) refer to iterative learning as a source of model quality improvement.
We use this idea in the form described below, which we call self-training .
Fig.
2 presents the framework which we use in this work.
Sentri j model retrieves top- k passages from Wikipedia for each question from the initial
QA training set.
Then we select the positive ( p + ) passages for each question (we know the ones for the initial training set) and treat the rest as hard [Table] negative ( p −− ) examples.
Afterwards, we train new iteration Sentri j +1 model using the passages marked up previously.
In contrast to (Asai et al., 2021b) we train iteratively only a retrieval part of the whole system.
At stage 0 when there is no trained model available, we use well-known BM25 model (Sanderson, 2010) as a retriever in our experiments.
The important feature of this model is that it does not need any kind of training, thus it could be used to retrieve documents from collections in languages with little or no training data.
For Sentri system we report the results for the second stage of self-training below.
For BiEncoder we report results for stages 1 and 2 adding the specifying index.
3. Answer Generation
We have experimented with both extractive and abstractive answer generation and found out that abstractive is more profitable.
Here we describe the abstractive reader approach we use as primary one.
We decided to use the FiD model (Izacard and
Grave, 2020a) as a reader model in Sentri for end to end question answering task XOR-Full since it allows us to exclude the translator from a pipeline and to aggregate information in a cross-lingual setup.
Since the original FiD model is monolingual, we present extension of this work, multilingual version which we call MFiD .
To train MFiD, we use several QA datasets, listed in Sec. 2, namely XORFull, XOR TyDi, Natural Questions, and Trivia
QA datasets, the same ones used for training the retriever part of the Sentri model.
For each question from the QA datasets we retrieve top-50 passages from multi-language knowledge source using our retriever model.
And then use it for training
MFiD as cross-lingual fusion reader.
We also experimented with standard extractive reader.
The details on extractive approach could be found in
Sec. 5.
Experiments
We have conducted a series of experiments with number of models, namely these are Sentri model combined with different reader parts and BiEncoder model with one or two stages of selftraining.
Bi-Encoder model is using standard extractive reader (plus machine translation where applicable).
The main difference between Sentri and
Bi-Encoder, that the latter is based on classic dualencoder architecture, while the former is using single encoder for questions and paragraphs.
4. Results on XOR TyDi
Tables 1 and 2 contain results for our system in retrieval and end-to-end setups, XOR-Retrieve and
XOR-Full
respectively.
These two tables contain the results of the models’ evaluation on the development and test parts of the XOR TyDi dataset.
It is important to mention that we use name Sentri for our model in both tasks, while in XOR-Retrieve task the reader part is not used, since it is essentially a passage ranking evaluation.
Also, you can find results for our models titled as Bi-Encoder 1 and
Bi-Encoder 2 (for first and stages of self-training respectively).
As one can see Sentri significantly outperforms these baseline models and existing state-of-the-art models.
We provide more detailed analysis in section Ablation Study.
Tab. 1 displays recall scores for 2000 andfirst tokens ( R@2kt and R@5kt respectively).
That means that we expect to find an answer span in the first l tokens.
This metric was proposed
[Table] [Table]
DPR + BM
[Table]
MT
[Table]
CORA (Asai et al., 2021b)
[Table]
reader
[Table] [Table]
Da
De
Es
Fr
He
Hu
It
Km
Ms
Nl
No
CORA (Asai et al., 2021b)
[Table]
BM
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
Sentri 57.
[Table]
[Table] [Table] [Table]
[Table]
Pl
[Table]
Th
[Table]
CORA (Asai et al., 2021b)
[Table]
[Table]
[Table]
BM25 + MT ∗ 46.
[Table]
Bi-Encoder
[Table]
Sentri
[Table]
[Table]
in (Asai et al., 2021a) as alternative to more common Recall @ N in purpose to make more fair comparison across various models with different passage size used.
Our system outperforms the previous state-ofthe-art system in both R@2kt and R@5kt metrics by a wide margin on four languages, namely Arabic, Japanese, Russian, and Telugu.
More importantly, our system outperforms the previous system on average for all the languages.
Interestingly, selftraining improves the results in all the languages, with the intriguing exception of the Russian language.
This fact requires an additional investigation, we leave it as future work for now.
Table 2 displays F1, Exact Match (EM), and
BLEU scores for the end-to-end setup where given a question in target language L i and Wikipedia in both English and L i , a system is required to generate an answer in the target language.
F1 measure is computed per token for an answer span.
Exact Match compares the golden answer span with the system output for exact equality.
BLEU metric, defined as in (Papineni et al., 2002), computes the number of overlapping n -gram between the golden answer and the system output.
In this experiment, we see a somewhat different behaviour of the model.
Our model outperform previous state of the art system for all languages, with exception for Telugu where CORA model (Asai et al., 2021b) shows insignificantly higher score.
4. Zero-Shot Cross-Lingual Transfer
We investigated the transferability across the languages for the trained system.
We used the MKQA dataset in similar to the XOR-Retrieve setup, i.e. we retrieved the passages from English Wikipedia, extracted the answer from the top-ranked passage, and translated it with a machine translation model.
Here we again use M2M100 model for machine translation task.
As a baseline for this task, we utilized BM25 with extractive reader and the machine translation model at the end of pipeline.
We selected from MKQA such unseen languages that were not presented to the system during the training process.
The achieved results presented in Tab.show that even in such a zero-shot setting our sys- tem significantly outperforms both the strong baseline and previous approaches in all languages.
Additional details on zero-shot transfer could be found in Appendix.
Ablation Study
Sentri model has five important features which differentiate it from the previous work: self-training, a single encoder model for passage and question processing as a retriever, a generative model as a reader, in-batch negative filtering, usage of the machine translated data during the training process.
(I)
The effect of the first of mentioned features could be analysed basing on Tab. {+
+}
(upper part of the table, showing results on development set) and 2.
Self-training for Bi-Encoder model improves results by 3% on average for the XOR-Full task and about 5% for the XOR-Retrieve task (BiEncoder 1 vs Bi-Encoder 2 ).
(II)
The usage of a single encoder could be
estimated
as again for the XOR-Full task and
about 6% for the
XOR-Retrieve task Sentri + ext.
reader vs. Bi-Encoder 2 ).
In Tab. 4 we demonstrate key motivation of using shared encoder model.
We observe that the single encoder approach superior to the Bi-Encoder in terms of memory efficiency and overall performance.
With the same size (less than 2% difference) it achieves more than 12% relative improvement or ∼ 6-7 difference in absolute points in retrieval task.
Note that Sentri and BiEncoder 2 trained in the same setting and same QA datasets except used base model variants ( base and large respectively).
We do it for the sake of the matching number of parameters, matched memory consumption and similar training time of both [Table] [Figure] [Table]
[Figure] [Table]
of single and biencoder models. The single encoder approach significantly outperform dual-encoder one using almost the same memory amount. Most
[Table] effective [Table] Sentri + ext.
reader + MT
[Table]
Sentri + CORA Reader
[Table]
Sentri + MFiD
[Table]
models.
(III)
The replacement of standard extractive reader, i.e. span-tagging model, with a generative one, MFiD model in our case, turned out to add up to 34% of F1 measure (for Japanese) and 25% on average.
We observe that with the number of contexts more than 5 performance of extractive reader degrades.
Contrary that using more contexts for answer generation can significantly improve model quality.
We further evaluate generative reader usage by adding the one described in (Asai et al., 2021b), it uses only 15 top-ranked contexts due to memory constraints.
Unlike that, MFiD can use up to 100 top-ranked contexts thanks to the independent processing of passages in the reader’s encoder.
The results are presented in Tab. 5.
(IV & V)
The importance of the last two features could be estimated basing on Tab. 6.
While the former one adds up to 2 per cent, the latter is of crucial importance adding up to 20% depending on task and measure.
We could conclude that all the features are important for our approach to present state-of-the-art results in retrieval and end-to-end tasks. 5. Languages Used It is also interesting to know if the system is actually using for answer generation.
In other words, if our model is truly cross-lingual.
The analysis of the top-100 paragraphs for all the questions in validation set of XOR TyDi, namely the breakdown on the languages used is shown on Tab. 7. As one can see, there are several interesting features could be spotted in the table.
The highest self-usage (i.e. when a question and a paragraph are in the same language) percentage is shown by Korean language (almost entirely, 97.6%), with negligible usage of other languages, except English with 1.4%. On the contrary, Finnish language is shown the lowest self-usage of 52.9% with 34.6% usage of English. These two facts could be explained as Korean having unique writing system, thus it has almost non-existent intersection with other languages in terms. In the contrary Finnish is Latin-based and is in close contact with Swedish language since the Middle Ages.
While Swedish is close German language to English.
But this speculation has its downside: the second most-used language for
Finnish is Japanese (7.4%), which is both unrelated and uses other script.
Interestingly, Japanese language is the second language by usage for almost
R@2kt
R@5kt
MKQA
XOR
MKQA
XOR
Sentri 53.
Sentri w/o false negative filtering 52.
Sentri w/o weight-sharing (Bi-Encoder approach) 45.
Sentri w/o multi-language translations of training set 41.
all the languages, including Korean
but with only 0.7%.
For Telugu the second most used language is Bengali, which related to it.
But surprisingly, for
Bengali Telugu is almost used.
Another peculiar feature is that Russian language is significant in usage for almost all the languages with exception for Korean.
We hypothesise that it is due to the proportion of Russian data in the training set, this language being the second by size in the dataset.
It is important to mention that the largest present language is Arabic, but its influence is lower than
Russian.
The influence of Arabic, Russian, and
Japanese need more in-depth future investigation.
Related Work
Datasets
The cross-lingual question answering datasets were scarce before recent years.
Fortunately, these years left us with several publicly available datasets.
Lewis et al.
(2020) introduced
MLQA dataset.
It consists of parallel QA pairs in several languages.
Liu et al.
(2019) have presented
XQA dataset, with training set in English and validation and test sets in the other languages.
Crosslingual Question Answering Dataset (XQuAD) benchmark presented in Artetxe et al.
It consists of a subset of 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 (Rajpurkar et al., 2016) together with their translations into ten languages.
Systems
Open-domain question answering task assumes answering factoid questions without a predefined domain (Kwiatkowski et al., 2019).
Recent research was focused on creating non-English question answering datasets and applying crosslingual transfer learning techniques, from English to other languages.
Until recently, the availability of appropriate train and test datasets has been a key factor in the development of the field: however, in recent years, many works have focused on the collection of loosely aligned data obtained through automatic translation or by parsing similar multilingual sources.
Artetxe et al.
(2020) studied cross-lingual transferability of monolingual representations of a transformer-based masked language model.
In most previous approaches the authors use extractive models to generate the actual answer.
This could be explained by the mental inertia from
SQuAD-like datasets.
By SQuAD-like we mean a dataset where labelled data includes an explicitly stated question, a passage, containing an answer, and a span markup for the answer.
Such markup was presented for the question answering task called SQuAD in (Rajpurkar et al., 2016).
But recently there were presented cross-lingual generation of answers from raw texts.
Kumar et al. (2019); Chi et al.
(2019) studied cross-lingual question generation.
Shakeri et al.
(2020) proposed a method to generate multilingual question and answer pairs by a generative model (namely, a fine-tuned multilingual T5 model), it is based on automatically translated samples from English to the target domain.
Generative question answering was mostly considered in previous work for long answers datasets.
However, FiD model (Izacard and Grave, 2020b) archives competitive results on
SQuAD-like datasets, where an answer is supposed to be short text span.
For open domain question answering, one of the first approaches named RAG used generative models was presented in (Lewis et al., 2021).
A key idea of this RAG model is to process several (top k) passages from the retriever in the encoder simultaneously.
The produced dense representations of the passages are used in the decoder for the answer generation, this process is called fusion.
Processing the passages independently in the encoder allows a model to scale to many contexts, as it only runs self-attention over one context at a time.
FiD model follows this paradigm further improving the results in question generation. [Table]
Conclusion
Nowadays multi-lingual and cross-lingual problems are coming to the stage once the natural language models become more and more powerful.
One of these problems is that where the systems answer the questions using various mutually disjoint language data, as it stated in XOR TyDi task.
This task is based on a specific XOR TyDi dataset (Asai et al., 2021a) [Table] which ensured such information asymmetry in the different language data.
We introduced the cross-lingual system to solve the XOR task.
While the XOR TyDi is a challenging test that stimulates cross-linguality in NLP systems [Table] we have outperformed the existing models in two subtasks: XOR-Retrieve and XOR-Full without using external APIs.
The first task is a classical passage retrieval task, while the second one is an end-to-end question-answering task.
Besides showing the state of the art results on these two subtasks, our system is demonstrated the ability the transfer to the unseen languages in retrieval task, including the languages which were not presented in the pre-trained language model we use as an encoder for the retriever part of our Sentri model.
And last we found that the previous works ignored the existence of the morphology in XOR TyDi presented languages, thus missing many results in information retrieval.
We propose to solve this issue by using stemming or lemmatization for such languages.
Our system has five differentiating features, which are self-training (using the output of the previously trained models), single encoder (allowing us to reduce the number of parameters about twice in retriever), usage of a generative model to get the question from retrieved passages, in-batch negative filtering, and usage of the machine translated data during the training process.
All of these features are proved to make a share in the achieved significant quality improvement demonstrated by our model.
Although, our system has several flaws, e.g. passage selection strategy and stemming for the languages, we consider these flaws as our future work.
But we hope that current study will foster research in cross-lingual question answering tasks.
A Implementation Details
For our system, we have adapted models from the Huggingface Transformers library (Wolf et al., 2019).
We trained the question and passage encoders using the in-batch negative sampling with a batch size of 16, one hard negative per question.
We trained the system for 40 epochs with a learning rate of 10 − 5 using Adam, linear scheduling with warm-up and dropout rate of 0 . 1 .
For training and validation, we used NQ, Trivia, XOR and TyDi QA datasets.
The number of hard negative passages was 32 and 50 for the first and second stages respectively.
All experiments were carried out on four nVIDIA V100 GPUs (with 32Gb RAM each).
B Data Distribution
The overall statistics on mined data from the mentioned datasets are available in Tab. 8. As one can see, the data acquired is following the rough pattern: most of the samples come from TriviaQA, the second source is NQ, while the rest comes from XOR TyDi QA.
In the perspective of languages the pattern follows the XOR TyDi distribution, with
Arabic being the largest non-English language, and
Korean being the smallest one.
NQ
TriviaQA
XOR QA
Total
Ar
Bn
Fi
Ja
[Table]
C Effects of Normalisation
Since we are using normalisation for the retrieval [Table] we decided to look into the evaluation process.
The metrics used for the evaluation, namely pertoken F1, Exact Match, and BLEU, are based on simple token comparison.
Such comparison is inefficient for the languages with rich morphology, like Russian or Japanese.
So we applied the same normalisation as in retrieval for the generated and gold answers.
Table 9 shows the achieved results.
As one can see normalisation helps to achieve less strict and thus more informative comparison for the morphology-rich languages.
Since
Japanese and Finnish are both synthetic agglutinative languages, we suppose that the results on them could also be improved with usage of the stemming/lemmatization, thus improving the crosslingual average further.
Given that we think the usage of some kind of normalisation should be recommended for any cross-lingual QA task including morphology-rich languages.
[Table]
D Zero-Shot Cross-Lingual Transfer
In addition to the MKQA results on which were presented in main contents [Table] we used M2M100 model to translate the MKQA English subset to all known to it languages, thus extending MKQA to 98 languages.
We provide the results on the this extended dataset in Tab. 10.
E Unseen Languages
We thoroughly analysed the (partially) unseen languages and found out that our system performs reasonably well even for those languages, which are not present in the pre-trained XLM Roberta used as an encoder in our model.
The results are presented in Tab. 11.
We also provide aggregated results for the unseen on training stage languages and seen in the training stage ones.
As one could see, our model performs surprisingly well on the languages which were not presented on any training stage, although the training improves Recall by 12 andper cent on average, while pre-training adds 13.8% of Recall.
We think this is another evidence of the great generalizability of the pre-trained language models.
[Table]
[Table]