Ask Me Anything in Your Native Language (2022)

Abstract: Cross-lingual question answering is a thriving field in the modern world, helping people to search information on the web more efficiently. One of the important scenarios is to give an answer even there is no answer in the language a person asks a question with. We present a novel approach based on single encoder for query and passage for retrieval from multi-lingual collection, together with cross-lingual generative reader. It achieves a new state of the art in both retrieval and end-to-end tasks on the XOR TyDi dataset outperforming the previous results up to 10% on several languages. We find that our approach can be generalized to more than 20 languages in zero-shot approach and outperform all previous models by 12%.

## introduction question answering (qa) is an important tool for information search on the internet. since it is natural for a person to ask a question to get some information, the qa systems are designed to meet this requirement. qa provides a wide range of tasks, from engineering to cornerstone scientific tasks. open-domain qa, in this direction, is an interesting example of a problem that connects both: multilingual knowledge sources form differing knowledge and supplement gaps in each specific language. the requirement for modern language models to be cross-lingual is gradually becoming more and more important and is being incorporated into popular benchmarks such as xglue @xcite or xtreme @xcite , where the systems are evaluated not only by their performance metrics on single language tasks but the ones on many languages. when and how did tumbleweeds get to america? a: se saapui pellavansiementen kanssa etelä-venäjältä 1870 it arrived from in the with flaxseed 1870s southern russia sentri encoder солянка сорная [...] перекати-поле. распространен широко на юге россии. [...] в северной америке как интродуцент широко распространился. образует шаровидные кусты, которые осенью отрываются от ## datasets in this work, we use xor tydi and mkqa to evaluate our system onto and several datasets to (pre-)train it. xor tydi @xcite ) is a multilingual open-retrieval qa dataset that enables crosslingual answer retrieval. the dataset, based on questions from tydi qa @xcite , articulates three new tasks that involve finding documents in different languages using multilingual and english resources. it consists of questions written by information-seeking native speakers in 7 typologically diverse languages: arabian, bengali, finnish, japanese, korean, russian and telugu. answer annotations are retrieved from multilingual document collections. xor-retrieve is a crosslingual retrieval task where a question is written in the target language (e.g., japanese), and a system is required to retrieve an english document that answers the question. xor-english span is a cross-lingual retrieval task where a question is written in the target language (e.g., japanese), and a system is required to output a short answer in english. xor-full is a cross-lingual retrieval task where a question is written in the target language (e.g., japanese), and a system is required to output a short answer in the target language. in our work, we concentrate on xor-retrieve and xor-full tasks and use xor tydi to train and evaluate our system. ## method the open domain question answering task heavily relies on retrieval from some (possibly more than one) document collections. in the case of the crosslingual variant of this task, the usage of several (at least two -in english and in a target language) document collections is almost inevitable. we evaluate our model in two cross-lingual setups: using english wikipedia (w eng ) to search for a relevant passage containing the answer to the question or using collection of multilingual reference passages from arabic, russian, english, finnish, telugu, bengali, japanese, korean wikipedia (w multi ). more formally, given a question q in language l i , a system retrieves the documents from w eng or w multi , and formulates an answer a. thus the system could be virtually split to retriever, which creates a list of relevant documents, and reader, which generates an answer using the most relevant documents. the sample of the system output is presented on fig. 1 . ## experiments we have conducted a series of experiments with number of models, namely these are sentri model combined with different reader parts and bi-encoder model with one or two stages of selftraining. bi-encoder model is using standard extractive reader (plus machine translation where applicable). the main difference between sentri and bi-encoder, that the latter is based on classic dualencoder architecture, while the former is using single encoder for questions and paragraphs. ## ablation study sentri model has five important features which differentiate it from the previous work: self-training, a single encoder model for passage and question processing as a retriever, a generative model as a reader, in-batch negative filtering, usage of the machine translated data during the training process. we could conclude that all the features are important for our approach to present state-of-the-art results in retrieval and end-to-end tasks. ## conclusion nowadays multi-lingual and cross-lingual problems are coming to the stage once the natural language models become more and more powerful. one of these problems is that where the systems answer the questions using various mutually disjoint language data, as it stated in xor tydi task. this task is based on a specific xor tydi dataset @xcite , which ensured such information asymmetry in the different language data. we introduced the cross-lingual system to solve the xor task. while the xor tydi is a challenging test that stimulates cross-linguality in nlp systems, we have outperformed the existing models in two subtasks: xor-retrieve and xor-full without using external apis. the first task is a classical passage retrieval task, while the second one is an end-to-end question-answering task. besides showing the state of the art results on these two subtasks, our system is demonstrated the ability the transfer to the unseen languages in retrieval task, including the languages which were not presented in the pre-trained language model we use as an encoder for the retriever part of our sentri model. and last we found that the previous works ignored the existence of the morphology in xor tydi presented languages, thus missing many results in information retrieval. we propose to solve this issue by using stemming or lemmatization for such languages. our system has five differentiating features, which are self-training (using the output of the previously trained models), single encoder (allowing us to reduce the number of parameters about twice in retriever), usage of a generative model to get the question from retrieved passages, in-batch negative filtering, and usage of the machine translated data during the training process. all of these features are proved to make a share in the achieved significant quality improvement demonstrated by our model. although, our system has several flaws, e.g. passage selection strategy and stemming for the languages, we consider these flaws as our future work. but we hope that current study will foster research in cross-lingual question answering tasks. in tab. 11. we also provide aggregated results for the unseen on training stage languages and seen in the training stage ones. as one could see, our model performs surprisingly well on the languages which were not presented on any training stage, although the training improves recall by 12 and 13 per cent on average, while pre-training adds 13.8% of recall. we think this is another evidence of the great generalizability of the pre-trained language models. query metric query metric language r@2kt r@5kt language r@2kt r@5kt