Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering (2021)

Abstract: Human knowledge is collectively encoded in the roughly 6500 languages spoken around the world, but it is not distributed equally across languages. Hence, for information-seeking question answering (QA) systems to adequately serve speakers of all languages, they need to operate cross-lingually. In this work we investigate the capabilities of multilingually pretrained language models on cross-lingual QA. We find that explicitly aligning the representations across languages with a post-hoc finetuning step generally leads to improved performance. We additionally investigate the effect of data size as well as the language choice in this fine-tuning step, also releasing a dataset for evaluating cross-lingual QA systems.

## introduction information seeking question answering, where a user asks a question to get a related passage or short text as answer, is a widely studied area @xcite @xcite , inter alia) 2 that has been successfully deployed in user-facing applications such as conversational assistants @xcite . for example, an english computer science student that asks apple siri, amazon alexa, or google assistant the question "where did joan clarke work?" 3 will receive the answer "bletchley park", an answer based on the english wikipedia entry for joan clarke automatically retrieved by the system. now, consider another student, this time based in greece or bangladesh, asking effectively the same question, "πού δούλευε η joane clarke;" or "joane clarke েকাথায় কাজ করত?", but this time in their native language greek or bengali. however, as of july 1 code and dataset are publicly available here: https:// github.com/ffaisal93/aligned_qa 2 see @xcite for a thorough survey of the field. 3 joan clarke was the only female computer scientist at bletchley park during the second world war. details: https://en.wikipedia.org/wiki/joan_ clarke 2021, there is no wikipedia article for joane clarke in greek or in bengali. 4 for the qa system to adequately serve these students, it will need to function in a cross-lingual setting, retrieving the english (or any of the other available languages) article and producing an answer given the question in a different language @xcite . throughout this paper we will refer to a setting where the question and the context are in different languages as "cross-lingual" qa. multilingually-pretrained language models such as mbert @xcite or xlm-r @xcite are widely used as the base of modern qa systems and have shown promise for performing the task in zero-shot @xcite or few-shot manner @xcite as well as in cross-lingual settings @xcite . at the same time, these models are not without drawbacks; their pre-training objectives did not explicitly require that their representations align across languages for semantically similar words/phrases/sentences, with adverse effects especially for languages written in scripts other than the latin one @xcite . a recent line of work @xcite @xcite in fact shows improvements on a variety of natural language tasks by explicitly or implicitly aligning the models' representations across languages leveraging parallel data. in this work, we explore the effect of such representation alignment techniques on the performance of qa systems in cross-lingual settings, concluding that they can be very beneficial. our contributions can be summarized as follows: ## cross-lingual tydi-qa dataset to effectively study the cross-lingual capabilities of extractive qa systems, we construct a dataset with questions and potential contexts in different languages. we base our cross-lingual qa dataset on the tydi-qa dataset @xcite . tydi-qa dataset covers 11 languages with 204k questionanswer pairs. unlike other multilingual datasets like mlqa @xcite and xquad @xcite , tydi-qa dataset is collected from native speakers without the aid of translation. we select 5 typologically diverse language: bengali, korean, arabic, swahili, and english from tydi-qa dataset for all of our experiments. similar to other tydi-qa-derivative datasets we use the publicly available development set as our test set, and sample a few examples from the training data to build a custom development set (these are examples are discarded from the training set). we will refer to this (original) dataset as tydi-qa-mono to denote that all training/test instances are in a monolingual setting. we augment the dataset both automatically by generating translations of questions and by collecting human-created translations for some test sets. we describe these two datasets below: synthetic translations we create a crosslingually augmented version of our training set in two ways. first, we augment the english portion by automatically translating english questions to the other four languages (q en translate -------→ q ar,sw,ko,bn ), keeping the context (and answers) in english. similarly, we augment the bengali, swahili, korean, and arabic portions of the dataset by translating their questions to english (q ar,sw,ko,bn translate -------→ q en ), still keeping the context and answers in the original language. 5 we will refer to this version of the 5 while english is not always the best pivot language for cross-lingual transfer (anastasopoulos and neubig, 2020, inter dataset as tydi-qa+aug. for all translations we use the google translate api 6 but we will make them publicly available to ensure the reproducibility of our experiments. the same process is applied to our custom development and test sets. ## alignment-based fine-tuning language models like mbert and xlm-r are trained without the inclusion of any parallel data. @xcite use a small amount of parallel data to align the representations of similar words across different languages, an approach that leads to improvements in cross-lingual inference tasks. this method relies on the availability of k parallel corpora: c 1 , c 2 , ...c k where each corpus c (ls,lt) = (s 1 , t 1 ), ...(s n , t n ) is a set of parallel sentences in the source-target (l s , l t ) languages. using any word alignment technique, we can obtain word-level alignments α(s, t) = (p 1 , q 1 )....(p m , q m ) for each sentence pair, such that for each (p, q) pair, word s p of source sentence s is translated to word t q in target sentence t. we can devise an objective function that minimizes the distance of the contextualized embeddings of aligned words, by penalizing the distance of their representations. denoting with f ϕ (s i ) the contextual embedding of word in position i for sentence s given by a model parameterized by ϕ, we get this fine-tuning process runs the risk of leading the encoder's parameters too far from their initial values ϕ 0 , which could lead to catastrophic forgetting @xcite @xcite ). hence, following @xcite we additionally use a regularization metric to penalize the excessive drifting from the initial encoder state ϕ 0 for each of the languages. the final objective is simply the sum of the two components, similar to the approaches for retrofitting static embeddings @xcite : ## experimental setup model and baselines our extractive qa model architecture is similar to @xcite . in this implementation, both the question and the context are encoded to contextual representations using pre-trained language models such as bert, and a dedicated classification head produces answer depending on the two types of tasks: 1) the index of the passage answering the questions or null (if no such answer exist) for passage selection task. 2) a yes/no answer or the answer span using these representations or null for minimal answer span task. we train baseline models using both tydi-qa-mono and tydi-qa+aug training datasets. all models are trained multilingually, training using all languages jointly. we report results on the two tydi-qa primary tasks, passage selection (given a question and a number of candidate passages, this task is to return the index of the passage containing the answer) and minimal answer selection (a question and a number of passages are given; the task is to return the start and end byte of a short text span containing the answer). we preform experiments with both mbert and xlm-r. our baselines use the pre-trained models, fine-tuned as above on the qa task (without any intermediate alignment-based fine-tuning). our models, in turn, first perform alignment-based finetuning on the pre-trained models, and then train them on the qa task. alignment-based fine-tuning due to the lack of n-way parallel corpora, all parallel corpora we use have english as one of the two languages. we obtain word-level alignments using awesome-align @xcite . we use data from wikimatrix @xcite , wikimedia (tiedemann, 2012) and cc-aligned @xcite to prepare various versions of our parallel corpus. the size of these data varies from 260k to less than 1k for each language based on experiment type. to study the effect of using different language samples and data sizes, we experiment with different models (summarized in beyond these models, we also explore the following settings: ## related work a significant amount of work is devoted to studying and improving the cross-lingual capabilities of qa models. tydi-qa @xcite ) is a notable recent dataset focusing on the inclusion of 11 typologically diverse languages. xor-qa @xcite builds on top of tydi-qa, exploring open domain qa systems, where the search for an answer to a question unanswerable in the original language integrates translated resources from relevant english wikipedia pages, in a task reflective of our setting. we view our work as orthogonal to xor-qa, as @xcite put more emphasis on the retrieval of relevant passages rather than the cross-lingual capabilities of the qa ("reader") model per se. there exist a number of multilingual qa benchmarks, including mlqa @xcite , mkqa @xcite and xquad @xcite . mlqa translates original english questions to 7 other languages to train a multilingual qa model. mkqa comprises of questions from 26 diverse languages. xquad uses translated questions from squad @xcite (originally in english) to prepare a widely used and easily adaptable multilingual benchmark of squad baseline. in sd-qa @xcite from which our work is inspired, the authors prepare a naturally spoken version of tydi-qa over 5 languages. in this work, we further expand the scope to study the cross-lingual abilities of qa systems for these 5 languages. using cross-lingual objectives that leverage parallel language data is a promising direction towards improving the cross-lingual abilities of lan ## conclusion and future work in this work, we have studied the cross-lingual extractive qa setting where the question and contextto-search are in different languages. through experiments on synthetic and newly collected data in 4 languages, we have shown that data augmentation along with alignment-based fine-tuning can lead to big improvements in downstream performance. in future work, we plan to collect a larger dataset covering more languages for such cross-lingual settings. we also aim to extend this study using even more parallel data whenever available, as well as to investigate the feasibility of using language-specific tools