Pivot Through E nglish: Reliably Answering Multilingual Questions without Document Retrieval (2022)

Abstract: Existing methods for open-retrieval question answering in lower resource languages (LRLs) lag significantly behind English. They not only suffer from the shortcomings of non-English document retrieval, but are reliant on language-specific supervision for either the task or translation. We formulate a task setup more realistic to available resources, that circumvents document retrieval to reliably transfer knowledge from English to lower resource languages. Assuming a strong English question answering model or database, we compare and analyze methods that pivot through English: to map foreign queries to English and then English answers back to target language answers. Within this task setup we propose Reranked Multilingual Maximal Inner Product Search (RM-MIPS), akin to semantic similarity retrieval over the English training set with reranking, which outperforms the strongest baselines by 2.7% on XQuAD and 6.2% on MKQA. Analysis demonstrates the particular efficacy of this strategy over state-of-the-art alternatives in challenging settings: low-resource languages, with extensive distractor data and query distribution misalignment. Circumventing retrieval, our analysis shows this approach offers rapid answer generation to many other languages off-the-shelf, without necessitating additional training data in the target language.

## introduction open-retrieval question answering (orqa) has seen extensive progress in english, significantly outperforming systems in lower resource languages (lrls). this advantage is largely driven by the scale of labelled data and open source retrieval tools that exist predominantly for higher resource languages (hrls) -usually english. to remedy this discrepancy, recent work leverages english supervision to improve multilingual systems, either by simple translation or zero shot figure 1 : cross-lingual pivots (xlp): we introduce the "cross lingual pivots" task, formulated as a solution to multilingual question answering that circumvents document retrieval in low resource languages (lrl). to answer lrl queries, approaches may leverage a question-answer system or database in a high resource language (hrl), such as english. in practice, we believe this assumption can be challenging to meet. a new document index can be expensive to collect and maintain, and an effective retrieval stack typically requires languagespecific labelled data, tokenization tools, manual heuristics, and curated domain blocklists @xcite @xcite . consequently, we discard the common assumption of robust non-english document retrieval, for a more realistic one: that there exists a high-quality english database of query-answer string pairs. we motivate and explore the cross-lingual pivots (xlp) task (section 2), which we contend will accelerate progress in lrl question answering by reflecting these practical considerations. this pivot task is figure 2 : reranked multilingual maximal inner product search (rm-mips): for the cross-lingual pivots task, we propose an approach that maps the lrl query to a semantically equivalent hrl query, finds the appropriate hrl answer, then uses knowledge graph or machine translation to map the answer back to the target lrl. specifically, the first stage (in blue) uses multilingual single encoders for fast maximal inner product search (mips), and the second stage (in red) reranks the top k candidates using a more expressive multilingual cross-encoder that takes in the concatenation of the lrl query and candidate hrl query. similar to "translate test" and "mt-in-the-middle" paradigms @xcite @xcite except for the availability of the high-resource language database, which allows for more sophisticated pivot approaches. figure 1 illustrates a generalized version of an xlp, where lrl queries may seek knowledge from any hrl with its own database. for this task we combine and compare state-ofthe-art methods in machine translation ("translate test") and cross-lingual semantic similarity, in order to map lrl queries to english, and then english answers back to the lrl target language. in particular we examine how these methods are affected by certain factors: (a) whether the language is high, medium or low resource, (b) the magnitude of data in the hrl database, and (c) the degree of query distribution alignment between languages (i.e., the number of lrl queries that have matches in the hrl database). lastly we propose an approach to this task, motivated by recent dense nearest neighbour (knn) models in english which achieve strong results in qa by simply searching for similar questions in the training set (or database in our case) @xcite . we leverage nearest neighbor semantic similarity search followed by cross-encoder reranking (see figure 2 ), and refer to the technique as reranked multilingual maximal inner product search (rm-mips). not only does this approach significantly improve upon "translate test" (the most common pivot technique) and state-of-the-art paraphrase detection baselines, our analysis demonstrates it is more robust to lower resource languages, query distribution misalignment, and the size of the english database. by circumventing document retrieval and taskspecific supervision signals, this straightforward approach offers reliable answer generation to many of the languages present in pretraining, off-the-shelf. furthermore, it can be re-purposed to obtain reliable training data in the target language, with fewer annotation artifacts, and is complementary to a standard end-to-end question answering system. we hope this analysis complements existing multilingual approaches, and facilitates adoption of more practical (but effective) methods to improve knowledge transfer from english into other languages. we summarize our contributions as: the open-retrieval question answering (orqa) task evaluates models' ability to answer information-seeking questions. in a multilingual setting, the task is to produce answers in the same language as the query. in some cases, queries may only find answers, or sufficient evidence, in a different language, due to informational asymmetries @xcite . to address this, @xcite propose cross-lingual open-retrieval question answering (xorqa), similar to the cross-lingual information retrieval (clir) task, where a model needs to leverage intermediary information found in other languages, in order to serve an answer in the target language. in practice, this intermediary language tends to be english, with the most ample resources and training data. building on these tasks, we believe there are other benefits to pivoting through high resource languages that have so far been overlooked, and consequently limited research that could more rapidly improve non-english qa. these two benefits are (i) large query-answer databases have already been collected in english, both in academia @xcite and in industry @xcite , and (ii) it is often very expensive and challenging to replicate robust retrieval and passage reranking stacks in new languages @xcite @xcite . 1 as a result, the english capabilities of question answering systems typically exceed those for non-english languages by large margins @xcite @xcite . we would note that prior work suggests even without access to an english query-answer database, translation methods with an english document index and retrieval outperforms lrl retrieval for open-retrieval qa (see the end-to-end xor-full results in @xcite ). this demonstrates the persistent weakness of non-english retrieval, and motivates alternatives approaches such as cross-lingual pivots. to remedy this disparity, we believe attending to these two considerations would yield a more realistic task setup. like multilingual orqa, or xorqa, the task of xlps is to produce an answer âlrl in the same "target" language as question q lrl , evaluated by exact match of f1 tokenoverlap with the real answer a lrl . instead of assuming access to a lrl document index or retrieval system (usually provided by the datasets), 1 while it is straightforward to adapt question answering "reader" modules with zero-shot learning @xcite , retrieval can be quite challenging. not only is the underlying document index costly to expand and maintain for a new language @xcite , but supervision signals collected in the target language are particularly important for dense retrieval and reranking systems which both serve as bottlenecks to downstream multilingual qa @xcite . additionally, real-world qa agents typically require human curated, language-specific infrastructure for retrieval, such as regular expressions, custom tokenization rules, and curated website blocklists. ## re-ranked multilingual maximal inner product search for the first stage of the xlp task, our goal is to find an equivalent english query for a lrl query: "query matching". competing approaches include single encoders and cross encoders, described further in section 4.2. single encoders embed queries independently into a latent vector space, meaning each query q en from the english database q en can be pre-embedded offline. at inference time, the low resource query q lrl is embedded, then maximal inner product search (mips) finds the approximate closest query q en among all q en by cosine similarity. by comparison, cross encoders leverage cross-attention between q lrl and candidate match q en at inference time, thus requiring o(|q en |) forward passes at inference time to find the best paraphrase. while usually more accurate this is computationally infeasible for a large set of candidates. we propose a method that combines both single encoders and cross encoders, which we refer to as reranked mulilingual maximal inner product search (rm-mips). the process, shown in figure 2 , first uses a multilingual sentence embedder with mips to isolate the top-k candidate similar queries, then uses the cross encoder to rerank the candidate paraphrases. this approach reflects the retrieve and read paradigm common in or-qa, but applies it to a multilingual setting for semantic similarity search. the model first queries the english database using the multilingual single encoder se(q i ) = z i to obtain the k-nearest english query neighbors n q lrl ⊆ q en to the given query q lrl by cosine similarity. then, it uses the multilingual cross encoder ce(q 1 , q 2 ) to score the remaining set of queries n qlrl to obtain the final prediction. ## experiments we compare systems that leverage an english qa database to answer questions in lower resource languages. figure 1 illustrates a cross-lingual pivot (xlp), where the task is to map an incoming query from a low resource language to a query in the high resource language database (@xmath0, discussed in 4.2), and then a high resource language answer to a low resource language answer (@xmath1, discussed in 4.3). ## conclusion in conclusion, we formulate a task to cross-lingual open-retrieval question answering more realistic to the constraints and challenges faced by practitioners expanding their systems' capabilities beyond english. leveraging access to a large english training set our method of query retrieval followed by reranking greatly outperforms strong baseline methods. our analysis compares multiple methods of leveraging this english expertise and concludes our two-stage approach transfers better to lower resource languages, and is more robust in the presence of extensive distractor data and query distribution misalignment. circumventing retrieval, this approach offers fast online or offline answer generation to many languages straight off-the-shelf, without necessitating additional training data in the target language.