M c C rolin: Multi-consistency Cross-lingual Training for Retrieval Question Answering (2024)

Abstract: Automated question answering (QA) systems are increasingly relying on robust cross-lingual retrieval to identify and utilize information from multilingual sources, ensuring comprehensive and contextually accurate responses. Existing approaches often struggle with consistency across multiple languages and multi-size input scenarios. To address these challenges, we propose McCrolin, a Multi-consistency Cross-lingual training framework, leveraging multi-task learning to enhance cross-lingual consistency, ranking stability, and input-size robustness. Experimental results demonstrate that McCrolin achieves state-of-the-art performance on standard cross-lingual retrieval QA datasets. Furthermore, McCrolin outperforms competitors when dealing with various input sizes on downstream tasks. In terms of generalizability, results from further analysis show that our method is effective for various encoder architectures and sizes.

## introduction automated question answering is becoming more common thanks to the rapid advancement of large language models (llms). for reliability, these systems utilize a mechanism to integrate external knowledge into question answering, which, in turn, relies on robust retrieval capability. @xcite show that the integration of external knowledge can substantially improve the performance of qa tasks. a modern qa system must contend with users with diverse language preferences while utilizing documents written in different languages. for example, a user might ask in english, "what was the function of surabaya pre-colonial?" or in german, "was war die funktion von surabaya * equal contributions ♠ work was conducted while peerat limkonchotiwat was a phd candidate at vistec vor der kolonialzeit?" the system may retrieve relevant documents found in indonesian historical records, as well as memoirs of missionaries and explorers written in dutch and portuguese from different perspectives. robust cross-lingual retrieval is crucial to modern qa systems, as it provides the capability to identify and utilize information from diverse multilingual sources, forming the basis for comprehensive and contextually rich answers. cross-lingual representation learning is a common approach to obtaining the cross-lingual retrieval qa capability. in particular, one can employ bi-encoder learning, where a pre-trained language model (plm) serves as the encoder for queries and documents. the training loss attempts to maximize the cosine similarity between the representation vectors of the query and its corresponding document @xcite @xcite . despite the rapid advances in representation learning, previous works @xcite have revealed that cross-lingual retrieval for qa requires further improvements in terms of consistency when dealing with multiple languages and input with different sizes. to address this limitation, existing methods utilize a larger plm (330m parameters or over) and fine-tune it on extensive multilingual corpora @xcite . while these approaches perform well in their training data distribution, they often lack robustness in handling unseen languages and out-of-domain scenarios. in this paper, we propose a framework designed to address the challenges of language diversity and varying input sizes in cross-lingual retrieval qa, called multi-consistency cross-lingual training framework @xcite . our approach leverages the reciprocal nature of multi-task learning, where the concurrent optimization of multiple interrelated objectives enhances the overall performance of the retrieval model. the crux of our proposed method lies in multi-task learning objectives derived from the three properties vital to robust qa retrieval in a cross-lingual environment as follows: to evaluate the efficacy of our proposed framework, mccrolin, we conducted extensive experiments comparing it with 7 models across 5 experimental setups. the experimental results from crosslingual retrieval qa demonstrate that mccrolin outperforms competitors in the average score case. when evaluating multi-candidate retrievals, our framework achieves state-of-the-art performance in mean reciprocal rank at 10 (mrr@10). moreover, our framework enables the base model to effectively handle multi-input levels, encompassing sentence and paragraph texts, improving retrieval performance at all levels. the main contributions of this work include: ## experimental results in this section, we present a series of experiments on cross-lingual retrieval question answering (sections 5.1) and a downstream task, namely machine reading comprehension (section 5.2). additionally, we investigate the impact of our framework on different plms in section 5.3. ## conclusion in this paper, we introduce mccrolin, a multiconsistency cross-lingual training framework designed to address key challenges in cross-lingual qa retrieval, including cross-lingual consistency, ranking stability, and input-size robustness. we incorporate the desired properties into the base encoder using three loss components: cross-lingual language consistency l xlc , learning-to-rank l rank , and hierarchical alignment objectives l ha . our component-wise analysis stresses the backbone status of l xlc , and the combination of all three loss components yields the best results. mccrolin's superior performance in handling various input sizes on downstream tasks further underscores its robustness. notably, analysis of various encoders demonstrates the generalizability of mccrolin across different encoder architectures and sizes, providing significant improvements when applied to various plms, such as mbert, xlm-r, and me5. this versatility enhances the cross-lingual consistency and robustness of qa systems, confirming the effectiveness of our approach.