Datasets for Multilingual Answer Sentence Selection (2024)

Abstract: Answer Sentence Selection (AS2) is a critical task for designing effective retrieval-based Question Answering (QA) systems. Most advancements in AS2 focus on English due to the scarcity of annotated datasets for other languages. This lack of resources prevents the training of effective AS2 models in different languages, creating a performance gap between QA systems in English and other locales. In this paper, we introduce new high-quality datasets for AS2 in five European languages (French, German, Italian, Portuguese, and Spanish), obtained through supervised Automatic Machine Translation (AMT) of existing English AS2 datasets such as ASNQ, WikiQA, and TREC-QA using a Large Language Model (LLM). We evaluated our approach and the quality of the translated datasets through multiple experiments with different Transformer architectures. The results indicate that our datasets are pivotal in producing robust and powerful multilingual AS2 models, significantly contributing to closing the performance gap between English and other languages.

## introduction answer sentence selection (as2) represents a crucial component in many qa systems in both academic and industrial settings. the role of this component is to select the correct answer for a given question among a pool of candidate sentences. while in recent years significant progress has been made in developing models and datasets for as2 @xcite @xcite @xcite , most of these are designed and evaluated in english. by contrast, less attention has been paid to other medium resource languages, such as french, german, italian, portuguese, and spanish, for which researchers struggle to obtain sufficient amounts of quality data to train their models. recently, machine translation (mt) proved to be an effective approach to address the challenges of lowresource language qa systems @xcite @xcite . for the as2 task, researchers have released a plethora of as2 datasets in english, such as asnq @xcite , wikiqa @xcite , and trec-qa @xcite , but there remains a gap for lower-resource languages that still needs to be filled. in this work, we contribute to this research area by introducing three new large multilingual as2 corpora named masnq, mwikiqa, and mtrec-qa for the most common european languages, comprising over 100 million question-answer pairs. we prepared these datasets by translating existing datasets (asnq, wikiqa, and trec-qa) into five european languages (french, german, italian, portuguese, and spanish) using a recent state-of-theart translation model @xcite . to validate the effectiveness of our approach, we trained several models using the masnq foot_0 , mwikiqa foot_1 , and mtrec-qa foot_2 datasets and evaluated their performance. our results demonstrate that these new datasets can be reliably used to train robust rankers for lower resource languages, yielding higher performance levels than the ones achieved by other competitors. this contribution helps to reduce the language barrier and provides valuable assets for researchers working in low and medium resource languages. ## as2 translated datasets for dataset translation, we use the largest version of the nllb (team et al., 2022) model foot_3 , which has 3.3 billion parameters. we consider three datasets: trec-qa @xcite , wikiqa @xcite , and asnq @xcite . for each of them, we translate both the questions and the answers. our translation process can be described as a two-step procedure. in the first step, we leverage the nllb model to translate the source datasets into five languages: french, german, italian, portuguese, and spanish. in the second step, we employ two techniques to evaluate the quality of the translations by identifying poor translations and then correcting any misleading sentences. firstly, we use a cross-language semantic similarity model released by @xcite 5 to assess the quality of the translated sentences. this model compares the semantic similarity between the original sentences in english and their translated versions in the target language. by measuring the similarity, we can identify bad translations that deviate significantly from the original sentences in english, due to translation errors and artifacts. secondly, we target these errors by applying a set of heuristics to correct misleading sentences. these heuristics are designed to correct errors, improve clarity, remove non-original text, and enhance the overall quality of the translated datasets. ## experiments in this section, we measure the benefits of our datasets applied to existing multilingual models. with these experiments, we aim to verify and prove the effectiveness of our contributions. specifically, we want to show that the translated data could be used to train state-of-the-art as2 models in multiple languages. for each considered language, we to verify these hypotheses, we consider an existing multilanguage pre-trained cross-encoder transformer model, which is xlm-roberta base foot_5 , and bert-multilingual foot_6 . following the tanda approach @xcite , we perform a two-stage training for each model. this technique consists of a two-stage training paradigm, where the first training stage, named transfer step, involves training the models on asnq to teach them to recognize and solve the as2 tasks. in the second step, named adaptation step, the transferred models are fine-tuned on the final target as2 datasets. in our setting, we apply this paradigm by first training and doing a separate transfer step on each language of masnq. secondly, we finetune the obtained models on mwikiqa and mtrec-qa. to have a comprehensive perspective of how our approach behaves, we measure the performance on three test sets from (i) mwikiqa, (ii) mtrec-qa, and (iii) xtr-wikiqa foot_7 . with the first two datasets, we aim to show the performance of our models in a controlled environment where the translation pipeline and the heuristics are the same as those used on masnq. the third dataset, instead, allows us to prove that (i) our approach is robust in a zero-shot setting and (ii) can be extended to datasets translated using different pipelines. to conduct our experiments, we considered the hyperparameters and the settings described in appendix b. in addition, we propose additional extensive experiments performing the tanda transfer step on the original asnq in appendix c and using different multilingual models in appendix d and several ablations described in section 5. ## ablation studies we conducted several experiments to estimate the benefits provided by our multilingual datasets, assessing their impact on different aspects of model performance. cross-lingual: models trained on the masnq dataset consistently outperformed those trained on asnq, demonstrating higher map and p@1 scores across all languages. this confirms the effectiveness of masnq in enhancing cross-lingual model performance (appendix e). ranks correlation: models trained on masnq and mwikiqa showed strong positive correlations in their ranking outputs compared to those trained on asnq and wikiqa. this indicates consistent translation quality and robust model performance (appendix f). passage ranking: models trained on mmarco outperformed those trained on msmarco, emphasizing the significant advantages provided by adapting models trained on our multilingual datasets for various tasks (appendix g). ## conclusion our study tackles the language barrier in qa systems by focusing on european languages such as italian, german, portuguese, spanish, and french. we introduced new large multilingual as2 datasets (masnq, mwikiqa, and mtrec-qa) by translating existing english as2 datasets using a stateof-the-art translation model. this approach provides valuable resources for lower-resource languages. our extensive experiments demonstrated the effectiveness of these datasets in training robust as2 rankers across various languages, achieving performance comparable to english datasets. this contributes significantly to reducing the language barrier, making as2 more accessible and effective across different linguistic contexts. to support further research, we released our new multilingual as2 datasets to the research community. we hope our work inspires future studies to address language diversity challenges in qa, leading to more inclusive and effective solutions for global users.