Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages (2023)

Abstract: While impressive performance has been achieved on the task of Answer Sentence Selection (AS2) for English, the same does not hold for languages that lack large labeled datasets. In this work, we propose Cross-Lingual Knowledge Distillation (CLKD) from a strong English AS2 teacher as a method to train AS2 models for low-resource languages in the tasks without the need of labeled data for the target language. To evaluate our method, we introduce 1) Xtr-WikiQA, a translation-based WikiQA dataset for 9 additional languages, and 2) TyDi-AS2, a multilingual AS2 dataset with over 70K questions spanning 8 typologically diverse languages. We conduct extensive experiments on Xtr-WikiQA and TyDi-AS2 with multiple teachers, diverse monolingual and multilingual pretrained language models (PLMs) as students, and both monolingual and multilingual training. The results demonstrate that CLKD either outperforms or rivals even supervised fine-tuning with the same amount of labeled data and a combination of machine translation and the teacher model. Our method can potentially enable stronger AS2 models for low-resource languages, while TyDi-AS2 can serve as the largest multilingual AS2 dataset for further studies in the research community.

## introduction answer sentence selection (as2) is the task of ranking a given set of answer candidates according to their probability of correctly answering a given question. this is a core task for retrieval-based web question answering (qa) systems. indeed, as2 models applied to the sentences of documents relevant to a question, e.g., retrieved by a search engine, provide accurate answers. figure 1 : cross-lingual knowledge distillation (clkd) in two different scenarios: (top) using unlabeled english as2 dataset for target low-resource language lacking any data and (bottom) using unlabeled original low-resource language as2 dataset. clkd enables student as2 models to learn from english teacher as2 models without human-annotated datasets. while as2 has been extensively studied for english @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite , much less research has been devoted to other languages. this is despite the rapidly increasing importance of multilingual qa with the proliferation of conversational agents and voice assistants using multilingual content from the web to target locales across the world @xcite . a major barrier to achieving similar performance obtained with english in other languages is the lack of large labeled datasets. however, labeling as2 datasets for every language will be prohibitively expensive as even a single as2 instance can contain hundreds of candidate answers per question (see in this work, we propose cross-lingual knowledge distillation (clkd) as a method to use readily available and highly-accurate english as2 models to train as2 models for low-resource languages lacking labeled data. clkd can use english datasets to train as2 models for languages lacking any data and can further leverage unlabeled original target language data without the need for costly manual annotation. figure 1 illustrates a high-level description of our approach. clkd works similarly to classic knowledge distillation (kd) @xcite in that a student model is trained to mimic a teacher. the main novelty of our approach is the fact that the teacher and student models operate in different languages, namely the source and target languages. thus, the input question-answer pairs are translated into both the source and the target languages. additionally, to allow use of original target language data, which is typically unlabeled, we use only the soft labels obtained from the teacher even when gold labels are available, i.e., given an unlabeled input question and candidate answer pair (q, a), the student is trained using the kullback-leibler divergence loss between the probability scores of the teacher and the student when applied to (q, a). to evaluate our approach for diverse languages, we construct two new multilingual as2 datasets: xtr-wikiqa 3 (10 languages) and tydi-as2 (8 languages). xtr-wikiqa consists of 10-language par-3 xtr-wikiqa: x-translated-wikiqa allel corpora, including the original english corpus from wikiqa @xcite . 4 to enable evaluation with original target language data, we further create tydi-as2 by converting tydi-qa @xcite ), foot_1 a multilingual qa dataset to an as2 dataset. we also translate the english tydi-as2 dataset to all the other tydi-as2 languages to build an additional translationese corpus. using the above datasets, we perform extensive experiments with multiple teacher models, about 20 different student models of varying sizes, including both monolingual and multilingual plms, and both monolingual and multilingual training. additionally, to evaluate the utility of our method for both languages lacking any data and for those with some unlabeled data, we experiment with both using only english data (xtr-wikiqa and english tydi-as2) and only unlabeled target language data (tydi-as2). we show that clkd consistently either rivals or outperforms even supervised finetuning with the same amount of gold-labeled data demonstrating the benefit of clkd using soft labels obtained from a strong english as2 teacher model. in particular, we show that clkd using original language unlabeled data outperforms 1) fine-tuning with gold-labeled translationese and 2) for larger students, even the mt+english as2 model pipeline, demonstrating the importance of original target language data. we expect that the ability of clkd to train as2 models without the need for costly annotation process will enable stronger as2 performance for the world's many low-resource languages. to support further studies on as2 tasks for such languages, we will make the datasets introduced in this work and our trained models publicly available. ## related work we briefly summarize the related studies. ## experimental setup for a rigorous assessment of the efficacy of clkd, we design various experiments with different teachers, students, and training data. ## evaluations we now describe the results of our experiments. in § 6.1, we have a problem setting where we assume that no target language data is available for training and we use translations of the english data instead. in § 6.2, we use the tydi-as2 dataset to experiment with the setting where some original target language unlabeled data is available. ## conclusion in this work, we proposed cross-lingual knowledge distillation (clkd) to leverage strong english as2 models to train accurate models for low-resource languages without the need for costly manual annotation. furthermore, we introduced 1) xtr-wikiqa, a machine-translated wikiqa dataset in 9 additional languages and 2) tydi-as2, a new multilingual as2 benchmark spanning 8 languages. we conducted comprehensive experiments involving various teachers, students, and training settings, to discuss the potential of clkd. our results demonstrate the benefits of using soft supervision from a strong english teacher to train a student model for low-resource languages, suggesting the importance of original target language data compared to translationese potentially due to cultural biases and noise introduced by machine translation. despite requiring no manual annotations, clkd leverages both strong english teachers and original target language data and outperforms or rivals strong baselines such as supervised finetuning with the same amount of data and direct usage of a strong teacher model on english translations. the results also suggest that clkd has a po-tential to greatly reduce the cost of training strong as2 models for languages lacking labeled training data. to engage studies on as2 for such languages, we publish xtr-wikiqa 1 and tydi-as2. 2