Cascading Adaptors to Leverage E nglish Data to Improve Performance of Question Answering for Low-Resource Languages (2021)

Abstract: Transformer based architectures have shown notable results on many down streaming tasks including question answering. The availability of data, on the other hand, impedes obtaining legitimate performance for low-resource languages. In this paper, we investigate the applicability of pre-trained multilingual models to improve the performance of question answering in low-resource languages. We tested four combinations of language and task adapters using multilingual transformer architectures on seven languages similar to MLQA dataset. Additionally, we have also proposed zero-shot transfer learning of low-resource question answering using language and task adapters. We observed that stacking the language and the task adapters improves the multilingual transformer models’ performance significantly for low-resource languages. Our code and trained models are available at: https://github.com/CALEDIPQALL/

## introduction last few years have seen emergence of transformer based pretrained models like bert @xcite , xlnet @xcite , t5 @xcite , xlm-roberta @xcite etc. the pretrained models have shown significant improvement in various downstream tasks like question answering, ner, machine translation and speech recognition @xcite @xcite @xcite @xcite @xcite . the emergence of multilingual models: mbert @xcite and xlm-roberta @xcite made it possible to leverage english * equal contribution † corresponding authors data to improve the performance of low-resource languages. in this paper, we continue to investigate the effectiveness of multilingual pretrained transformer models in improving the performance of question answering systems in a low-resource setup using the cascading of language and task adapters @xcite @xcite . our work contributes by evaluating cross-lingual performance in seven languages -hindi, arabic, german, spanish, english, vietnamese and simplified chinese. our models are evaluated on the combination of xquad @xcite and mlqa @xcite datasets which are similar to squad @xcite . to this end, our contributions are as follows: we first analyze the fine-tuned model by stacking language-specific adapter with the xlm-roberta base foot_2 . after fine-tuning the language-specific adapter we augment the task-specific adapter upon the previously finetuned language adapter. we analyze both the experiments separately and conclude that multiple adapters with the transformer-based model perform notably better. ## proposed approach in this section we describe our approach of training the task adapter and the language adapters in 4 different setup. ## experimental setups we have performed 4 different analysis as represented in figure 1 . details of all 4 setups are shown below: ## observations to study the impact of the task adapter and the language adapters, we have conducted experiments as shown in setup b and setup c. our observations from we have compared two task adapter architectures and noted that the usage of different task adapter architectures have negligible performance impact on majority of the languages. as a result, no clear distinction can be drawn from this observation, which can be used to guide future research. high-resource languages that use the right-to-left (rtl) scripting approach dominate the training of pretrained transformer models. the arabic language follows left-to-right(ltr) scripting the chinese language has a symbolic language structure and can be written in a variety of forms (left-to-right, or vertically top-to-bottom). the degraded findings in chinese compared to other lowresource languages are most likely due to the language's writing flexibility. ## conclusions we have investigated the efficacy of cascading adapters with transformer models to leverage highresource language to improve the performance of low-resource languages on the question answering task. we trained four variants of adapter combinations for -hindi, arabic, german, spanish, english, vietnamese, and simplified chinese languages. we demonstrated that by using the transformer model with the multi-task adapters, the performance can be improved for the downstream task. our results and analysis provide new insights into the generalization abilities of multilingual models for crosslingual transfer on question answering tasks.