Deep Exploration of Cross-Lingual Zero-Shot Generalization in Instruction Tuning (2024)

Abstract: Instruction tuning has emerged as a powerful technique, significantly boosting zero-shot performance on unseen tasks. While recent work has explored cross-lingual generalization by applying instruction tuning to multilingual models, previous studies have primarily focused on English, with a limited exploration of non-English tasks. For in-depth exploration of cross-lingual generalization in instruction tuning, we perform instruction tuning individually for two distinct language meta-datasets. Subsequently, we assess the performance on unseen tasks in the language different from the one used for training. To facilitate this investigation, we introduce a novel non-English meta-dataset named “KORANI” (Korean Natural Instruction), comprising 51 Korean benchmarks. Moreover, we design cross-lingual templates to mitigate discrepancies in language and instruction-format of the template between training and inference within the cross-lingual setting. Our experiments reveal consistent improvements through cross-lingual generalization in both English and Korean, outperforming baseline by average scores of 20.7% and 13.6%, respectively. Remarkably, these enhancements are comparable to those achieved by mono-lingual instruction tuning and even surpass them in some tasks. The result underscores the significance of relevant data acquisition across languages over linguistic congruence with unseen tasks during instruction tuning.

## introduction recent studies have highlighted instruction tuning, where large language models are fine-tuned using instructions (templates) for various tasks, resulting in significant improvements in zero-shot performance on unseen tasks @xcite @xcite @xcite @xcite . several works have investigated the effectiveness of instruction tuning to show cross-lingual zero-shot generalization. for example, @xcite ; @xcite ; @xcite ; @xcite apply instruction tuning to multilingual large language models @xcite @xcite , pre-trained with multiple languages, and demonstrate cross-lingual generalization ability by achieving meaningful performance enhancements for unseen tasks in other languages. however, prior investigations into cross-lingual generalization through instruction tuning, as observed in works by @xcite and @xcite , have predominantly focused on english and have limited diversity of tasks in non-english. this limited scope makes it challenging to thoroughly evaluate the effectiveness of cross-lingual zero-shot generalization, as direct comparisons with monolingual instruction tuning within the same language are not attainable. moreover, previous research lacks validation of crosslingual generalization of instruction tuning for languages other than english, and the evaluation of non-english datasets has been confined to specific tasks, offering only partial understanding. to fill these gaps, our study undertakes a comprehensive investigation of the cross-lingual zero-shot generalization in instruction tuning. we define a new cross-lingual setting as the case where the training and inference language differs. in the setting, we instruction tune for two languages metadataset separately and evaluate the other language's unseen tasks. specifically, to examine how effective cross-lingual zero-shot generalization is for a different language, we conduct a comparative analysis between cross-lingual instruction tuning and monolingual instruction tuning, where the latter indicates models trained and tested on the same language's meta-dataset. the collection of a non-english meta-dataset @xcite is imperative for the comprehensive examination of cross-lingual generalization of instruction tuning. however, collecting diverse supervised task datasets for non-english poses a substantial challenge due to the limited availability of open-source data in non-english languages compared to english. to address this issue, we propose a novel non-english language meta-dataset named korani, short for korean natural instruction. this meta-dataset comprises 51 diverse korean benchmarks, including 34 nlu benchmarks and 17 nlg benchmarks. notably, korani surpasses the quantity of non-english benchmarks explored in previous multilingual research @xcite and approaches the size of p3 datasets @xcite , which we employ as english benchmarks in our study. furthermore, in the cross-lingual setting, the language and instructional format of templates are different during the training and inference phase. these discrepancies in the template may contribute to suboptimal model performance @xcite @xcite . to address this issue, we construct cross-lingual templates to align the template between the training and inference phases. our experiments show that cross-lingual instruction tuning consistently improves the zero-shot performance of unseen tasks in both english and korean. surprisingly, these cross-lingual performances are comparable to those of monolingual instruction-tuned models across various tasks, and they even surpass some tasks when cross-lingual templates are applied. these findings suggest that learning relevant tasks, even in different languages, is more crucial for performance improvement than ensuring linguistic congruence within unseen tasks in instruction tuning. furthermore, our findings reframe the traditional view that cross-lingual instruction tuning merely enhances performance in low-resource languages, suggesting that it could serve as a viable alternative to monolingual instruction tuning. our contributions are summarized as follows: ## measuring cross-lingual zero-shot generalization to investigate the effect of cross-lingual zero-shot generalization in instruction tuning, we perform instruction tuning independently for both english and korean. we then measure the cross-lingual generalization by evaluating the models' performance on unseen tasks in the other language, respectively. ## conclusion our research contributes to a deep understanding of the cross-lingual zero-shot generalization effect and its benefits by leveraging the novel korani meta-dataset to compare cross-lingual and monolingual instruction tuning directly. the experimental results indicate that cross-lingual instruction tuning can match or even exceed the performance of monolingual tuning. our findings highlight the importance of training relevant data across diverse languages rather than strictly maintaining linguistic consistency in unseen tasks. the successful application of cross-lingual templates, which ensure consistency in both language and format, further validates the potential of cross-lingual instruction tuning. these discoveries present cross-lingual instruction tuning not just as an auxiliary strategy but as a potential alternative to monolingual methods, especially in low-resource language scenarios.