Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning (2024)

Abstract: Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks, but focus on conversational tasks has been rather limited. This is partly due to the high cost of obtaining non-English conversational data, which results in limited coverage. In this work, we introduce for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset that we created by translating the English-only Schema-Guided Dialogue (SGD) dataset (Rastogi et al., 2020) into 105 other languages. XSGD contains about 330k utterances per language. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts. We also investigate two different classifiers: NLI-based and vanilla classifiers, and test cross-lingual capability enabled by the aligned prompts. We evaluate our model’s cross-lingual generalization capabilities on two conversation tasks: slot-filling and intent classification. Our results demonstrate strong and efficient modeling ability of NLI-based classifiers and the large cross-lingual transfer improvements achieved by our aligned prompts, particularly in few-shot settings. We also conduct studies on large language models (LLMs) such as text-davinci-003 and ChatGPT in both zero- and few-shot settings. While LLMs exhibit impressive performance in English, their cross-lingual capabilities in other languages, particularly low-resource ones, are limited.

## introduction it has long been known that nlp research and applications are concentrated on high-resource languages such as english, french, and japanese. this limitation introduces bias and prevents people in minority language groups from accessing recent nlp technologies. driven by advances in large-scale training, there has been an increase in the number of approaches that attempt to learn general-purpose multilingual representations, which aim to capture shared knowledge across languages. jointly trained multilingual language models such as xlm-r @xcite and mbart @xcite , coupled with supervised fine-tuning in the source (english) language, have been quite successful in transferring linguistic and task knowledge from one language to another without using any task labels in the target language, a.k.a. zero-shot transfer. despite their effectiveness, studies @xcite @xcite have also highlighted key factors for successful transfer which include structural similarity between languages and the tasks under consideration. when it comes to conversational tasks, studies on cross-lingual zeroshot transfer have been limited to only few domains and languages. to investigate the cross-lingual transfer ability on conversational tasks, we create the xsgd dataset by translating data from the english-only schema-guided dialogue or sgd @xcite , which is currently the largest multi-domain dialogue corpora. while previous work such as multi 2 woz @xcite has also tried to expand monolingual datasets into multiple languages, it is primarily a translation of development and test dialogues from the english-only multiwoz dataset @xcite into arabic, chinese, german, and russian. in contrast, xsgd comprises 106 languages (in-cluding english), with roughly 330k utterances and 10 domains per language, as compared to the 7 domains and 29.5k utterances per language in multi 2 woz. recently, several studies @xcite @xcite have shown the potential of prompt tuning. in particular, @xcite observed that prompt tuning can achieve much better cross-lingual transfer than model fine-tuning across multiple xtreme tasks @xcite using significantly fewer parameters. in this work, we propose an efficient prompt-tuning-based method that utilizes soft prompts to obtain stronger cross-lingually aligned representations on the xsgd dataset. the aligned prompts enable models to learn cross-lingual representations that can improve cross-lingual retrieval. additionally, we compare the performance of vanilla and nli-based formulations on intent classification task. the latter utilizes label descriptions or label names in conjunction with utterances for entailment prediction. we find that it exhibits stronger few-shot cross-lingual generalization capability for english-only tuning. finally, our experimental results on intent classification and slot filling demonstrate consistent performance improvements with our learned aligned prompts, especially in few-shot settings. our contributions are summarized as follows: our study also showns the benefits of our approach, even when compared to large language models (llms) such as text-davinci-003 and chatgpt. ## xsgd dataset prior work has focused on enhancing pre-trained language models (plms) for either deeper understanding of conversational contexts or improved cross-lingual generalization. for example, @xcite and @xcite have explored adapting general-purpose english plms @xcite by applying conversation-specific training objectives on largescale english conversational corpus. one of the main challenges to achieve crosslingual conversational capability is the lack of paired multi-lingual conversational corpus. in this work, we take the initiative on this challenge and create a multi-lingual dataset xsgd on top of the sgd dataset @xcite . to this end, we leverage google translate api foot_1 and translate the original sgd dataset into 105 languages. it is a context-aware translation. because of the limitations of the translation api, the maxim context is set to 100 utterances in a dialogue per api call. a complete list of the 105 languages can be found in appendix a. we follow the same train, development, and test splits as in the original sgd dataset. human evaluation our parallel dataset is the largest multilingual tod corpus (330k per language), however, it inherits noise from the translation api. it is prohibitively expensive to do fullscale manual quality control because of its scale across 106 languages foot_2 . ## method in the zero-shot cross-lingual setting, models are fine-tuned solely on english and then evaluated on other languages. however, their performance on non-english languages, especially low-resource ones, tend to deteriorate @xcite . previous works, specifically tod-bert @xcite (with mlm loss) and convfit @xcite (with multiple negatives ranking loss), employ fine-tuning methods, where all model parameters are tuned. this process is not efficient for large pretrained models. the primary focus of our work is the exploration of efficient tuning methods. to address this issue, we propose a prompttuning-based method that utilizes translation data to learn aligned prompts, which can lead to improved cross-lingual transfer performance, especially when task data in english is limited. sequence pairs our dialogue corpus consists of dialogues with approximately 20 turns each. to reduce the sequence length of each dialogue during training, we randomly select consecutive turns within each dialogue in each epoch and concatenate them into a sequence. we repeat this process for the corresponding turns in the target language. we use this way to construct translation pairs dynamically during training, and then use the resulting translation pairs ⟨x i , y i ⟩ from two different languages to learn aligned representations for an improved cross-lingual generalization capability foot_5 . ## aligned prompts results in section 4, we propose a method that learns aligned prompts on conversation pair data in order to improve cross-lingual transfer ability. in this section, we show some aligned prompts results. ## conclusion in this paper, we present xsgd, a large-scale parallel multilingual conversation corpus that can be used for aligned cross-lingual transfer. additionally, we propose a prompt-tuning method to learn alignment prompts, which can further improve the efficiency of the cross-lingual transfer. we evaluate our approach on intent classification and slot-filling tasks, and our experiments demonstrate its effectiveness. we also study popular llms and find that their performance on non-english languages remain to be improved.