Towards Zero-Shot Multilingual Transfer for Code-Switched Responses (2023)

Abstract: Recent task-oriented dialog systems have had great success in building English-based personal assistants, but extending these systems to a global audience is challenging due to the need for annotated data in the target language. An alternative approach is to leverage existing data in a high-resource language to enable cross-lingual transfer in low-resource language models. However, this type of transfer has not been widely explored in natural language response generation. In this research, we investigate the use of state-of-the-art multilingual models such as mBART and T5 to facilitate zero-shot and few-shot transfer of code-switched responses. We propose a new adapter-based framework that allows for efficient transfer by learning task-specific representations and encapsulating source and target language representations. Our framework is able to successfully transfer language knowledge even when the target language corpus is limited. We present both quantitative and qualitative analyses to evaluate the effectiveness of our approach.

## introduction recent task-oriented dialog systems (tod) have achieved great success in intelligently communicating with humans in natural languages @xcite . they are designed to fully assist users with widely heralded applications such as music playing, ticket ordering, or customer servicing @xcite . however, most tod systems are primarily established for english due to its ubiquity and the abundance of high-quality human annotations @xcite . extending these services to global users may take tremendous efforts, especially in low-resource languages where the collection of training corpus is labor-intensive. on the other hand, given a sizeable english dialog corpus with standard dialog features shared across other languages, it is possible to transfer the knowledge and logic between languages via machine translation or cross-task alignments. datadriven approaches @xcite perform standard supervised training with translated dialogs, known as translate-train. different pseudo-data pairs could be leveraged to enhance the multilingual model's robustness. nevertheless, a fine-grained machine translation system may not exist in an extremely low-resource language. the translation errors of entities in ground truth annotations (e.g. indian could be translated in chinese either to an adjective of indian or indian people in different contexts.) can drastically influence how model is supervised. this primarily happens in dialog tasks like dialog state tracking (dst) or natural language response generation (nlg) with language-sensitive outputs. another line of approaches instead investigates cross-lingual transfer directly in pretrained multilingual language models @xcite . in particular, the multilingual sequence-to-sequence model family (mseq2seq), which learns to encode the hidden representation of a given input and generates relevant outputs, can achieve promising multi-task performance in different languages. pretraining a multilingual encoder with machine translation @xcite , mask context learning @xcite or learning a word alignment matrix @xcite could possibly transfer knowledge between languages. these methods mostly share languageagnostic outputs @xcite for typical classification tasks to avert off-target problem, i.e., models (partially) translate its prediction into the wrong language during zero-shot transfer due to spurious correlation @xcite , ⃝ e&e setting will have english sentences with english entities and dialog acts. 2 ⃝ f&e setting will still have entire english dialog acts but foreign responses with english entities embedded. 3 ⃝ f&f will have all foreign responses but with code-switched dialog acts. to this end, our contributions are the following: ## related work response generation is one of critical components in tod systems. extensive works have proposed to enhance response quality with rnns @xcite , large pretrained models @xcite , augmentation @xcite or new learning objectives @xcite . they are either dealing with monolingual data or still require large amounts of annotated data which cannot allow few-shot foreign language generation -a vast majority of existing multilingual systems mostly consider language-agnostic task outputs like semantic parsing or ignore real codeswitched sentences in real cases @xcite . instead, deltalm @xcite pretrains interleaved multilingual decoders for text summarization and question generation and csrl @xcite learns language-agnostic structure-aware representations for semantic role labeling. often, due to the high cost of collecting low-resource taskoriented dialog annotations, data-based @xcite @xcite and model-based transfer approaches @xcite are popular to take advantage of high-resource language corpus for cross-lingual transfer. nevertheless, few-shot response generation is yet largely unexplored to induce cross-lingual transfer. the most related prior work is @xcite which extends the seq2seq models for response generation with private and local memory to accommodate new languages, which nevertheless cannot learn good memory modules when language data is highly imbalanced. our work continues to explore the possibility of cross-lingual response generation with large seq2seq models under low-resource language constraint more effectively.