Improving Zero-shot Cross-lingual Dialogue State Tracking via Contrastive Learning (2023)

Abstract: “Recent works in dialogue state tracking (DST) focus on a handful of languages, as collectinglarge-scale manually annotated data in different languages is expensive. Existing models addressthis issue by code-switched data augmentation or intermediate fine-tuning of multilingual pre-trained models. However, these models can only perform implicit alignment across languages. In this paper, we propose a novel model named Contrastive Learning for Cross-Lingual DST(CLCL-DST) to enhance zero-shot cross-lingual adaptation. Specifically, we use a self-builtbilingual dictionary for lexical substitution to construct multilingual views of the same utterance. Then our approach leverages fine-grained contrastive learning to encourage representations ofspecific slot tokens in different views to be more similar than negative example pairs. By thismeans, CLCL-DST aligns similar words across languages into a more refined language-invariantspace. In addition, CLCL-DST uses a significance-based keyword extraction approach to selecttask-related words to build the bilingual dictionary for better cross-lingual positive examples. Experiment results on Multilingual WoZ 2.0 and parallel MultiWoZ 2.1 datasets show that ourproposed CLCL-DST outperforms existing state-of-the-art methods by a large margin, demon-strating the effectiveness of CLCL-DST.”

## introduction dialogue state tracking is an essential part of task-oriented dialogue systems @xcite , which aims to extract user goals or intentions throughout a dialogue process and encode them into a compact set of dialogue states, i.e., a set of slot-value pairs. in recent years, dst models have achieved impressive success with adequate training data. however, most models are restricted to monolingual scenarios since collecting and annotating task-oriented dialogue data in different languages is time-consuming and costly @xcite . it is necessary to investigate how to migrate a high-performance dialogue state tracker to different languages when no annotated target language dialogue data are available. previous approaches are generally divided into the following three categories: (1) data augmentation methods with neural machine translation system @xcite . although translating dialogue corpora using machine translation is straightforward, it has inherent limitation of heavily depending on performance of machine translation. (2) pre-trained cross-lingual representation @xcite . the approach applies a cross-lingual pre-trained model, such as mbert @xcite , xlm @xcite and xlm-roberta (xlm-r) @xcite as one of the components of the dst architecture and then is trained with task data directly. however, the approach does not introduce cross-lingual information during the training process. (3) code-switched data augmentation @xcite @xcite . the method replaces words randomly from the source language to the target language with a bilingual dictionary as a way to achieve data augmentation. nevertheless, a synonym substitution with some meaningless words may introduce noise that impairs the semantic coherence of the sentence. besides, the model only use the code-switched corpus as the training data, ignoring the interaction between the original and code-switched sentences. consequently, these models can not sufficiently learn the semantic representation of the corpus. to address the above-mentioned issues, we propose a novel model named contrastive learning for cross-lingual dst (clcl-dst), which utilizes contrastive learning (cl) for cross-lingual adaptation. clcl-dst first captures comprehensive cross-lingual information from different perspectives and explores the consistency of multiple views through contrastive learning @xcite . simultaneously, as dialogue state tracking is to predict the state of slots in each turn of the dialogue, we consider it as a token-level task and then employ the same fine-grained cl. specifically, we obtain the encoded feature representation of each slot in the original sentence and the corresponding code-switched sentence from the multilingual pre-trained model, respectively. we then employ fine-grained cl to align the representations of slot tokens in different views. by introducing cl, our model is able to distinguish between the code-switched utterance and a set of negative samples, thus encouraging representations of similar words in different languages to align into a language-invariant feature space (subsection 3.1). furthermore, clcl-dst introduces a significance-based keyword extraction approach to obtain taskrelated keywords with high significance scores in different domains. for example, in the price range domain, some words like "cheap", "moderate" and "expensive" are more likely to have higher significance scores than background words, such as "a", "is" and "do". specifically, our approach obtains the semantic representation of sentences and corresponding subwords by encoder. then the approach gets the significance scores of the words by calculating the cosine similarity and get the keywords of the dataset based on the scores. we then replace these keywords with the corresponding words in the target language to generate multilingual code-switched data pairs. these code-switched keywords can be considered as cross-lingual views sharing the same meaning, allowing the shared encoder to learn some direct bundles of meaning in different languages. thus, our keyword extraction approach facilitates the transfer of cross-lingual information and strengthens the ties across different languages (subsection 3.2). we evaluate our model on two benchmark datasets. for the multilingual woz 2.0 dataset @xcite which is single-domain, our model outperforms the existing state-of-the-art model by 4.1% and 4.8% slot accuracy for german (de) and italian (it) under the zero-shot setting, respectively. for the parallel multiwoz 2.1 dataset @xcite which is multi-domain, our method outperforms the current state-of-the-art by 22% and 38.7% in joint goal accuracy and slot f1 for chinese (zh), respectively. moreover, further experiments show that introducing fine-grained cl performs better than coarse-grained cl. we also investigate the impact of different keyword extraction approaches on the model to demonstrate the superiority of our extraction approach. our main contributions can be summarized as follows: ## methodology in this section, we set up the notations that run throughout the paper first, before describing our clcl-dst model which explicitly uses contrastive learning to achieve cross-lingual alignment in dialogue state tracking. then, we introduce a significance-based code-switching approach on how to select taskrelated keywords in the utterance and code-switch the input sentence dynamically in detail. the main c c l 2 0 2 3 architecture of our model is illustrated in figure 1. notation. suppose the dialogue has t turns. we define the dialogue utterance at turn t as d @xmath0; ⊕u t ⊕ [sep], where r t and u t (1 ≤ @xmath1. ⊕ denotes token concatenation, and the semicolon ; is a separation symbol, while [sep] marks the end boundary of the dialogue. besides, we represent the dialogue states as @xmath2, ..., b t }, where b @xmath3... ⊕ [slot] i ⊕ b i t denotes i states combination at the t-th turn. i is the total number of slots. the i-th slot-value pair b i t is defined as: ## ablation studies we conduct ablation experiments to explore the effect of fine-grained contrastive learning and the significance-based keyword extraction approach on the overall performance for the multilingual woz c c l 2 0 2 3 2.0 dataset. ## conclusion in this paper, we propose a novel zero-shot adaptation method clcl-dst for cross-lingual dialogue state tracking. our approach leverages fine-grained contrastive learning to explicitly align representations across languages. besides, we introduce the significance-based code-switching approach to replace taskrelevant words with target language for generating code-switched sentences on downstream tasks. our method obtains new state-of-the-art results on multilingual woz dataset and parallel multiwoz dataset, which demonstrates its effectiveness. in the future, we would investigate better training objectives for cross-lingual dst task, especially on multi-domain area, to further boost the dialogue system on multilingual scenarios. we would also explore better positive and negative samples when applying contrastive learning on dst task.