Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation (2024)

Abstract: Zero-shot dialogue state tracking (DST) seeks to enable dialogue systems to transition to unfamiliar domains without manual annotation or extensive retraining. Prior research has approached this objective by embedding prompts into language models (LMs). Common methodologies include integrating prompts at the input layer or introducing learnable variables at each transformer layer. Nonetheless, each strategy exhibits inherent limitations. Prompts integrated at the input layer risk underutilization, with their impact potentially diminishing across successive transformer layers. Conversely, the addition of learnable variables to each layer can complicate the training process and increase inference latency. To tackle the issues mentioned above, this paper proposes Dual Low-Rank Adaptation (DualLoRA), a plug-and-play architecture designed for zero-shot DST. DualLoRA incorporates two distinct Low-Rank Adaptation (LoRA) components, targeting both dialogue context processing and prompt optimization, to ensure the comprehensive influence of prompts throughout the transformer model layers. This is achieved without incurring additional inference latency, showcasing an efficient integration into existing architectures. Through rigorous evaluation on the MultiWOZ and SGD datasets, DualLoRA demonstrates notable improvements across multiple domains, outperforming traditional baseline methods in zero-shot settings.

## introduction task-oriented dialogue (tod) systems, designed for specific tasks like restaurant reservations or travel bookings, rely on dialogue state tracking (dst) to interpret user intents as slot-value pairs across dialogue turns, essential for managing multidomain conversations @xcite ; luo i am looking for a college type attraction. there are 18 colleges i have found, would you prefer 1 in town centre or in the west? i would like to visit on in town centre please. sure, we have thirteen options, 10 of which are free. may i suggest king's college, or hughes hall? okay, may i have their postcode, entrance fee, and phone number? sure, the post code to king's college is cb21st, the entrance fee is free, and phone number 01223331100. can you find an indian restaurant for me that is also in the town centre? figure 1 : example of a multi-domain dialogue from multiwoz dataset. @xcite . figure 1 shows the change of dialogue states in a multi-turn dialogue crossing multiple domains. an ideal system seamlessly transitions to new domains with minimal training, overcoming the resource-intensive process of data collection and annotation for new domains. zero-shot learning emerges as a solution, enabling tod systems to adapt to unseen domains by leveraging existing knowledge, thus bypassing the need for extensive domain-specific data. the advancement in language models (lms) @xcite has significantly enhanced the exploration of zero-shot learning approaches within the context of dialogue state tracking tasks. adaptation to novel domains is frequently achieved through the incorporation of prompts into transformer-based models @xcite . the current method can be classified into two categories. layer 1 layer n ... dialogue context prompt transformer (a) type i: incorporating prompts in the input layer. layer 1 layer n ... dialogue context prompt transformer prefix prefix (b) type ii: adding prefixes in every layer. layer 1 layer n ... dialogue context prompt transformer slot prompt (c) duallora: encoding prompts with a separate lora. dialogue context prompt dialogue context prompt (a) attention map in the first encoder layer. dialogue context prompt dialogue context prompt (b) attention map in the last encoder layer. e.g. slot description, in the first transformer layer @xcite @xcite . a common practice within this approach involves the concatenation of prompts subsequent to the input, as shown in fig 2a . despite the straightforward nature of their implementation, these strategies exhibit suboptimal utilization of prompts. this refers to the phenomenon where the influence of prompts on the model's output progressively diminishes with each additional transformer layer. figure 3 shows an example, where deep transformer layers, compared with shallow transformer layers, exhibit a reduced allocation of attention to other tokens within the dialogue context or prompt, in the token encoding process. this issue is further exacerbated in the context of dialogue state tracking, where the dialogue's progression causes a rapid escalation in the context length -often extending to 300-400 tokens or more, in stark contrast to the prompts' modest length of 10-20 tokens. such a diminishing effect of prompts can severely impair the system's performance in zero-shot learning scenarios, where the ability to discern and leverage domain-specific knowledge becomes critically important. the second category of methodologies incorpo-rates a learnable vector, initialized by prompts, into each layer of the transformer architecture @xcite , as shown in fig 2b . this technique facilitates the model's capacity to consistently engage with the prompt information across all layers, ensuring the prompt's considerations are integrated throughout the encoding process. however, these methods present certain limitations. direct concatenation of prompts at each layer might introduce extraneous noise, adversely affecting the efficacy of model training. furthermore, this method of concatenation can result in augmented inference latency of the model. consequently, devising a strategy to effectively utilize prompts without impeding the standard input-output dynamics of the model and without exacerbating inference time emerges as a critical challenge in the field. to address the aforementioned challenges, we propose a novel plug-and-play framework designed for zero-shot dialogue state tracking, termed dual low-rank adaptation (duallora). this architecture is characterized by the integration of two low-rank adaptation (lora) components: one dedicated to processing dialogue context and the other to refining prompts, as shown in fig 2c . dual-lora operates by receiving prompts and applying modifications within the attention layers of the transformer model, subsequently integrating this modified output with the transformer's original output. through the implementation of this framework across each layer, duallora ensures that the influence of the prompts is perpetuated throughout the entire depth of the model. notably, this architecture does not incur additional latency during the inference phase, maintaining efficiency in model performance. furthermore, duallora requires few parameter adjustments and demonstrates exceptional transferability across different domains. comprehensive experimental evaluations were performed utilizing the multiwoz and sgd datasets to assess the efficacy of the proposed methodology. comparative analyses reveal that our approach yields enhanced joint goal accuracy (jga) metrics across various domains within both the multiwoz and sgd datasets, relative to established baseline methodologies. the structure of this paper is organized as follows: section 2 offers an overview of the literature pertinent to this research area. section 3 delineates a detailed exposition of our proposed model. section 4 presents a detailed account of the experimental outcomes and the specifics of the implementation. section 5 encapsulates the conclusions derived from this study. ## dual low-rank adaptation this paper proposes duallora, a plug-and-play architecture for dialogue state tracking. duallora consists of a context lora and a prompt lora. the context lora is used to optimize the original dialogue context input, while the prompt lora is used to optimize the prompt input. both the context lora and prompt lora are based on lora. by employing two distinct loras, the task of zeroshot learning can be divided into two subtasks: tuning the original input and tuning the prompt input. additionally, to avoid impacting the model's inference latency, the parameters of context lora and prompt lora can be merged into the pre-trained parameters, which ensures the efficiency of dual-lora. ## conclusion in summary, this paper introduces duallora based on a parameter-efficient transfer-learning method for zero-shot dialogue state tracking. dual-lora consists of two parts, namely context lora and prompt lora. duallora can simultaneously accept two inputs: normal input and prompt input. through experiments, we demonstrate that this method improves the joint goal accuracy (jga) metric for multiwoz and sgd datasets without introducing inference latency. additionally, we conduct a series of experiments to show the effectiveness of deploying duallora and how to fuse two different types of information best. our analysis indicates that the effectiveness of duallora stems from the model's ability to extract and utilize prompt information efficiently. future work will focus on applying duallora to various domains, expanding the method to multimodal zero-shot dialogue state tracking, or researching better fusion methods for more effective integration of prompts and dialogue context.