Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System (2023)

Abstract: Task-Oriented Dialogue (TOD) systems are designed to carry out specific tasks by tracking dialogue states and generating appropriate responses to help users achieve defined goals. Recently, end-to-end dialogue models pre-trained based on large datasets have shown promising performance in the conversational system. However, they share the same parameters to train tasks of the dialogue system (NLU, DST, NLG), so debugging each task is challenging. Also, they require a lot of effort to fine-tune large parameters to create a task-oriented chatbot, making it difficult for non-experts to handle. Therefore, we intend to train relatively lightweight and fast models compared to PLM. In this paper, we propose an End-to-end TOD system with Task-Optimized Adapters which learn independently per task, adding only small number of parameters after fixed layers of pre-trained network. We also enhance the performance of the DST and NLG modules through reinforcement learning, overcoming the learning curve that has lacked at the adapter learning and enabling the natural and consistent response generation that is appropriate for the goal. Our method is a model-agnostic approach and does not require prompt-tuning as only input data without a prompt. As results of the experiment, our method shows competitive performance on the MultiWOZ benchmark compared to the existing end-to-end models. In particular, we attain state-of-the-art performance on the DST task of 2.2 dataset.

## introduction task-oriented dialogue systems are trained to achieve specific goal to enhance efficiency and convenience in various fields such as customer service centers and healthcare information retrieval. task-oriented dialogue systems are divided into key components: understanding the user's intent (nlu), tracking the current dialogue states (dst), and generating responses based on previous sessions @xcite . pipeline-based systems separately train each component, so they have the advantage of optimizing each module and raising the performance of a given task. user feedback is, however, difficult to propagate to each module, and inputs to the component is dependent on the result of the previous module @xcite . recently, dialogue systems have been trained in an end-toend manner with transfer learning or pre-training networks with large dialogue corpora. however, building efficient end-to-end tod systems requires a large amount of data and has some limitations due to parameter sharing. end-to-end models backpropagate to transfer the gradients of the output and end back to the entire neural network. they pose an issue of parameter efficiency as updating all parameters for every downstream scenario. also, it is challenging to debug each task and take task-flow characteristics into account. therefore, we propose a simple structure of adding adapters to the core modules (nlu, dst, nlg) of the tod system as shown in figure 1 . by using the adapters, it is possible to optimize each task with only a small amount of training parameters, remaining the pre-trained model's parameters fixed. additionally, it is safe from the catastrophic forgetting problem @xcite , which causes pre-trained models to lose important skills acquired during the pre-training process. the key is to apply a transfer learning strategy that yields compact and extensible downstream models in the dialogue system @xcite . this makes it easy for people to train large-scale end-to-end tod models. also, by applying reinforce @xcite , we attempt to reduce the expected score gap caused by the small parameters of adapter compared to the full fine-tuning. specifically, we use @xcite i need a place to dine in the center that s expensive. ## background pipeline-based task-oriented dialogue system conventional task-oriented dialogue systems usually based on the pipeline method, consisted of language understanding (nlu), dialogue state tracking (dst), policy learning (pol), and language generation (nlg). this kind of modularization allows for each component to be optimized independently, making it easier to update and understand how the model is working. pipeline-based systems, however, have several limitations. each of the modules train sequentially, so it is hard to align modules to the common optimization targets @xcite . this makes the system more complex and harder to backpropagate cumulated errors. the performance of the previous components affects the next modules, so if upper modules perform poorly, errors that occurred earlier may propagate and be amplified in downstream components @xcite . end-to-end task-oriented dialogue system endto-end task-oriented dialogue systems, on the other hand, are easier to optimize and train to directly map the input to output in a single model. they can leverage large amounts of data for robust learning and the entire system can be optimized under end-to-end settings, which leads to better performance. a general approach for building end-to-end systems is to fine-tune pre-trained language models @xcite . this approach utilizes the strength of pre-trained networks, which help the models to leverage the pre-trained knowledge while also adapting to task-specific data. for example, simpletod (hosseini-asl et al., 2020) solved task-oriented dialogue as causal language modeling task using several versions of gpt (radford and narasimhan, 2018; @xcite . pre-training of dialogue language model recently, methods with pre-training dialogue language model @xcite @xcite @xcite , instead of fine-tuning pre-trained networks have outperformed the previous baselines on the benchmark. for instance, space-3 @xcite captures the contextualized knowledge from largescale dialogue corpora by pre-training the unified language model. however, there are still some issues with these methods. a large amount of parameters is required for training backbone models like bert, t5 @xcite , gpt, and unilm @xcite . as shown in adapter tuning for nlp since the pre-trained models for nlp tasks have become mainstream, they are mainly used for transfer learning downstream tasks. however, a parameter efficiency issue has been raised because updating all plm parameters is expensive for every downstream scenario. to address the issue, the adapter module is proposed to transfer plm like bert with parameterefficient tuning @xcite and shows comparable performance to full fine-tuning. the main idea behind the adapter is to train the network to the downstream task with task-specific parameters while maintaining the original pre-trained parameters. the module is composed of two feedforward layers and a non-linear layer, which can be inserted into the transformer blocks of an end-toend model. it projects d-dimensional input features into a smaller dimension m, and then projects back into the original dimension, so the total of parameters add per layer with biases is 2md + d + m. the number of additional parameters per task can be restricted by setting @xmath0. in most of the previous research, the adapter was used for efficient learning. but in this approach, we adapt the adapter modules not just for efficient learning ( stickland and murray, 2019 ), but also task-optimized learning as recent studies that have mainly focused on separating parameters @xcite @xcite . reinforcement learning for text generation although token-based supervised learning is a widely adopted training method in text generation tasks, as highlighted by ranzato et al., 2016, there are two major problems associated with this approach. the first problem is the exposure bias problem, where during training, the model is exposed to the ground-truth outputs, thus allowing it to learn to generate text that is similar to the training data. however, during evaluation, the model is not exposed to the ground-truth outputs and instead, generates texts based on the previous words generated by the model itself. this can lead to errors and a significant deviation in the generated text from the training data. the second problem is the tokenlevel loss problem, where the training objective function is based solely on individual words' prediction, neglecting the predicted token sequence's overall coherence and fluency. to address these problems, researchers have applied reinforcement learning to text generation tasks @xcite @xcite @xcite using metrics such as bleu @xcite or rouge @xcite as rewards for sequence-level training. in our study, we also use the reinforce method and task-oriented dialogue metrics to continually train our model after supervised learning, not only to address these challenges but also to mitigate performance degradation caused by the use of an adapter. nlg) in a task-oriented dialogue system. as illustrated in figure 2 , the adapters are inserted after the feed-forward layer following the multi-head attention layer of the transformer blocks of the end-to-end model. it enables the model to learn task-specific representations while preserving the shared parameters learned during pre-training. as described in the output of the j th feed-forward layer with residual connection in the transformer block is represented as h @xmath2, where n is the input dimension, and d is the hidden dimension. as shown in figure 2 , the overall architecture of the adapter module includes multiple feed-forward projections, referred to as down-projection and up-projection, followed by layer normalization (ln). the downprojection with w @xmath3, which is passed by the relu activation function, and then the up-projection with w @xmath4. the bottleneck dimension h is a hyperparameter to project the original input to a smaller dimension. and each adapter has a residual connection to avoid vanishing gradient @xcite . ## conclusion we propose toatod, task-optimized adapters for an end-to-end task dialogue system. by adapting task-optimized adapters, we utilize the end-to-end models without updating the pre-trained parameters and enabling debugging per task, which is different from previous research. in addition, we apply reinforce algorithm with metric-aware reward function directly not only on the nlg task but also dst task to prevent score degradation. as a result, we attain comparable performance to the previous sota models on every benchmark with very small number of trainable parameters. for the dst task of multiwoz2.2, our toatod model outperforms the current sota systems. 51.59. it is important to carefully choose the bottleneck dimension when using the adapter module in the task-oriented dialogue system. as described in the