Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System (2024)

Abstract: Intelligent task-oriented dialogue systems (ToDs) are expected to continuously acquire new knowledge, also known as Continual Learning (CL), which is crucial to fit ever-changing user needs. However, catastrophic forgetting dramatically degrades the model performance in face of a long streamed curriculum. In this paper, we aim to overcome the forgetting problem in ToDs and propose a method (HESIT) with hyper-gradient-based exemplar strategy, which samples influential exemplars for periodic retraining. Instead of unilaterally observing data or models, HESIT adopts a profound exemplar selection strategy that considers the general performance of the trained model when selecting exemplars for each task domain. Specifically, HESIT analyzes the training data influence by tracing their hyper-gradient in the optimization process. Furthermore, HESIT avoids estimating Hessian to make it compatible for ToDs with a large pre-trained model. Experimental results show that HESIT effectively alleviates catastrophic forgetting by exemplar selection, and achieves state-of-the-art performance on the largest CL benchmark of ToDs in terms of all metrics.

## introduction serving as a core technique of smart assistants, task-oriented dialogue systems (tods) are expected to continuously acquire new knowledge through time regarding user needs @xcite , e.g., adding fresh slot-value pairs or handling dissimilar tasks. this ability is also known as continual learning (cl) @xcite , which has recently attracted a surge of interest in nlp community @xcite , as well as other machine learning techniques @xcite . in this setting, the central problem of cl is catastrophic forgetting @xcite , as the data is streamed and the neural model inevitably forgets previously learned knowledge when fitting new training data in the sequential order. such a phenomenon is particularly conspicuous in tods tasks due to the obvious distributional shift between task scenarios, e.g., setting an alarm clock and booking a flight are completely irrelevant tasks. as a result, it is challenging for a single model that is learned from a unidirectional curriculum to simultaneously handle multi-domain tasks. to mitigate the problem of forgetting, existing efforts @xcite follow the three lines: (i) adding regularization terms to consolidate learned knowledge, (ii) developing dynamic architectures for the task-specific domain, and (iii) applying historical data rehearsal. previous works @xcite have demonstrated that regularization methods gradually lose effectiveness if faced with a long curriculum, and we confirm this belief with a relative experiment. dynamic architecture methods (e.g. adapter tuning) are usually considered as an unfair baseline @xcite @xcite due to multiple sets of parameters, and it requires a further step during inference to determine the correct parameters resulting in high latency for tods. therefore, this work focuses on rehearsal methods (fig. 1 (a) ) @xcite that store a small number of past samples (a.k.a. exemplars) in episodic memory and replay them periodically, which is simple but effective. then we raise our research question: which utterances are suitable to be selected as rehearsal exemplars for tods? although statistic methods (e.g., reservoir algorithm @xcite ) conduce to select better exemplars than random sampling, according to the accredited influence chain "data-model-performance" @xcite , an exemplar selection strategy should further consider the general performance of models during testing. in other words, we should select the exemplars that positively influence the performance of the trained network on unseen data, as shown in fig. 1 (b) . to explore the influence of training data from the performance perspective, one pioneering work uses the influence functions (ifs) based on the derivation chain rule @xcite to quantify the contribution of an individual training sample, by observing its impact on the test loss when removing it. though ifs have shown effectiveness in image classification tasks, this method requires estimating the inverse hessian matrix, leading to high computing costs @xcite and unstable results for large pre-trained models that are widely used in tods. furthermore, ifs only measure the contribution around the model parameters at the final training epoch, which fails to trace the data contribution in the dynamic optimization process @xcite . in this paper, we propose a rehearsal-based cl method, named hesit (hyper-gradient-based exemplar selection by influence tracing) to effectively overcome the catastrophic forgetting in tods. specifically, instead of only focusing on the model parameters of the final epoch, hesit traces and analyzes the hyper-gradient of training examples in the complicated optimization process. more importantly, its exemplar selection strategy roots in model performance on unseen data to measure the influence of these traced samples. in this way, influential and representative exemplars can be determined and stored, and then utilized to remind the model of learned knowledge. moreover, hesit is hessian-free to avoid the instability when estimating hessian matrix of a large pre-trained models, which has been widely used in tods. experimental results demonstrate that hesit achieves state-ofthe-art performance on the largest cl benchmark of tods @xcite including 37 domain tasks in terms of all metrics. furthermore, comparative experiments show that the exemplar selection in hesit surpasses all other selection strategies from other perspectives. the main contributions of this paper can be summarized as follows: continual learning in tods. early work on cl for tods was introduced by @xcite , where the elastic weight consolidation (ewc) method was utilized to alleviate catastrophic forgetting on 3 sequential domains. @xcite studied cl for dst sub-task on the multiwoz dataset, where several baselines have been compared. @xcite combined data rehearsal and ewc on nlg sub-task and expanded the learned knowledge to 13 task domains. furthermore, @xcite developed a large benchmark for all intent, dst, and nlg tasks, where mainstream cl methods (e.g., l2, agem, and adapter) were compared on a total of 37 domains. however, it shows that regularization-based methods lose effectiveness when tods encounter dozens of tasks @xcite . meanwhile, architecture-based methods require a further step to determine which set of parameters to use @xcite , thus blocking the real-time response from tods. in this work, we focus on rehearsal-based cl methods as it is simple and effective, with an acceptable cost of extra memory. ## conclusion in this paper, we propose hesit to address the catastrophic forgetting in tods. it selects exemplar for rehearsal-based cl methods that examines training examples in each domain from a performance perspective. furthermore, hesit tracks the hyper-gradient of training examples in an optimization method that is hessian-free and compatible with large pre-trained models. experiments show that hesit effectively overcomes catastrophic forgetting and delivers state-of-the-art performance on the largest tods benchmark for cl.