Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding (2021)

Abstract: Domain classification is the fundamental task in natural language understanding (NLU), which often requires fast accommodation to new emerging domains. This constraint makes it impossible to retrain all previous domains, even if they are accessible to the new model. Most existing continual learning approaches suffer from low accuracy and performance fluctuation, especially when the distributions of old and new data are significantly different. In fact, the key real-world problem is not the absence of old data, but the inefficiency to retrain the model with the whole old dataset. Is it potential to utilize some old data to yield high accuracy and maintain stable performance, while at the same time, without introducing extra hyperparameters? In this paper, we proposed a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. Specifically, we utilize Fisher information to select exemplars that can “record” key information of the original model. Also, a novel scheme called dynamical weight consolidation is proposed to enable hyperparameter-free learning during the retrain process. Extensive experiments demonstrate baselines provide fluctuated performance which makes them useless in practice. On the contrary, our proposed model significantly and consistently outperforms the best state-of-the-art method by up to 20% in average accuracy, and each of its component contributes effectively to overall performance.

## introduction catastrophic forgetting is the well-known achilles' heel of deep neural networks, that the knowledge learned from previous tasks will be forgotten when the networks are retrained to adapt to new tasks. although this phenomenon has been noticed as early as the birth of neural networks @xcite , it didn't attract much attention until deep neural networks have achieved impressive performance gains in various applications @xcite . domain classification is the task that mapping the spoken utterances to natural language understanding domains. it is widely used in intelligent personal digital assistant (ipda) devices, such as amazon alexa, google assistant, and microsoft cortana. as many ipdas now allow third-party developers to build and integrate new domains @xcite , these devices are eager to continual learning technologies that can achieve high performance stably @xcite . however, most traditional ipads only work with wellseparated domains built by specialists @xcite or customized designed for specific datasets @xcite . there is still a lack of continual learning methods that capable of general domain classification. most previous approaches capable of continual learning focus on the scenario that the new model should be retrained without any access to old data @xcite @xcite . however, these methods often involve more parameters that require extensive efforts in expert tuning. and when data distributions of new tasks are obviously different from the original task (e.g. class-incremental learning), these approaches can hardly maintain good accuracy for both tasks and may suffer from fluctuations in performance. on the other hand, old data are not unavailable in many practical cases. the main concerns arise from the tremendous cost in memory and computation resources, if the model is updated with huge previous datasets. also, most continual learning approaches are applied to image data that little attention has been paid to text data. is it possible to develop a desirable model capable of continual learning that satisfies the following qualities? 1) high accuracy with limited old data. compared to the extreme cases that no access or full access to old data, it is more practical to put models under the setting with a limited amount of old data available (e.g., no more than 10% of original data). in this case, models can achieve good performance without too much additional cost in physical resources and can be conveniently renewed with periodical system updates. 2) high stability with zero extra parameters. many continual learning models can perform well only under restricted experiment settings, such as specific datasets or carefully chosen parameters. however, practical datasets are usually noisy and imbalanced distributed, and inexperienced users can't set suitable parameters in real-world applications. therefore, it is desirable to develop a hyperparameter-free model that can work stably under various experimental environments. to achieve these goals, we proposed a continous learning model based on weight consolidation and fisher information sampling (ccfi), with application to domain classification. the main challenge is how to "remember" information from original tasks, not only the representative features from data, but also the learned parameters of the model itself. this is a non-trivial contribution since "uncontrollable changes" will happen to neural network parameters whenever the feature changes. to avoid such "uncontrollable changes", previous work icarl even discards deep neural networks as its final classifier and turns to k-nearest neighbors algorithm for actual prediction @xcite . our work demonstrates that these changes are "controllable" with exemplar selected by fisher information and parameters learned by dynamical weight consolidation. our contributions can be summarized as follows. ## related work although most of the existing approaches are not directly applicable to our problem, several main branches of research related to this work can be found: exemplar selection, regularization of weights, and feature extraction or fine-tune method based on pre-trained models. our problem is closest to the setting of exemplar selection methods @xcite . these approaches store examples from original tasks, and then combine them together with the new domain data for retraining. icarl @xcite discards the classifier based on neural network to prevent the catastrophic forgetting, and turns to traditional k-nearest neighbor classifier. to avoid large changes of important parameters, regularization models @xcite @xcite add constraints to the loss function. they usually introduce extra parameters requiring careful initialization. and it has been shown that their performance will drop significantly if the new tasks are drawn from different distributions @xcite . on the contrary, our proposed ccfi is a parameterfree model that can produce stable performance under various experimental environments. feature extraction methods utilize pre-trained neural networks to calculate features of input data @xcite . they make little modifications to the original network but often result in a limited capacity for learning new tasks. fine-tuning models @xcite can modify all the parameters in order to achieve better performance in new tasks. although starting with a small learning rate can indirectly preserve the knowledge learned from the original task, fine-tuning method will eventually tend to new tasks. adapter tuning @xcite can be viewed as the hybrid of fine-tune and feature extraction. unlike our model that makes no changes to the backbone model, the adapter tuning method increases the original model size and introduces more parameters by inserting adapter modules to each layer. ## our ccfi model given data stream @xmath0, y i } n @xmath1, the classification tasks in deep learning neural networks are equal to find the best parameter set ⇥ that can maximize the probability of the data p(d|⇥). namely, the classifier can make predictions ŷ that best reproduce the ground truth labels y . under the continual learning setting, new data d n of additional classes will be added to the original data stream d o in an incremental form. our goal is to update the old parameters ⇥ o (trained on original data stream d o ) to the new parameters ⇥ n , which can work well on both new data d n and old data d o . in this paper, the initial model is trained with the original data set d o , and will output the trained model ⇥ o . in this training process, fisher information sampling is conducted to select the most representative examples that can help to "remember" the parameters of the initial trained model. in the retraining process, the renewed model is learned based on dynamical weight consolidation, and evaluated on the training set consisted of new classes and the old exemplars. ## conclusion this paper proposes a hyperparameter-free model called ccfi for continuous domain classification. ccfi can record information of old models via exemplars selected by fisher information sampling, and conduct efficient retraining through dynamical weight consolidation. the comparison against the existing models reveals ccfi is the best performer under various experimental environments, without additional efforts in hyperparameter searching.