Continual Learning Long Short Term Memory (2020)

Abstract: Catastrophic forgetting in neural networks indicates the performance decreasing of deep learning models on previous tasks while learning new tasks. To address this problem, we propose a novel Continual Learning Long Short Term Memory (CL-LSTM) cell in Recurrent Neural Network (RNN) in this paper. CL-LSTM considers not only the state of each individual task’s output gates but also the correlation of the states between tasks, so that the deep learning models can incrementally learn new tasks without catastrophically forgetting previously tasks. Experimental results demonstrate significant improvements of CL-LSTM over state-of-the-art approaches on spoken language understanding (SLU) tasks.

## introduction the whole ai community has enjoyed a superior performance boost from the emerging of deep learning technologies, thanks to the availability of big data and computing technologies. one of the most recent, realistic and emerged challenges for deep learning models on streaming data is continual learning capability. when new data is available, re-training brand new models with all the old and new data is the ideal way to achieve high performance on both tasks. however, there are several factors preventing saving old data for the entire lifetime, such as the memory restriction and data governance. when learning without all the old data, the performance on old tasks will drop dramatically, this phenomenon is called catastrophic forgetting @xcite . catastrophic forgetting occurs in neural networks due to the stability-plasticity dilemma @xcite , where the network requires sufficient plasticity to capture new tasks, but large weights variations may disrupt previous learned representations. continual learning methods are proposed to prevent catastrophic forgetting, when only a limited size of old data is available. several approaches have been proposed to solve this problem in deep learning field @xcite @xcite @xcite @xcite . a popular trend is to use expandable networks to store/learn old/new knowledge then acquire a task id to select one from all the tasks during the inference stage. @xcite @xcite . in contrast, only a few attempts have been made to address catastrophic forgetting in natural language forgetting (nlp) field. elastic weight consolidation (ewc) @xcite has been adapted to visual question answering @xcite and language modeling @xcite . progressive neural network proposed in reinforcement learning @xcite has been adopted to semantic slot filling in @xcite . a continual learning architecture preventing catastrophic forgetting via block-sparsity and orthogonality constraints is presented in @xcite on diverse sentence-pair classification tasks. to our best knowledge, none of the previous works in nlp considers the interactions between tasks at the lstm cell level. moreover, the requirement of task ids in inference is infeasible and impractical in the real scenarios as shown in fig. 1 . therefore, a novel continual learning long short term memory (cl-lstm) cell is proposed to prevent catastrophic forgetting. the contributions of the paper are: (a) a novel lstm cell for continual learning is proposed. the proposed cl-lstm includes separate modules for different tasks; (b) each task further has a broadcast module to send its hidden states to all of the old tasks, and a collect module to take hidden states as inputs from all of the old tasks. therefore, the output gates of each task integrates information from all tasks; (c) the proposed model doesn't require task ids to perform inference, which is more practical in real-world scenarios. we evaluate the proposed cl-lstm on both slot filling and intent detection of spoken language understanding. experimental results show that the proposed cl-lstm outperforms state-of-the-arts by a large margin. code is available at https: //github.com/ibm-gcdo/emnlp-cl-lstm. figure 2 : cl-lstm with three tasks. for the third task, old modules are frozen (grey) and m 3 , m c 3 , m b 3 (yellow) are trained for information sharing. h specifically, it incorporates a gating mechanism, taking the form: the outputs c t and h t can be obtained from: ## experiments in this section, cl-lstm is evaluated on spoken language understanding (slu) tasks in continual learning framework. slu mainly includes two goals: slot filling and intent detection. slot filling is a sequence labelling problem which maps each sentence to a sequence of slot labels with the same length, while intent detection is a classification problem where each sentence has one intent label. ## conclusion in this paper, we propose a novel cl-lstm cell to alleviate catastrophic forgetting problem in continual learning frameworks. experimental results have demonstrated that adding broadcast and collect modules can help keeping old knowledge as well as learning new knowledge. superior performance is achieved by cl-lstm over other related works on spoken language understanding tasks.