Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers (2021)

Abstract: This paper investigates continual learning for semantic parsing. In this setting, a neural semantic parser learns tasks sequentially without accessing full training data from previous tasks. Direct application of the SOTA continual learning algorithms to this problem fails to achieve comparable performance with re-training models with all seen tasks because they have not considered the special properties of structured outputs yielded by semantic parsers. Therefore, we propose TotalRecall, a continual learning method designed for neural semantic parsers from two aspects: i) a sampling method for memory replay that diversifies logical form templates and balances distributions of parse actions in a memory; ii) a two-stage training method that significantly improves generalization capability of the parsers across tasks. We conduct extensive experiments to study the research problems involved in continual semantic parsing and demonstrate that a neural semantic parser trained with TotalRecall achieves superior performance than the one trained directly with the SOTA continual learning algorithms and achieve a 3-6 times speedup compared to re-training from scratch.

## introduction in the recent market research report published by marketsandmarkets (inc., 2020), it is estimated that the smart speaker market is expected to grow from usd 7.1 billion in 2020 to usd 15.6 billion by 2025. commercial smart speakers, such as alexa and google assistant, often need to translate users' commands and questions into actions. therefore, semantic parsers are widely adopted in dialogue systems to map natural language (nl) utterances to executable programs or logical forms * corresponding author (lfs) @xcite . due to the increasing popularity of such speakers, software developers have implemented a large volume of skills for them and the number of new skills grow quickly every year. for example, as of 2020, the number of alexa skills exceeds 100,000 and 24 new skills are introduced per day in 2020 @xcite . although machine learning-based semantic parsers achieve the state-of-the-art performance, they face the following challenges due to the fast growing number of tasks. given new tasks, one common practice is to retrain the parser from scratch on the training data of all seen tasks. however, it is both economically and computationally expensive to re-train semantic parsers because of a fast-growing number of new tasks @xcite . to achieve its optimal performance, training a deep model on all 8 tasks of nlmap @xcite takes approximately 14 times longer than training the same model on one of those tasks. in practice, the cost of repeated re-training for a commercial smart speaker is much higher, e.g. alexa needs to cope with the number of tasks which is over 10,000 times more than the one in nlmap foot_0 . in contrast, continual learning provides an alternative cost-effective training paradigm, which learns tasks sequentially without accessing full training data from the previous tasks, such that the computational resources are utilized only for new tasks. privacy leakage has gradually become a major concern in many artificial intelligence (ai) applications. as most computing environments are not 100% safe, it is not desirable to always keep a copy of the training data including identifiable personal information. thus, it is almost not feasible to assume that complete training data of all known tasks is always available for re-training a semantic parser @xcite . for the semantic parser of a privacy-sensitive ai system, e.g. personalized social robot, continual learning provides a solution to maintain the knowledge of all learned tasks when the complete training data of those data is not available anymore due to security reasons. a major challenge of continual learning lies in catastrophic forgetting that the (deep) models easily forget the knowledge learned in the previous tasks when they learn new tasks @xcite . another challenge is to learn what kind of knowledge the tasks share in common and support fast adaptation of models for new tasks. methods are developed to mitigate catastrophic forgetting @xcite and facilitate forward knowledge transfer @xcite . instead of directly measuring speedup of training, those methods assume that there is a small fixed-size memory available for storing training examples or parameters from the previous tasks. the memory limits the size of training data thus proportionally reduces training time. however, we empirically found that direct application of those methods to neural semantic parsers leads to a significant drop of test performance on benchmark datasets, in comparison to re-training them with all available tasks each time. in this work, we investigated the applicability of existing continual learning methods to semantic parsing in-depth, and we have found that most methods have not considered the special properties of structured outputs, which distinct semantic parsing from the multi-class classification problem. therefore, we propose total recall (tr), a continual learning method that is especially designed to address the semantic parsing specific problems from two perspectives. first, we customize the sampling algorithm for memory replay, which stores a small sample of examples from each previous task when continually learning new tasks. the corresponding sampling algorithm, called diversified logical form selection (dlfs), diversifies lf templates and maximizes the entropy of the parse action distribution in a memory. second, motivated by findings from cognitive neuroscience @xcite , we facilitate knowledge transfer between tasks by proposing a two-stage training procedure, called fast slow continual learning (fscl). it updates only unseen action embeddings in the fast-learning stage and updates all model parameters in the follow-up stage. as a result, it significantly improves generalization capability of parsing models. our key contributions are as follows: ## base parser a semantic parser learns a mapping π θ : @xmath0. most sota neural semantic parsers formulate this task as translating a word sequence into an output sequence, whereby an output sequence is either a sequence of lf tokens or a sequence of parse ac-tions that construct an lf. for a fair comparison between different continual learning algorithms, we adopt the same base model for them, as commonly used in prior works @xcite @xcite . similar to @xcite , the base parser converts the utterance x into a sequence of actions @xmath1, ..., a t }. as an lf can be equivalently parsed into an abstract syntax tree (ast), the actions a sequentially construct an ast deterministically in the depth-first order, wherein each action a t at time step t either i) expands an intermediate node according to the production rules from a grammar, or ii) generates a leaf node. as in @xcite , the idioms (frequently occurred ast fragments) are collapsed into single units. the ast is further mapped back to the target lf. the parser employs the attention-based sequence-to-sequence (seq2seq) architecture @xcite for estimating action probabilities. encoder. the encoder in seq2seq is a standard bidirectional long short-term memory (lstm) network @xcite , which encodes an utterance x into a sequence of contextual word representations. decoder. the decoder applies an lstm to generate action sequences. at time t, the decoder produces an action representation s t , which is yielded by concatenating the hidden representation h t produced by the lstm and the context vector o t produced by the soft attention @xcite . we maintain an embedding for each action in the embedding table. the probability of an action a t is estimated by: ## continual semantic parsing problem formulation. we consider a widely adopted continual learning setting that a parser π θ is trained continually on a sequence of k distinct tasks {t (1) , t (2) ,...,t (k) } @xcite . in both training and testing, we know which task an example belongs to. as the definition of tasks is application specific and parallel data of semantic parsing is often created by domain experts, it is easy to identify the task of an example in practice. we further assume that there is a fixed-size memory m k associated with each task t (k) for e.g. storing a small amount of replay instances, as adopted in @xcite . this setting is practical for personalized conversational agents because it is difficult for them to re-collect past information except reusing the ones in the memories. ## conclusion we conducted the first in-depth empirical study to investigate continual learning for semantic parsing. to cope with the catastrophic forgetting and facilitate knowledge transfer between tasks, we propose total recall, consisting of a sampling method specifically designed for semantic parsing and a two-stage training method implementing an inductive bias for continual learning. the resulted parser achieves superior performance over the existing baselines on three benchmark settings. the ablation studies also demonstrate why it is effective.