Continual Learning for Natural Language Generations with Transformer Calibration (2022)

Abstract: Conventional natural language process (NLP) generation models are trained offline with a given dataset for a particular task, which is referred to as isolated learning. Research on sequence-to-sequence language generation aims to study continual learning model to constantly learning from sequentially encountered tasks. However, continual learning studies often suffer from catastrophic forgetting, a persistent challenge for lifelong learning. In this paper, we present a novel NLP transformer model that attempts to mitigate catastrophic forgetting in online continual learning from a new perspective, i.e., attention calibration. We model the attention in the transformer as a calibrated unit in a general formulation, where the attention calibration could give benefits to balance the stability and plasticity of continual learning algorithms through influencing both their forward inference path and backward optimization path. Our empirical experiments, paraphrase generation and dialog response generation, demonstrate that this work outperforms state-of-the-art models by a considerable margin and effectively mitigate the forgetting.

## introduction sequence-to-sequence (seq2seq) generation has been widely applied in artificial learning (ai) system to deal with various challenging tasks, e.g., paraphrase, dialogue system @xcite , machine translation, etc. in addition, powerful representation learning (e.g., transformer) have been used in seq2seq models, which have taken the state-of-the-art of generation models to a new level. generally, nature language generation (nlg) models leverage an encoder to create a vector representation for source inputs, and then pass this representation into a decoder so as to output a target sequence word by word. for example, bart @xcite is such a transformer-based nlg architecture that is equipped with the bert-type net-work structure @xcite as its encoder and with the gpt-type structure as the decoder. despite the remarkable ability on sequence generation, the conventional paradigm aims to learn a seq2seq model on the whole available dataset, which limits its ability in accumulating knowledge in continual learning scenario. when switching to a new task from some previously learned ones, the fine-tuned model on the new task sometimes faces a significant performance drop on previous learned data, where such a phenomenon is also referred to as catastrophic forgetting @xcite @xcite . in contrast, humans and animals exhibit remarkable ability to deal with new tasks by effectively adapting their acquired knowledge without forgetting the previously learned skills. if one desires to build a human-like nlg model, continual learning ability is a necessary skill for achieving this goal. the existing replay-based continual learning approaches have taken into account of different perspectives of the model training process to remedy the catastrophic forgetting dilemma, such as regularizing the parameter change during training @xcite , selective memory storage or replay @xcite , bayesian and variational bayesian training @xcite , and task-specific parameterization of the model @xcite . in this paper,we tackle the problem from a novel angle that is distinct to all the aforementioned attempts, i.e., seeking a better balance between stability and plasticity with neuron calibration. specifically, we refer to neuron calibration as a process of mathematically adjusting the transformation functions in various layers of transformer-based architecture. in this way, the neuron calibration is able to prioritize both model parameter and feature map that are suitable to new tasks. in detail, our proposed neuron calibration approach regularizes the param-eter update against catastrophic forgetting via posing a trainable soft mask on the attention and feature maps, which then influences both the model inference process and the model training process through the forward inference path and the backward optimization path. the contributions of our work are three-fold: (i) we introduce a general and light-weight feature calibration approach to tackle task-incremental continual learning problems where the models are formulated as feed-forward transformer-based function approximations; (ii) we formulate a novel task-incremental learning paradigm to train the calibrated model with an interleaved optimization scheme to mitigate the forgetting issue; (iii) we indicate through extensive empirical experiments that the proposed method could outperform the recent continual learning algorithms on seq2seq language generation applications. ## empirical experiments we evaluated the proposed algorithm on seq2seq generation tasks. we applied the algorithms on two datasets for seq2seq generation tasks in the continual learning. we also conducted the ablation study with respect to attention calibration and feature calibration to evaluate the robustness and effectiveness of the proposed calibration techniques. ## conclusions we propose an efficient seq2seq generation model with the calibration on the transformer, where a fixed architecture network after calibration can dynamically adjust the function with respect to each individual task. to optimize our method, we further propose a reproductive learning equipped with an iterative optimization objective that trade-off between plasticity and stability. moreover, our calibration module is very light-weight without introducing any task-specific parameters. extensive empirical experiments indicate that our approach outperforms the baselines and achieves a promising result. we also indicate that the calibration module and interleaved optimization play a vital role to boost the performance. finally, extending the calibration module to multi-lingual pre-trained model is a promising future research direction.