F - MALLOC : Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation (2024)

Abstract: In the evolving landscape of Neural Machine Translation (NMT), the pretrain-then-finetune paradigm has yielded impressive results. However, the persistent challenge of Catastrophic Forgetting (CF) remains a hurdle. While previous work has introduced Continual Learning (CL) methods to address CF, these approaches grapple with the delicate balance between avoiding forgetting and maintaining system extensibility. To address this, we propose a CL method, named \textbf{F-MALLOC} ( \textbf{F} eed-forward \textbf{M} emory \textbf{ALLOC} ation). F-MALLOC is inspired by recent insights highlighting that feed-forward layers emulate neural memories and encapsulate crucial translation knowledge. It decomposes feed-forward layers into discrete memory cells and allocates these memories to different tasks. By learning to allocate and safeguard these memories, our method effectively alleviates CF while ensuring robust extendability. Besides, we propose a comprehensive assessment protocol for multi-stage CL of NMT systems. Experiments conducted following this new protocol showcase the superior performance of F-MALLOC, evidenced by higher BLEU scores and almost zero forgetting.

## introduction in the pursuit of achieving state-of-the-art results in neural machine translation (nmt), the reliance on large-scale parallel corpora has been pivotal @xcite . however, practical application scenarios often present challenges, especially when translation is necessitated for specific domains with limited data resources @xcite . typically, the prevalent paradigm involves the initial pretraining of models on expansive general domain corpus, followed by finetuning for the target domain @xcite . despite the efficacy of this pretrain-then-finetune paradigm, it has been demonstrated that fine-tuning on the target domain can result in significant performance degradation in the general domain, a phenomenon known as catastrophic forgetting (cf) @xcite . in response to this challenge, various continual learning (cl) approaches have emerged to address cf in nmt systems. existing efforts primarily rely on regularization-based techniques to constrain the divergence of model parameters from their previous values @xcite @xcite . while these methods are mathematically elegant, they still face challenges related to forgetting. alternatively, some approaches take an architecturebased framework, isolating parameters specific to different tasks to prevent forgetting @xcite @xcite . however, they require prior information on task numbers to allocate parameters and rely on external storage of model or mask matrices, limiting its extendibility and applicability. in summary, the demand for a cl method for nmt systems that is both extendable and effective in preventing forgetting is pressing. to this end, we introduce a new cl method termed f-malloc (feed-forward memory allocation), which is inspired by recent insights that feed-forward layers emulate neural memories and encapsulate crucial translation knowledge @xcite . therefore, we facilitate new knowledge learning and mitigate cf by allocating and protecting these memories. f-malloc first leverages a structural pruning method to trim the feedforward layers of a pretrained nmt model, preserving memories that encapsulate crucial general domain knowledge. subsequently, f-malloc proceeds to learn a set of non-exclusive task masks @xcite , automatically allocating the 'writable' memory capacity to upcoming tasks. the memories allocated in this manner are then designated as 'read-only'. f-malloc strategically blocks gradient flows through these 'read-only' memories, effectively mitigating the risk of forgetting. meanwhile, conventional cl evaluation protocols in the nmt area typically focus on a single stage of training, lacking a holistic perspective over multiple stages. therefore, we introduce a comprehensive evaluation protocol for multi-stage cl in the nmt scenario. our protocol incorporates metrics assessing forgetting mitigation and adaptation to novel tasks. to enhance robustness, we conduct tests with random task sequences, reducing biases from specific orders. this protocol provides a nuanced understanding of f-malloc and competing methods' performance over time in nmt. experiments conducted following the proposed protocol highlight the superior performance of f-malloc with high robustness. additional analysis of f-malloc's memory allocation strategy reveals its effective utilization of task information, such as inherent difficulty or inter-task similarities, resulting in enhanced performance. in summary, the contributions of this paper are as follows: ## experiments in our multi-stage cl experiments for nmt systems, we finetune a pretrained general domain model on t new domains successively foot_1 . the pretrained model is based on the wmt'19 german-english news translation task winner @xcite . to neutralize the impact of task order, we randomly generate five task order sequences and report the average result. ## results and analysis regrading the non-cl baselines, although still trailing behind the upper bound performance, f-malloc demonstrates comparable performance to the strong baseline method, adapter. these results collectively underscore the effectiveness of f-malloc in continual learning scenarios for transformer-based neural machine translation (nmt) systems. for a more comprehensive analysis and comparison of various cl methods, the following subsections will use task sequence 0: @xmath0. ## conclusion this paper introduces f-malloc, a pioneering method for cl in nmt systems. by decomposing feed-forward layers into memory cells and implementing a strategic memory allocation approach, f-malloc proves effective in simultaneously enhancing new knowledge acquisition and alleviating forgetting. evaluation with a specialized protocol for cl in nmt, positions f-malloc as a superior performer, showcasing substantial improvements, robustness, and extensibility compared to existing approaches. the method's ability to leverage task difficulty and inter-task similari-ties for enhanced performance represents a significant advancement not seen in previous methods. f-malloc not only contribute to the field of cl in nmt but also pave the way for more efficient and adaptable neural network architectures.