Orthogonal Subspace Learning for Language Model Continual Learning (2023)

Abstract: Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

## introduction learning tasks sequentially is crucial for developing real-world nlp models @xcite , as it enables continuous evolution when encountering new tasks or knowledge. although pre-trained models @xcite @xcite have achieved tremendous success on static tasks @xcite , learning multiple tasks sequentially, commonly referred to as continual learning, remains challenging @xcite . as a model learns new tasks, it tends to forget or lose the knowledge it had acquired for earlier tasks, leading to a phenomenon known as catastrophic forgetting @xcite . ## orthogonal low-rank adaptation in this section, we introduce o-lora, illustrated in figure 2 . first, we adopt instruction tuning as our training paradigm. then, we incrementally learn new tasks within an orthogonal subspace while keeping the lora parameters fixed for past tasks. lastly, we conduct a comparative analysis of our method compared to existing approaches. ## conclusion in this paper, we introduce o-lora, a novel approach that leverages orthogonal subspace learning for continual learning in language models. o-lora systematically addresses catastrophic forgetting by adopting an incremental learning strategy within orthogonal subspaces. distinguished by its data privacy considerations, efficient model parameter utilization, and robust generalization to novel tasks, our method stands out. empirical evaluations underscore o-lora's efficacy in tackling the intricacies of continual learning.