Original: I TERATIVE R ELAXING G RADIENT P ROJECTION FOR C ONTINUAL L EARNING
Revised: I TERATIVE R ELAXING G RADIENT P ROJECTION FOR C ONTINUAL L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A critical capability for intelligent systems is to continually learn given a sequence of tasks.
Revised: A critical capability for intelligent systems is to continually learn given a sequence of tasks.

Original: An ideal continual learner should be able to avoid catastrophic forgetting and effectively leverage past learned experiences to master new knowledge.
Revised: An ideal continual learner should be able to avoid catastrophic forgetting and effectively leverage past learned experiences to master new knowledge.

Original: Among different continual learning algorithms, gradient projection approaches impose hard constraints on the optimization space for new tasks to minimize task interference, yet hinder forward knowledge transfer at the same time.
Revised: Among different continual learning algorithms, gradient projection approaches impose hard constraints on the optimization space for new tasks to minimize task interference, yet hinder forward knowledge transfer at the same time.

Original: Recent methods use expansion-based techniques to relax the constraints, but a growing network can be computationally expensive.
Revised: Recent methods use expansion-based techniques to relax the constraints, but a growing network can be computationally expensive.

Original: Therefore, it remains a challenge whether we can improve forward knowledge transfer for gradient projection approaches using a fixed network architecture .
Revised: Therefore, it remains a challenge whether we can improve forward knowledge transfer for gradient projection approaches using a fixed network architecture .

Original: In this work, we propose the Iterative
Revised: In this work, we propose the Iterative

Original: Relaxing Gradient Projection (IRGP) framework.
Revised: Relaxing Gradient Projection (IRGP) framework.

Original: The basic idea is to iteratively search for the parameter subspaces most related to the current task and relax these parameters, then reuse the frozen spaces to facilitate forward knowledge transfer while consolidating previous knowledge.
Revised: The basic idea is to iteratively search for the parameter subspaces most related to the current task and relax these parameters, then reuse the frozen spaces to facilitate forward knowledge transfer while consolidating previous knowledge.

Original: Our framework requires neither memory buffers nor extra parameters.
Revised: Our framework requires neither memory buffers nor extra parameters.

Original: Extensive experiments have demonstrated the superiority of our framework over several strong baselines.
Revised: Extensive experiments have demonstrated the superiority of our framework over several strong baselines.

Original: We also provide theoretical guarantees for our iterative relaxing strategies.
Revised: We also provide theoretical guarantees for our iterative relaxing strategies.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A critical capability for intelligence systems is to continually learn given a sequence of tasks (Thrun & Mitchell, 1995; McCloskey & Cohen, 1989).
Revised: A critical capability for intelligence systems is to continually learn given a sequence of tasks (Thrun & Mitchell, 1995; McCloskey & Cohen, 1989).

Original: Unlike human beings, vanilla neural networks straightforwardly update parameters regarding current data distribution when learning new tasks, suffering from catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990; Kirkpatrick et al., 2017).
Revised: Unlike human beings, vanilla neural networks straightforwardly update parameters regarding current data distribution when learning new tasks, suffering from catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990; Kirkpatrick et al., 2017).

Original: As a result, continual learning is gaining increasing attention in recent years (Kurle et al., 2019; Ehret et al., 2020;
Revised: As a result, continual learning is gaining increasing attention in recent years (Kurle et al., 2019; Ehret et al., 2020;

Original: Ramesh & Chaudhari, 2021; Liu & Liu, 2022; Teng et al., 2022).
Revised: Ramesh & Chaudhari, 2021; Liu & Liu, 2022; Teng et al., 2022).

Original: An ideal continual learner is expected to not only avoid catastrophic forgetting but also facilitating forward knowledge transfer (Lopez-Paz & Ranzato, 2017), which is to leverage past learning experiences to master new knowledge efficiently and effectively (Parisi et al., 2019;
Revised: An ideal continual learner is expected to not only avoid catastrophic forgetting but also facilitating forward knowledge transfer (Lopez-Paz & Ranzato, 2017), which is to leverage past learning experiences to master new knowledge efficiently and effectively (Parisi et al., 2019;

Original: Finn et al., 2019).
Revised: Finn et al., 2019).

Original: Several types of methods have been proposed for continual learning.
Revised: Several types of methods have been proposed for continual learning.

Original: Replay-based methods (LopezPaz & Ranzato, 2017; Shin et al., 2017) alleviate catastrophic forgetting by storing some old samples in the memory as they are inaccessible when new tasks come, while expansion-based methods (Rusu et al., 2016; Yoon et al., 2017; 2019) expand the model structure to accommodate incoming knowledge.
Revised: Replay-based methods (LopezPaz & Ranzato, 2017; Shin et al., 2017) alleviate catastrophic forgetting by storing some old samples in the memory as they are inaccessible when new tasks come, while expansion-based methods (Rusu et al., 2016; Yoon et al., 2017; 2019) expand the model structure to accommodate incoming knowledge.

Original: However, these methods require either extra memory buffers (Parisi et al., 2019) or a growing network architecture as new tasks continually arrive (Kong et al., 2022), which always results in expensive computation costs (De Lange et al., 2021).
Revised: However, these methods require either extra memory buffers (Parisi et al., 2019) or a growing network architecture as new tasks continually arrive (Kong et al., 2022), which always results in expensive computation costs (De Lange et al., 2021).

Original: In order to maintain a fixed network capacity, regularization-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018) penalize the transformation of parameters regarding the corresponding plasticity via regularization terms.
Revised: In order to maintain a fixed network capacity, regularization-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018) penalize the transformation of parameters regarding the corresponding plasticity via regularization terms.

Original: While these regularization terms are applied to individual neurons, recent gradient projection methods (Zeng et al., 2019; Saha et al., 2021; Wang et al., 2021) modify the gradients in the feature space by constraining the directions of gradient update, which achieves outstanding performance.
Revised: While these regularization terms are applied to individual neurons, recent gradient projection methods (Zeng et al., 2019; Saha et al., 2021; Wang et al., 2021) modify the gradients in the feature space by constraining the directions of gradient update, which achieves outstanding performance.

Original: However, although gradient projection methods effectively mitigate forgetting within a fixed network capacity (Zeng et al., 2019), the capability of learning new tasks is hindered by the limited optimization space, resulting in insufficient forward knowledge transfer.
Revised: However, although gradient projection methods effectively mitigate forgetting within a fixed network capacity (Zeng et al., 2019), the capability of learning new tasks is hindered by the limited optimization space, resulting in insufficient forward knowledge transfer.

Original: In other words, constraining the directions of gradient update fails on the plasticity in the stability-plasticity dilemma (French, 1997).
Revised: In other words, constraining the directions of gradient update fails on the plasticity in the stability-plasticity dilemma (French, 1997).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Trust Region Gradient Projection (Lin et al., 2022) tackles this problem by expanding the selected subspace of old tasks as trust regions with scaled weight projection, similar to other expansion-based methods (Yoon et al., 2019).
Revised: Trust Region Gradient Projection (Lin et al., 2022) tackles this problem by expanding the selected subspace of old tasks as trust regions with scaled weight projection, similar to other expansion-based methods (Yoon et al., 2019).

Original: In spite of substantial improvement, these methods are computationally expensive as a result of growing network architecture (Wang et al., 2021).
Revised: In spite of substantial improvement, these methods are computationally expensive as a result of growing network architecture (Wang et al., 2021).

Original: Therefore, insufficient forward knowledge transfer remains a key challenge for gradient projection methods.
Revised: Therefore, insufficient forward knowledge transfer remains a key challenge for gradient projection methods.

Original: To address this challenge, we propose the Iterative Relaxing Gradient Projection (IRGP) framework to facilitate forward knowledge transfer within a fixed network capacity.
Revised: To address this challenge, we propose the Iterative Relaxing Gradient Projection (IRGP) framework to facilitate forward knowledge transfer within a fixed network capacity.

Original: We design a simple yet effective strategy to find the critical subspace within the frozen space.
Revised: We design a simple yet effective strategy to find the critical subspace within the frozen space.

Original: During the training phase, we iteratively reuse the parameters within the selected subspace.
Revised: During the training phase, we iteratively reuse the parameters within the selected subspace.

Original: Instead of strictly freezing those parameters, our method explores a larger optimization space, which allows better forward knowledge transfer and thus achieves better performance on new tasks.
Revised: Instead of strictly freezing those parameters, our method explores a larger optimization space, which allows better forward knowledge transfer and thus achieves better performance on new tasks.

Original: The procedure of our approach is illustrated in Figure 1.
Revised: The procedure of our approach is illustrated in Figure 1.

Original: Extensive experiments on various continual learning benchmarks demonstrate that our IRGP framework promotes forward knowledge transfer and achieves better classification performance compared with related state-of-the-art approaches.
Revised: Extensive experiments on various continual learning benchmarks demonstrate that our IRGP framework promotes forward knowledge transfer and achieves better classification performance compared with related state-of-the-art approaches.

Original: Moreover, our framework performs can also be extended as an expansion-based methods by storing the parameters of the selected relaxing subspace, universally surpassing TRGP (Lin et al., 2022) and other expansion-based approaches.
Revised: Moreover, our framework performs can also be extended as an expansion-based methods by storing the parameters of the selected relaxing subspace, universally surpassing TRGP (Lin et al., 2022) and other expansion-based approaches.

Original: We also provide theoretical proof to guarantee the efficiency of our relaxing strategy.
Revised: We also provide theoretical proof to guarantee the efficiency of our relaxing strategy.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we review the representative approaches for continual learning and briefly analyze their differences from our method.
Revised: In this section, we review the representative approaches for continual learning and briefly analyze their differences from our method.

Original: Conceptually, these approaches can be roughly divided into the following four categories.
Revised: Conceptually, these approaches can be roughly divided into the following four categories.

Original: Replay-based methods: These methods maintain a complementary memory for old samples, which are replayed during learning novel tasks.
Revised: Replay-based methods: These methods maintain a complementary memory for old samples, which are replayed during learning novel tasks.

Original: GEM (Lopez-Paz & Ranzato, 2017) constrains gradients concerning previous samples and Chaudhry et al.
Revised: GEM (Lopez-Paz & Ranzato, 2017) constrains gradients concerning previous samples and Chaudhry et al.

Original: (2018) further propose to estimate with random samples to accelerate.
Revised: (2018) further propose to estimate with random samples to accelerate.

Original: While past samples are commonly not accessible in the real world, auxiliary deep generative models are thus deployed to synthesize pseudo data (Chenshen et al., 2018;
Revised: While past samples are commonly not accessible in the real world, auxiliary deep generative models are thus deployed to synthesize pseudo data (Chenshen et al., 2018;

Original: Cong et al., 2020).
Revised: Cong et al., 2020).

Original: Recent approaches (PourKeshavarzi et al., 2021; Choi et al., 2021) leverage a single model for both classification and pseudo data generation.
Revised: Recent approaches (PourKeshavarzi et al., 2021; Choi et al., 2021) leverage a single model for both classification and pseudo data generation.

Original: However, including extra data into the current task introduces excessive training time (De Lange et al., 2021), especially on long task sequence.
Revised: However, including extra data into the current task introduces excessive training time (De Lange et al., 2021), especially on long task sequence.

Original: Our approach requires no previous data, in other words, is a replay-free method.
Revised: Our approach requires no previous data, in other words, is a replay-free method.

Original: Expansion-based methods
Revised: Expansion-based methods

Original: : Expansion-based methods dynamically allocate new parameters or modules to learn new tasks.
Revised: : Expansion-based methods dynamically allocate new parameters or modules to learn new tasks.

Original: Rusu et al.
Revised: Rusu et al.

Original: (2016) propose to incrementally introduce additional subnetworks with a fixed capacity.
Revised: (2016) propose to incrementally introduce additional subnetworks with a fixed capacity.

Original: DEN (Yoon et al., 2017) selectively retrains the frozen model and expands only with necessary neurons.
Revised: DEN (Yoon et al., 2017) selectively retrains the frozen model and expands only with necessary neurons.

Original: Moreover, Li et al.
Revised: Moreover, Li et al.

Original: (2019) perform an explicit network architecture search to decide where to expand.
Revised: (2019) perform an explicit network architecture search to decide where to expand.

Original: APD (Yoon et al., 2019) further decomposes the network and utilizes sparse task-specific parameters.
Revised: APD (Yoon et al., 2019) further decomposes the network and utilizes sparse task-specific parameters.

Original: However, these methods face capacity explosion inevitably after learning a long sequence of tasks.
Revised: However, these methods face capacity explosion inevitably after learning a long sequence of tasks.

Original: In contrast, our approach maintains a fixed network architecture to avoid expensive model growth.
Revised: In contrast, our approach maintains a fixed network architecture to avoid expensive model growth.

Original: Regularization-based methods: Methods in this category introduce extra regularization terms to the objective function to penalize the modification of parameters.
Revised: Regularization-based methods: Methods in this category introduce extra regularization terms to the objective function to penalize the modification of parameters.

Original: EWC (Kirkpatrick et al., 2017) first proposes to constrain the change based on the importance weight approximated by Fisher Information Matrix.
Revised: EWC (Kirkpatrick et al., 2017) first proposes to constrain the change based on the importance weight approximated by Fisher Information Matrix.

Original: MAS (Aljundi et al., 2018) measures the importance of the sensitivity of model outputs under an unsupervised setting.
Revised: MAS (Aljundi et al., 2018) measures the importance of the sensitivity of model outputs under an unsupervised setting.

Original: Other methods, also called parameter-isolation methods, defy catastrophic forgetting via freezing the gradient updates of particular parameters (De Lange et al., 2021).
Revised: Other methods, also called parameter-isolation methods, defy catastrophic forgetting via freezing the gradient updates of particular parameters (De Lange et al., 2021).

Original: PackNet (Mallya & Lazebnik, 2018) iteratively prunes and allocates parameters subset to corresponding tasks, whereas HAT (Serra et al., 2018) learns task-based hard attention to identify important parameters.
Revised: PackNet (Mallya & Lazebnik, 2018) iteratively prunes and allocates parameters subset to corresponding tasks, whereas HAT (Serra et al., 2018) learns task-based hard attention to identify important parameters.

Original: Instead of restricting individual parameters with estimated importance, the main idea of our approach is constraining the direction of gradients.
Revised: Instead of restricting individual parameters with estimated importance, the main idea of our approach is constraining the direction of gradients.

Original: Gradient projection methods : Gradient projection methods directly constrain the gradients to overcome catastrophic forgetting, and our approach belongs to this category.
Revised: Gradient projection methods : Gradient projection methods directly constrain the gradients to overcome catastrophic forgetting, and our approach belongs to this category.

Original: Mehta et al.
Revised: Mehta et al.

Original: (2021) implicitly expands the model with respect to the frozen space.
Revised: (2021) implicitly expands the model with respect to the frozen space and GEM (Lopez-Paz & Ranzato, 2017) utilizes complementary memory to restrict the update.

Original: While our approach requires neither storing old samples nor expanding the network.
Revised: While our approach requires neither storing old samples nor expanding the network.

Original: OWM (Zeng et al., 2019) first proposed to modify the gradients upon projector matrices.
Revised: OWM (Zeng et al., 2019) first proposed to modify the gradients upon projector matrices.

Original: OGD (Farajtabar et al., 2020) keeps the gradients orthogonal to the space spanned by previous gradients, whereas GPM (Saha et al., 2021) computes the frozen space based on old data.
Revised: OGD (Farajtabar et al., 2020) keeps the gradients orthogonal to the space spanned by previous gradients, whereas GPM (Saha et al., 2021) computes the frozen space based on old data.

Original: Wang et al. (2021) and Kong et al. (2022) construct the frozen space with covariance matrices instead.
Revised: NCL (Kao et al., 2021) combines the idea of gradient projection and Bayesian weight regularization to mitigate catastrophic forgetting.

Original: In spite of minimizing backward interference, these approaches suffer poor forward knowledge transfer and lack plasticity (Kong et al., 2022).
Revised: In spite of minimizing backward interference, these approaches suffer poor forward knowledge transfer and lack plasticity (Kong et al., 2022).

Original: TRGP (Lin et al., 2022) expands the model with trust regions based on task relationship to achieve better performance on new tasks.
Revised: TRGP (Lin et al., 2022) expands the model with trust regions based on task relationship to achieve better performance on new tasks.

Original: In contrast, we focus on facilitating forward knowledge transfer within a fixed capacity network by iteratively relaxing frozen regions with constraints.
Revised: In contrast, we focus on facilitating forward knowledge transfer within a fixed capacity network by iteratively relaxing frozen regions with constraints.

Original:  I TERATIVE R ELAXING G RADIENT P ROJECTION 3. P RELIMINARIES
Revised:  I TERATIVE R ELAXING G RADIENT P ROJECTION 3. P RELIMINARIES

Original: In a continual learning setting, we consider T tasks arriving as a sequence.
Revised: In a continual learning setting, we consider T tasks arriving as a sequence.

Original: The datasets are denoted as D ( t ) =
Revised: The datasets are denoted as D ( t ) =

Original: { x ( t ) i , y ( t ) i } N t
Revised: { x ( t ) i , y ( t ) i } N t

Original: i =1 , where N t is the number of samples.
Revised: i =1 , where N t is the number of samples.

Original: When learning the current task, the datasets of old tasks are inaccessible.
Revised: When learning the current task, the datasets of old tasks are inaccessible.

Original: We use an L -layer neural network with fixed capacity, and parameters defined as W =
Revised: We use an L -layer neural network with fixed capacity, and parameters defined as W =

Original: { W l } Ll =1 , where W l denotes the parameters in the l -th layer.
Revised: { W l } Ll =1 , where W l denotes the parameters in the l -th layer.

Original: The model is optimized by minimizing the objective function (1) and L t is the loss function for task t .
Revised: The model is optimized by minimizing the objective function (1) and L t is the loss function for task t .

Original: [Equation]
Revised: [Equation]

Original: Gradient projection methods mitigate catastrophic forgetting by only updating the model in the orthogonal direction to frozen spaces.
Revised: Gradient projection methods mitigate catastrophic forgetting by only updating the model in the orthogonal direction to frozen spaces.

Original: Saha et al.
Revised: Saha et al.

Original: (2021) proposed to compute the frozen spaces based on the inputs of each layer.
Revised: (2021) proposed to compute the frozen spaces based on the inputs of each layer.

Original: For task t , the frozen gradient spaces for the first t − 1 tasks are denoted as U t − 1 =
Revised: For task t , the frozen gradient spaces for the first t − 1 tasks are denoted as U t − 1 =

Original: { U lt − 1 } Ll =1 , where U lt is the frozen space of layer l for task t .
Revised: { U l t − 1 } L l =1 , where U l t is the frozen space of layer l for task t .

Original: During the training phase, for each layer l , gradients g Lt are constrained to be orthogonal to U lt − 1 .
Revised: During the training phase, for each layer l , gradients g lt are constrained to be orthogonal to U lt − 1 .

Original: Particularly, assuming
Revised: Particularly, assuming

Original: B lt =
Revised: B lt =

Original: [ u lt − 1 , 1 , ..., u lt − 1 ,N ] as the total N basis for U lt − 1 , gradients g t are modified as:
Revised: [ u lt − 1 , 1 , ..., u lt − 1 ,N ] as the total N basis for U lt − 1 , gradients g t are modified as:

Original: [Equation]
Revised: [Equation]

Original: After getting the learned model W t = { W lt } Ll =1 , for each layer l , we record the intermediate representation h lt,j of the j -th input and stack them to obtain the representation matrix H lt =
Revised: After getting the learned model W t = { W lt } Ll =1 , for each layer l , we record the intermediate representation h lt,j of the j -th input and stack them to obtain the representation matrix H lt =

Original: [ h lt, 1 , ..., h lt,N t ] .
Revised: [ h lt, 1 , ..., h lt,N t ] .

Original: Then compress the representation matrices by performing Singular Value Decomposition SVD( H lt )
Revised: Then compress the representation matrices by performing Singular Value Decomposition SVD( H lt )

Original: =
Revised: =

Original: U lH,t Σ lt ( V lt ) T .
Revised: U lH,t Σ lt ( V lt ) T .

Original: Given the threshold ϵ lth , select the first k vectors in U lH,t based on the following criteria:
Revised: Given the threshold ϵ lth , select the first k vectors in U lH,t based on the following criteria:

Original: [Equation]
Revised: [Equation]

Original: to construct the significant representation space R l t = span { U l H,t [0 : k ] } , where || · || 2 F denotes
Revised: to construct the significant representation space R lt = span { U lH,t [0 : k ] } , where || · || 2 F denotes

Original: Frobenius norm here.
Revised: Frobenius norm here.

Original: The significant representation spaces, considered as the frozen spaces for current task t , are then merged into the whole frozen gradient spaces for the first t tasks:
Revised: The significant representation spaces, considered as the frozen spaces for current task t , are then merged into the whole frozen gradient spaces for the first t tasks:

Original: [Equation]
Revised: [Equation]

Original: Although freezing gradients update significantly mitigates catastrophic forgetting, limited optimization space hinders the forward knowledge transfer, compromising the performance of new tasks.
Revised: Although freezing gradients update significantly mitigates catastrophic forgetting, limited optimization space hinders the forward knowledge transfer, compromising the performance of new tasks.

Original: TRGP (Lin et al., 2022) tackles this problem by selecting old tasks relevant to the current task and expanding the corresponding frozen spaces as the trust regions.
Revised: TRGP (Lin et al., 2022) tackles this problem by selecting old tasks relevant to the current task and expanding the corresponding frozen spaces as the trust regions.

Original: The scaled weight projection is further designed for memory-efficient updating and storing the parameters within the trust regions by scaling the basis, instead of directly changing the parameters.
Revised: The scaled weight projection is further designed for memory-efficient updating and storing the parameters within the trust regions by scaling the basis, instead of directly changing the parameters.

Original: Considering that task i is selected as the trust region, the scaled weight projection is shown as:
Revised: Considering that task i is selected as the trust region, the scaled weight projection is shown as:

Original: [Equation]
Revised: [Equation]

Original: where S li denotes the scale matrix.
Revised: where S li denotes the scale matrix.

Original: The parameters in the trust regions are retrained with the scaled weight projection and the learnt scale matrices are stored in the memory for the inference phase.
Revised: The parameters in the trust regions are retrained with the scaled weight projection and the learnt scale matrices are stored in the memory for the inference phase.

Original: Particularly, during the forward transfer, the parameters are modified with the scale matrices as:
Revised: Particularly, during the forward transfer, the parameters are modified with the scale matrices as:

Original: [Equation]
Revised: [Equation]

Original: where ( · ) ⊥ denotes the orthogonal complemented subspace.
Revised: where ( · ) ⊥ denotes the orthogonal complemented subspace.

Original: However, as tasks come, increasing extra parameters are introduced by storing the scaling matrices.
Revised: However, as tasks come, increasing extra parameters are introduced by storing the scaling matrices.

Original: Our experiments demonstrate that
Revised: Our experiments demonstrate that

Original: TRGP requires around 5000% amount of the parameters regarding the network architecture after learning 20 tasks on MiniImageNet, see Figure 3-(c).
Revised: TRGP requires around 5000% amount of the parameters regarding the network architecture after learning 20 tasks on MiniImageNet, see Figure 3-(c).

Original: Therefore, we propose our Iterative Relaxing
Revised: Therefore, we propose our Iterative Relaxing

Original: Gradient Projection framework to facilitate forward knowledge transfer while maintaining a fixed network capacity by wisely reusing parameters within the frozen space.
Revised: Gradient Projection framework to facilitate forward knowledge transfer while maintaining a fixed network capacity by wisely reusing parameters within the frozen space.

Original: 3. R ELAXING S UBSPACE S EARCHING
Revised: 3. R ELAXING S UBSPACE S EARCHING

Original: We first design a searching strategy to determine which part of the frozen space to relax based on the estimated importance characterized by the angle from the representation space spanned by current gradients g lt .
Revised: We first design a searching strategy to determine which part of the frozen space to relax based on the estimated importance characterized by the angle from the representation space spanned by current gradients g lt .

Original: The angle between a given space and a vector is defined in definition 3.1.
Revised: The angle between a given space and a vector is defined in definition 3.1.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: (Angle between vector and space)
Revised: (Angle between vector and space)

Original: We denote the angle between two inputs as Θ ( · ) and the inner product between two vectors as ⟨·⟩ .
Revised: We denote the angle between two inputs as Θ ( · ) and the inner product between two vectors as ⟨·⟩ .

Original: The angle between a vector v ∈ R n and a space
Revised: The angle between a vector v ∈ R n and a space

Original: U n × c ⊂ R n is defined as the minimum angle between the given vector v and any unit vector u ∈ U :
Revised: U n × c ⊂ R n is defined as the minimum angle between the given vector v and any unit vector u ∈ U :

Original: [Equation]
Revised: [Equation]

Original: Moreover, given the threshold γ lt , we define that a vector d is relaxable when:
Revised: Moreover, given the threshold γ lt , we define that a vector d is relaxable when:

Original: [Equation]
Revised: [Equation]

Original: where R lg,t is constructed by compressing g lt with Equation (3).
Revised: where R lg,t is constructed by compressing g lt with Equation (3).

Original: For task t , we aim to find the relaxing subspace V lt ⊆
Revised: For task t , we aim to find the relaxing subspace V lt ⊆

Original: U lt − 1 spanned all by relaxable vectors from U lt − 1 .
Revised: U lt − 1 spanned all by relaxable vectors from U lt − 1 .

Original: Particularly, we implement with the modulus of the projection, namely:
Revised: Particularly, we implement with the modulus of the projection, namely:

Original: [Equation]
Revised: [Equation]

Original: where ζ lt = cos γ lt is the threshold and U l,ct − 

Revised: where ζ lt = cos γ lt is the threshold and U l,ct −

Original: =
Revised: =

Original: U lt − 1 \ V lt denotes the complemented subspace of V lt with respect to U lt − 1 .
Revised: U lt − 1 \ V lt denotes the complemented subspace of V lt with respect to U lt − 1 .

Original: Above criterion guarantees max u ∈ V lt Θ ( u, R lg,t [Equation] ≤ γ lt and min v ∈ U l,ct − 1 Θ ( v, R lg,t ) > γ lt .
Revised: Above criterion guarantees max u ∈ V lt Θ ( u, R lg,t ) ≤ γ lt and min v ∈ U l,ct − 1 Θ ( v, R lg,t ) > γ lt .

Original: However, it is hard to construct V lt directly from U lt − 1 .
Revised: However, it is hard to construct V lt directly from U lt − 1 .

Original: Therefore, we propose a simple yet efficient strategy to find the relaxing subspaces.
Revised: Therefore, we propose a simple yet efficient strategy to find the relaxing subspaces.

Original: With V lt initiated as ∅ , we select the closest vector to R lg,t within U l,ct − 1 by arg min [Equation] l
Revised: With V lt initiated as ∅ , we select the closest vector to R lg,t within U l,ct − 1 by arg min d ∈ U l,ct −Θ

Original: [Equation]

Original: −Θ ( d, R lg,t ) .
Revised: ( d, R lg,t ) .

Original: The selected vector d is then appended into V lt as basis if it satisfies criterion (8).
Revised: The selected vector d is then appended into V lt as basis if it satisfies criterion (8).

Original: We repeat this procedure until no relaxable vector left to get the target V lt .
Revised: We repeat this procedure until no relaxable vector left to get the target V l t .

Original: The pseudo-code of our searching strategy is provided in Algorithm 1.
Revised: The pseudo-code of our searching strategy is provided in Algorithm 1.

Original: Considering the scope of our procedure, we further provide theoretical analysis on the upper bound of the dimension of the selected subspace V lt , which is also the number of iterations.
Revised: Considering the scope of our procedure, we further provide theoretical analysis on the upper bound of the dimension of the selected subspace V lt , which is also the number of iterations.

Original: Here we introduce Lemma 3.2 and Theorem 3.3, which guarantee that the dimension of V lT is no more than of the representation matrix.
Revised: Here we introduce Lemma 3.2 and Theorem 3.3, which guarantee that the dimension of V lT is no more than of the representation matrix.

Original: Detailed proof is provided in Appendix A.1 and A.2.
Revised: Detailed proof is provided in Appendix A.1 and A.2.

Original: Lemma 3.2.

Original: Denote the relaxed subspace as V = span { v 1 , v 2 , ..., v N } , where v N is the last base included in V .

Original: Given representation subspace U , ∀ v ∈ V , we have Θ ( v, U ) ≤ Θ ( v t , U ) .

Original: Input

Original: Output

Original: 1: for do 2:

Original: Construct the significant representation space R lg,t from gradients g lt by Equation (3).
Revised: Construct the significant representation space R lg t from gradients g lt by Equation (3).

Original: V lt ← ∅ 4: repeat 5: d ← arg min d ∈ U l,ct −Θ
Revised: V lt ← ∅ 4: repeat 5: d ← arg min d ∈ U l,ct −Θ

Original: (d, R lg,t ) 6: if Θ ( d
Revised: ( d

Original: R lg,t ) ≤ γ lt then 7:
Revised: R ,t ) 6: if Θ t then 7:

Original: Vlt ← V lt ∪ d 8: 
Revised: V ← V lt ∪ d 8: ct

Original: U l,ct − 1 ← U lt
Revised: Lemma 3.2. Denote the relaxed subspace as V = span { v 1 , v 2 , ..., v N } , where v N is the last base included in V .

Original: 1 \ V 9: end if 10: > lt 11: end for
Revised: Given representation subspace U , ∀ v ∈ V , we have Θ ( v, U ) ≤ Θ ( v t , U ) .

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Denote k p as the dimension of the representation subspace and k l as the dimension of the relaxed subspace, the upper bound of k l is k p , regardless of the frozen subspace.
Revised: Denote k p as the dimension of the representation subspace and k l as the dimension of the relaxed subspace, the upper bound of k l is k p , regardless of the frozen subspace.

Original: Moreover, according to Theorem 3.4, we figure that our strategy guarantees to find the maximum space within the whole solution set satisfying criterion (9), which further substantiates the efficiency of our searching strategy.
Revised: Moreover, according to Theorem 3.4, we figure that our strategy guarantees to find the maximum space within the whole solution set satisfying criterion (9), which further substantiates the efficiency of our searching strategy.

Original: We also include the corresponding proof in Appendix A.3.
Revised: We also include the corresponding proof in Appendix A.3.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: The relaxed subspace obtained by Algorithm 1 takes up the maximum subspace of the whole solution set.
Revised: The relaxed subspace obtained by Algorithm 1 takes up the maximum subspace of the whole solution set.

Original: To further validate our searching strategy, we propose IRGP-
Revised: To further validate our searching strategy, we propose IRGP-

Original: Exp, a modified version of our proposed IRGP, directly storing the parameters in the relaxed subspaces.
Revised: Exp, a modified version of our proposed IRGP, directly storing the parameters in the relaxed subspaces.

Original: For task t , we retrieve the corresponding relaxed subspaces { V lt } Ll =1 and the scale matrices { S lt } Ll =1 during the inference phase similar to TRGP.
Revised: For task t , we retrieve the corresponding relaxed subspaces { V lt } Ll =1 and the scale matrices { S lt } Ll =1 during the inference phase similar to TRGP.

Original: The modified parameters W lt,I used for inference on task t is:
Revised: The modified parameters W lt,I used for inference on task t is:

Original: [Equation]
Revised: [Equation]

Original: (10) where W l denotes the parameters of layer l of current network.
Revised: where W l denotes the parameters of layer l of current network.

Original: Replacing the parameters in the relaxed subspaces with the parameters optimized in task t , the model achieves better performance.
Revised: Replacing the parameters in the relaxed subspaces with the parameters optimized in task t , the model achieves better performance.

Original: 3. I TERATIVE M ODIFYING THE S CALE M ATRIX WITH C ONSTRAINTS
Revised: 3. I TERATIVE M ODIFYING THE S CALE M ATRIX WITH C ONSTRAINTS

Original: After getting the relaxed subspaces, we want to retrain inside parameters while consolidating previous knowledge, to facilitate forward knowledge transfer within a fixed network capacity.
Revised: After getting the relaxed subspaces, we want to retrain inside parameters while consolidating previous knowledge, to facilitate forward knowledge transfer within a fixed network capacity.

Original: One direct way is to fine-tune the parameters with regularization such as EWC (Kirkpatrick et al., 2017).
Revised: One direct way is to fine-tune the parameters with regularization such as EWC (Kirkpatrick et al., 2017).

Original: However, regularization terms are designed for explicit parameters, which are not applicable for implicit subspace in our framework.
Revised: However, regularization terms are designed for explicit parameters, which are not applicable for implicit subspace in our framework.

Original: Therefore, we introduce the scaled weight projection (Lin et al., 2022) to modify explicit parameters instead.
Revised: Therefore, we introduce the scaled weight projection (Lin et al., 2022) to modify explicit parameters instead.

Original: With scaled weight projection, we fine-tune the parameters within V lt by adding regularization term on scaling matrices S t = { S lt } Ll =1 instead of direct on target parameters.
Revised: With scaled weight projection, we fine-tune the parameters within V lt by adding regularization term on scaling matrices S t = { S lt } Ll =1 instead of direct on target parameters.

Original: Specifically, the objective function of task t is:
Revised: Specifically, the objective function of task t is:

Original: [Equation]
Revised: [Equation]

Original:  where 1 ( · ) denotes the identity matrix with the size of the rank of the input matrix and β l is the weight of the regularization term for layer l .
Revised: (11) where 1 ( · ) denotes the identity matrix with the size of the rank of the input matrix and β l is the weight of the regularization term for layer l .

Original: During back propagation, gradients within frozen space U lt − 1 are eliminated and parameters within V lt are modified by S lt .
Revised: During back propagation, gradients within frozen space U lt − 1 are eliminated and parameters within V lt are modified by S lt .

Original: Generally, in our Iterative
Revised: Generally, in our Iterative

Original: Relaxing Gradient Projection framework, we adopt our searching strategy to determine the relaxed subspaces and modify those parameters with constraints on the scaling matrices.
Revised: Relaxing Gradient Projection framework, we adopt our searching strategy to determine the relaxed subspaces and modify those parameters with constraints on the scaling matrices.

Original: However, during the training phase, the direction of gradients shifts sharply and frequently due to the steep learning scope of deep neural networks.
Revised: However, during the training phase, the direction of gradients shifts sharply and frequently due to the steep learning scope of deep neural networks.

Original: Diverse subspaces would be selected in different training phases.
Revised: Diverse subspaces would be selected in different training phases.

Original: Therefore, we iteratively execute Algorithm 1 during training until no extra subspace is required.
Revised: Therefore, we iteratively execute Algorithm 1 during training until no extra subspace is required.

Original: Particularly, for each task, our model is optimized for limited epochs first.
Revised: Particularly, for each task, our model is optimized for limited epochs first.

Original: Then we search for the target relaxing subspace and examine whether there exists a new subspace within the remaining frozen space.
Revised: Then we search for the target relaxing subspace and examine whether there exists a new subspace within the remaining frozen space.

Original: If extra frozen subspace is released, the scale of the scaling matrix requires to be modified, while TRGP maintains a fixed-size scaling matrix throughout training.
Revised: If extra frozen subspace is released, the scale of the scaling matrix requires to be modified, while TRGP maintains a fixed-size scaling matrix throughout training.

Original: Thus, to accommodate the increasing relaxing subspace, we propose to expand the scaling matrix with the identity matrix of the corresponding size as:
Revised: Thus, to accommodate the increasing relaxing subspace, we propose to expand the scaling matrix with the identity matrix of the corresponding size as:

Original: [Equation]
Revised: [Equation]

Original: where V l, [Equation] t denotes the newly included relaxing subspace.
Revised: where V l,newt denotes the newly included relaxing subspace.

Original: If there is no extra relaxing subspace, we optimize our model thoroughly on current task.
Revised: If there is no extra relaxing subspace, we optimize our model thoroughly on current task.

Original: After learning the task, the parameters within the relaxed subspace are further consolidated by Equation (6) and the scaling matrices are emptied to be identical matrices.
Revised: After training, the parameters within V lt are further consolidated by Equation (6) and the scaling matrices are emptied to be identical matrices.

Original: The pseudo-code of our Iterative Relaxing Gradient Projection framework is provided inAlgorithm 2 in Appendix C.5.
Revised: The pseudo-code of our Iterative Relaxing Gradient Projection framework is provided in Appendix D.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Original: Datasets: We evaluate our framework on five datasets.
Revised: Datasets: We evaluate our framework on five datasets.

Original: Following Saha et al.
Revised: Following Saha et al.

Original: , we conduct experiments on CIFAR-100 Split (Krizhevsky & Hinton, 2009), MiniImageNet (Vinyals et al., 2016),
Revised: , we conduct experiments on CIFAR-100 Split (Krizhevsky & Hinton, 2009), MiniImageNet (Vinyals et al., 2016),

Original: Permuted MNIST (MNIST) (Kirkpatrick et al., 2017) and CIFAR-100 Sup (Yoon et al., 2019).
Revised: Permuted MNIST (MNIST) (Kirkpatrick et al., 2017) and CIFAR-100 Sup (Yoon et al., 2019).

Original: Moreover, Serra et al.
Revised: Moreover, Serra et al.

Original: (2018) first propose Mixture , consists of CIFAR-10 (Krizhevsky & Hinton, 2009), MNIST (LeCun et al., 1998), CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), FashionMNIST (Xiao et al., 2017),
Revised: (2018) first propose Mixture , consists of CIFAR-10 (Krizhevsky & Hinton, 2009), MNIST (LeCun et al., 1998), CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), FashionMNIST (Xiao et al., 2017),

Original: TrafficSigns (Stallkamp et al., 2011), FaceScrub (Ng &
Revised: TrafficSigns (Stallkamp et al., 2011), FaceScrub (Ng &

Original: Winkler, 2014) and NotMNIST (Bulatov, 2011).
Revised: Winkler, 2014) and NotMNIST (Bulatov, 2011).

Original: Here we evaluate our framework on Mixture with seven tasks as a sequence except TrafficSigns 1 .
Revised: Here we evaluate our framework on Mixture with seven tasks as a sequence except TrafficSigns 1 .

Original: Details and statistics of the datasets can be found in
Revised: Details and statistics of the datasets can be found in

Original: Appendix B.1.
Revised: Appendix B.1.

Original: Moreover, we include the details of network architectures in Appendix B.2.
Revised: Moreover, we include the details of network architectures in Appendix B.2.

Original: Baselines: We compare our approach with competitive and well-established approaches maintaining a fixed network capacity following Saha et al.
Revised: Baselines: We compare our approach with competitive and well-established approaches maintaining a fixed network capacity following Saha et al.

Original: We adopt ER Res (Chaudhry et al., 2019) and A-GEM (Chaudhry et al., 2018) as representative replay-based methods.
Revised: We adopt ER Res (Chaudhry et al., 2019) and A-GEM (Chaudhry et al., 2018) as representative replay-based methods: the memory buffer size for PMNIST, CIFAR-100 Split, MiniImageNet, and Mixture are 1000, 2000, 500 and 3000, respectively.

Original: For gradient projection approaches, we consider OWM (Zeng et al., 2019) and GPM (Saha et al., 2021).
Revised: For gradient projection approaches, we consider OWM (Zeng et al., 2019) and GPM (Saha et al., 2021).

Original: For regularization approaches, we compare against EWC (Kirkpatrick et al., 2017) and state-of-the-art HAT (Serra et al., 2018).
Revised: For regularization approaches, we compare against EWC (Kirkpatrick et al., 2017) and state-of-the-art HAT (Serra et al., 2018).

Original: We also include the “multitask” baseline jointly training all tasks in a single network, which is always considered as an upper bound for continual learning.
Revised: We also include the “multitask” baseline jointly training all tasks in a single network, which is always considered as an upper bound for continual learning.

Original: Details on the implementation of these baselines are provided in Appendix B.3. We do not consider expansion-based methods here because they introduce continually growing architecture, which is out of the scope of our main framework. However, we compared IRGP- Exp with those methods in complementary experiments.
Revised: Other implementation details are listed in Appendix B.3.We exclude expansion-based methods in the main experiments as they use continually growing architecture, which is out of the scope of our work.

Original: Metrics: We  employ two standard evaluation metrics: Average Accuracy (ACC) (Mirzadeh et al., 2020) and Backward Transfer (BWT) (Lopez-Paz & Ranzato, 2017).
Revised: Metrics: We first employ two standard evaluation metrics: Average Accuracy (ACC) (Mirzadeh et al., 2020) and Backward Transfer (BWT) (Lopez-Paz & Ranzato, 2017).

Original: Denote A
Revised: Denote A

Original: i,j as the test accuracy of task j after learning task i .
Revised: i,j as the test accuracy of task j after learning task i .

Original: ACC is the average test accuracy evaluated after learning all tasks, defined as ACC = 1 T Σ [-Ti =
-]
Revised: ACC is the average test accuracy evaluated after learning all tasks, defined as ACC = 1 T Σ [-Ti =
-]T i =

Original:  A T,i .
Revised: 1 A T,i .

Original: BWT is the average accuracy decrease after learning following tasks, defined as BWT = 1 T − 1 Σ T
Revised: BWT is the average accuracy decrease after learning following tasks, defined as BWT = 1 T − 1 Σ T

Original: −
Revised: −

Original: i =

Revised: i =

Original: ( A T,i − A
Revised: ( A T,i − A

Original: i,i ) .
Revised: i,i ) .

Original: We further introduce Forward Transfer (FWT) proposed by Lopez-Paz & Ranzato (2017) to analyze the forward knowledge transfer.
Revised: To evaluate forward knowledge transfer, we further introduce Forward Transfer (FWT) (Lopez-Paz & Ranzato, 2017) and Ω new (Kemker et al., 2018).

Original: FWT is initially defined as FWT = 1 T − 1 Σ Ti =2 ( A i,i − b i ) where b i denotes test accuracy of task i at random initialization.
Revised: FWT reflects the influence of the observed tasks on new tasks in a zero-shot manner, while Ω new indicates the capability of acquiring new tasks. The detailed definitions are provided in Appendix B.4.

Original: For simplicity, as b i stays still across different approaches, we consider FWT [-= 1 T − 1 Σ Ti =
 A i,i .-]
Revised: In this paper, we mainly focus on Ω new among the three metrics and results on FWT [-= 1 T − 1 Σ Ti =
 A i,i .-]are provided as well.

Original: Generally, the larger ACC, the better the approach.
Revised: Generally, the larger ACC, the better the approach.

Original: FWT and BWT evaluate the capability of learning and consolidating new tasks respectively.
Revised: Forward and backward knowledge transfer evaluate the capability of learning and memorizing respectively.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: We show the comparative results on four benchmarks in Table 1.
Revised: We show the comparative results on four benchmarks in Table 1.

Original: The experiments on Mixture are implemented by us, while other results are  from (Saha et al., 2021).
Revised: The experiments on Mixture are implemented by us, while other results are reported from (Saha et al., 2021).

Original: We run each experiment five times and report the mean results.
Revised: We run each experiment five times and report the mean results.

Original: We include implementation details in Appendix B.3 and detailed results including standard deviation can be found in Appendix C.1.
Revised: We include implementation details in Appendix B.3 and detailed results including forward transfer can be found in Appendix C.2.

Original: As shown in Table 1, our approach obtain the best accuracy with comparable forgetting across all datasets.[Table] [Table] [Table]
Revised: As shown in Table 1, our approach obtain the best accuracy with comparable forgetting across all datasets.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: IRGPGPM

Original: [Figure]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: [-IRGPGPM 1 2 3 4 5 6 7 
-]
Revised: [-IRGPGPM 1 2 3 4 5 6 7 
-]

Original: IRGPGPM [Figure] [Table] As shown in Table 1, compared with replay-based methods A-GEM and ER Res, our approach achieves at least around 2% higher ACC with less forgetting.
Revised: Compared with replay-based methods A-GEM and ER Res, IRGP achieves at least around 2% higher ACC with less forgetting.

Original: For regularization-based methods, our approach significantly dominates EWC across all benchmarks and outperforms HAT on MiniImageNet and PMNIST.
Revised: For regularization-based methods, IRGP significantly dominates EWC across all benchmarks and outperforms HAT on MiniImageNet and PMNIST.

Original: Although HAT obtains less forgetting on the other two datasets, our approach gains 1% better ACC on average.
Revised: Although HAT obtains less forgetting on the other two datasets, IRGP gains 1% better ACC on average.

Original: For gradient projection methods, we observe that our method achieves around 1% higher ACC on CIFAR-100 Split and MiniImageNet than GPM with comparable forgetting.
Revised: For gradient projection methods, we observe that IRGP achieves around 1% higher ACC on CIFAR-100 Split and MiniImageNet than GPM with comparable forgetting.

Original: On PMNIST and Mixture benchmarks, our IRGP improves the accuracy with less forgetting, reducing 1% BWT than GPM.
Revised: On PMNIST and Mixture, IRGP improves the accuracy with less forgetting, reducing 1% BWT than GPM.

Original: The averaged accuracy after learning each task on CIFAR-100 Split exhibited in Figure 2-(a) further validates that IRGP universally outperforms GPM.
Revised: The averaged accuracy after learning each task on CIFAR-100 Split exhibited in Figure 2-(a) further validates that IRGP universally outperforms GPM.

Original: We include the detailed results on other benchmarks in Appendix C.2.
Revised: We include the detailed results on other benchmarks in Appendix C.1.

Original: Moreover, we compare the accuracy evolution of specific tasks during sequential tasks with GPM, which achieves the highest accuracy among selected baselines according to Table 1.
Revised: Moreover, we compare the accuracy evolution of specific tasks during sequential tasks with GPM, which achieves the highest accuracy among selected baselines according to Table 1.

Original: Here we show the results of the second task on CIFAR-100 Split in Figure 2-(b).
Revised: Here we show the results of the second task on CIFAR-100 Split in Figure 2-(b).

Original: We further present the results of three randomly selected tasks on all benchmarks in Appendix C.4.
Revised: We further present the results of three randomly selected tasks on all benchmarks in Appendix C.3.

Original: We notice that IRGP achieves better accuracy right after learning a new task, in other words, gains better FWT, which is also the purpose of our relaxing strategy.
Revised: We notice that IRGP achieves better accuracy right after learning a new task, in other words, gains better Ω new , which is also the purpose of our relaxing strategy.

Original: Without forward knowledge transfer, approaches may have limited performance even with less forgetting (Lopez-Paz & Ranzato, 2017).
Revised: Without forward knowledge transfer, approaches may have limited performance even with less forgetting (Lopez-Paz & Ranzato, 2017).

Original: Thus, we observe the accuracy tested after learning each task.
Revised: Thus, we observe the accuracy tested after learning each task.

Original: As shown in Figure 2-(c), our approach achieves 2.7% better average accuracy on CIFAR-100 Split setting than GPM.
Revised: As shown in Figure 2-(c), our approach achieves 2.7% better average accuracy on CIFAR-100 Split setting than GPM.

Original: Results on other benchmarks provided in Appendix C.3 further substantiate this phenomenon.
Revised: Results on other benchmarks provided in Appendix C.2 further substantiate this phenomenon.

Original: As tasks keep coming, accumulated frozen spaces lead to decreasing optimization spaces for GPM.
Revised: As tasks keep coming, accumulated frozen spaces lead to decreasing optimization spaces for GPM.

Original: In the contrast,
Revised: In the contrast,

Original: IRGP explores larger optimization spaces by relaxing previous frozen spaces.
Revised: IRGP explores larger optimization spaces by relaxing previous frozen spaces.

Original: Thus, IRGP achieves better forward knowledge transfer by implicitly reusing the weights within the relaxed subspaces.
Revised: Thus, IRGP achieves better forward knowledge transfer by implicitly reusing the weights within the relaxed subspaces.

Original: In brief, our approach universally outperforms selected baselines on all datasets in a fixed capacity.
Revised: In brief, our approach universally outperforms selected baselines on all datasets in a fixed capacity.

Original: With comparable forgetting, IRGP achieves better forward knowledge transfer with larger optimization spaces against GPM.
Revised: With comparable forgetting, IRGP achieves better forward knowledge transfer with larger optimization spaces against GPM.

Original: To validate the efficiency of our relaxing strategy, we further compare
Revised: To validate the efficiency of our relaxing strategy, we further compare

Original: IRGP-Exp with well-established and competitive expansion-based methods in the next section.
Revised: IRGP-Exp with well-established and competitive expansion-based methods in the next section.

Original: 4. C OMPARED WITH E XPANSION - BASED M ETHODS
Revised: 4. C OMPARED WITH E XPANSION - BASED M ETHODS

Original: The above experiments exhibit the outstanding performance of our approach maintaining a fixed network capacity.
Revised: The above experiments exhibit the outstanding performance of our approach maintaining a fixed network capacity.

Original: By allocating new neurons or modules, expansion-based methods significantly[Table] [Table] mitigate backward interference with increasing capacity.
Revised: By allocating new neurons or modules, expansion-based methods significantly mitigate backward interference with increasing capacity.

Original: Thus, to further validate our strategy, we compare IRGP and IRGP-
Revised: Thus, to further validate our strategy, we compare IRGP and IRGP-

Original: Exp with relative expansion-based methods in this section.
Revised: Exp with relative expansion-based methods in this section.

Revised: [Table]

Revised: [Table]

Original: Following Saha et al.
Revised: Following Saha et al.

Original: (2021), wefirst perform experiments on CIFAR-100 Sup (Yoon et al., 2019).
Revised: [Table] we perform experiments on CIFAR-100 Sup (Yoon et al., 2019).

Original: ACC results shown in Table 2 are averaged over 5runs with different sequence orders proposed by Yoon et al.
Revised: ACC results shown in Table 2 are averaged over 5 different sequence orders proposed by Yoon et al.

Original: We refer to the results of baselines from Saha et al.
Revised: We refer to the results of baselines from Saha et al.

Original: Capacity denotes the model capacity normalized with respect to the network used in GPM.
Revised: Capacity denotes the model capacity normalized with respect to the network used in GPM.

Original: Here we use the same LeNetarchitecture as GPM.
Revised: Here we use the same model as GPM.

Original: According to Table 2, IRGP outperforms all baselines including GPM with the smallest capactiy.
Revised: According to Table 2, IRGP outperforms all baselines including GPM with the smallest capactiy.

Original: We further include detailed results in Appendix C.

Original: Lin et al.
Revised: Lin et al.

Original: (2022) proposed TRGP to expand the limited optimization spaces by retraining parameters within the selected trust regions, achieving superior performance.
Revised: (2022) proposed TRGP to expand the limited optimization spaces by retraining parameters within the selected trust regions, achieving superior performance.

Original: During the inference phase, TRGP reuses the parameters in corresponding trust regions memorized after learning this task.
Revised: During the inference phase, TRGP reuses the parameters in corresponding trust regions memorized after learning this task.

Original: In contrast,
Revised: In contrast,

Original: GPM and our IRGP only store the representation of the frozen space.
Revised: GPM and our IRGP only store the representation of the frozen space.

Original: Therefore, although indeed a stable network capacity is allocated for each task, the entire memory size of TRGP grows continually.
Revised: Therefore, although indeed a stable network capacity is allocated for each task, the entire memory size of TRGP grows continually.

Original: As shown in Figure 3-(c), after learning the last task on MiniImageNet setting, TRGP requires around 5000% extra parameters with respect to the network capacity.
Revised: As shown in Figure 3-(c), after learning the last task on MiniImageNet setting, TRGP requires around 5000% extra parameters with respect to the network capacity.

Original: Results on other benchmarks provided in Appendix C.6 further substantiate that TRGP introduces a significant number of extra parameters.
Revised: Results on other benchmarks provided in Appendix C.5 further substantiate that TRGP introduces a significant number of extra parameters.

Original: Thus, we categorize TRGP as an expansion-based method here.
Revised: Thus, we categorize TRGP as an expansion-based method here.

Original: As mentioned in Section 3.2, we propose IRGP-
Revised: As mentioned in Section 3.2, we propose IRGP-

Original: Exp to further validate our searching strategy.
Revised: Exp to further validate our searching strategy.

Original: In this setting, the main difference between IRGP-Exp and TRGP is the strategy of deciding which part of the frozen space to reuse.
Revised: In this setting, the main difference between IRGP-Exp and TRGP is the strategy of deciding which part of the frozen space to reuse.

Original: We conduct experiments on two benchmarksCIFAR-100 Split and PMNIST against TRGP.
Revised: We conduct experiments on all four benchmarks against TRGP.

Original: The results of ACC are provided in the left of Table 3.
Revised: The results of ACC are provided in the left of Table 3.

Original: The percentages indicate the ratios of the rank of relaxing subspaces with respect to the corresponding frozen space.
Revised: The percentages indicate the ratios of the rank of relaxing subspaces with respect to the corresponding frozen space.

Original: We evaluate three constant ratios and further use the ratios in TRGP, denoted as T%.
Revised: We evaluate three constant ratios and further use the ratios in TRGP, denoted as T%.

Original: According to the left of Table 3, IRGP-Exp already outperforms TRGP with relaxing only 50% of the frozen spaces.
Revised: According to the left of Table 3, IRGP-Exp already outperforms TRGP with relaxing only 50% of the frozen spaces on CIFAR100-Split and

Original:  As TRGP selects the top 2 tasks as the trust regions, T% is larger than 80% most times.
Revised: PMNIST. As TRGP selects the top 2 tasks as the trust regions, T% is larger than 80% most times.

Original: Moreover, our approach gainsaround 0.6% better ACC on both benchmarks with a comparable size of relaxing subspaces, which substantiate the efficiency of our subspace searching strategy.
Revised: Moreover, our approach gains better ACC on all benchmarks with a comparable size of relaxing subspaces, which substantiate the efficiency of our subspace searching strategy.

Original: We further modify TRGP as TRGP-Reg with similar regularization terms on the scale matrices as our IRGP to compare the relaxing strategies.
Revised: We further modify TRGP as TRGP-Reg with similar regularization terms on the scale matrices as our IRGP to compare the relaxing strategies.

Original: We report the results on two benchmarks with three representative regularization weights w on TRGP-Reg in the right of Table 3.
Revised: We report the results on four benchmarks with three representative regularization weights w on TRGP-Reg in the right of Table 3.

Original: As shown in Table 3, IRGP significantly outperforms TRGP-Reg, especially on PMNIST, gaining over 20% ACC improvement.
Revised: As shown in Table 3, IRGP significantly outperforms TRGP-Reg, especially on PMNIST, gaining over 20% ACC improvement.

Revised: Generally, IRGP achieves better or comparable Ω new than TRGP under or without the constraint of a fixed network capacity.

Original: More experimental details are included in Appendix C.7.
Revised: Detailed results are included in Appendix C.7.

Original:  A NALYSIS AND D ISCUSSION
Revised:  A NALYSIS AND D ISCUSSION

Original: To have a deeper insight into IRGP, we investigate the trend of scales of the subspaces relaxed by our strategy.
Revised: To gain a deeper insight into IRGP, we investigate the trend of scales of the subspaces relaxed by our strategy.

Original: With the theoretical upper bound of the rank of the relaxed subspace provided in
Revised: With the theoretical upper bound of the rank of the relaxed subspace provided in

Original: Theorem 3.3, we further inspect the ratios of the relaxed subspaces concerning corresponding frozen spaces in practical.
Revised: Theorem 3.3, we further inspect the ratios of the relaxed subspaces concerning corresponding frozen spaces in practical.

Original: Results of the last layer on three different settings are provided in Figure 3-(a).
Revised: Results of the last layer on three different settings are provided in Figure 3-(a).

Original: [Figure] As shown in Figure 3-(a)
Revised: As shown in Figure 3-(a), relaxing ratios maintain a stable trend, fluctuating smoothly in a similar

Original: [Figure]
Revised: [Figure]

Original: relaxing ratios maintain a stable trend, fluctuating smoothly in a similar range over sequential tasks, on both benchmarks.
Revised: range over sequential tasks, on both benchmarks.

Original: As different tasks explore different optimization directions, ideal relaxing subspaces vary across tasks, in accord with the fluctuation of our results.
Revised: As different tasks explore different optimization directions, ideal relaxing subspaces vary across tasks, in accord with the fluctuation of our results.

Original: The dimension of the frozen spaces keeps growing as the tasks come, leading to expanding range for searching relaxing subspaces.
Revised: The dimension of the frozen spaces keeps growing as the tasks come, leading to expanding range for searching relaxing subspaces.

Original: Therefore, the computation complexity and time consumption are supposed to increase gradually.
Revised: Therefore, the computation complexity and time consumption are supposed to increase gradually.

Original: To investigate the practical efficiency of our approach, we further reported the time consumption of IRGP on CIFAR-100 Split and MiniImageNet against other baselines in Appendix C.5.
Revised: To investigate the practical efficiency of our approach, we further reported the time consumption of IRGP on CIFAR-100 Split and MiniImageNet against other baselines in Appendix C.4.

Original: According to Table 13, our approach takes around 60% more time than GPM on both settings.
Revised: According to Table 14, our approach takes around 60% more time than GPM on both settings.

Original: In general, the practical efficiency of our approach is accecptable.
Revised: In general, the practical efficiency of our approach is accecptable.

Original: To understand our relaxing strategy better [Figure] we further conduct experiments on different thresholds ϵ mentioned in Equation (3), which regulate the criterion of the frozen spaces.
Revised: To understand our relaxing strategy better, we further conduct experiments on different thresholds ϵ mentioned in Equation (3), which regulate the criterion of the frozen spaces.

Original: Saha et al.
Revised: Saha et al.

Original: (2021) argue that ϵ controls the scale of the frozen space to mediate the stability-plasticity dilemma, and thus is critical for GPM.
Revised: (2021) argue that ϵ controls the scale of the frozen space to mediate the stability-plasticity dilemma, and thus is critical for GPM.

Original: However, IRGP enables the frozen space to be dynamically regulated regarding the current task.
Revised: However, IRGP enables the frozen space to be dynamically regulated regarding the current task.

Original: Therefore, ϵ plays a much less important role in IRGP.
Revised: Therefore, ϵ plays a much less important role in IRGP.

Original: We present the performance of different ϵ on CIFAR-100 Split in Figure 3-(b).
Revised: We present the performance of different ϵ on CIFAR-100 Split in Figure 3-(b).

Original: As shown in Figure 3-(b), the performance of GPM drops significantly when ϵ ≥ 0 .
Revised: As shown in Figure 3-(b), the performance of GPM drops significantly when ϵ ≥ 0 .

Original: 97 , the optimal value reported in GPM, while IRGP consistently performs well even with ϵ = 0 .
Revised: 97 , the optimal value reported in GPM, while IRGP consistently performs well even with ϵ = 0 .

Original: Generally, IRGP is more robust on the threshold ϵ .
Revised: Generally, IRGP is more robust on the threshold ϵ .

Original: [Table]
Revised: [Table]

Original: In the contrast, IRGP mediates the stability-plasticity dilemma by controlling the dimension and flexibility of the relaxing space by ζ in Equation (9) and β in
Revised: In the contrast, IRGP mediates the stability-plasticity dilemma by controlling the dimension and flexibility of the relaxing space by ζ in Equation (9) and β in

Original: Equation (11) respectively.
Revised: Equation (11) respectively.

Original: We present the results on
Revised: We present the results on

Original: CIFAR-100 Split in Table 4, where ζ conv denotes the hyper-parameter for convolutional layers and ζ fc denotes the hyper-parameter for fully connected layers.
Revised: CIFAR-100 Split in Table 4, where ζ conv denotes the hyper-parameter for convolutional layers and ζ fc denotes the hyper-parameter for fully connected layers.

Original: As shown in Table 4, a larger ζ guarantees less forgetting, as a result of smaller relaxing subspaces.
Revised: As shown in Table 4, a larger ζ guarantees less forgetting, as a result of smaller relaxing subspaces.

Revised: Detailed results are provided in Appendix C.6.

Original: Similarly, we observe less forgetting on larger β , which constrains the update of parameters within the relaxing subspace more strictly.
Revised: Similarly, we observe less forgetting on larger β , which constrains the update of parameters within the relaxing subspace more strictly.

Original: However, strict constraints also lead to limited performance on new tasks as discussed in Section 4.
Revised: However, strict constraints also lead to limited performance on new tasks as discussed in Section 4.

Original: Generally, ζ and β work together to overcome  forgetting with better forwardknowledge transfer.
Revised: Generally, ζ and β work together to overcome catastrophic forgetting with better forward transfer.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a novel continual learning approach that facilitates the forward knowledge transfer in gradient projection methods with a fixed network capacity by iteratively searching and relaxing subspaces within the frozen space to expand the optimization space.
Revised: In this paper, we propose a novel continual learning approach that facilitates the forward knowledge transfer in gradient projection methods with a fixed network capacity by iteratively searching and relaxing subspaces within the frozen space to expand the optimization space.

Original: Extensive experiments demonstrate that our IRGP framework surpasses related state-of-the-art approaches on diverse benchmarks.
Revised: Extensive experiments demonstrate that our IRGP framework surpasses related state-of-the-art approaches on diverse benchmarks.

Original: Moreover, we propose a modified version expanding the architecture with relaxing subspaces, achieving better average accuracy than other expansion-based methods.
Revised: Moreover, we propose a modified version expanding the architecture with relaxing subspaces, achieving better average accuracy than other expansion-based methods.

Original: We further provide solid proof and analysis validating the efficiency of our algorithm.
Revised: We further provide solid proof and analysis validating the efficiency of our algorithm.

Original: A P ROOF A. P ROOF OF L EMMA 3. 
Revised: A P ROOF A. P ROOF OF L EMMA 3.

Original: For simplification, all vectors here are assumed to be unit vectors, namely ∥ v ∥ = 1 .
Revised: For simplification, all vectors here are assumed to be unit vectors, namely ∥ v ∥ = 1 .

Original: Depicting the angle by projection form, Lemma 3.2 can be expressed as:
Revised: Depicting the angle by projection form, Lemma 3.2 can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Denote B =
Revised: Denote B =

Original: [ u 1 , ..., u m ] as the representation matrix of the representation subspace U = span { u 1 , ..., u m } , where u i is the i -th normalized base of U .
Revised: [ u 1 , ..., u m ] as the representation matrix of the representation subspace U = span { u 1 , ..., u m } , where u i is the i -th normalized base of U .

Original: Lemma 3.2 can be expressed as:
Revised: Lemma 3.2 can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: As v i s are the basis iteratively appended by Algorithm 1, for any i ≤ j , we have:
Revised: As v i s are the basis iteratively appended by Algorithm 1, for any i ≤ j , we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, to prove Lemma 3.2, it suffices to prove the following lemma.
Revised: Therefore, to prove Lemma 3.2, it suffices to prove the following lemma.

Original: If there exists a vector v = Σ t +1 i =
Revised: If there exists a vector v = Σ t +1 i =

Original: 1 w i v i satisfying the condition v T BB T v < v Tt BB T v t , then we can find another vector v ′
Revised: 1 w i v i satisfying the condition v T BB T v < v Tt BB T v t , then we can find another vector v ′

Original: =
Revised: =

Original: Σ t +1 i = k w ′
Revised: Σ t +1 i = k w ′

Original: i v i such that v ′ T BB T v ′ > v Tk BB T v k , which contradicts
Revised: i v i such that v ′ T BB T v ′ > v Tk BB T v k , which contradicts

Original: Algorithm 1 where Θ ( v k , U ) ≤ Θ ( v, U ) for ∀ v ∈ span { v k , ..., v t +1 } .
Revised: Algorithm 1 where Θ ( v k , U ) ≤ Θ ( v, U ) for ∀ v ∈ span { v k , ..., v t +1 } .

Original: First, we consider a special case where the current relaxed subspace has only one base, denoted by
Revised: First, we consider a special case where the current relaxed subspace has only one base, denoted by

Original: V = span { v 1 } .
Revised: V = span { v 1 } .

Original: Assume there exists v = w 1 v 1 + w 2 v 2 that v T BB T v < v T 2 BB T v 2 , we have:
Revised: Assume there exists v = w 1 v 1 + w 2 v 2 that v T BB T v < v [Equation] 2 BB T v 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: Construct v ′ =
Revised: Construct v ′ =

Original: w 2 v 1 − w 1 v 2 , we have:
Revised: w 2 v 1 − w 1 v 2 , we have:

Original: [Equation]
Revised: [Equation]

Original:  which contradicts Θ ( v 1 , U ) ≤ Θ ( v, U ) for ∀ v ∈ span { v 1 , v 2 } .
Revised: 17) which contradicts Θ ( v 1 , U ) ≤ Θ ( v, U ) for ∀ v ∈ span { v 1 , v 2 } .

Original: Then we consider the general case V = span { v 1 , ..., v t } .
Revised: Then we consider the general case V = span { v 1 , ..., v t } .

Original: For ∀ v ∈ V , we have Θ ( v, U ) ≤ Θ ( v t , U ) .
Revised: For ∀ v ∈ V , we have Θ ( v, U ) ≤ Θ ( v t , U ) .

Original: After v t is included, we assume that there exists v ∈ V that Θ ( v, U ) >
Revised: After v t is included, we assume that there exists v ∈ V that Θ ( v, U ) >

Original: Θ ( v t +1 , U ) .
Revised: Θ ( v t +1 , U ) .

Original: Then we can find the minimum s satisfying that there exists v = Σ si =1 w i v i + w t
Revised: Then we can find the minimum s satisfying that there exists v = Σ si =1 w i v i + w t

Original: +

Revised: +

Original: v t +1 that Θ ( v, U ) >
Revised: v t +1 that Θ ( v, U ) >

Original: Θ ( v t +1 , U ) and for ∀
Revised: Θ ( v t +1 , U ) and for ∀

Original: v ′ =
Revised: v ′ =

Original: Σ s − 1 i =1 w
Revised: Σ s − 1 i =1 w

Original: i
Revised: i

Original: v i + w t
Revised: v i + w t

Original: +1 v t +1 , we have Θ ( v ′ , U )
Revised: +1 v t +1 , we have Θ ( v ′ , U )

Original: ≤ Θ ( v t +1 , U ) .
Revised: ≤ Θ ( v t +1 , U ) .

Original: When s = 1 , it is similar to the special case, so the proof is omitted.
Revised: When s = 1 , it is similar to the special case, so the proof is omitted.

Original: Thus, we consider the case where s ≥ 2 .
Revised: Thus, we consider the case where s ≥ 2 .

Original: For simplification, we express v as v =
Revised: For simplification, we express v as v =

Original: c
Revised: c

Original: 0 v 0 + c 1 v s + c 2 v t +1 with v 0 = Σ s − 1 i =1 a i v i , where c i s and
Revised: 0 v 0 + c 1 v s + c 2 v t +1 with v 0 = Σ s − 1 i =1 a i v i , where c i s and

Original: a i s are coefficients.
Revised: a i s are coefficients.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: As Θ ( w 1 v 0 + w 2 v s , U ) ≤ Θ ( v s , U ) ≤ Θ ( v t +1 , U ) , we have:
Revised: As Θ ( w 1 v 0 + w 2 v s , U ) ≤ Θ ( v s , U ) ≤ Θ ( v t +1 , U ) , we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  which is:
Revised: 19) which is:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly we have:
Revised: Similarly we have:

Original: [Equation]
Revised: [Equation]

Original: Then we can express Equation (18) as:
Revised: Then we can express Equation (18) as:

Original: [Equation]
Revised: [Equation]

Original: As ∥ v ∥ =
Revised: As ∥ v ∥ =

Original: ∥ v i ∥ = 1 , c 2 0 + c 21 + c 2 2 = 1 .
Revised: ∥ v i ∥ = 1 , c 20 + c 21 + c 22 = 1 .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Construct v ′ =
Revised: Construct v ′ =

Original: c 2 v s − c 1 v t 
Revised: c 2 v s − c 1 v t +1 √ c

Original: [Equation]

Original: 1 √ c 21 + c 22 , we have: 
Revised: + c 22 , we have: v

Original: [Equation]
Revised: [Equation]

Original: [Equation] which contradicts Θ ( v s , U ) ≤ Θ ( v, U ) for ∀ v ∈ span { v s , v s +1 , ..., v t , v t +1 } . Thus, for ∀ v ∈ V = span { v 
Revised: which contradicts Θ ( v s , U ) ≤ Θ ( v, U ) for ∀ v ∈ span { v s , v s +1 , ..., v t , v t +1 } . Thus, for ∀ v ∈ V = span { v 1 , ..., v t } , we have Θ ( v, U ) ≤ Θ ( v t , U ) .

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: Denote the relaxed subspace and the representation subspace as V and U = span { u 1 , ..., u k p } respectively.
Revised: Denote the relaxed subspace and the representation subspace as V and U = span { u 1 , ..., u k p } respectively.

Original: With Lemma 3.2, we have ∀ v ∈ V , Θ ( v, U ) ≤ Θ ( v t , U ) <
Revised: With Lemma 3.2, we have ∀ v ∈ V , Θ ( v, U ) ≤ Θ ( v t , U ) <

Original: π 2 .
Revised: π 2 .

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original: For the sake of contradiction, assume the dimension of V is larger than k p , namely dim( V ) >
Revised: For the sake of contradiction, assume the dimension of V is larger than k p , namely dim( V ) >

Original: dim( U ) =
Revised: dim( U ) =

Original: k p .
Revised: k p .

Original: denote U c as the complemented subspace of U with respect to the whole space R .
Revised: denote U c as the complemented subspace of U with respect to the whole space R .

Original: Obviously, dim( U c ) = n − k p , where n is the dimension of R .
Revised: Obviously, dim( U c ) = n − k p , where n is the dimension of R .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Thus, there exists v ′ ∈ V such that v ′ ∈
Revised: Thus, there exists v ′ ∈ V such that v ′ ∈

Original: U c too.
Revised: U c too.

Original: As U c is the complemented subspace, ∀ u ∈ U c , u ⊥ U .
Revised: As U c is the complemented subspace, ∀ u ∈ U c , u ⊥ U .

Original: Then we have v ′ ⊥ U , which contradicts Equation (25).
Revised: Then we have v ′ ⊥ U , which contradicts Equation (25).

Original: Therefore, the assumption is aborted.
Revised: Therefore, the assumption is aborted.

Original: The upper bound of the dimension of V is k p , namely the dimension of the representation subspace U .
Revised: The upper bound of the dimension of V is k p , namely the dimension of the representation subspace U .

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: Denote the whole solution set as S =
Revised: Denote the whole solution set as S =

Original: { u | Θ ( u, U ) ≤ γ and u ∈ U f } where U is the representation subspace, U f is the frozen space and γ is the threshold.
Revised: { u | Θ ( u, U ) ≤ γ and u ∈ U f } where U is the representation subspace, U f is the frozen space and γ is the threshold.

Original: Theorem 3.4 can be expressed as that all subspace V ′ ⊆
Revised: Theorem 3.4 can be expressed as that all subspace V ′ ⊆

Original: S satisfies dim( V ′ )
Revised: S satisfies dim( V ′ )

Original: ≤ dim( V ) , where V is the relaxed subspace obtained by
Revised: ≤ dim( V ) , where V is the relaxed subspace obtained by

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Similar to Theorem 3.3, we assume there exists V ′ ⊆ S that dim( V ′ ) >
Revised: Similar to Theorem 3.3, we assume there exists V ′ ⊆ S that dim( V ′ ) >

Original: dim( V ) .
Revised: dim( V ) .

Original: denote V cf as the complemented subspace of V with respect to the frozen space U f .
Revised: denote V cf as the complemented subspace of V with respect to the frozen space U f .

Original: According to
Revised: According to

Original: Algorithm 1, for ∀ v ∈ V cf , we have Θ ( v, U ) > γ .
Revised: Algorithm 1, for ∀ v ∈ V cf , we have Θ ( v, U ) > γ .

Original: We also have:
Revised: We also have:

Original: [Equation]
Revised: [Equation]

Original: Thus, there exists v ′ ∈ V ′ that v ′ ∈
Revised: Thus, there exists v ′ ∈ V ′ that v ′ ∈

Original: U cf , namely there exists v ′ ∈ S that Θ ( v ′ , U ) > γ , which is contradict.
Revised: U cf , namely there exists v ′ ∈ S that Θ ( v ′ , U ) > γ , which is contradict.

Original: Therefore, the relaxed subspace obtained by Algorithm 1 takes up the maximum subspace of the whole solution set.
Revised: Therefore, the relaxed subspace obtained by Algorithm 1 takes up the maximum subspace of the whole solution set.

Original: B E XPERIMENTAL S ETUP B. D ATASETS
Revised: B E XPERIMENTAL S ETUP B. D ATASETS

Original: Here we introduce the datasets we use for evaluation.
Revised: Here we introduce the datasets we use for evaluation.

Original: 1) CIFAR-100 Split Saha et al.
Revised: 1) CIFAR-100 Split Saha et al.

Original: (2021) constructed CIFAR-100 Split , by splitting CIFAR100 (Krizhevsky & Hinton, 2009) into 10 tasks where each task has 10 classes.
Revised: (2021) constructed CIFAR-100 Split , by splitting CIFAR100 (Krizhevsky & Hinton, 2009) into 10 tasks where each task has 10 classes.

Original: 2) MiniImageNet Following Saha et al.
Revised: 2) MiniImageNet Following Saha et al.

Original: (2021), we split MiniImageNet (Vinyals et al., 2016) into 20 sequential tasks with 5 classes each. 3)
Revised: (2021), we split MiniImageNet (Vinyals et al., 2016) into 20 sequential tasks with 5 classes each. 3)

Original: Permuted MNIST (PMNIST) PMNIST (Kirkpatrick et al., 2017) is a variant of MNIST (LeCun et al., 1998) where each task has a different permutation of inputting images, consists of 10 sequential tasks withclasses each.
Revised: Permuted MNIST (PMNIST) PMNIST (Kirkpatrick et al., 2017) is a variant of MNIST (LeCun et al., 1998) where each task has a different permutation of inputting images, consists of 10 sequential tasks withclasses each.

Original: 4) CIFAR-100 Sup Following Yoon et al.
Revised: 4) CIFAR-100 Sup Following Yoon et al.

Original: (2019), we adopt CIFAR-100 Sup consisting of 20 superclasses as sequential tasks.
Revised: (2019), we adopt CIFAR-100 Sup consisting of 20 superclasses as sequential tasks.

Original: 5) Mixture Serra et al.
Revised: 5) Mixture Serra et al.

Original: first proposed Mixture consisting of 8 datasets, including CIFAR-10 (Krizhevsky & Hinton, 2009), MNIST (LeCun et al., 1998), CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), FashionMNIST (Xiao et al., 2017),
Revised: first proposed Mixture consisting of 8 datasets, including CIFAR-10 (Krizhevsky & Hinton, 2009), MNIST (LeCun et al., 1998), CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), FashionMNIST (Xiao et al., 2017),

Original: TrafficSigns (Stallkamp et al., 2011), FaceScrub (Ng & Winkler, 2014), and NotMNIST (Bulatov, 2011), from which Ebrahimi et al.
Revised: TrafficSigns (Stallkamp et al., 2011), FaceScrub (Ng & Winkler, 2014), and NotMNIST (Bulatov, 2011), from which Ebrahimi et al.

Original: (2020) further constructed 5-Datasets.
Revised: (2020) further constructed 5-Datasets.

Original: Here we follow the original harder benchmark.
Revised: Here we follow the original harder benchmark.

Original: Particularly, we consider all tasks as a sequence except TrafficSigns (Stallkamp et al., 2011), which we failed to access.
Revised: Particularly, we consider all tasks as a sequence except TrafficSigns (Stallkamp et al., 2011), which we failed to access.

Revised: Among all evaluated datasets,

Revised: PMNIST is a benchmark under the domain-incremental scenario, while other four datasets are under the task-incremental scenario.

Original: Moreover, we provide the statistics of selected datasets in Table 5 and Table 6.
Revised: Moreover [Table] we provide the statistics of selected datasets in Table 5 and Table 6.

Original: For the Mixture benchmark, the images of MNIST, FashionMNIST, and notMNIST are replicated across all RGB channels following Serra et al.
Revised: For the Mixture benchmark, the images of MNIST, FashionMNIST, and notMNIST are replicated across all RGB channels following Serra et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B. M ODEL D ETAILS
Revised: B. M ODEL D ETAILS

Original: MLP architecture:
Revised: MLP architecture:

Original: We adopt a 3-layer model including two hidden layers with 100 neurons each for the PMNIST setting, the same as Lopez-Paz & Ranzato (2017).
Revised: We adopt a 3-layer model including two hidden layers with 100 neurons each for the PMNIST setting, the same as Lopez-Paz & Ranzato (2017).

Original: ReLU is used as the activate function here and for all other architectures.
Revised: ReLU is used as the activate function here and for all other architectures.

Original: Also, we use softmax with cross entropy loss on all settings.
Revised: Also, we use softmax with cross entropy loss on all settings.

Original: AlexNet architecture: For CIFAR-100 Split setting, we adopt the same network as Serra et al.
Revised: AlexNet architecture: For CIFAR-100 Split setting, we adopt the same network as Serra et al.

Original: (2018) with batch normalization, including two fully connected layers and three convolutional layers.
Revised: (2018) with batch normalization, including two fully connected layers and three convolutional layers.

Original: The convolutional layers have 4 × 4 , 3 × 3 , and 2 × 2 kernel sizes with 64, 128, and 256 filters respectively.
Revised: The convolutional layers have 4 × 4 , 3 × 3 , and 2 × 2 kernel sizes with 64, 128, and 256 filters respectively.

Original: After each convolutional layer, we add batch normalization and 2 × 2 max-pooling.
Revised: After each convolutional layer, we add batch normalization and 2 × 2 max-pooling.

Original: Each fully connected layer has 2048 units.
Revised: Each fully connected layer has 2048 units.

Original: For the first two layers, we use the dropout of 0.2, and for the rest layers, we use the dropout of 0.5.
Revised: For the first two layers, we use the dropout of 0.2, and for the rest layers, we use the dropout of 0.5.

Original: Modified LeNet-5 architecture:
Revised: Modified LeNet-5 architecture:

Original: For the CIFAR-100 Sup setting, a modified LeNet-5 architecture consisting of two convolutional layers and two fully connected layers is adopted, similar to Saha et al.
Revised: For the CIFAR-100 Sup setting, a modified LeNet-5 architecture consisting of two convolutional layers and two fully connected layers is adopted, similar to Saha et al.

Original: Max-pooling of 3 × 2 is used after each convolutional layer.
Revised: Max-pooling of 3 × 2 is used after each convolutional layer.

Original: The last two layers have 800 and 500 units respectively.
Revised: The last two layers have 800 and 500 units respectively.

Original: Reduced ResNet-18 architecture: We adopt the same reduced ResNet-18 architecture as Saha et al.
Revised: Reduced ResNet-18 architecture: We adopt the same reduced ResNet-18 architecture as Saha et al.

Original: (2021) for the MiniImageNet and Mixture settings, using 2 × 2 average-pooling before the classifier layer instead of the 4 × 4 average-pooling used by Lopez-Paz & Ranzato (2017).
Revised: (2021) for the MiniImageNet and Mixture settings, using 2 × 2 average-pooling before the classifier layer instead of the 4 × 4 average-pooling used by Lopez-Paz & Ranzato (2017).

Original: Moreover, we present the dimension of the representation space of each layer of our architectures in Table 7.[Table]
Revised: Moreover, we present the dimension of the representation space of each layer of our architectures in Table 7.

Original: B. I MPLEMENTATION D ETAILS
Revised: B. I MPLEMENTATION D ETAILS

Original: We use the official implementation of GPM (Saha et al., 2021), OWM (Zeng et al., 2019), HAT Serra et al.
Revised: We use the official implementation of GPM (Saha et al., 2021), OWM (Zeng et al., 2019), HAT Serra et al.

Original: (2018), and TRGP (Lin et al., 2022).
Revised: (2018), and TRGP (Lin et al., 2022).

Revised: Moreover, we implement A-GEM and ER Res with the

Revised: [Table]

Original: Moreover, we implement EWC with theofficial implementation by Serra et al.
Revised: official implementation by Chaudhry et al. and implement EWC with the implementation by Serra et al.

Original: Following Saha et al.
Revised: Following Saha et al.

Original: (2021) and Lin et al.
Revised: (2021) and Lin et al.

Original: , we run all experiments five times on an established seed without fixing the cuda settings for a fair comparison.
Revised: , we run all experiments five times on an established seed without fixing the cuda settings for a fair comparison.

Original: Particularly, we use five random seeds on PMNIST where there is no diversity on a single seed.
Revised: Particularly, we use five random seeds on PMNIST where there is no diversity on a single seed.

Original: For CIFAR-100 Sup, we use five different orders provided by Yoon et al. 
Revised: For CIFAR-100 Sup, we use five different orders provided by Yoon et al. Following Saha et al.

Revised: (2021), we report the experimental results of replay-base methods A-GEM and ER Res on the Mixture dataset with the same buffer size as GPM and our IRGP, which is 8.98M in term of the number of parameters for the Resnet18 architecture.

Original: On CIFAR-100 Split, MiniImageNet, and PMNIST, we follow the hyper-parameters utilized by
Revised: On CIFAR-100 Split, MiniImageNet, and PMNIST, we follow the hyper-parameters utilized by

Original: Saha et al.
Revised: Saha et al.

Original: (2021) and Lin et al.
Revised: (2021) and Lin et al.

Original: (2022), including learning rate, batch size, and the threshold ϵ .
Revised: (2022), including learning rate, batch size, and the threshold ϵ .

Original: On
Revised: On

Original: Mixture, as we adopt the same network architecture Saha et al.
Revised: Mixture, as we adopt the same network architecture Saha et al.

Original: (2021) use on their 5-Dataset setting, we follow the provided learning rate and batch size as well.
Revised: (2021) use on their 5-Dataset setting, we follow the provided learning rate and batch size as well.

Original: Moreover, for the threshold ϵ in GPM, we conduct experiments with ϵ in the range of 0 .
Revised: Moreover, for the threshold ϵ in GPM, we conduct experiments with ϵ in the range of 0 .

Original: 95 to 1 provided in (Saha et al., 2021) and report the best results whose ϵ is 0 .
Revised: 95 to 1 provided in (Saha et al., 2021) and report the best results whose ϵ is 0 .

Original: We further use ϵ = 0 .
Revised: We further use ϵ = 0 .

Original: 96 for all layers in our IRGP.
Revised: 96 for all layers in our IRGP.

Original: As discussed in Section 5, the threshold ζ = cos γ controls the criterion of the relaxing subspace.
Revised: As discussed in Section 5, the threshold ζ = cos γ controls the criterion of the relaxing subspace.

Original: For CIFAR100-Split and PMNIST, we use ζ = 0 .
Revised: For CIFAR100-Split and PMNIST, we use ζ = 0 .

Original: 95 for convolutional layers and ζ = 0 .
Revised: 95 for convolutional layers and ζ = 0 .

Original: 9 for fully connected layers.
Revised: 9 for fully connected layers.

Original: For MiniImageNet and Mixture, we use the same ζ for all layers, 0 .
Revised: For MiniImageNet and Mixture, we use the same ζ for all layers, 0 .

Original: 95 and 0 .respectively.
Revised: 95 and 0 .respectively.

Original: Furthermore, we set the regularization weight as 5 for the ResNet18 architecture andfor others.
Revised: Furthermore, we set the regularization weight as 5 for the ResNet18 architecture andfor others.

Original: Particularly, we run all the experiments on a single NVIDIA GeForce RTX 2080 Ti GPU.
Revised: Particularly, we run all the experiments on a single NVIDIA GeForce RTX 2080 Ti GPU.

Original: C E XPERIMENTAL R ESULTS C. S TANDARD D EVIATION
Revised: B. M ETRICS

Revised: Here we present the detailed definitions of the metrics evaluating the forward knowledge transfer. Ω new (Kemker et al., 2018). Denote b i as the test accuracy of task i at random initialization, FWT, first proposed by Lopez-Paz & Ranzato (2017), is defined as FWT

Revised: = 1 T − 1 Σ Ti =2 ( A i − 1 ,i − b i ) , evaluating the zero-shot performance of the initialization with respect to the observed tasks.

Original: We provide the performance of ACC and BWT with standard deviation on four datasets in Table 8.
Revised: While Ω new , first proposed by Kemker et al. (2018), is defined as Ω new = 1 T − 1 Σ Ti =2 ( A i,i − b i ) , reflecting the test accuracy on new tasks based on the learnt knowledge. As b i stays still across different approaches, we consider Ω new = 1 T − 1 Σ

Original: [Table]{+Ti =
 A+}
Revised: {+Ti =
 A+}

Revised: i,i for simplicity.

Original:  C. F INAL A CCURACY
Revised: C E XPERIMENTAL R ESULTS C. F INAL A CCURACY

Original: We provide the test accuracy after learning each task on other benchmarks here.
Revised: We provide the test accuracy after learning each task on other benchmarks here.

Original: As discussed in
Revised: As discussed in

Original: Section 4, our IRGP universally outperforms GPM over the task sequence on all benchmarks.[Figure] [Equation] r
Revised: Section 4, our IRGP universally outperforms GPM over the task sequence on all benchmarks.

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure] C. F ORWARD  T RANSFER
Revised: C. F ORWARD K NOWLEDGE T RANSFER

Original: We provide the detailed accuracy of each task after learning it on all  benchmarks here.
Revised: We provide the detailed forward knowledge transfer performance on all four benchmarks here.

Original: According to 
Revised: First, we present the results of Ω new and the detailed accuracy of each task after learning it in Table 8 to 11.

Original:  Table 11 and Table 12, IRGP achieves a similar forward knowledge transfer compared with GPM.
Revised: According to Table 10 and Table 11, IRGP achieves a similar forward knowledge transfer compared with GPM.

Original: For other benchmarks, IRGP improves FWT by 2.7% and 1.8% onaverage on CIFAR-100 Split and 
Revised: For other benchmarks, IRGP improves Ω new by 2.7% and 1.8% on CIFAR-100 Split and MiniImageNet respectively, as shown in Table 8 and Table 9. Detailed results including the standard deviation and other baselines are provided in Table 12. According to Table 12, our IRGP gains better forward knowledge transfer than all baselines without extra memory buffer on Mixture. [Table]

Revised: [Figure]

Revised: [Table]

Original: MiniImageNet respectively, as shown in Table 9 and Table
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Moreover

Original: [Table]
Revised: [Table]

Revised: we provide the result of FWT (using the definition in (Lopez-Paz & Ranzato, 2017)) on all benchmarks in Table 13.

Revised: According to Table

Original: [Table]
Revised: [Table]

Revised: although our method facilitates the forward knowledge transfer reflected by Ω new , IRGP achieves better FWT than GPM on all three taskincremental benchmarks

Original: [Table] 
Revised: [Table] [Table] +0. [Table]

Revised: 1.49 Ours (IRGP) +

Revised: [Table]

Revised: [Table]

Revised: +

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: +

Original: [Table]
Revised: [Table]

Original:  A CCURACY E VOLUTION
Revised: C. A CCURACY E VOLUTION

Original: Here we present the accuracy tested on three randomly selected tasks after learning them on four benchmarks.
Revised: Here we present the accuracy tested on three randomly selected tasks after learning them on four benchmarks.

Original: We select the 2nd, 4th, and 6th tasks for CIFAR-100 Split, MiniImageNet, and PMNIST.
Revised: We select the 2nd, 4th, and 6th tasks for CIFAR-100 Split, MiniImageNet [Figure] and PMNIST.

Original: As there are only 7 tasks in Mixture, we select the 1st, 3rd, and 5th tasks.
Revised: As there are only 7 tasks in Mixture, we select the 1st, 3rd, and 5th tasks.

Original: Generally, IRGP outperforms GPM on selected tasks over the sequence.
Revised: Generally, IRGP outperforms GPM on selected tasks over the sequence.

Original: We further notice that the improvement is more significant on later tasks as a result of larger relaxing subspaces, as discussed in Section 5. 
Revised: We further notice that the improvement is more significant on later tasks as a result of larger relaxing subspaces, as discussed in Section 5. [Figure] [Figure] r

Revised: [Figure]

Original: GPMIRGP
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Table]
Revised: [Figure]

Revised: ) 4th, and (c) 6th task on PMNIST.

Original: GPMIRGP       

Revised: GPMIRGP

Original: {+GPMIRGP     
+}
Revised: {+GPMIRGP     
+}

Original: GPMIRGP
Revised: GPMIRGP

Original: [Figure]
Revised: [Figure]

Original: T e s t
Revised: T e s t

Original: A cc u r ac y ( % )
Revised: A cc u r ac y ( % )

Original: Task ID
Revised: Task ID

Original: T e s t
Revised: T e s t

Original: A cc u r ac y ( % )
Revised: A cc u r ac y ( % )

Original: [Table] ID
Revised: Task ID

Original: [Table] ID
Revised: Task ID

Original: T e s t
Revised: T e s t

Original: A cc u r ac y ( % )
Revised: A cc u r ac y ( % )

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: GPMIRGP     6 98.

Original: C.{+GPMIRGP   
 GPMIRGP [Figure]+} T IME C ONSUMPTION
Revised: {+GPMIRGP   
 GPMIRGP [Figure]+} T e s t

Original: We report the time consumption of IRGP on two benchmarks compared with relative baselines.
Revised: A cc u r ac y ( % ) Task ID T e s t A cc u r ac y ( % )

Original: TRGP and IRGP are both evaluated on a single NVIDIA GeForce RTX 2080 Ti GPU
Revised: Task ID Task ID T e s t

Original: and we report [Figure]
Revised: A cc u r ac y ( % )

Original: [Figure] [Figure] [Figure]
Revised: We report the time consumption of IRGP on two benchmarks compared with relative baselines.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: TRGP and IRGP are both evaluated on a single NVIDIA GeForce RTX 2080 Ti GPU

Original: [Table] the results according to (Lin et al., 2022).
Revised: and we report the results according to (Lin et al., 2022).

Original: As discussed in Section 5, IRGP takes acceptable extra time compared with GPM on both datasets.
Revised: As discussed in Section 5, IRGP takes acceptable extra time compared with GPM on both datasets.

Original: For CIFAR-100 Split, IRGP tasks similar time as TRGP, which is similar to ER Res and HAT and much less than A-GEM and OWM.
Revised: For CIFAR-100 Split, IRGP tasks similar time as TRGP, which is similar to ER Res and HAT and much less than A-GEM and OWM.

Original: For MiniImageNet,
Revised: For MiniImageNet,

Original: IRGP tasks around 30% time than TRGP, and is similar to A-GEM.
Revised: IRGP tasks around 30% time than TRGP, and is similar to A-GEM.

Original: [Table]
Revised: [Table]

Original: C. M EMORY U SAGE
Revised: C. M EMORY U SAGE

Original: We provide a comparison between TRGP and IRGP on the ratio of the amount of extra parameters concerning the amount of the parameters of the initial network architecture.
Revised: We provide a comparison between TRGP and IRGP on the ratio of the amount of extra parameters concerning the amount of the parameters of the initial network architecture.

Original: According to Figure 
Revised: According to Figure 9,

Original: [Table] TRGP requires at least 200% of the number of extra parameters after learning all tasks on the other three benchmarks, while IRGP only stores the representation of the frozen space, which can further be released in the inference phase. 
Revised: TRGP requires at least 200% of the number of extra parameters after learning all tasks on the other three benchmarks, while IRGP only stores the representation of the frozen space, which can further be released in the inference phase. [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: C.O THER R ESULTS
Revised: C. R ELAXING R ATIO

Revised: We provide the results on CIFAR100-Split between the hyper-parameter ζ and the relaxing ratio of all five layers of the AlexNet architecture in Table 15.

Revised: And β is set to be 1.0 for all experiments here.

Revised: According to Table 15, generally, larger ζ guarantees smaller relaxing ratios.

Revised: As discussed in

Revised: Section 5, forgetting is mitigated by constraining the percentage of the relaxing weights, as a result of increasing ζ .

Revised: [Table]

Revised: C. O THER R ESULTS

Original: We provide the test accuracy over the task sequence in Figure 10.
Revised: We provide the test accuracy over the task sequence in Figure 10.

Original: As shown in Figure 10-(a) and
Revised: As shown in Figure 10-(a) and

Original: Figure 10-(b), our IRGP-Exp dominates TRGP on both benchmarks relaxing either 80% or T% of the frozen spaces under an expansion setting.
Revised: Figure 10-(b), our IRGP-Exp dominates TRGP on both benchmarks relaxing either 80% or T% of the frozen spaces under an expansion setting.

Original: We further compare IRGP with TRGP modified with the same regularization terms.
Revised: We further compare IRGP with TRGP modified with the same regularization terms.

Original: As shown in Figure 10-(c) and Figure 10-(d), the performance of
Revised: As shown in Figure 10-(c) and Figure 10-(d), the performance of

Original: TRGP drops significantly constrained in a fixed network capacity on both benchmarks
Revised: TRGP drops significantly constrained in a fixed network capacity on both benchmarks.

Revised: Detailed results between IRGP and TRGP are provided in Table 16 and 17.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: T e s t A cc u r ac y ( % ) Task ID
Revised: [Figure]

Original: T e s t
Revised: [Table]

Original: A cc u r ac y ( % ) Task ID
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ID
Revised: [Figure]

Original: T e s t
Revised: [Table]

Original: A cc u r ac y ( % )
Revised: [Figure]

Original: T e s t
Revised: [Table]

Original: A cc u r ac y ( % ) Task ID (d)
Revised: [Figure] [Table] [Table]

Original: D A LGORITHM
Revised: D A LGORITHM

Original: We present the pseudo-code of our Iterative Relaxing Gradient Projection here.
Revised: We present the pseudo-code of our Iterative Relaxing Gradient Projection here.

Original: 1: Initiate frozen subspaces U

Original: { U l 0 } Ll

Original: 1 as ∅ s and

Original: W 1 for

Original: 2: Compute frozen subspace U

Original: 3: for do 4:

Original: Initiate relaxing subspace { V lt

Original: Ll s 5:

Original: Initiate scaling matrices { S lt } Ll

Original: 1 as identity matrix 1 s 6: repeat 7:

Original: Fine-tune for pre-defined e t epochs with objective function8:

Original: Search additional relaxing subspace { V l,newt } Ll =1 by Algorithm 1 with current gradients{ g lt } Ll =1 , remaining frozen subspace { U l,restt = U lt \ V lt } Ll =1 and thresholds { ϵ lth , γ lt } Ll = 9: V lt ← V lt ∪ V l,newt 10: Expand S lt with Equation (12) 11: until V l,newt is ∅ 12:
Revised: current gradients

Original: Optimize W t , S t with objective function13: Update W lt ← W lt − Proj V lt ( W lt ) + Proj S t V lt ( W lt ) 14:
Revised: =

Original: Update frozen subspace U t with Equation (4) 15: end for