Original: L EARNING WITH A UXILIARY A CTIVATION
Revised: L EARNING WITH A UXILIARY A CTIVATION

Original: FOR
Revised: FOR

Original: M EMORY -E FFICIENT T RAINING
Revised: M EMORY -E FFICIENT T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While deep learning has achieved great success in various fields, a large amount of memory is necessary to train deep neural networks, which hinders the development of massive state-of-the-art models.
Revised: While deep learning has achieved great success in various fields, a large amount of memory is necessary to train deep neural networks, which hinders the development of massive state-of-the-art models.

Original: The reason is the conventional learning rule, backpropagation, should temporarily store input activations of all the layers in the network.
Revised: The reason is the conventional learning rule, backpropagation, should temporarily store input activations of all the layers in the network.

Original: To overcome this, recent studies suggested various memoryefficient implementations of backpropagation.
Revised: To overcome this, recent studies suggested various memoryefficient implementations of backpropagation.

Original: However, those approaches incur computational overhead due to the recomputation of activations, slowing down neural network training.
Revised: However, those approaches incur computational overhead due to the recomputation of activations, slowing down neural network training.

Original: In this work, we propose a new learning rule which significantly reduces memory requirements while closely matching the performance of backpropagation.
Revised: In this work, we propose a new learning rule which significantly reduces memory requirements while closely matching the performance of backpropagation.

Original: The algorithm combines auxiliary activation with output activation during forward propagation, while only auxiliary activation is used during backward propagation instead of actual input activation to reduce the amount of data to be temporarily stored.
Revised: The algorithm combines auxiliary activation with output activation during forward propagation, while only auxiliary activation is used during backward propagation instead of actual input activation to reduce the amount of data to be temporarily stored.

Original: We mathematically show that our learning rule can reliably train the networkswhose loss landscape is convex if the auxiliary activation satisfies certain conditions.
Revised: We mathematically show that our learning rule can reliably train the networks if the auxiliary activation satisfies certain conditions.

Original: Based on this observation, we suggest candidates of auxiliary activation that satisfy those conditions.
Revised: Based on this observation, we suggest candidates of auxiliary activation that satisfy those conditions.

Original: Experimental results confirm that the proposed learning rule achieves competitive performance compared to backpropagation in various models such as ResNet, Transformer, BERT, ViT, and MLP-Mixer.
Revised: Experimental results confirm that the proposed learning rule achieves competitive performance compared to backpropagation in various models such as ResNet, Transformer, BERT, ViT, and MLP-Mixer.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Backpropagation (1) is an essential learning rule to train deep neural networks and has proven its outstanding performance in diverse models and tasks.
Revised: Backpropagation (1) is an essential learning rule to train deep neural networks and has proven its outstanding performance in diverse models and tasks.

Original: In general, the wider and deeper the deep learning model, the better the training performance (2).
Revised: In general, the wider and deeper the deep learning model, the better the training performance (2).

Original: However, increasing model size unavoidably requires larger memory in training hardware such as GPU (3; 4).
Revised: However, increasing model size unavoidably requires larger memory in training hardware such as GPU (3; 4).

Original: This is because backpropagation needs to temporarily store input activations of all the layers in the network generated in forward propagation as they are later used to update weights in backward propagation.
Revised: This is because backpropagation needs to temporarily store input activations of all the layers in the network generated in forward propagation as they are later used to update weights in backward propagation.

Original: Consequently, state-of-the-art deep learning models require substantial memory resources due to the large amount of input activation to store.
Revised: Consequently, state-of-the-art deep learning models require substantial memory resources due to the large amount of input activation to store.

Original: In order to train very deep models with limited hardware resources, the batch size may be reduced (3; 4) or many GPUs can be used in parallel (5; 6; 7; 8; 9; 10; 11).
Revised: In order to train very deep models with limited hardware resources, the batch size may be reduced (3; 4) or many GPUs can be used in parallel (5; 6; 7; 8; 9; 10; 11).

Original: However, reducing the batch size causes a long training time, and the advantage of batch normalization (12) disappears.
Revised: However, reducing the batch size causes a long training time, and the advantage of batch normalization (12) disappears.

Original: Also, training huge models such as GPT-3 (13) still requires expensive GPU clusters with thousands of GPUs and incurs high
Revised: Also, training huge models such as GPT-3 (13) still requires expensive GPU clusters with thousands of GPUs and incurs high

Original: I/O costs even with parallelism.
Revised: I/O costs even with parallelism.

Original: Recently, a wide range of algorithms have been proposed to alleviate this memory requirement.
Revised: Recently, a wide range of algorithms have been proposed to alleviate this memory requirement.

Original: For instance, gradient checkpointing (14; 15; 16; 17; 18) does not store all input activations, but only  some of them during forward propagation.
Revised: For instance, a new optimizer (14; 15) or neural network architectures (16; 17; 18; 19; 20; 21) have been suggested to reduce the memory requirements. Gradient checkpointing (22; 23; 24; 25; 26) reduces memory space by only storing some of the input activation during forward propagation.

Original: Then, it restores the unsaved input activations through recomputation in backward propagation.
Revised: Then, it restores the unsaved input activations through recomputation in backward propagation.

Original: In-place activated batch normalization (19) merges a batch normalization layer and a leaky ReLU layer and stores the output activation of the merged layer in forward propagation.
Revised: In-place activated batch normalization (27) merges a batch normalization layer and a leaky ReLU layer and stores the output activation of the merged layer in forward propagation.

Original: In backward propagation, the layer input can be reconstructed for training because the leaky ReLU function is reversible.
Revised: In backward propagation, the layer input can be reconstructed for training because the leaky ReLU function is reversible.

Original: Similarly, RevNet (20) and Reformer (21) employ reversible neural network architectures, which allow for calculating input activation from output activation in backward propagation.
Revised: Similarly, RevNet (28), Momentum ResNet (29), and Reformer (30) employ reversible neural network architectures, which allow for calculating input activation from output activation in backward propagation.

Original: Gradient checkpointing and reversible network structures reduce training memory space because they partially store input activations (e.g., input activations of selected layers).
Revised: Gradient checkpointing and reversible network structures reduce training memory space because they partially store input activations (e.g., input activations of selected layers).

Original: However, these methods incur additional computational overhead because the unstored input activations must be recomputed during backward propagation.
Revised: However, these methods incur additional computational overhead because the unstored input activations must be recomputed during backward propagation.

Original: Alternatively, algo- rithms to approximate activation have been suggested (22; 23; 24; 25; 26; 27; 28), but they suffer from performance degradation or slow down training due to additional computations to quantize and dequantize activations.
Revised: Alternatively, algorithms to approximate activation have been suggested (31; 32; 33; 34; 35; 36; 37; 38), but they suffer from performance degradation or slow down training due to additional computations to quantize and dequantize activations.

Original: TinyTL (29) entirely avoids saving activations by updating only bias parameters while fixing weight parameters.
Revised: TinyTL (39) entirely avoids saving activations by updating only bias parameters while fixing weight parameters.

Original: However, it is only applicable to fine-tuning of a pre-trained model.
Revised: However, it is only applicable to fine-tuning of a pre-trained model.

Original: In this study, we propose a new learning rule, Auxiliary Activation Learning, which can significantly reduce memory requirements for training deep neural networks without sacrificing training speed.
Revised: In this study, we propose a new learning rule, Auxiliary Activation Learning, which can significantly reduce memory requirements for training deep neural networks without sacrificing training speed.

Original: We first introduce the concept of auxiliary activation in the training process.
Revised: We first introduce the concept of auxiliary activation in the training process.

Original: Auxiliary activations are combined with output activations and become the input activation of the next layer when processing forward propagation, but only the auxiliary activations are temporarily stored instead of actual input activation for updating weights in backward propagation.
Revised: Auxiliary activations are combined with output activations and become the input activation of the next layer when processing forward propagation, but only the auxiliary activations are temporarily stored instead of actual input activation for updating weights in backward propagation.

Original: To justify our algorithm, we prove that an alternate type of input activation could reliably train neural networkswhose landscapes are convex if auxiliary activation satisfies certain conditions.
Revised: To justify our algorithm, we prove that an alternate type of input activation could reliably train neural networks if auxiliary activation satisfies certain conditions.

Original: Then, we propose multiple candidates of auxiliary activations which meet this criterion.
Revised: Then, we propose multiple candidates of auxiliary activations which meet this criterion.

Original: Experimental results demonstrate that the proposed algorithm not only succeeds in training ResNet models (30) on ImageNet (31) with similar performance to backpropagation, but is also suitable for training other neural network architectures such as Transformer (32), BERT (33), ViT (34), and MLP-Mixer (35).
Revised: Experimental results demonstrate that the proposed algorithm not only succeeds in training ResNet models (40) on ImageNet (41) with similar performance to backpropagation, but is also suitable for training other neural network architectures such as Transformer (42), BERT (43), ViT (44), and MLP-Mixer (45).

Original:  A UXILIARY A CTIVATION L EARNING
Revised:  A UXILIARY A CTIVATION L EARNING

Original: [Figure]
Revised: [Figure]

Original: 2. M EMORY REQUIREMENTS OF BACKPROPAGATION
Revised: 2. M EMORY REQUIREMENTS OF BACKPROPAGATION

Original: To understand why backpropagation uses a large memory space for training, we describe how it trains a deep neural network.
Revised: To understand why backpropagation uses a large memory space for training, we describe how it trains a deep neural network.

Original: The training process of backpropagation can be expressed by the equations below.
Revised: The training process of backpropagation can be expressed by the equations below.

Original: [Equation]
Revised: [Equation]

Original: Here h l , W l +1 , and b l +1 denote the input activation, weight, and bias of the hidden layer l + 1 , respectively.
Revised: Here h l , W l +1 , and b l +1 denote the input activation, weight, and bias of the hidden layer l + 1 , respectively.

Original: ϕ , δ , and η represent the nonlinear function, backpropagated error, and learning rate, respectively.
Revised: ϕ , δ , and η represent the nonlinear function, backpropagated error, and learning rate, respectively.

Original: y l is the output of a linear or convolutional layer (i.e., y l = W l h l − 1 + b l ), which becomes the input of the nonlinear function ϕ .
Revised: y l is the output of a linear or convolutional layer (i.e., y l = W l h l − 1 + b l ), which becomes the input of the nonlinear function ϕ .

Original: In forward propagation, the input propagates through the network following equation (1).
Revised: In forward propagation, the input propagates through the network following equation (1).

Original: During this process, the generated input activations are stored in memory.
Revised: During this process, the generated input activations are stored in memory.

Original: In backward propagation, the error is backpropagated through the network based on equation (2).
Revised: In backward propagation, the error is backpropagated through the network based on equation (2).

Original: The weights are updated using the backpropagated errors and the activations stored in memory (equation (3)).
Revised: The weights are updated using the backpropagated errors and the activations stored in memory (equation (3)).

Original: Due to the time gap between forward and backward computations, all activations obtained in forward propagation must be temporarily stored in memory as they are later used for backward propagation.
Revised: Due to the time gap between forward and backward computations, all activations obtained in forward propagation must be temporarily stored in memory as they are later used for backward propagation.

Original: Therefore, as the model size increases, the amount of activation to be stored in memory increases accordingly.
Revised: Therefore, as the model size increases, the amount of activation to be stored in memory increases accordingly.

Original: To alleviate this issue, gradient checkpointing (14; 15; 16; 17; 18) stores only some of the activations in memory during forward propagation, and the rest of the activations are restored during backpropagation by recomputing equation (1).
Revised: To alleviate this issue, gradient checkpointing (22; 23; 24; 25; 26) stores only some of the activations in memory during forward propagation, and the rest of the activations are restored during backpropagation by recomputing equation (1).

Original: Similarly, activation compression methods (22; 23; 24; 26; 27; 28) quantize activations before saving them in memory to reduce memory requirements, and they are dequantized during backward propagation.
Revised: Similarly, activation compression methods (31; 32; 33; 35; 36; 37) quantize activations before saving them in memory to reduce memory requirements, and they are dequantized during backward propagation.

Original: However, these methods incur computational overhead and increase training time due to additional processes during forward and backward propagation as depicted in Fig. 1a.
Revised: However, these methods incur computational overhead and increase training time due to additional processes during forward and backward propagation as depicted in Fig. 1a.

Original: 2. A UXILIARY A CTIVATION L EARNING : STORE AUXILIARY ACTIVATION INSTEAD OF EXACT ACTIVATION
Revised: 2. A UXILIARY A CTIVATION L EARNING : STORE AUXILIARY ACTIVATION INSTEAD OF EXACT ACTIVATION

Original: For memory-efficient training with minimal overhead, here we propose a new learning rule, Auxiliary Activation Learning .
Revised: For memory-efficient training with minimal overhead, here we propose a new learning rule, Auxiliary Activation Learning .

Original: First, we introduce the concept of auxiliary activation in the training process.
Revised: First, we introduce the concept of auxiliary activation in the training process.

Original: Then, the algorithm trains the model following the equations below.
Revised: Then, the algorithm trains the model following the equations below.

Original: [Equation]
Revised: [Equation]

Original: Here a l represents the newly introduced auxiliary activation of layer l .
Revised: Here a l represents the newly introduced auxiliary activation of layer l .

Original: In forward propagation, each layer takes the sum of auxiliary activation a l and the previous layer’s output activation o l as input activation h l (equation (4)).
Revised: In forward propagation, each layer takes the sum of auxiliary activation a l and the previous layer’s output activation o l as input activation h l (equation (4)).

Original: The generated input activation propagates through the layer following equation (5).
Revised: The generated input activation propagates through the layer following equation (5).

Original: In backward propagation, the error δ l +1 is backpropagated by equation (6).
Revised: In backward propagation, the error δ l +1 is backpropagated by equation (6).

Original: Finally, the weights are updated only using auxiliary activation a l instead of the actual input activation h l following equation (7).
Revised: Finally, the weights are updated only using auxiliary activation a l instead of the actual input activation h l following equation (7).

Original: The learning process of our algorithm is depicted in Fig.
Revised: The learning process of our algorithm is depicted in Fig.

Original: 1b.
Revised: 1b.

Original: Now the memory needs to store the auxiliary activations a l , not real activations h l .
Revised: Now the memory needs to store the auxiliary activations a l , not real activations h l .

Original: Consequently, if the auxiliary activation is reused (e.g., a l = a l +1 ) or the amount by auxiliary activation itself is small, the required memory space can be reduced accordingly.
Revised: Consequently, if the auxiliary activation is reused (e.g., a l = a l +1 ) or the amount by auxiliary activation itself is small, the required memory space can be reduced accordingly.

Original: Furthermore, it does not require additional computations, avoiding training time increases.
Revised: Furthermore, it does not require additional computations, avoiding training time increases.

Original: The intuition behind only using auxiliary activation for weight updates is that the direction of a weight update obtained using auxiliary activation could be similar to that of backpropagation if we carefully choose auxiliary activation.
Revised: The intuition behind only using auxiliary activation for weight updates is that the direction of a weight update obtained using auxiliary activation could be similar to that of backpropagation if we carefully choose auxiliary activation.

Original: More specifically, we use the sum of the auxiliary activation a l and the previous layer’s output activation o l as the input activation h l (equation (4)).
Revised: More specifically, we use the sum of the auxiliary activation a l and the previous layer’s output activation o l as the input activation h l (equation (4)).

Original: If we take an existing value (e.g., activation from a different layer) as the auxiliary activation, this can be seen as making an additional connection to the layer.
Revised: If we take an existing value (e.g., activation from a different layer) as the auxiliary activation, this can be seen as making an additional connection to the layer.

Original: If we update weights using backpropagation as in equation (3), the input activation h l generated in forward propagation will be used for updating weights: ∆ W BPl +1 = − η δ l +1 h Tl = − η δ l +1 ( o l + a l ) T = − η δ l +1 o Tl − η δ l +1 a Tl .
Revised: If we update weights using backpropagation as in equation (3), the input activation h l generated in forward propagation will be used for updating weights: ∆ W BPl +1 = − η δ l +1 h Tl = − η δ l +1 ( o l + a l ) T = − η δ l +1 o Tl − η δ l +1 a Tl .

Original: Consequently, the weight gradient obtained by using our algorithm can be expressed by ∆ W AALl + = ∆ W BPl +1 + η δ l +1 o Tl .
Revised: Consequently, the weight gradient obtained by using our algorithm can be expressed by ∆ W AALl + = ∆ W BPl +1 + η δ l +1 o Tl .

Original: Therefore, it is expected that the weight update direction of our algorithm can be similar to that of backpropagation if the effect of the second term is ignorable, which will result in proper training of the neural network.
Revised: Therefore, it is expected that the weight update direction of our algorithm can be similar to that of backpropagation if the effect of the second term is ignorable, which will result in proper training of the neural network.

Original: In the next section, we prove that this assumption can be realized if we choose the auxiliary activation appropriately.
Revised: In the next section, we prove that this assumption can be realized if we choose the auxiliary activation appropriately.

Original:  A NALYSIS OF A UXILIARY A CTIVATION L EARNING 3. H OW AND WHEN USING ALTERNATIVE ACTIVATION CAN TRAIN DEEP NEURAL NETWORKS
Revised:  A NALYSIS OF A UXILIARY A CTIVATION L EARNING 3. H OW AND WHEN USING ALTERNATIVE ACTIVATION CAN TRAIN DEEP NEURAL NETWORKS

Original: This section shows how learning with an alternative form of activation can make the loss of nonlinear networks converge to a minimum value if a specific condition is satisfied.
Revised: This section shows how learning with an alternative form of activation can make the loss of nonlinear networks converge to a minimum value if a specific condition is satisfied.

Original: For mathematical analysis, we consider a loss function f ( x ) which isconvex, differentiable, and Lipschitz continuous.
Revised: For mathematical analysis, we consider a loss function f ( x ) whose gradient is Lipschitz continuous.

Original: h and r are column vectors representing real activations and random alternative activations, respectively.
Revised: h and r are column vectors representing real activations and random alternative activations, respectively.

Original: δ is a column vector of backpropagated errors, and W is a flattened vector of weights.
Revised: δ is a column vector of backpropagated errors, and W is a flattened vector of weights.

Original: When weights are updated by backpropagation using exact activationsh ,
Revised: When weights are updated by backpropagation using exact activations

Original: [Equation]
Revised: [Equation]

Original: where t denotes the current epoch.
Revised: where t denotes the current epoch.

Original: Contrarily, now we assume that the weights are updated using alternative activations r l as below, which represents a generalization of our algorithm.
Revised: Contrarily, now we assume that the weights are updated using alternative activations r l as below, which represents a generalization of our algorithm.

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: If r Tl (2 h l − r l ) > 0 in all layers and the step size η satisfies 0 ≤ η ≤ 1 Lwhere L = ∇ 2 f , then the loss function f ( W )  converges to a minimum value.
Revised: If r Tl (2 h l − r l ) > 0 in all layers, the step size η satisfies 0 < η ≤ 1 L , and the gradient of the loss function f ( W ) is Lipschitz continuous, then f ( W ) converges to a minimum value.

Original: Theorem 1 can be proven by applying equations (9)-(10) to inequality for a convex function with an L-Lipschitz continuous gradient.
Revised: Theorem 1 can be proven by applying equations (9)-(10) to the quadratic upper bound.

Original: A detailed proof is provided in Appendix A. Because condition 0 ≤ η ≤ 1 L can be easily satisfied if we use a small step size, r Tl (2 h l − r l ) ≥ 0 is the most important condition for successful learning that alternative activations should satisfy.
Revised: A detailed proof is provided in Appendix A. Because condition 0 < η ≤ 1 L can be easily satisfied if we use a small step size, r Tl (2 h l − r l ) ≥ 0 is the most important condition for successful learning that alternative activations should satisfy.

Original: We name this condition Learning Criterion .
Revised: We name this condition Learning Criterion .

Original: Furthermore, the larger the value of r Tl (2 h l − r l ) , the better the network converges because the amount of loss reduction in each epoch ( K n in Appendix A) increases.
Revised: Furthermore, the larger the value of r Tl (2 h l − r l ) , the better the network converges because the amount of loss reduction in each epoch ( K n in Appendix A) increases.

Original: One interesting observation is that r Tl (2 h 
Revised: One interesting observation is that r Tl (2 h l − r l ) =

Original: [Equation] − r [Equation]
Revised: 2 r l · h l −∥ r l ∥ 2 = −∥

Original:  reaches its maximum when r l = h l , which translates to a conventional backpropagation algorithm.
Revised: h l − r l ∥ 2 + ∥ h l ∥ 2 reaches its maximum when r l = h l , which translates to a conventional backpropagation algorithm.

Original: For convenience, we normalize the learning criterion inequality with respect to its maximum value ∥
Revised: For convenience, we normalize the learning criterion inequality with respect to its maximum value ∥

Original: h l ∥ 2 and define a learning indicator as below.
Revised: h l ∥ 2 and define a learning indicator as below.

Original: [Equation]
Revised: [Equation]

Original: In summary, alternative activation can train the network if the learning indicator in equation (11) is positive.
Revised: In summary, alternative activation can train the network if the learning indicator in equation (11) is positive.

Original: Moreover, the larger the learning indicator, the better the training performance.
Revised: Moreover, the larger the learning indicator, the better the training performance.

Original: Finally, the training performance is maximized when the learning indicator is 1 ( r l = h l , backpropagation).
Revised: Finally, the training performance is maximized when the learning indicator is 1 ( r l = h l , backpropagation).

Original: 3. C ONSTRUCTING AUXILIARY ACTIVATIONS
Revised: 3. C ONSTRUCTING AUXILIARY ACTIVATIONS

Original: [Figure]
Revised: [Figure]

Original: Theorem 1 suggests that using an alternate type of activation in calculating weight gradient could still reliably train the neural network.
Revised: Theorem 1 suggests that using an alternate type of activation in calculating weight gradient could still reliably train the neural network.

Original: However, in reality, it is cumbersome to meet the learning criterion if we use an arbitrary value as the alternate activation r l (see Appendix B).
Revised: However, in reality, it is cumbersome to meet the learning criterion if we use an arbitrary value as the alternate activation r l (see Appendix C).

Original: To address this, we modify the forward propagation path by adding auxiliary activation to the input activation as in equation (4) in the proposed Auxiliary Activation Learning.
Revised: To address this, we modify the forward propagation path by adding auxiliary activation to the input activation as in equation (4) in the proposed Auxiliary Activation Learning.

Original: Then, the input activation h l =
Revised: Then, the input activation h l =

Original: o l + a l and the alternate activation r l = a l share a common term a l , making it easier to meet the learning criterion.
Revised: o l + a l and the alternate activation r l = a l share a common term a l , making it easier to meet the learning criterion.

Original: Also, if we employ some values already existing in the network as auxiliary activation a l , this modification can be regarded as introducing an additional connection in the layer.
Revised: Also, if we employ some values already existing in the network as auxiliary activation a l , this modification can be regarded as introducing an additional connection in the layer.

Original: The learning indicator of Auxiliary Activation Learning is expressed by
Revised: The learning indicator of Auxiliary Activation Learning is expressed by

Original: [Equation]
Revised: [Equation]

Original: by applying equation (4) to equation (11).
Revised: by applying equation (4) to equation (11).

Original: For successful learning, the value of the learning indicator in equation (12) should be positive.
Revised: For successful learning, the value of the learning indicator in equation (12) should be positive.

Original: To satisfy this learning criterion, we suggest two candidates of auxiliary activation as below.
Revised: To satisfy this learning criterion, we suggest two candidates of auxiliary activation as below.

Original: Auxiliary Residual Activation First, we can use the previous layer’s input activations as auxiliary activations and combine them with output activations, which is identical to having a residual connection.
Revised: Auxiliary Residual Activation First, we can use the previous layer’s input activations as auxiliary activations and combine them with output activations, which is identical to having a residual connection.

Original: Then, the learning process is expressed by
Revised: Then, the learning process is expressed by

Original: [Equation]
Revised: [Equation]

Original: Using auxiliary residual activations can save training memory if we only store h l and use it for updating W l +1 using backpropagation as well as for updating W l +2 and W l +3 using Auxiliary
Revised: Using auxiliary residual activations can save training memory if we only store h l and use it for updating W l +1 using backpropagation as well as for updating W l +2 and W l +3 using Auxiliary

Original: Activation Learning as depicted in Fig.
Revised: Activation Learning as depicted in Fig.

Original: 2a and equation (14).
Revised: 2a and equation (14).

Original: Contrarily, backpropagation needs to store h l , h l +1 , and h l +2 for updating these three layers.
Revised: Contrarily, backpropagation needs to store h l , h l +1 , and h l +2 for updating these three layers.

Original: Auxiliary Sign Activations
Revised: Auxiliary Sign Activations

Original: Another option is using the sign of the output activation as auxiliary activations.
Revised: Another option is using the sign of the output activation as auxiliary activations.

Original: Then, the weight update process is represented by the equations below.
Revised: Then, the weight update process is represented by the equations below.

Original: [Equation]
Revised: [Equation]

Original: The auxiliary sign activation ( s l = ϵ sign( o l ) ) can be guaranteed to satisfy the learning criterion.
Revised: The auxiliary sign activation ( s l = ϵ sign( o l ) ) can be guaranteed to satisfy the learning criterion.

Original: Because 2 a Tl o l = 2 ϵ sign( o l ) T o l ≥ 0 , the learning indicator in equation (12) is always positive.
Revised: Because 2 a Tl o l = 2 ϵ sign( o l ) T o l ≥ 0 , the learning indicator in equation (12) is always positive.

Original: Using auxiliary sign activation can enable memory-efficient training by storing only a 1-bit sign of activation instead of actual input activation in high precision as shown in Fig. 2b.
Revised: Using auxiliary sign activation can enable memory-efficient training by storing only a 1-bit sign of activation instead of actual input activation in high precision as shown in Fig. 2b.

Original: In equations (15) and (16), we multiply the auxiliary activation by a hyperparameter ϵ to make the two types of activation comparable in magnitude since the output of the sign function is -1 or 1.
Revised: In equations (15) and (16), we multiply the auxiliary activation by a hyperparameter ϵ to make the two types of activation comparable in magnitude since the output of the sign function is -1 or 1.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In experiments, we trained various deep neural networks using the proposed learning rule and compared its training performance to that of backpropagation.
Revised: In experiments, we trained various deep neural networks using the proposed learning rule and compared its training performance to that of backpropagation.

Original: We observed if the selected auxiliary activations satisfied the learning criterion during training and also measured the amount of memory savings and training time.
Revised: We observed if the selected auxiliary activations satisfied the learning criterion during training and also measured the amount of memory savings and training time.

Original: Our algorithm was used to train ResNet (30) as well as transformer and its variants (32; 33; 34; 35), which are widely used architectures in vision and NLP-related applications.
Revised: Our algorithm was used to train ResNet (40) as well as transformer and its variants (42; 43; 44; 45), which are widely used architectures in vision and NLP-related applications.

Original: 4. T RAINING R ES N ET
Revised: 4. T RAINING R ES N ET

Original: We applied the proposed Auxiliary Activation Learning to ResNet training using two types of auxiliary activation as discussed above: Auxiliary Residual Activation (ARA) and Auxiliary Sign Activation (ASA).
Revised: We applied the proposed Auxiliary Activation Learning to ResNet training using two types of auxiliary activation as discussed above: Auxiliary Residual Activation (ARA) and Auxiliary Sign Activation (ASA).

Original: For ARA, we replaced the first convolutional layer, which receives a residual connection from the previous block, with an ARA-Conv layer which performs weight updates using auxiliary residual activations instead of actual activations (Fig. 3a).
Revised: For ARA, we replaced the first convolutional layer, which receives a residual connection from the previous block, with an ARA-Conv layer which performs weight updates using auxiliary residual activations instead of actual activations (Fig. 3a).

Original: Hence, memory overhead can be reduced for these ARA-Conv layers.
Revised: Hence, memory overhead can be reduced for these ARA-Conv layers.

Original: The blocks in ResNet (30) can be categorized into different sets based on their dimension.
Revised: The blocks in ResNet (40) can be categorized into different sets based on their dimension.

Original: For example, ResNet-50 consists of blocks with an input channel dimension of 256, 512, 1024, or 2048 as displayed in Fig.
Revised: For example, ResNet-50 consists of blocks with an input channel dimension of 256, 512, 1024, or 2048 as displayed in Fig.

Original: 3b.
Revised: 3b.

Original: In experiments with Auxiliary Activation Learning, the blocks are replaced with ARA blocks with different strides (”sharing stride”) in each set of blocks as shown in Fig.
Revised: In experiments with Auxiliary Activation Learning, the blocks are replaced with ARA blocks with different strides (”sharing stride”) in each set of blocks as shown in Fig.

Original: 3c and 3d.
Revised: 3c and 3d.

Original: For instance, ARA(3,4,4,2) has a sharing stride of 3, 4, 4, and 2 in the sets with a dimension of 256, 512, 1024, and 2048, respectively (Fig. 3d).
Revised: For instance, ARA(3,4,4,2) has a sharing stride of 3, 4, 4, and 2 in the sets with a dimension of 256, 512, 1024, and 2048, respectively (Fig. 3d).

Original: Therefore, the larger the sharing stride, the more blocks that are replaced by ARA blocks and the larger the amount of memory savings.
Revised: Therefore, the larger the sharing stride, the more blocks that are replaced by ARA blocks and the larger the amount of memory savings.

Original: Similarly, for experiments with auxiliary sign activation, the first convolutional layer was replaced by an ASA-Conv layer, which employs auxiliary sign activation
Revised: Similarly, for experiments with auxiliary sign activation, the first convolutional layer was replaced by an ASA-Conv layer, which employs auxiliary sign activation

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: for weight updates.
Revised: for weight updates.

Original: Other details of the experiments including hyperparameters can be found in
Revised: Other details of the experiments including hyperparameters can be found in

Original: Appendix
Revised: Appendix

Original: We first trained ResNet-18 on CIFAR-10, CIFAR-100 (36), and Tiny-ImageNet (37) by using BP,
Revised: We first trained ResNet-18 on CIFAR-10, CIFAR-100 (46), and Tiny-ImageNet (47) by using BP,

Original: ARA, and ASA to confirm the effectiveness of Auxiliary Activation Learning and the correlation between training performance and the value of the learning indicator.
Revised: ARA, and ASA to confirm the effectiveness of Auxiliary Activation Learning and the correlation between training performance and the value of the learning indicator.

Original: Fig.
Revised: Fig.

Original: 4a shows that ARA not only satisfies the learning criterion (i.e., keeps learning indicator values positive), but also results in relatively large learning indicator values close to 1.
Revised: 4a shows that ARA not only satisfies the learning criterion (i.e., keeps learning indicator values positive), but also results in relatively large learning indicator values close to 1.

Original: This is consistent with the training performance displayed in Table 1; ARA achieves high accuracy and even outperforms backpropagation in some cases.
Revised: This is consistent with the training performance displayed in Table 1; ARA achieves high accuracy and even outperforms backpropagation in some cases.

Original: Fig.
Revised: Fig.

Original: 4b shows that ASA satisfies the learning criterion, but the values of the indicator are much lower than that of ARA.
Revised: 4b shows that ASA satisfies the learning criterion, but the values of the indicator are much lower than that of ARA.

Original: Accordingly, ASA achieves slightly lower accuracy than BP and ARA.
Revised: Accordingly, ASA achieves slightly lower accuracy than BP and ARA.

Revised: More detailed comparisons between ARA and ASA can be founds in Appendix B.

Original: In addition, the experiments in Appendix B show that the performance is significantly degraded when the learning criterion is unsatisfied due to using modifications of auxiliary activation.
Revised: In addition, the experiments in Appendix C show that the performance is significantly degraded when the learning criterion is unsatisfied due to using modifications of auxiliary activation.

Original: These results suggest that the proposed learning indicator has a strong correlation with training performance.
Revised: These results suggest that the proposed learning indicator has a strong correlation with training performance.

Original: In the next experiment, we applied ARA to training ResNet-50 and ResNet-152 on ImageNet with various sharing stride values.
Revised: In the next experiment, we applied ARA to training ResNet-50 and ResNet-152 on ImageNet with various sharing stride values.

Original: Experimental results are summarized in Table 2.
Revised: Experimental results are summarized in Table 2.

Original: ARA achieved competitive performance close to BP in all cases, and ARA(3,4,2,2) even achieved slightly higher 
Revised: ARA achieved competitive performance close to BP in all cases, and ARA(3,4,2,2) even achieved slightly higher performance than BP in ResNet-152.

Revised: Since the mechanism of our algorithm is entirely different

Original: [Table]
Revised: [Table]

Original: performance than BP in ResNet-152. Since the mechanism of our algorithm is entirely different from other memory-saving techniques, our algorithm can be employed in addition to those conventional approaches for even larger memory savings.
Revised: from other memory-saving techniques, our algorithm can be employed in addition to those conventional approaches for even larger memory savings.

Original: For experiments, we trained the same networks using gradient checkpointing (GCP) (16) and ActNN (25) which is per-group activation compression training.
Revised: For experiments, we trained the same networks using gradient checkpointing (GCP) (24) and ActNN (34) which is per-group activation compression training.

Original: Then, we additionally applied our ARA to these learning rules.
Revised: Then, we additionally applied our ARA to these learning rules.

Original: More specifically, GCP was applied to the layers that did not employ ARA, whereas ActNN was applied to all layers including
Revised: More specifically, GCP was applied to the layers that did not employ ARA, whereas ActNN was applied to all layers including

Original: ARA Conv layers (see Appendices C.1 and C.2 for details).
Revised: ARA Conv layers (see Appendices H.1 and H.2 for details).

Original: Experimental results clearly show that our algorithm allows for larger memory savings with a very small increase in training time.
Revised: Experimental results clearly show that our algorithm allows for larger memory savings with a very small increase in training time.

Original: This is in contrast with GCP and ActNN, which noticeably slows down training due to additional computations for reconstructing activations during backward propagation.
Revised: This is in contrast with GCP and ActNN, which noticeably slows down training due to additional computations for reconstructing activations during backward propagation.

Original: For instance, applying ActNN alone to ResNet-50 reduces memory space by 11.8x while increasing training time from 18m to 

Revised: For instance, applying ActNN alone to ResNet-50 reduces memory space by 11.8x while increasing training time from 18m to

Original: m compared to BP.
Revised: m compared to BP.

Original: Contrarily, additionally applying ARA(3,4,6,2) increases the compression rate to 15.01x without impacting training speed.
Revised: Contrarily, additionally applying ARA(3,4,6,2) increases the compression rate to 15.01x without impacting training speed.

Revised: We observed a similar trend in experiments under identical compression rates (see Appendix D).

Revised: Furthermore, we additionally compared our ARA with

Revised: Momentum ResNet (29), which is reversible network for saving training memory in Appendix E.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. T RAINING T RANSFORMER , V I T, AND MLP-M IXER
Revised: T RAINING T RANSFORMER , V I T, AND MLP-M IXER

Original: Transformer (32) achieved state-of-the-art performance in NLP
Revised: Transformer (42) achieved state-of-the-art performance in NLP

Original: [Table]
Revised: [Table]

Original: and the models inspired by Transformer such as BERT (33), ViT (34)
Revised: and the models inspired by Transformer such as BERT (43), ViT (44)

Original: [Table]
Revised: [Table]

Original: and MLP-Mixer (35) are producing promising results in computer vision.
Revised: and MLP-Mixer (45) are producing promising results in computer vision.

Original: The structure of these models is quite distinct from ResNet models (30; 38).
Revised: The structure of these models is quite distinct from ResNet models (40; 48).

Original: Hence, we applied our Auxiliary Activation Learning to Transformer and its variants to verify the scalability of our algorithm.
Revised: Hence, we applied our Auxiliary Activation Learning to Transformer and its variants to verify the scalability of our algorithm.

Original: We trained Transformer on IWSLT (39) and BERT-Large on  MNLI (40) datasets.
Revised: We trained Transformer on IWSLT (49) and BERT-Large on MRPC (50) and MNLI (51) datasets.

Original: For

Original:  ViT-Large and MLP-Mixer-Large, we used CIFAR-100 (36).
Revised: For ViT-Large and MLP-Mixer-Large, we used CIFAR-100 (46).

Original: Note that ARA was not used in this experiment since residual connections are connected to layer normalization layers (41) rather than to fully-connected layers, whereas ARA requires residual connections to directly connect to fullyconnected layers to realize equation (13).
Revised: Note that ARA was not used in this experiment since residual connections are connected to layer normalization layers (52) rather than to fully-connected layers, whereas ARA requires residual connections to directly connect to fully-connected layers to realize equation (13).

Original: Instead, we applied ASA to linear layers in multi-head attention and feedforward network.
Revised: Instead, we applied ASA to linear layers in multi-head attention and feedforward network.

Original: Fig.
Revised: Fig.

Original: 5 depicts how ASA is applied to those layers.
Revised: 5 depicts how ASA is applied to those layers.

Original: Since ASALinear layers store only 1-bit auxiliary sign activation instead of exact activation, memory overhead can be mitigated.
Revised: Since ASA-Linear layers store only 1-bit auxiliary sign activation instead of exact activation, memory overhead can be mitigated.

Original: We trained networks with BP and Mesa (27) which is activation compression training for Transformer-like models.
Revised: We trained networks with BP and Mesa (36) which is activation compression training for Transformer-like models.

Original: Then, we additionally applied ASA to these algorithms (see[Figure] Appendix C.2 for details).
Revised: Then, we additionally applied ASA to these algorithms (see Appendix H.2 for details).

Original: Mesa stores 8-bit quantized activation for linear layers; hence, applying
Revised: Mesa stores 8-bit quantized activation for linear layers; [Figure]

Original:  ASA to these linear layers can further reduce memory space as ASA-Linear layers only store 1-bit sign activations.
Revised: hence, applying ASA to these linear layers can further reduce memory space as ASA-Linear layers only store 1-bit sign activations.

Original: Because Mesa is designed for the networks with self-attention, it is not applied to MLP-Mixer which does not employ self-attention layers.
Revised: Because Mesa is designed for the networks with self-attention, it is not applied to MLP-Mixer which does not employ self-attention layers.

Original: Experimental results are displayed in Table 3.
Revised: Experimental results are displayed in Table 3.

Original: When we apply ASA only, it closely matches or even outperforms BP while using smaller memory space.
Revised: When we apply ASA only, it closely matches or even outperforms BP while using smaller memory space.

Original: Since ASA needs to extract 1-bit sign from actual activation to construct auxiliary activation, training time is slightly increased.
Revised: Since ASA needs to extract 1-bit sign from actual activation to construct auxiliary activation, training time is slightly increased.

Original: However,[Table] it is negligible compared to Mesa which requires quantization and dequantization during training.
Revised: However, it is negligible compared to Mesa which requires quantization and dequantization during training.

Original: For example, when training BERT-Large on MNLI, BP and ASA consumed 2h 37m and 2h 41m,  respectively, while Mesa takes 3h 51m. We can maximize memory savings by applying Mesa and our algorithm (ASA) simultaneously.
Revised: For example, when training BERT-Large on MNLI, BP and ASA consumed 2h 37m and 2h 41m, [Table] respectively, while Mesa takes 3h 51m. We can maximize memory savings by applying Mesa and our algorithm (ASA) simultaneously.

Original: If we train ViT-Large on CIFAR-100 using Mesa only, we can obtain a 3.0x compression rate.
Revised: If we train ViT-Large on CIFAR-100 using Mesa only, we can obtain a 3.0x compression rate.

Original: On the other hand, applying both Mesa and ASA4 simultaneously results in the compression rate of 4.3x.
Revised: On the other hand, applying both Mesa and ASA4 simultaneously results in the compression rate of 4.3x.

Original: Furthermore, the training time is reduced by 1m 13s because linear layers are replaced with ASA-Linear layers, avoiding costly quantization and dequantization processes for these layers.
Revised: Furthermore, the training time is reduced by 1m 13s because linear layers are replaced with ASA-Linear layers, avoiding costly quantization and dequantization processes for these layers.

Revised: Experimental results under identical compression rates can be found in Appendix D, which confirm that ASA enables faster training than Mesa with identical memory savings.

Original: 4. U SAGE CASE OF A UXILIARY A CTIVATION L EARNING
Revised: 4. U SAGE CASE OF A UXILIARY A CTIVATION L EARNING

Original: The proposed Auxiliary Activation Learning algorithm allows for training deep neural networks on the systems with limited hardware resources by alleviating memory overheads with minimal computational cost.
Revised: The proposed Auxiliary Activation Learning algorithm allows for training deep neural networks on the systems with limited hardware resources by alleviating memory overheads with minimal computational cost.

Original: For example, one may use a larger batch size using the same  number of GPUs, which enables faster training.
Revised: For example, one may increase the width or depth of a neural network to improve its accuracy. In experiments, we scaled ResNet-152 and BERT-Large using the same batch size to confirm the effectiveness of our algorithm. For ResNet-152, we fixed other parameters and increased the number of layers or the width of the bottleneck block following the scheme suggested by Chen et al. We also scaled BERT-Large by increasing the number of transformer blocks or the hidden size following the scheme proposed by Liu et al.

Original: For experiments, we trained various networks using only three RTX3090 GPUs to confirm the effectiveness of our algorithm.
Revised: The results are summarized in Table 4. [Table] Using ARA, we can train 13% deeper or 22% wider models compared to using BP only.

Original: The experimental results for ResNetand ViT-Large are displayed in Tables 4 and 5. Using memory-saving techniques allow for using a larger batch size for a given GPU memory space.
Revised: Similarly, additionally applying ARA allows for using 15% deeper or 11% wider models compared to using

Original: Interestingly, although conventional algorithms such as GCP, ActNN, and Mesa enables a larger batch size, it does not translate to faster training speed due to additional computations.
Revised: ActNN only. For BERT-Large, we can train 20% deeper or 8% wider models by using ASA. Additionally applying ASA allows for using 9% deeper or 4% wider model compared to using Mesa only.

Original: In contrast, our ARA and ASA algorithms take full advantage of larger batch size and improve training speed. [Table] [Table]
Revised: Our algorithm may be used to increase the batch size for faster training using identical GPU resources, and experimental results are provided in Appendix F.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we proposed a new learning rule, Auxiliary Activation Learning, which effectively alleviates memory overhead in deep neural network training.
Revised: In this work, we proposed a new learning rule, Auxiliary Activation Learning, which effectively alleviates memory overhead in deep neural network training.

Original: The algorithm employs an alternate form of activations for weight updates, and hence reduces the amount of data to be stored in memory without additional computations.
Revised: The algorithm employs an alternate form of activations for weight updates, and hence reduces the amount of data to be stored in memory without additional computations.

Original: We proved that the algorithm can train neural networkswhose landscape is convex if an alternative form of activations satisfies a certain condition and also provided a performance indicator that can anticipate the training performance of the selected auxiliary activation type.
Revised: We proved that the algorithm can train neural networks if an alternative form of activations satisfies a certain condition and also provided a performance indicator that can anticipate the training performance of the selected auxiliary activation type.

Original: Experimental results confirmed that the proposed algorithm successfully trains various deep neural networks for vision and NLP tasks with smaller memory without training speed reduction.
Revised: Experimental results confirmed that the proposed algorithm successfully trains various deep neural networks for vision and NLP tasks with smaller memory without training speed reduction.

Original: Furthermore, since our algorithm is orthogonal to existing memory saving algorithms, it is possible to increase the compression rate as much as possible without reduction in training speed by applying existing algorithms and our algorithm simultaneously.
Revised: Furthermore, since our algorithm is orthogonal to existing memory saving algorithms, it is possible to increase the compression rate as much as possible without reduction in training speed by applying existing algorithms and our algorithm simultaneously.

Original: Since our study aims to save training memory in linear and convolutional layers, it currently cannot be applied to layers that perform other types of operations such as normalization and activation functions.
Revised: Since our study aims to save training memory in linear and convolutional layers, it currently cannot be applied to layers that perform other types of operations such as normalization and activation functions.

Original: In future work, we will try to address those issues to realize a more memory-efficient learning algorithm and expand its applicability.
Revised: In future work, we will try to address those issues to realize a more memory-efficient learning algorithm and expand its applicability.

Original: REPRODUCIBILITY
Revised: REPRODUCIBILITY

Original: We presented the rationale for our algorithm in Section 3.
Revised: We presented the rationale for our algorithm in Section 3.

Original: And it was proven through the experimental results in Section 4.
Revised: And it was proven through the experimental results in Section 4.

Original: We have detailed various points for experimental reproduction in 4 and Appendix C. Furthermore, we support code to reproduce our results as follow: https://github.com/viqpldem/Auxiliary Activation Learning.
Revised: We have detailed various points for experimental reproduction in 4 and Appendix H. Furthermore, we support code to reproduce our results as follow: https://github.com/viqpldem/Auxiliary Activation Learning.

Original: A PPENDICES A P ROOF OF THEOREMS
Revised: A PPENDICES A P ROOF OF THEOREMS

Original: Theorem 1.
Revised: Theorem 1.

Original: If r Tl (2 h l − r l ) > 0 in all layers and the step size η satisfies 0 ≤ η ≤ 1 Lwhere L = ∇ 2 f , then the loss function f ( W )  converges to a minimum value.
Revised: If r Tl (2 h l − r l ) > 0 in all layers, the step size η satisfies 0 < η ≤ 1 L , and the gradient of the loss function f ( W ) is Lipschitz continuous, then f ( W ) converges to a minimum value.

Original: Proof.
Revised: Proof.

Original: Since  f ( x )is convex, differentiable, and Lipschitz continuous, it satisfies f ( y ) ≤ f ( x ) + ∇ f ( x ) T ( y − x ) + 12 L ∥x − y  ∥ 2 .
Revised: Since a gradient of f ( x ) Lipschitz continuous (i.e. ∥∇ f ( y ) − ∇ f ( x ) ∥ 2 ≤ L ∥ y − x ∥ 2 ), it satisfies f ( y ) ≤ f ( x ) + ∇ f ( x ) T ( y − x ) + 12 L ∥ y − x ∥ 2 .

Original: By substituting W t for x and W t +1 for y , we obtain
Revised: By substituting W t for x and W t +1 for y , we obtain

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: because we assumed that the step size is 0 ≤ η ≤ 1 L .
Revised: because we assumed that the step size is 0 < η ≤ 1 L .

Original: By applying equations (8) and (9) [Equation] we obtain
Revised: By applying equations (8) and (9), we obtain

Original: [Equation]
Revised: [Equation]

Original: We can show that Vec( AB T ) T Vec( AC T ) =
Revised: We can show that Vec( AB T ) T Vec( AC T ) =

Original: ∥
Revised: ∥

Original: A ∥ 2 C T B when A , B , and C are column vectors, and B and C have the same dimension as below.
Revised: A ∥ 2 C T B when A , B , and C are column vectors, and B and C have the same dimension as below.

Original: [Equation]
Revised: [Equation]

Original: Consequently, by applying this equality to the inequality above,
Revised: Consequently, by applying this equality to the inequality above,

Original: [Equation]
Revised: [Equation]

Original: because r Tl (2 h l − r l ) > 0 in all layers by the assumption above.
Revised: because r Tl (2 h l − r l ) > 0 in all layers by the assumption above.

Original: Therefore, 0 ≤ f ( W t +1 ) ≤ f ( W t ) − ηK t ≤ f ( W t − 1 ) − ηK t − 1 − ηK t ≤ f ( W 0 ) − η (cid:80) tn
Revised: Therefore, 0 ≤ f ( W t +1 ) ≤ f ( W t ) − ηK t ≤ f ( W t − 1 ) − ηK t − 1 − ηK t ≤ f ( W 0 ) − η (cid:80) tn

Original: =0 K n , which suggests that the value of f ( W ) becomes smaller as the number of epochs t increases because K n ≥ 0 .
Revised: =0 K n , which suggests that the value of f ( W ) becomes smaller as the number of epochs t increases because K n ≥ 0 .

Original: Consequently, because f ( W ) decreases and is bounded below by 0, f ( W ) converges to zero during training even when using alternative activations for updating weights by the monotone convergence theorem.
Revised: Consequently, because f ( W ) decreases and is bounded below by 0, f ( W ) converges to zero during training even when using alternative activations for updating weights by the monotone convergence theorem.

Original: Furthermore, if the values of r Tl (2 h l − r l ) increases, K n , which is the amount of loss reduction in each epoch, also increases and therefore the network converges well.
Revised: Furthermore, if the values of r Tl (2 h l − r l ) increases, K n , which is the amount of loss reduction in each epoch, also increases and therefore the network converges well.

Original: B D OES ADDING AUXILIARY  ACTIVATIONREALLY HELP ?
Revised: B C OMPARISONS BETWEEN AUXILIARY RESIDUAL ACTIVATION AND AUXILIARY SIGN ACTIVATION

Revised: [Figure] Table 5 shows that the training memory of ASA is slightly larger than that of ARA since ASA has to obtain and store 1-bit auxiliary sign activation, while ARA does not need to store actual activation and just reuses auxiliary activation. We also measured the mean values of the learning indicator.

Revised: The mean values of the learning indicator of ARA are significantly higher than those of ASA.

Revised: Therefore, from Theorem 1, we can expect that the loss of the network would converge better by using ARA compared to ASA.

Revised: This is consistent with the experimental results in Table 5; ARA achieves higher accuracy than ASA in all experiments.

Revised: In addition, both train and validation losses converge better when using ARA, as shown in Fig.

Revised: The processing time for one epoch of ASA is slightly larger than that of ARA since ASA needs to pack 1-bit auxiliary sign activation, whereas ARA employs auxiliary residual activation as is.

Revised: [Table]

Original: Theorem 1 suggests that an alternative type of activation can replace actual input activation in backward propagation, as long as it satisfies the learning criterion.
Revised: Theorem 1 suggests that an alternative type of activation can replace actual input activation in backward propagation, as long as it satisfies the learning criterion.

Original: However, it is difficult to meet this condition using arbitrary values, so we proposed to add the alternate activation to real activation in forward propagation.
Revised: However, it is difficult to meet this condition using arbitrary values, so we proposed to add the alternate activation to real activation in forward propagation.

Original: Then, the input activation h l =
Revised: Then, the input activation h l =

Original: o l + a l in forward propagation and the alternate activation a l in backward propagation share a common term, which is expected to help satisfy the learning criterion.
Revised: o l + a l in forward propagation and the alternate activation a l in backward propagation share a common term, which is expected to help satisfy the learning criterion.

Original: In this section, we experimentally verify this claim by comparing ARA and ASA to baselines where the alternate activations are not added to real activation in forward propagation.
Revised: In this section, we experimentally verify this claim by comparing ARA and ASA to baselines where the alternate activations are not added to real activation in forward propagation.

Original: In baselines, we replace real activation with the previous layer’s activation and 1-bit sign of original activation in backward propagation, while they are not added to real activation in forward propagation.
Revised: In baselines, we replace real activation with the previous layer’s activation and 1-bit sign of original activation in backward propagation, while they are not added to real activation in forward propagation.

Original: They are denoted by Residual Activation (RA) and Sign Activation (SA), respectively.
Revised: They are denoted by Residual Activation (RA) and Sign Activation (SA), respectively.

Original: SA is identical to simply quantizing activation into 1 bit in conventional backpropagation.
Revised: SA is identical to simply quantizing activation into 1 bit in conventional backpropagation.

Original: [Figure]
Revised: [Figure]

Original: Distributions of learning indicator values of RA and ARA in ResNetFigs.
Revised: Figs.

Original: 6 and 7  display learning indicator values of ResNet-18 trained on CIFAR-10, CIFAR-100, and Tiny ImageNet.
Revised: 7 and 8 display learning indicator values of ResNet-18 trained on CIFAR-10, CIFAR-100, and Tiny ImageNet.

Original: These results show that adding auxiliary activation to real activation in forward propagation significantly improves the distributions of learning indicator values by positively skewing them toward 1.
Revised: These results show that adding auxiliary activation to real activation in forward propagation significantly improves the distributions of learning indicator values by positively skewing them toward 1.

Original: This is consistent with training performance comparisons provided in Table 6.
Revised: This is consistent with training performance comparisons provided in Table 6.

Original: ARA and ASA noticeably outperform RA and SA in all experiments, and the performance improvement is more significant for larger datasets.
Revised: ARA and ASA noticeably outperform RA and SA in all experiments, and the performance improvement is more significant for larger datasets.

Original: Therefore, we can conclude that adding auxiliary activation to real activation in forward propagation does help neural network training. 
Revised: Therefore, we can conclude that adding auxiliary activation to real activation in forward propagation does help neural network training. [Table]

Original: [Figure]
Revised: [Figure]

Revised: D C OMPARISONS BETWEEN A UXILIARY A CTIVATION L EARNING AND OTHER MEMORY SAVING ALGORITHMS UNDER IDENTICAL COMPRESSION RATES

Revised: In this section, we compare our Auxiliary Activation Learning with other memory saving algorithms under identical compression rates.

Revised: We apply GCP to the last Conv-BN layer of residual blocks in Fig.

Revised: 3a because the dimension of input activation in the last batch normalization layer is equal to that of the ARA-Conv layer, which is not stored in memory during ARA.

Revised: The input activation of the batch normalization layer is not stored in Conv-BN layers, but it is recomputed from the input activation of the convolutional layer during backward propagation.

Revised: Through this, an identical amount of memory saving can be achieved by using GCP.

Revised: For ActNN, we applied it to first convolutional layer of residual blocks as we did for ARA in Fig.

Revised: 3a, which resulted in an identical compression rate.

Revised: While our ASA stores 1-bit auxiliary sign activation, Mesa stores 8-bit compressed activation.

Revised: Therefore, we cannot achieve the same memory saving by applying Mesa in the same way as ASA.

Revised: To overcome this issue, we additionally apply Mesa to the batch normalization and layer normalization layers to obtain the same compression rate to ASA.

Revised: Tables 7 and 8 show that ARA and ASA train the networks faster than other memory-saving algorithms with identical compression rates as they do not require additional costly computations.

Revised: [Table]

Revised: [Table]

Revised: E C OMPARISONS BETWEEN A UXILIARY R ESIDUAL A CTIVATION AND M OMENTUM R ES N ET .

Revised: In this section

Revised: [Table]

Revised: we compare ARA with Momentum ResNet (MResNet) (29), which is a reversible network that can be applied to diverse networks with residual connections.

Revised: For comparisons, we trained ResNet-50 on CIFAR-100 with a 5e-2 learning rate for 100 epochs with 10 warm-up steps and cosine annealing.

Revised: The results are displayed in Table 9.

Revised: First, ARA closely matches BP in accuracy, but MResNet exhibits a small accuracy degradation.

Revised: While ARA requires larger training memory than MResNet, ARA is significantly faster since MResNet needs to recalculate the input of residual blocks along with additional forward propagation during backward propagation.

Original: [Table]
Revised: [Table]

Revised: F M AXIMUM B ATCH SIZE AND TRAINING TIME WITH A UXILIARY A CTIVATION L EARNING .

Revised: We trained various networks using only three RTX-3090 GPUs to confirm the effectiveness of our algorithm.

Revised: The experimental results for ResNet-152 and ViT-Large are displayed in Tables 10 and 11.

Revised: Using memory-saving techniques allows for a larger batch size for a given GPU memory space.

Revised: Interestingly, although conventional algorithms such as GCP, ActNN, and Mesa enables a larger batch size, it does not translate to faster training speed due to additional computations.

Revised: In contrast, our ARA and ASA algorithms take full advantage of larger batch size and improve training speed.

Revised: [Table]

Original:  C E XPERIMENTAL DETAILS C. H OW WE APPLY GRADIENT CHECKPOINITING IN EACH EXPERIMENT
Revised: [Table] G C OMPARISONS BETWEEN A UXILIARY A CTIVATION L EARNING AND SM  In this section [Table]

Revised: we compare ARA with SM3 (14)

Revised: [Table]

Revised: which is a memory-efficient optimizer.

Revised: While our algorithm aims to reduce the amount of input activation to be stored, SM3 aims to reduce the memory required for storing states generated by accumulating previous gradients of parameters

Original: Chen et al.(16) suggested that when there is a BN-ReLU-Conv layer in the network, applying gradient checkpointing to those layers can reduce memory requirements without incurring significant computational overhead.
Revised: [Table] For fair comparisons, we measured the total memory allocated to a GPU instead of the memory required to store activations. Experimental results are summarized in Table 12. The results show that our ASA4 algorithm requires smaller memory space but takes more time to process one epoch compared to SM3.

Original: More specifically, only the input activation of the BN layer is stored during forward propagation
Revised: This is because the SM3 optimizer is faster than the Adam optimizer that we used for all experiments. Since the theoretical backgrounds behind these two methods are quite different, they can be combined together to obtain even larger savings. As displayed in Table 12, if we apply both methods simultaneously, we can obtain the largest memory savings with minimal impact on the accuracy.

Original: [Table]
Revised: [Table]

Original:  and the rest is recalculated in backward propagation.
Revised: et al.(24) suggested that when there is a BN-ReLU-Conv layer in the network, applying gradient checkpointing to those layers can reduce memory requirements without incurring significant computational overhead. More specifically, only the input activation of the BN layer is stored during forward propagation, and the rest is recalculated in backward propagation.

Original: Therefore, while the BN and ReLU layers have to be recomputed during backward propagation to generate the input activation of the Conv layer, there is no need to recompute the Conv layer itself which may lead to large computational overhead.
Revised: Therefore, while the BN and ReLU layers have to be recomputed during backward propagation to generate the input activation of the Conv layer, there is no need to recompute the Conv layer itself which may lead to large computational overhead.

Original: We applied this method to BN-ReLU-Conv3x3 layers in Fig.
Revised: We applied this method to BN-ReLU-Conv3x3 layers in Fig.

Original: 3a to reduce memory overhead without a significant amount of recomputation.
Revised: 3a to reduce memory overhead without a significant amount of recomputation.

Original: In addition, we apply gradient checkpointing to BN-ReLU-Conv1x1-BN layers in Fig. 3a.
Revised: In addition, we apply gradient checkpointing to BN-ReLU-Conv1x1-BN layers in Fig. 3a.

Original: Although we have to recompute Conv1x1, its computational overhead is relatively small compared to Conv3x3 layers.
Revised: Although we have to recompute Conv1x1, its computational overhead is relatively small compared to Conv3x3 layers.

Original: We also applied gradient checkpointing to the first Conv-BN-ReLU-MaxPool layers in ResNet.
Revised: We also applied gradient checkpointing to the first Conv-BN-ReLU-MaxPool layers in ResNet.

Original: Here only the input of the first Conv layer is stored during forward propagation, and the input activation of the other layers is recomputed in backward propagation.
Revised: Here only the input of the first Conv layer is stored during forward propagation, and the input activation of the other layers is recomputed in backward propagation.

Original: Although the first Conv layer has to be recomputed, a large amount of training memory can be saved because the amount of the input activation in the first Conv, BN, and MaxPool layers is large.
Revised: Although the first Conv layer has to be recomputed, a large amount of training memory can be saved because the amount of the input activation in the first Conv, BN, and MaxPool layers is large.

Original: C. H OW WE APPLY ACTIVATION COMPRESSION TRAINING IN EACH EXPERIMENT
Revised: H. H OW WE APPLY ACTIVATION COMPRESSION TRAINING IN EACH EXPERIMENT

Original:  ActNN (25) stores 2-bit compressed activations instead of actual activations for all layers except for
Revised: [Figure] ActNN (34) stores 2-bit compressed activations instead of actual activations for all layers except for

Original: ReLU and pooling which only needs 1-bit sign activations.
Revised: ReLU and pooling which only needs 1-bit sign activations.

Original: We applied ActNN (L3) to all layers,[Figure] which performs per-group quantization with fine-grained mixed precision.
Revised: We applied ActNN (L3) to all layers, which performs per-group quantization with fine-grained mixed precision.

Original: The per-group quantization partitions the activation into several groups, and then quantization is performed by groups.
Revised: The per-group quantization partitions the activation into several groups, and then quantization is performed by groups.

Original: The fine-grained mixed precision includes two processes: per-sample allocation and per-layer allocation.
Revised: The fine-grained mixed precision includes two processes: per-sample allocation and per-layer allocation.

Original: In per-sample allocation, the sensitivity, which is related to gradient variance, is calculated for each group and then an optimal number of bits for each group is determined using this sensitivity.
Revised: In per-sample allocation, the sensitivity, which is related to gradient variance, is calculated for each group and then an optimal number of bits for each group is determined using this sensitivity.

Original: Then, per-layer allocation calculates the sensitivity of each layer and then an optimal number of bits for each layer is obtained.
Revised: Then, per-layer allocation calculates the sensitivity of each layer and then an optimal number of bits for each layer is obtained.

Original: When ARA and ActNN are applied simultaneously, we perform 2-bit groupwise quantization and per-sample allocation on auxiliary residual activation.
Revised: When ARA and ActNN are applied simultaneously, we perform 2-bit groupwise quantization and per-sample allocation on auxiliary residual activation.

Original: However, per-layer allocation is not applied to auxiliary residual activation because it is not straightforward to calculate the sensitivity of ARA layers.
Revised: However, per-layer allocation is not applied to auxiliary residual activation because it is not straightforward to calculate the sensitivity of ARA layers.

Original: While ActNN needs to quantize and store activations of all layers, we can skip this process for ARA Conv layers when using ActNN with ARA (Fig. 8).
Revised: While ActNN needs to quantize and store activations of all layers, we can skip this process for ARA Conv layers when using ActNN with ARA (Fig. 9).

Original: [Figure]
Revised: [Figure]

Original: Pan et al.
Revised: Pan et al.

Original: (27) pointed out that conventional activation compression algorithms do not consider heterogeneous activation distributions in a multi-head attention layer and therefore they are not applicable to Transformer-like models.
Revised: (36) pointed out that conventional activation compression algorithms do not consider heterogeneous activation distributions in a multi-head attention layer and therefore they are not applicable to Transformer-like models.

Original: Therefore, they proposed Mesa which performs head-wise quantization instead of per-tensor quantization.
Revised: Therefore, they proposed Mesa which performs head-wise quantization instead of per-tensor quantization.

Original: When we combine Mesa and ASA, we replace linear layers with ASA-Linear layers as shown in Fig.
Revised: When we combine Mesa and ASA, we replace linear layers with ASA-Linear layers as shown in Fig.

Original: C. D ATASETS
Revised: H. D ATASETS

Original: CIFAR-10 and CIFAR-

Revised: CIFAR-10 and CIFAR-

Original: The CIFAR-10 dataset (36) consists of 50,000 training images and 10,000 test images which are 32x32 RGB images for 10-class image classification.
Revised: The CIFAR-10 dataset (46) consists of 50,000 training images and 10,000 test images which are 32x32 RGB images for 10-class image classification.

Original: Likewise, the
Revised: Likewise, the

Original: CIFAR-100 dataset (36) includes 50,000 training images and 10,000 test images with the same resolution for 100-class image classification.
Revised: CIFAR-100 dataset (46) includes 50,000 training images and 10,000 test images with the same resolution for 100-class image classification.

Original: Tiny ImageNet
Revised: Tiny ImageNet

Original: The Tiny-ImageNet dataset (37) consists of images of 200 classes and each class has 500 images for training.
Revised: The Tiny-ImageNet dataset (47) consists of images of 200 classes and each class has 500 images for training.

Original: It also contains 10,000 test images.
Revised: It also contains 10,000 test images.

Original: All images included in Tiny
Revised: All images included in Tiny

Original: ImageNet are selected from ImageNet and downsized to 64x64.
Revised: ImageNet are selected from ImageNet and downsized to 64x64.

Original: ImageNet The ImageNet dataset (31) is a representative large-scale dataset for image classification.
Revised: ImageNet The ImageNet dataset (41) is a representative large-scale dataset for image classification.

Original: It consists of 1,281,167 training images, 50,000 validation images, and 100,000 test images which are all RGB images.
Revised: It consists of 1,281,167 training images, 50,000 validation images, and 100,000 test images which are all RGB images.

Original: The average image size on ImageNet is 469x387, but they are randomly cropped to 224x224 for image classification.
Revised: The average image size on ImageNet is 469x387, but they are randomly cropped to 224x224 for image classification.

Original: IWSLT 

Revised: IWSLT

Original: The IWSLT 2016 dataset (39) is constructed for spoken language translation task.
Revised: The IWSLT 2016 dataset (49) is constructed for spoken language translation task.

Original: We choose IWSLT2016 De-En dataset which is used for German-to-English translation.
Revised: We choose IWSLT2016 De-En dataset which is used for German-to-English translation.

Original: It has 1, talks, 197K sentences, and 3.96M tokens as a training set while the validation set consists of 13 talks, 1.1K sentences, and 21K tokens.
Revised: It has 1, talks, 197K sentences, and 3.96M tokens as a training set while the validation set consists of 13 talks, 1.1K sentences, and 21K tokens.

Revised: MRPC The Microsoft Research Paraphrase Corpus (MRPC) dataset (50) consists of 5081 sentence pairs from newswire articles.

Revised: In this dataset, the training set contains 4076 sentence pairs with 2753 paraphrases and the test set contains 1725 pairs with 1147 paraphrases.

Original: MNLI
Revised: MNLI

Original: The Multi-Genre Natural Language Inference (MultiNLI) dataset (40) consists of 433k sentence pairs for ten genres (Face-to-Face, Telephone, etc.).
Revised: The Multi-Genre Natural Language Inference (MultiNLI) dataset (51) consists of 433k sentence pairs for ten genres (Face-to-Face, Telephone, etc.).

Original: The dataset is divided into 392702 train sets, 9815 validation matched sets which are subsets of train sets, and 9832 validation mismatched sets which are included in train sets.
Revised: The dataset is divided into 392702 train sets, 9815 validation matched sets which are subsets of train sets, and 9832 validation mismatched sets which are included in train sets.

Original: C. H YPERPARAMETES
Revised: H. H YPERPARAMETES

Original: ResNet For training ResNet-18 from scratch on CIFAR-10, CIFAR-100, and Tiny ImageNet, we set the batch size and the total number of epochs to 512 and 200, respectively.
Revised: ResNet For training ResNet-18 from scratch on CIFAR-10, CIFAR-100, and Tiny ImageNet, we set the batch size and the total number of epochs to 512 and 200, respectively.

Original: We applied stochastic gradient descent with momentum (42) along with weight decaying (36) to our experiments.
Revised: We applied stochastic gradient descent with momentum (53) along with weight decaying (46) to our experiments.

Original: The momentum and weight decay rate was set to 0.9 and 1e-4, respectively.
Revised: The momentum and weight decay rate was set to 0.9 and 1e-4, respectively.

Original: The learning rate of the layers except for ASA layers was scheduled by cosine annealing (43) with a 0.1 initial learning rate during 200 epochs.
Revised: The learning rate of the layers except for ASA layers was scheduled by cosine annealing (54) with a 0.1 initial learning rate during 200 epochs.

Original: In comparison, we used a 100x higher learning rate for ASA layers to make the magnitude of weight updates comparable to those of other layers.
Revised: In comparison, we used a 100x higher learning rate for ASA layers to make the magnitude of weight updates comparable to those of other layers.

Original: We also set ϵ in equations (15) and (16) to 0.01.
Revised: We also set ϵ in equations (15) and (16) to 0.01.

Original: We used a nearly identical set of hyperparameters for ImageNet experiments.
Revised: We used a nearly identical set of hyperparameters for ImageNet experiments.

Original: However, we set the batch size and the number of epochs to 512 and 90, respectively, and the learning rate was decreased by 10x every 30 epochs instead of cosine annealing.
Revised: However, we set the batch size and the number of epochs to 512 and 90, respectively, and the learning rate was decreased by 10x every 30 epochs instead of cosine annealing.

Original: Transformer
Revised: Transformer

Original: In Transformer, the batch size and the number of epochs are set to 4096 and 20, respectively.
Revised: In Transformer, the batch size and the number of epochs are set to 4096 and 20, respectively.

Original: We trained Transformer from scratch on IWSLT and used the Adam optimizer (44) by setting β 1 to 0.9, β 2 to 0.98, and ϵ to 1e-9.
Revised: We trained Transformer from scratch on IWSLT and used the Adam optimizer (55) by setting β 1 to 0.9, β 2 to 0.98, and ϵ to 1e-9.

Original: In addition, the learning rate was updated by the warm-up learning rate scheduler (43), and the number of warm-up steps was set to 4,000.
Revised: In addition, the learning rate was updated by the warm-up learning rate scheduler (54), and the number of warm-up steps was set to 4,000.

Original: We applied a 100x higher learning rate to ASA layers as in ResNet experiments.
Revised: We applied a 100x higher learning rate to ASA layers as in ResNet experiments.

Original: ϵ in equations (15) and (16) for ASA was set to 0.01.
Revised: ϵ in equations (15) and (16) for ASA was set to 0.01.

Original: The model dimension was 512, the hidden layer dimension was 2048, the number of heads was 8, and the dropout rate (45) was 0.2.
Revised: The model dimension was 512, the hidden layer dimension was 2048, the number of heads was 8, and the dropout rate (56) was 0.2.

Original: Furthermore, we applied label smoothing with a 0.1 label smoothing rate (46) to all experiments in Transformer.
Revised: Furthermore, we applied label smoothing with a 0.1 label smoothing rate (57) to all experiments in Transformer.

Original: BERT-Large We used Bert-Large (uncased) pretrained on BookCorpus (47) and English
Revised: BERT-Large We used Bert-Large (uncased) pretrained on BookCorpus (58) and English

Original: Wikipedia for fine-tuning on  MNLI.
Revised: Wikipedia for fine-tuning on MRPC and MNLI.

Original: The batch size was set to 32, and we trained BERT-Large for three epochs.
Revised: The batch size was set to 32, and we trained BERTLarge for three epochs.

Original: The Adam optimizer (44) was used with 0.9 β 1 , 0.999 β 2 , 1e-6 ϵ , and 0.01 weight decay rate.
Revised: The Adam optimizer (55) was used with 0.9 β 1 , 0.999 β 2 , 1e-6 ϵ , and 0. weight decay rate.

Original: We also employed warm-up in the learning rate up to 10% of the total step, and cosine annealing is used for the remaining epochs (43).
Revised: We also employed warm-up in the learning rate up to 10% of the total step, and cosine annealing is used for the remaining epochs (54).

Original: The learning rate of cosine annealing was set to 1.5e-5.
Revised: The learning rate of cosine annealing was set to 1.5e-5.

Original: These learning rate was increased by 100x for ASA layers in ViT experiments.
Revised: These learning rate was increased by 100x for ASA layers in ViT experiments.

Original: ϵ in equations (15) and (16) was set to 0.01.
Revised: ϵ in equations (15) and (16) was set to 0.01.

Original: The other hyperparameters were selected according to the original BERT paper (33).
Revised: The other hyperparameters were selected according to the original BERT paper (43).

Original: ViT-Large We fine-tuned a ViT-Large model pretrained on ImageNet-21k.
Revised: ViT-Large We fine-tuned a ViT-Large model pretrained on ImageNet-21k.

Original: The model was finetuned using 32 batch size on CIFAR-100.
Revised: The model was finetuned using 32 batch size on CIFAR-100.

Original: We set the training batch size to 512 and experimented with different learning rates of 3e-2, 1e-1, and 3e-1 to achieve the highest performance.
Revised: We set the training batch size to 512 and experimented with different learning rates of 3e-2, 1e-1, and 3e-1 to achieve the highest performance.

Original: For training, learning rate was adjusted by stochastic gradient descent with momentum (42) and a warm-up cosine scheduler (43) was applied.
Revised: For training, learning rate was adjusted by stochastic gradient descent with momentum (53) and a warm-up cosine scheduler (54) was applied.

Original: The model was trained for 1000 steps with 100 warm-up steps.
Revised: The model was trained for 1000 steps with 100 warm-up steps.

Original: For
Revised: For

Original: Linear-ASA layers, 100x larger learning rate was used.
Revised: Linear-ASA layers, 100x larger learning rate was used.

Original: We also set ϵ in equations (15) and (16) to 0.01.
Revised: We also set ϵ in equations (15) and (16) to 0.01.

Original: The remaining hyperparameters were set according to the original ViT paper (34).
Revised: The remaining hyperparameters were set according to the original ViT paper (44).

Original: MLP-Mixer-Large The MLP-Mixer-
Revised: MLP-Mixer-Large The MLP-Mixer-

Original: Large model was fine-tuned from a pretrained model on
Revised: Large model was fine-tuned from a pretrained model on

Original: ImageNet-21k with 16 patch size.
Revised: ImageNet-21k with 16 patch size.

Original: We trained the model on CIFAR-10 for 2000 steps withbatch size.
Revised: We trained the model on CIFAR-10 for 2000 steps withbatch size.

Original: Like ViT, the SGD optimizer (42) was applied to these experiments.
Revised: Like ViT, the SGD optimizer (53) was applied to these experiments.

Original: We set the weight decay rate and momentum to 0 and 0.9, respectively.
Revised: We set the weight decay rate and momentum to 0 and 0.9, respectively.

Original: During the first 200 steps, the learning rate warmed up and cosine annealing (43) was employed in the remaining epochs.
Revised: During the first 200 steps, the learning rate warmed up and cosine annealing (54) was employed in the remaining epochs.

Original: We set the learning rate to 3e-2, 1e-1, and 3e-1.
Revised: We set the learning rate to 3e-2, 1e-1, and 3e-1.

Original: In the ASA layers, however, the learning rate was increased by 100x like Transformer and ViT-Large.
Revised: In the ASA layers, however, the learning rate was increased by 100x like Transformer and ViT-Large.

Original: ϵ of ASA layers in equations (15) and (16) was set to 0.01.The other hyperparameters followed the original MLP-Mixer paper (35).
Revised: ϵ of ASA layers in equations (15) and (16) was set to 0.01.The other hyperparameters followed the original MLP-Mixer paper (45).

Original: C. D EVICES AND MEMORY MEASUREMENTS
Revised: H. D EVICES AND MEMORY MEASUREMENTS

Original: In all experiments, we used Nvidia GeForce RTX 3090 GPUs.
Revised: In all experiments, we used Nvidia GeForce RTX 3090 GPUs.

Original: While one GPU was utilized to perform most of the experiments, we used six GPUs to train ResNet on ImageNet to reduce training time.
Revised: While one GPU was utilized to perform most of the experiments, we used six GPUs to train ResNet on ImageNet to reduce training time.

Original: We calculated the amount of training memory using torch.cuda.memory allocated function in PyTorch instead of nvidia-smi because nvidia-smi includes unused memory space handled by the memory allocator in the report 1 .
Revised: We calculated the amount of training memory using torch.cuda.memory allocated function in PyTorch instead of nvidia-smi because nvidia-smi includes unused memory space handled by the memory allocator in the report 1 .

Original: D M EMORY AND COMPUTE COMPLEXITY OF A UXILIARY A CTIVATION L EARNING
Revised: I M EMORY AND COMPUTE COMPLEXITY OF A UXILIARY A CTIVATION L EARNING

Original: [Table]
Revised: [Table]

Original: In this section, we compare the compute and memory complexity of different learning algorithms.
Revised: In this section, we compare the compute and memory complexity of different learning algorithms.

Original: For the sake of simplicity, we assume that the computational cost of forward propagation and backward propagation of a single layer is 1.
Revised: For the sake of simplicity, we assume that the computational cost of forward propagation and backward propagation of a single layer is 1.

Original: The amount of activations of a single layer is also assumed to be 1.
Revised: The amount of activations of a single layer is also assumed to be 1.

Original: Then we calculate the computational cost and memory space required for training a neural network with N identical layers.
Revised: Then we calculate the computational cost and memory space required for training a neural network with N identical layers.

Original: BP would need N computations for forward propagation, N computations for error propagation, and N computations for calculating weight updates, which results in 3 N computations in total.
Revised: BP would need N computations for forward propagation, N computations for error propagation, and N computations for calculating weight updates, which results in 3 N computations in total.

Original: For memory space, BP has to store N activations generated during forward propagation.
Revised: For memory space, BP has to store N activations generated during forward propagation.

Original: On the other hand, GCP requires N additional computations for the recomputation of activations.
Revised: On the other hand, GCP requires N additional computations for the recomputation of activations.

Original: However, it can reduce the memory complexity to 2 √ N when the network is divided into √ N blocks as suggested in (16).
Revised: However, it can reduce the memory complexity to 2 √ N when the network is divided into √ N blocks as suggested in (24).

Original: In our Auxiliary Activation Learning (AAL), we have to store auxiliary activations instead of actual activations.
Revised: In our Auxiliary Activation Learning (AAL), we have to store auxiliary activations instead of actual activations.

Original: We assume that our algorithm is applied to A layers in the network.
Revised: We assume that our algorithm is applied to A layers in the network.

Original: Then, we have to store actual activation for the rest of the layers, which translates to a memory complexity of N − A .
Revised: Then, we have to store actual activation for the rest of the layers, which translates to a memory complexity of N − A .

Original: For the layers employing our algorithm, we only have to store auxiliary activations and its amount is negligible compared to real activations.
Revised: For the layers employing our algorithm, we only have to store auxiliary activations and its amount is negligible compared to real activations.

Original: For example, Auxiliary Residual Activation (ARA) uses the previous layer’s activation as auxiliary activation, and hence there is no need to store this auxiliary activation since it is already stored in memory during forward propagation of the previous layer.
Revised: For example, Auxiliary Residual Activation (ARA) uses the previous layer’s activation as auxiliary activation, and hence there is no need to store this auxiliary activation since it is already stored in memory during forward propagation of the previous layer.

Original: Auxiliary Sign Activation (ASA) has to store a 1-bit sign of actual activations, respectively, but they are also significantly smaller than real activations.
Revised: Auxiliary Sign Activation (ASA) has to store a 1-bit sign of actual activations, respectively, but they are also significantly smaller than real activations.

Original: Therefore, our algorithm has N − A memory complexity and 3 N compute complexity.
Revised: Therefore, our algorithm has N − A memory complexity and 3 N compute complexity.

Original: To further reduce memory complexity, we can apply gradient checkpointing to the layers where we do not apply our AAL algorithm.
Revised: To further reduce memory complexity, we can apply gradient checkpointing to the layers where we do not apply our AAL algorithm.

Original: Thus, the memory complexity of those layers would decrease to 2 √ N − A and the compute complexity becomes 4 N due to recomputation.
Revised: Thus, the memory complexity of those layers would decrease to 2 √ N − A and the compute complexity becomes 4 N due to recomputation.

Original: E C OMPARISONS TO ALGORITHMS TO APPROXIMATE BACKPROPAGATION
Revised: J C OMPARISONS TO ALGORITHMS TO APPROXIMATE BACKPROPAGATION

Original: Previous studies have suggested various algorithms to approximate backpropagation (48; 49; 50).
Revised: Previous studies have suggested various algorithms to approximate backpropagation (59; 60; 61).

Original: Our algorithm may be considered in line with such approaches as it replaces actual input activation with auxiliary activation.
Revised: Our algorithm may be considered in line with such approaches as it replaces actual input activation with auxiliary activation.

Original: Furthermore, Uniform Sign-Concordant Feedback (50), which uses the sign of forward weights as feedback weights, looks similar to the concept of ASA which uses the sign of activation as auxiliary activation.
Revised: Furthermore, Uniform Sign-Concordant Feedback (61), which uses the sign of forward weights as feedback weights, looks similar to the concept of ASA which uses the sign of activation as auxiliary activation.

Original: However, the motivation is quite different.
Revised: However, the motivation is quite different.

Original: In prior works on approximate backpropagation, the authors aim to remove the need for symmetric feedforward and feedback weights (“weight transport problem”) by approximating the feedback path for better bio-plausibility.
Revised: In prior works on approximate backpropagation, the authors aim to remove the need for symmetric feedforward and feedback weights (“weight transport problem”) by approximating the feedback path for better bio-plausibility.

Original: Contrarily, our algorithm reduces memory requirements by using an alternate form of activations, but it does not necessarily improve bio-plausibility as we still need symmetric forward and backward paths.
Revised: Contrarily, our algorithm reduces memory requirements by using an alternate form of activations, but it does not necessarily improve bio-plausibility as we still need symmetric forward and backward paths.

Original: F C ONVERGENCE SPEED OF A UXILIARY A CTIVATION L EARNING
Revised: K C ONVERGENCE SPEED OF A UXILIARY A CTIVATION L EARNING

Original: [Figure]
Revised: [Figure]