F ORTUITOUS F ORGETTING IN C ONNECTIONIST N ETWORKS
A BSTRACT
Forgetting is often seen as an unwanted characteristic in both human and machine learning.
However, we propose that forgetting can in fact be favorable to learning.
We introduce forget-and-relearn as a powerful paradigm for shaping the learning trajectories of artiﬁcial neural networks.
In this process, the forgetting step selec tively removes undesirable information from the model and the relearning step reinforces features that are consistently useful under different conditions.
The forget and-relearn framework uniﬁes many existing iterative training algorithms in the image classiﬁcation and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forget ting of undesirable information.
We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations.
Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.
I NTRODUCTION
Forgetting is an inescapable component of human memory.
It occurs naturally as neural synapses get removed or altered over time (Wang et al., 2020), and is often thought to be an undesirable characteristic of the human mind.
Similarly, in machine learning, the phenomena of catastrophic forgetting is often blamed for certain failure modes in the performance of neural networks (McCloskey & Cohen, 1989; Ratcliff, 1990; French, 1999) 1 .
However, substantial evidence in psychology and neuroscience have shown that forgetting and learning have a symbiotic relationship (Bjork & Bjork, 2019; Gravitz, 2019).
A well-known example is the “spacing effect”, which refers to the observation that long-term recall is enhanced by spacing, rather than massing, repeated study sessions.
Bjork & Allen (1970) demonstrated that the key to the spacing effect is the decreased accessibility of information in-between sessions.
In this work, we study a general learning paradigm that we refer to as forget-and-relearn , and show that forgetting can also beneﬁt learning in artiﬁcial neural networks.
To generalize to unseen data, we want our models to capture generalizable concepts rather than purely statistical regularities, but these desirable solutions are a small subset of the solution space and often more difﬁcult to learn naturally (Geirhos et al., 2020).
Recently, a number of training algorithms have been proposed to improve generalization by iteratively reﬁning the learned solution.
Knowledge evolution (Taha et al., 2021) improves generalization by iteratively reinitializing one part of the network while continuously training the other.
Iterative magnitude pruning (Frankle & Carbin, 2019; Frankle et al., 2019) removes weights through an iterative pruning-retraining process, and outperforms unpruned models in certain settings.
Hoang et al.
(2018) iteratively utilize synthetic machine translation corpus through back-translations of monolingual data.
Furlanello et al.
; Xie et al.
(2020) employ iterative self-distillation to outperform successive teachers.
Iterated learning 2 (Kirby, 2001) has been shown to improve compositionality in emergent languages (Ren et al., 2020; Vani et al., 2021) and prevent language drift (Lu et al., 2020).
We propose that many existing iterative algorithms are instances of a more general forget-and-relearn process, and that their success can be understood by studying the shared underlying mechanism.
Forget-and-relearn is an iterative training paradigm which alternates between a forgetting stage and a relearning stage.
At a high level, we deﬁne a forgetting stage as any process that results in a decrease in training accuracy.
More speciﬁcally, let D = {
( X i , Y i ) } i ∈ [ n ] be the training dataset.
Let U represent uniform noise sampled from (0 , 1) , which we use as a source of randomness for stochastic functions.
Given a neural architecture, let F be the set of functions computable by a neural network with that architecture, and for any N ∈ F , let Acc ( N ) = 1 n (cid:80) ni
=
I { N ( X i ) =
Y i } be the training accuracy of N .
Let N t be a network trained on D , and let C :
= E [ Acc ( ˜ N )] represent the performance of a randomly initialized classiﬁer on D .
We say that f : F × (0 , 1) → F is a forgetting operation if two conditions hold: (i) P ( Acc ( f ( N t , U )) <
Acc ( N t ) | Acc ( N t ) > C ) = 1 ; (ii) the mutual information
I ( f ( N t , U ) , D ) is positive.
The ﬁrst criterion ensures that forgetting will allow for relearning due to a decrease in accuracy.
The second criterion captures the idea that forgetting equates to a partial removal of information rather than a complete removal.
For standard neural networks, it is sufﬁcient to show that training accuracy is lower than the model prior to forgetting, but higher than chance accuracy for the given task.
The Forget-and-Relearn Hypothesis.
Given an appropriate forgetting operation, iterative retraining after forgetting will amplify unforgotten features that are consistently useful under different learning conditions induced by the forgetting step.
A forgetting operation that favors the preservation of desirable features can thus be used to steer the model towards those desirable characteristics.
We show that many existing algorithms which have successfully demonstrated improved generalization have a forgetting step that disproportionately affects undesirable information for the given task.
Our hypothesis stresses the importance of selective forgetting, and shows a path towards better algorithms.
We illustrate the power of this perspective by showing that we can signiﬁcantly improve upon existing work in their respective settings through the use of more targeted forgetting operations.
Our analysis in Section 5 sheds insight on the mechanism through which iterative retraining leads to parameter values with better generalization properties.
E XISTING A LGORITHMS AS I NSTANCES OF F ORGET - AND -R ELEARN
In this section and the rest of the paper, we focus on two settings where iterative training algorithms have been proposed to improve performance: image classiﬁcation and language emergence.
We summarize and recast a number of these iterative training algorithms as instances of forget-andrelearn.
We also relate our notion of forgetting to other ones in the literature in Appendix A5.
Image Classiﬁcation.
A number of iterative algorithms for image classiﬁcation uses a reinitialization and retraining scheme, and these reinitialization steps can be seen as a simple form of forgetting.
Iterative magnitude pruning (IMP) (Frankle & Carbin, 2019; Frankle et al., 2019) removes weights based on ﬁnal weight magnitude, and does so iteratively by removing an additional percentage of weights each round.
Each round, small magnitude weights are set to zero and frozen, while the remaining weights are rewound to their initial values.
Zhou et al.
(2019a) show that the masking and weight rewinding 3 procedure still retains information from the previous training round by demonstrating that the new initialization achieves better-than-chance accuracy prior to retraining.
Thus, we can view IMP as a forget-and-relearn process with an additional sparsity constraint.
RIFLE (Li et al., 2020) shows that transfer learning performance can be improved by periodically reinitializing the ﬁnal layer during the ﬁnetuning stage, which brings meaningful updates to the low-level features in earlier layers.
We can view this reinitialization step as a forgetting operation that enables relearning.
Knowledge evolution (KE) (Taha et al., 2021) splits the network’s weights into a ﬁt-hypothesis and a reset-hypothesis , and trains the network in generations in order to improve generalization performance.
At the start of each generation of training, weights in the reset-hypothesis are reinitialized, while weights in the ﬁt-hypothesis are retained.
Taha et al.
(2021) suggest that KE is similar to dropout where neurons learn to be more independent.
However, it is hard to translate the dropout intuition to the case of a single subnetwork.
Another intuition compares to ResNet, since KE can induce a zero mapping of the reset-hypothesis without reducing capacity.
But this zero-mapping always occurs with sufﬁcient training, and it is unclear why it would beneﬁt generalization.
Language Emergence.
Several existing algorithms in the language emergence subﬁeld can also be seen as forget-and-relearn algorithms.
Emergent communication is often studied in the context of a multi-agent referential game called the Lewis game (Lewis, 2008;
Foerster et al., 2016; Lazaridou et al., 2016).
In this game, a sender agent must communicate information about the attributes of a target object to a receiver agent, who then has to correctly identify the target object.
Compositionality is a desirable characteristic for the language that agents use, as it would allow for perfect out-of-distribution generalization to unseen combinations of attributes.
Li & Bowling (2019) hypothesize that compositional languages are easier to teach, and propose an algorithm to exert an “ease-of-teaching” pressure on the sender through periodic resetting of the receiver.
We can instead view the receiver resetting as partial forgetting in the multi-agent model.
Ren et al.
(2020) use iterated learning to improve the compositionality of the emergent language.
Their algorithm consists of (i) an interacting phase where agents learn to solve the task jointly, (ii) a transmitting phase where a temporary dataset is generated from the utterances of a sender agent, and (iii) an imitation phase where a newly-initialized sender is pretrained on a temporary dataset of the previous sender’s utterances, before starting a new interacting phase, and this process iterates.
A crucial component during imitation is to limit the number of training iterations, which creates a learning bottleneck that prevents the new model from fully copying the old model.
This restricted transfer of information can be seen as a forgetting operation, and showcases an alternative approach to selective forgetting that removes information that is harder to learn.
P ARTIAL W EIGHT P ERTURBATION AS T ARGETED F ORGETTING
In this section, we show that common forms of partial weight perturbation disproportionately affect information that can hinder generalization in both image classiﬁcation and language emergence settings, and can thus be seen as a targeted forgetting operation under the forget-and-relearn hypothesis.
We provide full details on the perturbation method studied in Appendix A1.1.
3. I MAGE C LASSIFICATION
What information is undesirable depends on the task at hand.
For the issue of overﬁtting in image classiﬁcation, we want to favor features that are simple, general and beneﬁcial for generalization (Valle-P´erez et al., 2019).
Several works have shown that easy examples are associated with locally simpler functions and more generalizable features, while difﬁcult examples require more capacity to ﬁt and are often memorized (Baldock et al., 2021;
Yosinski et al., 2014; Kalimeris et al., 2019; Arpit et al., 2017).
Thus, as a proxy, we deﬁne undesirable information here as features associated with memorized examples.
In order to evaluate whether partial weight perturbation selectively forgets undesirable information, we create two types of example splits for our training data: (i) easy or difﬁcult examples, and (ii) true or mislabeled examples.
At each training iteration, we make a copy of the model and perform a forgetting operation on the copy.
We then evaluate both models and measure separately the training accuracy on the two groups of examples.
We perform these experiments using a two-layer MLP trained on MNIST 4 .
We further extend to a 4-layer convolution model and a ResNet18 on CIFAR-10, and to ResNet50 on ImageNet in Appendix A1.2.
For the deﬁnition of easy or difﬁcult examples, we follow prior work (Jiang et al., 2020; Baldock et al., 2021) and use the output margin as a measure of example difﬁculty.
The output margin is deﬁned as the difference between the largest and second-largest logits.
We label 10% of training examples with the smallest output margin in the current batch as difﬁcult examples.
In the noisy labels experiment, we sample 10% of the training data 5 and assign random labels to this group.
[Figure]
[Figure] [Figure]
s t
[Figure]
F o r g e t E x a c t
M a t c h (b) Perturb through setting masked weights
[Figure]
We consider KE-style perturbation to be the reinitialization of a random subset of the weights, and
IMP-style perturbation to be the zeroing of small magnitude weights and rewinding of the remaining weights.
More details are provided in Table A1.
Figure 1 shows that both types of weight perturbation adversely impact difﬁcult or mislabeled example groups more severely than the easy or true labeled examples, even when both example groups reach 100% train accuracy.
Related to our observations, Hooker et al.
(2019) show that pruned models disproportionately affect tails of the data distribution.
There are also theoretical works tying robustness-to-parameter-noise to compressibility, ﬂat minima, and generalization (Zhou et al., 2019b; Arora et al., 2018; Foret et al., 2021).
3. C OMPOSITIONAL LANGUAGE EMERGENCE
In the version of the Lewis game we study, the objective is to identify objects with two attributes:
SHAPE and COLOR .
The sender receives a one-hot representation of the SHAPE and COLOR attributes of an object and produces a discrete message.
The receiver takes this message as input and needs to classify the correct object from a set of candidate objects.
The two agents are typically parameterized as LSTMs (Hochreiter & Schmidhuber, 1997) and trained simultaneously using REINFORCE (Williams, 1992) or Gumbel-softmax (Jang et al., 2017; Maddison et al., 2017).
A compositional language would have a consistent and independent attribute-symbol mapping for all input objects.
This is desirable for out-of-distribution generalization, since unseen combinations of attributes will still have a meaningful language output.
Thus, the undesirable information in this task corresponds to a non-compositional language.
We deﬁne two candidate languages as designed by Li & Bowling
(2019), and use them to provide possible targets to the model.
One language is compositional, where each attribute (4 shapes and 8 colors) is represented by a distinct and consistent symbol-position.
The other language is non-compositional, which is formed by randomly permuting the mappings between messages and objects from the compositional language.
In this case, each object still has a unique message, but the individual symbols do not correspond to a consistent meaning for shape or color.
An example of a compositional and permuted language is provided in Table A2 and A3.
Since the sender controls the language output, we simplify the Lewis game here and train only the sender model to ﬁt a given language through supervised learning.
The target language is formed by mixing the two pre-speciﬁed languages.
For the 32 objects in the dataset, half of them are mapped to the compositional language and half to the non-compositional language.
Similar to the experiments in Section 3.1, we make a copy of the model at each training iteration, and perform the forgetting operation on the copy.
We evaluate both models and measure the accuracy of the two example groups separately.
For the weight perturbations, we consider either a mask based on initial weight magnitude or a random mask.
For both mask criteria, we either reinitialize the masked weights or set them to 0 .
More details can be found in Table A4.
As shown in Figure 2, we ﬁnd that all forms of weight perturbation under study lead to more forgetting of the non-compositional language.
T ARGETED F ORGETTING I MPROVES P ERFORMANCE
According to the forget-and-relearn hypothesis, the forgetting step should strike a balance between enabling relearning in the next generation and retaining desirable information from the previous generation.
We show in this section that through targeted forgetting of undesirable information and preserving of desirable information, we can signiﬁcantly improve upon existing algorithms.
4. R EDUCING O VERFITTING IN I MAGE C LASSIFICATION
We build upon the KE algorithm since it is widely applicable and leverages a large number of iterative training generations.
Following Taha et al.
(2021), we study tasks that have a small number of training examples per class, where the network is prone to overﬁtting, and adopt the same experimental framework unless otherwise speciﬁed.
Detailed hyperparameters, training procedure, and dataset descriptions can be found in Appendix A2.1-A2.2.
As described by Taha et al. (2021), in KE the mask criteria M is determined prior to training and generated randomly for weight-level splitting ( WELS ), or based on the location of the kernel for kernel-level splitting ( KELS ).
Given a split-rate s r , s r % of weights (for WELS ) or kernels (for
KELS ) in each layer l has a corresponding M l value of 1 .
For a neural network parameterized by Θ , we deﬁne the ﬁt-hypothesis as M (cid:12) Θ and reset-hypothesis as (1 − M ) (cid:12) Θ .
At the start of each generation, the weights in the ﬁt-hypothesis are kept the same, and the weights in the resethypothesis are reinitialized.
Then, the network is trained from this new initialization for e epochs, where e is the same for each generation.
Can we perform the forgetting step in a way that more explicitly targets difﬁcult examples in order to improve generalization performance?
Baldock et al.
(2021) introduce prediction depth as a measure of example difﬁculty in a given model.
Prediction depth refers to the layer after which an example’s activations can generate the same predictions using a KNN probe as the model’s ﬁnal predictions.
They show that increasing prediction depth corresponds to increasing example difﬁculty.
They study this correspondence using various notions of example difﬁculty, including output margin, adversarial input margin, and speed of learning for each data point.
Based on the observations in Baldock et al.
(2021), we hypothesize that by reinitializing the later layers of the neural network, we can remove information associated with difﬁcult examples more precisely than the mask criteria used in KE.
Thus, we propose a new forgetting procedure called later-layer forgetting (LLF) .
Given a layer threshold L , we deﬁne the LLF mask criterion for each layer l as:
[Equation]
We note that the concept of difﬁcult examples is used only as a proxy for information in the network that is more complex, brittle, and specialized.
We do not suggest that these examples should not be learned.
In fact, memorization of examples in the tails of the data distribution are often important for generalization (Feldman & Zhang, 2020), which we also achieve in the relearning stage.
Following Taha et al.
(2021), we perform LLF on top of non-iterative loss objectives designed to reduce overﬁtting.
We report performance with label-smoothing (S MTH )
(M¨uller et al., 2019; Szegedy et al., 2016) with α = 0 . 1 , and the CS-KD regularizer (Yun et al., 2020) with T = 4 and λ cls = 3 .
Additionally, we add a long baseline which trains the non-iterative method for the same total number of epochs as the corresponding iterative methods.
We present results for ResNet18 (He et al., 2016) in Table 1 and DenseNet169 (Huang et al., 2017) in Table A10.
We ﬁnd that LLF substantially outperforms both the long baseline and the KE models across all datasets for both model architectures.
Taha et al.
(2021) introduced KE to speciﬁcally work in the low data regime.
LLF as a direct extension of KE is expected to work in the same regime.
For completeness, we also evaluate LLF using ResNet50 on Tiny-ImageNet (Le & Yang, 2015), and WideResNet (Zagoruyko & Komodakis, 2016) and DenseNet-BC on CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) in Appendix A2.3.
We see a signiﬁcant gain on Tiny-ImageNet, but not on CIFAR-10 or CIFAR-100.
Analysis of Related Work.
Surprisingly, we ﬁnd that when compared to the long baselines with equal computation cost, KE actually underperforms in terms of generalization performance, which is contrary to the claims in Taha et al.
We can understand this low performance by looking at the ﬁrst few training epochs in each generation in Figure 3a.
We see that after the ﬁrst generation, the training accuracy remains near 100% after the forgetting operation.
This lack of forgetting suggests that KE would not beneﬁt from retraining under the forget-and-relearn hypothesis.
We also observe that KE experiences optimization difﬁculties post forgetting, as evidenced by a decrease in training accuracy in the ﬁrst epochs of each generation.
This could explain why it under-performs the long baseline.
Although KE can enable a reduction in inference cost by using only the slim ﬁt-hypothesis, our analysis reveals that this beneﬁt comes at the cost of lower performance.
Concurrent work (Alabdulmohsin et al., 2021) studies various types of iterative reinitialization approaches and proposes a layer-wise reinitialization scheme, which proceeds from bottom to top and reinitializing one fewer layer each generation.
We refer to this method as LW and provide a comparison to their method in Table 1.
We ﬁnd that LLF outperforms LW across all datasets we consider.
More discussion on this comparison can be found in Appendix A2.4.
Analysis of LLF.
In order to view the success of LLF and other similar iterative algorithms in terms of the selective forgetting of undesirable information, we should also observe that selective forgetting of desirable information should lead to worse results.
We can test this by reversing the
LLF procedure: instead of reinitializing the later layers, we can reinitialize the earlier layers.
The standard LLF for Flower reinitializes all layers starting from block 4 in ResNet18, which amounts to reinitializing 75% of parameters.
The reverse versions starting from block 4 and block 3 reinitializes 25% and 6% of weights respectively.
We see in Figure 3b that the reverse experiments indeed perform worse than both LLF and the long baseline.
Additionally, by looking at how prediction depth changes with the long baseline compared to LLF training in Figure A3, we can observe that LLF pushes more examples to be classiﬁed in the earlier layers.
This further suggests that forget-and-relearn with LLF encourages difﬁcult examples to be relearned using simpler and more general features in the early layers.
4. I NCREASING C OMPOSITIONALITY IN E MERGENT L ANGUAGES
In order to promote compositionality in the emergent language, Li & Bowling (2019) propose to periodically reinitialize the receiver during training.
They motivate this by showing that it is faster for a new receiver to learn a compositional language, and hypothesize that forcing the sender to teach the language to new receivers will encourage the language to be more compositional.
As is common in this line of work (Brighton & Kirby, 2006;
Lazaridou et al., 2018; Li & Bowling, 2019;
Ren et al., 2020), compositionality of the language is measured using topographic similarity ( ρ ), which is deﬁned as the correlation between distances of pairs of objects in the attribute space and the corresponding messages in the message space.
Li & Bowling (2019) show that ρ , measured using Hamming distances for objects and messages, is higher with the ease-of-teaching approach than training without resetting the receiver.
[Table]
We challenge this “ease-of-teaching” interpretation in Li & Bowling (2019) and offer an alternative explanation.
Instead of viewing the resetting of the receiver as inducing an ease-of-teaching pressure on the sender, we can view it as an asymmetric form of forgetting in the two-agent system.
This form of forgetting is sub-optimal, since we care about the quality of the messages , which is controlled by the sender.
This motivates a balanced forgetting approach for the Lewis game, whereby both sender and receiver can forget non-compositional aspects of their shared language.
Therefore, we propose an explicit forgetting mechanism that removes the asymmetry from the Li & Bowling (2019) setup.
We do this by partially reinitializing the weights of both the sender and the receiver and refer to this method as partial balanced forgetting (PBF) .
We use the “same weight reinit” perturbation method from Section 3.2 with 90% of weights masked.
Further training details are found in Appendix A3.
We show in Figure 4c that this method, which does not exert an “ease-of-teaching” pressure, signiﬁcantly outperforms both no resetting and resetting only the receiver in terms of ρ .
U NDERSTANDING THE F ORGET - AND -R ELEARN DYNAMIC
In Section 4, we discussed the importance of targeted forgetting in the forget-and-relearn dynamic.
In this section, we study the mechanism through which iterative retraining shapes learning.
5. I MAGE C LASSIFICATION
We consider two hypotheses for why iterative retraining with LLF helps generalization.
We note that these hypotheses are not contradictory and can both be true.
Later layers improve during iterative retraining with LLF.
Due to co-adaptation during training, the later layers may be learning before the more stable early layer features are fully developed (notably in early epochs of the ﬁrst generation), which might lead to more overﬁtting in the later
[Figure]
layers in some settings.
Reinitializing the later layers might give them an opportunity to be relearned by leveraging fully developed features from the earlier layers.
Early layers improve during iterative retraining with LLF.
Since the early layers are continuously reﬁned through iterative training, the features which are consistently useful under new learning conditions may be ampliﬁed.
These features may be beneﬁcial to generalization.
To test hypothesis 1, we freeze the early layer after the initial training generation and reinitialize the later layers for retraining in subsequent generations.
If the later layers improve as a result of learning on top of well-developed early layer features, we should expect a performance improvement during relearning even when early layers are frozen.
As shown in Figure 3c under “freeze early layers”, we observe no signiﬁcant performance improvement in this setting, suggesting that simply relearning later layers has little effect on generalization performance.
To test hypothesis 2, we design a reverse experiment where the later layers (except for the output
FC layer) are reinitialized and then frozen at each subsequent relearning generation.
The early layers are trained continuously in each generation.
As shown in Figure 3c under “freeze later layers”, we ﬁnd that even with a random frozen layer in the middle, iterative retraining of the early layers still improves performance signiﬁcantly.
To test the interpretation that early layers improve through learning under different conditions, we perform the same experiment as before, except we keep the later layers frozen at the same initialization in each generation.
This reduces the amount of variability seen during iterative retraining.
As shown in Figure 3c under “freeze ﬁxed later layers”, we ﬁnd the performance of this to be much worse than the version with a different reinitialization each generation, demonstrating the importance of having variable conditions for relearning.
These analyses suggest that the main function of iterative retraining is to distill features from the unforgotten knowledge that are consistently useful under different learning conditions.
5. C OMPOSITIONALITY
The analysis in Section 5.1 supports the hypothesis that features that are consistently learned across generations are strengthened in the iterative retraining process.
If this hypothesis explains why iterative algorithms improve compositionality in the Lewis game, it would predict that (i) compositional mappings are consistently learned across generations, (ii) the compositional mapping is strengthened during repeated retraining, and (iii) ρ increases during the retraining stage.
For iterated learning, Ren et al.
(2020) show that by limiting the number of training iterations in the imitation phase, the new sender can learn a language with better ρ than the previous one, arguing that the compositional components of the language are learned ﬁrst.
The success of iterated learning has primarily been attributed to this increase in compositionality during the imitation phase, which follows the explanation in the cognitive science theories from which iterated learning derives (Kirby, 2001).
However, when viewed as an instance of forget-and-relearn, our hypothesis predicts that the
[Figure]
improved performance in iterated learning is primarily driven by the interaction (retraining) phase.
The retraining phase reinforces features that are consistently learned, and we conjecture that these correspond to the compositional mappings.
We show in Figure 4b that the interaction phase is indeed primarily responsible for the improvement in ρ .
By looking at ρ during both the imitation and interaction phases using optimal hyperparameters under the setting in Ren et al.
(2020), we see that increases in ρ occur during the interaction phase, while the optimal imitation phase does not even need to result in a higher ρ than the end of the previous generation.
To show that the compositional components of the language are repeatedly learned and strengthened during retraining, we visualize the learned mappings from each input dimension to message dimension in Appendix A4.
We see that the same compositional mappings are retained across generations while the non-compositional aspects ﬂuctuate and decrease in magnitude.
We also observe that the probability of compositional mappings increase with increasing number of generations, illustrating the strengthening effects of iterative retraining.
C ONCLUSION
We introduce forget-and-relearn as a general framework to unify a number of seemingly disparate iterative algorithms in the literature.
Our work reveals several insights into what makes iterative training a successful paradigm.
We show that various forms of weight perturbation, commonly found in iterative algorithms, disproportionately forget undesirable information.
We hypothesize that this is a key factor to the success of these algorithms, and support this conjecture by showing that how well we can selectively forget undesirable information corresponds to the performance of the resulting algorithms.
Although we focus discussion on the forgetting of undesirable information, our analysis in Section 5 implies that this is in service of preserving desirable information.
We demonstrate that the relearning stage ampliﬁes the unforgotten information, and distills from it features that are consistently useful under different initial conditions induced by the forgetting step.
This work has implications in understanding and designing new iterative algorithms.
It places emphasis on what is forgotten rather than what is learned, and demonstrates that designing new methods for targeted information removal can be a fruitful area of research.
It can often be easier to deﬁne and suppress unwanted behavior than to delineate good behavior, and we illustrate two ways of doing so.
Our results suggest that there exists a Goldilocks zone of forgetting .
Forget too little and the network could easily retrain to the same basin; forget too much and the network would fail to accumulate progress over multiple relearning rounds.
Where this zone lies raises interesting questions in our understanding of neural network training that can be explored in future work.
Finally, the idea that features that are consistently useful under different learning conditions is reminiscent of the line of work in invariant predictions (Arjovsky et al., 2019; Mitrovic et al., 2021;
Zhang et al., 2021;
Parascandolo et al., 2021).
Understanding this dynamic in the iterative regime may point the way to creating algorithms that capture the same beneﬁt without the expensive iterative training process.
A CKNOWLEDGMENTS
The authors would like to thank Kevin Guo, Jason Yosinski, Sara Hooker, Evgenii Nikishin and
Ibrahim Alabdulmohsin for helpful discussions and feedback on this work.
The authors would also like to acknowledge Ahmed Taha, Fushan Li and Yi Ren for open sourcing their code, which this work heavily leverages.
This work was ﬁnancially supported by CIFAR, Hitachi, and Samsung.
R EPRODUCIBILITY
We make our code available at https://github.com/hlml/fortuitous_forgetting .
P ARTIAL W EIGHT P ERTURBATION AS T ARGETED F ORGETTING A1. D ETAILS ON W EIGHT P ERTURBATION M ETHODS
We can categorize this class of forgetting method in terms of both a mask criteria and a mask action .
The mask criteria is a function m : R d → { 0 , 1 } d , which determines which weights will be changed during the forgetting step by producing a binary mask with the same dimensions as the parameters of the neural network.
The mask action s : R d → R d determines how the replacement values of the weights are computed.
The mask-1 action s 1 operates on weights with mask value
[Table]
and mask-0 action s 0 operates on weights with mask value 0 .
For a neural network parameterized by Θ , forgetting through partial weight perturbation can then be deﬁned by the update rule
[Equation]
[Table] [Table]
A example of a non-compositional language adopted from Li & Bowling (2019).
black blue green grey pink purple red yellow circle ys gc lc pa bc ut ua rs square ga la yc ra dc pt rc dt star
yt ls ba bs bt lt uc gs triangle pc ps da ds rt us
ya gt
A1. E XTENSION TO CIFAR-10 AND I MAGE N ET
We extend our analyses of partial weight perturbation to CIFAR-10 (Krizhevsky et al., 2009) on a 4-layer convolutional model (Conv4) and a ResNet18 model, and to ImageNet (Deng et al., 2009) on ResNet50.
All CIFAR-10 models are evaluated using a 50% reset rate.
We ﬁnd reliable trends for both KE-style and IMP-style forgetting for all datasets.
For CIFAR-10, we follow the exact same procedure as the MNIST experiments.
Following the setup in Frankle & Carbin (2019), Conv4 model is trained using the Adam optimizer, and ResNet
[Table]
[Figure]
[Table]
[Figure]
is trained using SGD.
We train on 10% of training data for fast convergence.
Results on easy and hard examples are shown in Figure A1.
We ﬁnd that KE-style forgetting results in consistent and signiﬁcant difference between easy and hard examples.
However, the IMP-style forgetting operation renders the post-forgetting accuracy close to chance for CIFAR-10 models, and there is only a small difference between easy and hard example groups.
In the lottery ticket literature, it has long been observed that “late resetting” is needed to get performant sparse models as datasets and models get larger (Frankle et al., 2019).
In late resetting, we would rewind kept weights to the value a few epochs after training, rather than to the initial untrained values.
We perform a variant of IMPstyle forgetting where weights are rewound to epoch 3.
We ﬁnd that IMP-style forgetting with late resetting signiﬁcantly increases accuracy for both groups, and much more so for the easy group than the hard group.
We also show the same trend with true and random labeled example groups in
Figure A2.
These results demonstrate that IMP with late resetting achieves more targeted forgetting of undesirable information, which can be seen through the larger gaps between example groups.
These observations suggest another possible reason for the increased performance when using late resetting:
IMP with late resetting better leverages the beneﬁts of forget-and-relearn.
For ImageNet experiments, given computational constraints, we use a pretrained ResNet50 model on ImageNet and separate examples into easy and hard groups based on output margin.
We only use training examples that are correctly classiﬁed as to not disadvantage the difﬁcult examples.
This means that both group has 100% accuracy pre-forgetting.
We then perform various forgetting operations on top of the pretrained model and measure the accuracy of the two groups post-forgetting.
For KE-style forgetting, we randomly reinitialize 20% of weights on the same pretrained model and aggregate results over 5 samples.
For IMP-style forgetting, we set varying percentages of small weights to zero as a proxy given we are using a pretrained model and do not have access to initial or intermediate weights.
Results are shown in Table A5, and we ﬁnd that hard examples are more severely affected in all cases.
[Figure]
We measure the accuracy of the two groups after performing various forgetting operations on the pretrained models.
Random reinit values are the mean and standard error over 5 samples.
[Table]
A T ARGETED F ORGETTING I MPROVES P ERFORMANCE A2. A RCHITECTURES AND T RAINING D ETAILS
All networks are trained with stochastic gradient descent (SGD) with momentum of 0 . 9 and weight decay of 10 − 4 .
We also use a cosine learning rate schedule (Loshchilov & Hutter, 2017).
Taha et al.
(2021) use an initial learning rate of 0 . 256 , but we ﬁnd that a smaller learning rate than what is used in Taha et al.
(2021) to be beneﬁcial for certain datasets, thus we consider a learning rate in { 0 . 1 , 0 . 256 } for all experiments and report the setting with the better validation performance.
Models are trained with a batch size of 32 for 200 epochs each generation.
No early stopping is used.
For the layer threshold L , we determine the value through hyperparameter tuning on a validation set.
In the settings we consider, we ﬁnd that it is sufﬁcient to sweep through the starting layer or middle layer of model blocks, beginning from the second block.
This allows us to search for L within a restricted set of 6 of fewer values, making it practical even in deep models.
All ResNet18 models use a default layer threshold of 10 (start of block 3), with exception of the Flower dataset which uses a threshold of 14 (start of block 4) on the S MTH experiments.
All DenseNet169 models use a default layer threshold of 40 (start of denseblock 3), except for the Aircraft dataset which uses a threshold of 68 (middle of denseblock 3). [Table]
[Table]
A
[Table]
D ATASET S UMMARY
Taha et al.
(2021) uses different resizing of images for different datasets, and the speciﬁc resizing parameters are not speciﬁed in the paper.
For consistency, we resize all datasets to (256 , 256) .
We use a validation split to tune hyperparameters and report results on the test set when available.
A2. E XTENSION TO L ARGER D ATASETS AND M ODELS
In this section
[Table]
we evaluate LLF on larger datasets and models than what was used in the core comparison with KE.
We train ResNet50 on Tiny-ImageNet (Le & Yang, 2015), which consists of 200 classes with 500 training examples each.
We adopt the training setup from Ma et al.
and reset layers starting from the third block ( L = 23 ) during LLF.
As illustrated in Table A7, we ﬁnd
LLF to outperform the baselines on Tiny-ImageNet.
Furthermore, we also adopt two highly competitive baselines for CIFAR-10 and CIFAR-
(Krizhevsky et al., 2009) in the literature: WideResNet-28-10 (Zagoruyko & Komodakis, 2016) and DenseNet-BC ( k = 12 ) (Huang et al., 2017).
CIFAR-10 contains 10 classes with 5000 training examples each, while CIFAR-100 contains 100 classes with 500 examples each.
For WideResNet28-10, we reset from the second block ( L = 10 ), and for DenseNet-BC, we reset only the fullyconnected output layer ( L = 99 ) during LLF.
We report our results in Table A8.
In these settings, we do not see any improvements from LLF over our baselines.
For these dataset extensions, we follow the same simple heuristic that was used for the ﬁve small datasets in the main text.
Namely, we take existing training setups and hyperparameters from the literature as is and treat it as one generation, and search for L over a restricted set of values (6 or fewer).
Better results may be obtained through ﬁner hyperparameters tuning.
A2. C OMPARISON TO LW
Concurrent work (Alabdulmohsin et al., 2021) studies various types of iterative reinitialization approaches and ﬁnds that reinitializing entire layers produced the best performance.
They propose a layer-wise reinitialization scheme, proceeding from bottom to top and reinitializing one fewer layer each generation.
In addition, Alabdulmohsin et al.
(2021) performs weight rescaling and activation normalization for each layer that is not reinitialized.
We refer to this method as LW and add
[Table]
a comparison to their method in Table 1.
We ﬁnd that LLF outperforms LW across all datasets we consider.
However, we note that LLF introduces an additional hyperparameter for the layer threshold, which LW does not require.
Additionally, we ﬁnd that under our experimental settings, several choices proposed in LW actually harms performance.
We remove the addition of weight rescaling and normalization, and do not reinitialize batchnorm parameters, in order to achieve the best performance under the LW schedule.
The forget-and-relearn hypothesis suggests that it is unnecessary to reinitialize the ﬁrst few layers.
Doing so in LW hinders their progress in the ﬁrst few generations of training, and also leads to poorer generalization performance even after completing the full sequential schedule.
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
A D ETAILS ON P ARTIAL B ALANCED F ORGETTING
Our experimental settings follows Li & Bowling (2019).
We use one-layer LSTMs with a hidden size of 100 to parameterize both the sender and receiver agents.
We train using batch size of 100 , and use the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0 .
The language contains a vocabulary size of 8 and a message length of 2 .
We use 4 SHAPE and 8 COLOR attributes, and the object set consist of all 32 combinations of these attributes
The agents are trained with REINFORCE and entropy regularization using the following objectives:
[Equation]
[Table]
CS-KD
[Table]
CS-KD
long
[Table]
CS-KD
[Table]
KE
[Table]
CS-KD
[Table]
LLF
[Table]
N3)
[Table]
Ours) 67.
CS-KD
long
[Table]
CS-KD
[Table]
KE
[Table]
CS-KD
[Table]
Ours) 74.
CS-KD [Table] CS-KD [Table]
NoNorm NoRescale
[Table]
CS-KD
[Table]
LW(
[Table]
BN) [Table] CS-KD [Table]
LW(
[Table]
BN) NoNorm NoRescale (N8) 73.
CS-KD + LLF
[Table]
N8)
[Table]
Ours) 73.
∇ θ
[Equation]
=
E π
[Equation]
[Equation]
We use λ S = 0 .
1 and λ R = 0 .
1 in all our experiments, which differs slightly from Li & Bowling (2019) that use λ R = 0 .
05 , however we observe that the difference is small between these settings.
We train the “no reset” and “reset receiver” baselines for 6000 iterations per generation for 50 generations, following the hyperparameters in Li & Bowling (2019).
We train our PBF method with “same weight reinit” for 3000 iterations per generation for 100 generations for best performance.
However we note that even using the same 50 generation hyperparameter, PBF signiﬁcantly outperforms its baselines.
A E MERGENT LANGUAGE VISUALIZATIONS
In this section, we aim to visualize the languages learned by the sender in a Lewis game in the settings of Ren et al.
and Li & Bowling (2019).
Despite all the runs having near-perfect training accuracy , the degree of compositionality exhibited by the emergent languages varies signiﬁcantly depending on the use of forget-and-retrain as evident in Figure 4.
In both these settings, each object has two attributes that can take values in A 1 = { 1 , . . .
, r 1 } and A 2 = { 1 , . . . , r 2 } respectively, and each message has two tokens that can take values in M 1 = { 1 , . . .
, s 1 } and M 2 = { 1 , . . .
, s 2 } respectively.
Let us represent a ( j ) i as
if attribute
i has the value j , and 0 otherwise.
Similarly, we represent m ( j ) i as 1 if message token i has the value j , and 0 otherwise.
When r 1 + r 2 = s
1 + s
[Table]
CS-KD + LLF (N10) (Ours) 72.
( i.e. the total number of input attribute values is equal to the total number of message token values), a compositional language with maximum topographic similarity ( ρ ) would have each input attribute activation a ( l ) k mapped to a unique message token activation m ( q ) p .
Thus, we can represent the inputs and the messages both as concatenations of two one-hot vectors a = { a (1)1 , . .
, a ( r 1 ), a (1)2 , . . .
, a ( r 2 )}
and m = { m (1)1 , . . .
, m ( s 1 ), m (1)2 , . . .
, m ( s 2 )} respectively.
In Figures A4-A8, we illustrate the mappings from input dimensions a to message dimensions m .
Visualized values closer to 1 indicate a more consistent mapping across the entire dataset, indicating a compositional mapping.
Concretely, for each active input dimension a ( l ) k , we vary A (cid:54) =
k across all its values and collect the corresponding message token probabilities ˆ m =
ˆ m 1(1) , . .
, ˆ m 1( s 1 ) , ˆ m 2(1) , . .
, ˆ m 2( s 2 ) } produced by the sender via two softmax operations.
Then, the mean of these message probability vectors E A (cid:54) =
k [ ˆ m ] across A (cid:54) =
k is used as the row corresponding to input dimension a ( l ) k .
For a perfectly compositional mapping, the visualized matrix resembles a permutation matrix.
For iterated learning in Figure A4 and PBF in Figure A6, we see the emergence of a compositional mapping.
The compositional component is retained across generations whereas the noncompositional components are disproportionately affected in the forgetting step and are more likely to get remapped.
Finally, during the retraining phases, we see the compositional mappings get strengthened.
A C ONNECTIONS TO O THER T YPES OF F ORGETTING
In the continual learning literature, catastrophic forgetting (McCloskey & Cohen, 1989; Ratcliff, 1990) refers to the decrease in performance on older tasks due to the learning of new tasks.
There is a large body of literature focused on avoiding catastrophic forgetting (Ratcliff, 1990; Robins, 1995;
French, 1999; Kirkpatrick et al., 2017; Serra et al., 2018).
In this work, we view forgetting in a positive light, to say that it aides learning.
It is conceivable that catastrophic forgetting can even be leveraged to help learning with a well-chosen secondary task in a forget-and-relearn process.
It is worth noting that the continual learning problem is in some sense the opposite of what we consider
[Figure]
[Figure] [Figure]
[Figure] [Figure]
in forget-and-relearn.
Continual learning seeks to learn different tasks without forgetting old ones, while we explicitly assume that the tasks are available for iterative relearning.
Shwartz-Ziv & Tishby (2017) studies the progression of mutual information between each layer and the input and target variables throughout the training of deep neural networks.
They reveal that the converged values lie close to the information bottleneck theoretical bound, which implies a decrease in the mutual information with the input.
In a sense, the information bottleneck leads to the “forgetting” of unnecessary information.
It would be interesting to explore whether there are fundamental similarities between the information loss caused by SGD training dynamics and the effect resulting from forget-and-relearn.
Barrett & Zollman (2009) studies forgetting in the Lewis game from the perspective of suboptimal equilibria.
They show that using a learning rule that remembers the entire past (i.e. Herrnstein reinforcement learning), the model converges to a worse equilibrium than learning rules that discard past experience.
Yalnizyan-Carson & Richards (2021) also studies the beneﬁts of forgetting in the reinforcement learning formulation.
These works further point to the possibility of designing forgetting mechanisms that are built in to the learning process itself.
Although we claim many iterative training algorithms as instances of forget-and-relearn, there are also many that we do not, even though they may be related.
Algorithms like born-again networks (Furlanello et al., 2018) and self-training with noisy students (Xie et al., 2020) rely on iterative self-distillation, and do not have distinct forgetting and relearning stages.
Instead, knowledge is passed to the next generation through pseudo-labels.
A number of papers have tried to understand why students trained through self-distillation often generalize better than their teachers (Phuong &
Lampert, 2019; Tang et al., 2020; Allen-Zhu & Li, 2020; Stanton et al., 2021); perhaps viewing these methods through the lens of forgetting undesirable information offers an alternative path.