Original: Usable Information and Evolution of Optimal Representations During Training
Revised: Usable Information and Evolution of Optimal Representations During Training

Original: Abstract
Revised: Abstract

Original: We introduce a notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training, and how they adapt to different tasks.
Revised: We introduce a notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training, and how they adapt to different tasks.

Original: We use this to characterize the transient dynamics of deep neural networks on perceptual decisionmaking tasks inspired by neuroscience literature.
Revised: We use this to characterize the transient dynamics of deep neural networks on perceptual decisionmaking tasks inspired by neuroscience literature.

Original: In particular, we show that both the random initialization and the implicit regularization from Stochastic Gradient
Revised: In particular, we show that both the random initialization and the implicit regularization from Stochastic Gradient

Original: Descent play an important role in learning minimal sufﬁcient representations for the task.
Revised: Descent play an important role in learning minimal sufﬁcient representations for the task.

Original: If the network is not randomly initialized, we show that the training may not recover an optimal representation, increasing the chance of overﬁtting.
Revised: If the network is not randomly initialized, we show that the training may not recover an optimal representation, increasing the chance of overﬁtting

Original:  Introduction
Revised:  Introduction

Original: An important open question for the theory of deep learning is why highly overparametrized neural networks learn solutions that generalize well even though models can in principle memorize the entire training set.
Revised: An important open question for the theory of deep learning is why highly overparametrized neural networks learn solutions that generalize well even though models can in principle memorize the entire training set.

Original: Some have speculated that neural networks learn minimal but sufﬁcient representations of the input through implicit regularization of Stochastic Gradient Descent (SGD) (Shwartz-Ziv & Tishby, 2017; Achille & Soatto, 2018), and that the minimality of the representations relates to generalizability.
Revised: Some have speculated that neural networks learn minimal but sufﬁcient representations of the input through implicit regularization of Stochastic Gradient Descent (SGD) (Shwartz-Ziv & Tishby, 2017; Achille & Soatto, 2018), and that the minimality of the representations relates to generalizability.

Original: Follow-up work has disputed some of these claims (Saxe et al., 2018), leading to an ongoing debate on the optimality of representations and how they are learned during training.
Revised: Follow-up work has disputed some of these claims (Saxe et al., 2018), leading to an ongoing debate on the optimality of representations and how they are learned during training.

Original: Here, we design a simple task to empirically study how representations are formed during training, and how implicit regularization from SGD and initializations affect the resulting representations in deep networks.
Revised: Here, we design a simple task to empirically study how representations are formed during training, and how implicit regularization from SGD and initializations affect the resulting representations in deep networks.

Original: We then validate these results on a variant of an MNIST classiﬁcation task to assess how

Original: SGD affects the minimality of representations.

Original: Investigations into the optimality of representations have typically used information-theoretic reasoning.
Revised: Investigations into the optimality of representations have typically used information-theoretic reasoning.

Original: Most previous information-theoretic studies of deep learning have simultaneously studied the amount of information a representation contains about the input and output using mutual information.
Revised: Most previous information-theoretic studies of deep learning have simultaneously studied the amount of information a representation contains about the input and output using mutual information.

Original: However, when the mapping from input to representation is deterministic, the mutual information between the representation and input is degenerate (Saxe et al., 2018; Goldfeld et al., 2018).
Revised: However, when the mapping from input to representation is deterministic, the mutual information between the representation and input is degenerate (Saxe et al., 2018; Goldfeld et al., 2018).

Original: Rather than study the mutual information in a neural network, here we instead deﬁne and study the “usable information” in the network, which measures the amount of information that can be extracted from the representation by a learned decoder, and is scalable to high dimensional realistic tasks.
Revised: Rather than study the mutual information in a neural network, here we instead deﬁne and study the “usable information” in the network, which measures the amount of information that can be extracted from the representation by a learned decoder, and is scalable to high dimensional realistic tasks.

Original: We use this notion to quantify how relevant and irrelevant information is represented across layers of the network throughout the training process.
Revised: We use this notion to quantify how relevant and irrelevant information is represented across layers of the network throughout the training process.

Original: Studies into the optimality of representations typically assume that the parameter initializations are random.

Original: Random initializations do not encode information in the dataset.

Original: But often, for instance in transfer learning, initializations are not random and the weights from a related task are used as an initialization for a new task.

Original: The belief is that the initialization and representations from a related task will be useful for the new task.

Original: How does the initialization affect the solution found by SGD?

Original: How do the representations of inputs change?

Original: Does SGD always lead to equivalent representations, or does

Original: SGD trace a path through parameter space that leverages structure present in the initialization?

Original: Here, we propose simple tasks that allow us to characterize the usable information that representations contain, enabling us to address these questions.
Revised: Here, we propose simple tasks that allow us to characterize the usable information that representations contain, enabling us to address these questions.

Original: Our simple task was inspired by decision-making tasks in neuroscience, where inputs and outputs are carefully designed to probe speciﬁc information processing phenomena.
Revised: Our simple task was inspired by decision-making tasks in neuroscience, where inputs and outputs are carefully designed to probe speciﬁc information processing phenomena.

Original: In particular, we primarily focus on a particular task we refer to as the checkerboard (CB) task (Chandrasekaran et al., 2017;
Revised: In particular, we primarily focus on a particular task we refer to as the checkerboard (CB) task (Chandrasekaran et al., 2017;

Original: Kleinman et al., 2019).
Revised: Kleinman et al., 2019).

Original: In the CB task, one discerns the dominant color of a checkerboard ﬁlled with red and green squares.
Revised: In the CB task, one discerns the dominant color of a checkerboard ﬁlled with red and green squares.

Original: The subject then makes a reach to a left or right target whose color matches  the dominant color in the checkerboard (Fig 1a).
Revised: The subject then makes a reach to a left or right target whose color matches Trial 1: [Figure] the dominant color in the checkerboard (Fig 1a).

Original: This task therefore involves making two binary choices: a color decision (i.e., reach to the red or green target) and a direction decision (i.e., reach to left or right).
Revised: This task therefore involves making two binary choices: a color decision (i.e., reach to the red or green target) and a direction decision (i.e., reach to left or right).

Original: Critically, the target orientation (left red, right green; or left green, right red) is random on every trial.
Revised: Critically, the colors of the targets (red left, green right; or green left, red right) is random on every trial.

Original: The  output istherefore conditionally independent of the colordecision given the direction decision, as detailed further in Fig 1b and section A.5.
Revised: The direction decision output is conditionally independent of the color decision, as detailed further in Fig 1b and section C.4, even though the color information needs to be used to solve the task.

Original: We believe this task, albeit simple, captures key structure from deep learning tasks. For example, in image classiﬁcation, consider classifying an image as a car, which take on various colors. A representation in the last layer is typically conditionally independent of irrelevant input variations (i.e., the representation does not change based on differences in color) given a representation of the output class (i.e., a representation that the object is a car).
Revised: This task allows us to evaluate how both of these components of information are represented through training and across layers.

Original: We used this task and extensions to study the evolution of minimal representations during training.
Revised: We used this task and extensions to study the evolution of minimal representations during training.

Original: If a representation is sufﬁcient and minimal, we refer to this representation as optimal (Achille &
Revised: If a representation is sufﬁcient and minimal, we refer to this representation as optimal (Achille &

Original: Soatto, 2018).
Revised: Soatto, 2018).

Original: Our contributions are the following.
Revised: Our contributions are the following.

Original: (1) We introduce the notion of usable information for studying representations  in deep networks (Section 2).
Revised: (1) We introduce a notion of usable information for studying representations and training dynamics in deep networks (Section 2).

Original: (2) We used this notion to characterize the transient training dynamics in deep networks by studying the amount of usable relevant and irrelevant information in deep network layers.
Revised: (2) We used this notion to characterize the transient training dynamics in deep networks by studying the amount of usable relevant and irrelevant information in deep network layers.

Original: We deﬁne relevant and irrelevant information in the CB task in Appendix A.5.

Original: We found that training with SGD led to network solutions with minimal representations in later layers (Section 3.1 and 3.4).
Revised: We found that training with SGD led to network solutions with minimal representations in later layers (Section 3.1.

Original: This adds to literature suggesting that SGD results in minimal representations of input information (Achille & Soatto, 2018; ShwartzZiv & Tishby, 2017).
Revised: This adds to literature suggesting that SGD results in minimal representations of input information (Achille & Soatto, 2018; Shwartz-Ziv & Tishby, 2017).

Original: (3) We examined how pretraining on a related but different task affected asymptotic network representations (Section 3.2).
Revised: (3) We examined how pretraining on a related but different task affected asymptotic network representations (Section 3.2).

Original: When the network was initialized to contain usable information in later layers about a quantity it did not need to represent, SGD did not result in a minimal representation.
Revised: When the network was initialized to contain usable information in later layers about a quantity it did not need to represent, SGD did not result in a minimal representation.

Original: Rather, SGD leveraged the existing representations to solve the new task, leading to representations that were similar to the initializiation.
Revised: Rather, SGD leveraged the existing representations to solve the new task, leading to representations that were similar to the initialization.

Original: (4) We found that the minimality of the representation correlated with generalization performance and depended on the amount of pretraining (Section 3.3).

Original: Overall, we introduce a notion of usable information, and use it to study how optimal representations are formed during training, ﬁnding that SGD and random initializations play an important role.

Original: If the network is not randomly initialized, but is initialized with structure from a related but different task, we ﬁnd that networks may not recover an optimal representation and have worse generalization performance.

Original:  Usable information in a representation
Revised:  Usable information in a representation

Original: A deep neural network consists of a set of (cid:96) layers, with each layer forming a successive representation of the input.
Revised: A deep neural network consists of a set of (cid:96) layers, with each layer forming a successive representation of the input.

Original: A representation Z (cid:96) may store information in a variety of ways.
Revised: A representation Z (cid:96) may store information in a variety of ways.

Original: It may be that a complex transformation is required to readout the information, or it may be that a simple linear decoder could readout the information.
Revised: It may be that a complex transformation is required to readout the information, or it may be that a simple linear decoder could readout the information.

Original: In both cases, from an information-theoretic perspective, the same information is contained in the representation, however, there is an important distinction regarding how “usable” this information is.
Revised: In both cases, from an information-theoretic perspective, the same information is contained in the representation, however, there is an important distinction regarding how “usable” this information is.

Original: Information is usable if later layers, which comprise afﬁne transformations and element-wise nonlinearities, can use the representation to solve the task.
Revised: Information is usable if later layers, which comprise afﬁne transformations and element-wise nonlinearities, can use the representation to solve the task.

Original: Equivalently, usable information should be decodable by a separate neural network also employing afﬁne transformations and element-wise nonlinearities.
Revised: Equivalently, usable information should be decodable by a separate neural network also employing afﬁne transformations and element-wise nonlinearities.

Original: Formally, we deﬁne the usable information that a representation Z contains about a quantity Y , which may refer to the output or a component of the input, as:
Revised: Formally, we deﬁne the usable information that a representation Z contains about a quantity Y , which may refer to the output or a component of the input, as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, H ( Y ) is the entropy, or uncertainty, of Y , and L CE is the cross-entropy loss on the test set of a discriminator network q ( y | z ) trained to approximate the true distribution p ( y | z ) .
Revised: Here, H ( Y ) is the entropy, or uncertainty, of Y , and L CE is the cross-entropy loss on the test set of a discriminator network q ( y | z ) trained to approximate the true distribution p ( y | z ) .

Original: Our deﬁnition is motivated in the following manner.
Revised: Our deﬁnition is motivated in the following manner.

Original: The test set cross-entropy loss approximates how much uncertainty there is in the output Y given Z and the discriminator.
Revised: The test set cross-entropy loss approximates how much uncertainty there is in the output Y given Z and the discriminator.

Original: A low loss implies that there is low uncertainty in Y given Z , or that the discriminator can extract a lot of “information” about Y from Z .
Revised: A low loss implies that there is low uncertainty in Y given Z , or that the discriminator can extract a lot of “information” about Y from Z .

Original: If the logarithm in the cross-entropy loss is base 2 , it has the units of bits.
Revised: If the logarithm in the cross-entropy loss is base 2 , it has the units of bits.

Original: If nearly all of the output classes Y were the same, there would be little uncertainty in Y to begin with, so it is important to know the amount of uncertainty in Y given Z with respect to the initial uncertainty in Y .
Revised: If nearly all of the output classes Y were the same, there would be little uncertainty in Y to begin with, so it is important to know the amount of uncertainty in Y given Z with respect to the initial uncertainty in Y .

Original: What is most relevant is the amount of remaining uncertainty in Y given Z .
Revised: What is most relevant is the amount of remaining uncertainty in Y given Z .

Original: Thus we use the difference in uncertainty H ( Y ) − L CE as the amount of “usable information” that Z contains about Y , as shown in our deﬁnition in Equation 1.
Revised: Thus we use the difference in uncertainty H ( Y ) − L CE as the amount of “usable information” that Z contains about Y , as shown in our deﬁnition in Equation 1.

Revised: This deﬁnition is appealing to study representations, in part, because it can be computed from samples of Z and Y , and is a quantity that is comparable through network training.

Original: We estimate L CE using a small neural network that learns a distribution q ( y | z ) .
Revised: We estimate L CE using a small neural network that learns a distribution q ( y | z ) .

Original: To train the network, we sample activations Z and the quantity Y and learn q ( y | z ) by minimizing the cross entropy loss on a training set.
Revised: To train the network, we sample activations Z and the quantity Y and learn q ( y | z ) by minimizing the cross entropy loss on a training set.

Original: We then evaluate the L CE on the test set (Equation 1).
Revised: We then evaluate the L CE on the test set (Equation 1).

Original: Details of training the neural network we used for decoding are in Appendix A.2.
Revised: Details of training the neural network we used for decoding are in Appendix C.1.

Original: We also show in the Appendix that the usable information is a lower bound on the mutual information (Appendix A.1).
Revised: We also show in the Appendix that the usable information is a lower bound on the mutual information (Appendix B.1).

Revised: Importantly, usable information also is not constrained by the data processing inequality; that is, information can be made more “usable” by transformations to later layers, consistent with the representation learning view that later layers are forming improved representations of the inputs (Xu et al., 2020).

Original:  Experiments
Revised:  Experiments

Original: Our goal was to characterize how optimal representations are formed through SGD training and impacted by an initialization.
Revised: Our goal was to characterize how optimal representations are formed through SGD training and impacted by an initialization.

Original: We trained multiple network architectures on tasks and assessed the usable information in representations across layers and training epochs.
Revised: We trained multiple network architectures on tasks and assessed the usable information in representations across layers and training epochs.

Original: Within an architecture and task, all hyperparameters were kept constant throughout experiments.
Revised: Within an architecture and task, all hyperparameters were kept constant throughout experiments.

Original: We trained three different network architectures, ‘Small FC’: 5 layers, with 10 − 7 − 5 − 4 − 3 units in each layer, ‘Medium FC’: 100 − 20 − 20 − 20 , and ‘Large FC’: 1000 − 20 − 20 − 20 .
Revised: We trained three different network architectures, ‘Small FC’: 5 layers, with 10 − 7 − 5 − 4 − 3 units in each layer, ‘Medium FC’: 100 − 20 − 20 − 20 , and ‘Large FC’: 1000 − 20 − 20 − 20 .

Original: Small
Revised: Small

Original: FC and Large FC were networks used in recent literature (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018).
Revised: FC and Large FC were networks used in recent literature (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018).

Original: Our networks were fully-connected and used the relu activation.
Revised: Our networks were fully-connected and used the relu activation.

Original: We trained the networks using SGD with a constant learning rate to perform the CB task, described in detail in Appendix A.3.
Revised: We trained the networks using SGD with a constant learning rate to perform the CB task, described in detail in Appendix C.2.

Original: The hyperparameters used for all the experiments are listed in Appendix A.4.
Revised: The hyperparameters used for all the experiments are listed in Appendix C.3.

Original: In our CB task experiments, we quantiﬁed the usable color and direction information in the hidden representation, Z (cid:96) .
Revised: In our CB task experiments, we quantiﬁed the usable color and direction information in the hidden representation, Z (cid:96) .

Original: In the n = 2 CB task, the color information represents half of the input information.
Revised: In the n = 2 CB task, the color information represents half of the input information.

Original: We emphasize that, unless otherwise speciﬁed, the network was only trained to output the correct direction choice, so given a representation of the direction, a representation of the color choice is irrelevant.
Revised: We emphasize that, unless otherwise speciﬁed, the network was only trained to output the correct direction choice, so given a representation of the direction, a representation of the color choice is irrelevant.

Original: Therefore, a minimal representation should not include information about the color choice, since it is not necessary to represent given a representation of the direction decision.
Revised: Therefore, a minimal representation should not include information about the color choice, since it is not necessary to represent given a representation of the direction decision.

Original: To make the task more complex, we also generalized the CB task to have n = 10 and n = 20 targets.
Revised: To make the task more complex, we also generalized the CB task to have n = 10 and n = 20 targets.

Original: 3.1 SGD with random initialization results in minimal sufﬁcient representations 
Revised: 3.1 SGD with random initialization results in minimal sufﬁcient representations in the CB task

Original: We ﬁrst assessed the optimality of network representations with random weight initializations by training Small FC networks on the CB task using n = 2 colors (Fig 3a).
Revised: We ﬁrst assessed the optimality of network representations with random weight initializations by training Small FC networks on the CB task using n = 2 colors (Fig 2a).

Original: In random initializations, the initial weights do not contain information about the dataset.
Revised: In random initializations, the initial weights do not contain information about the dataset.

Original: We computed the usable color and direction information across layers of the neural network and epochs of training.
Revised: We computed the usable color and direction information across layers of the neural network and epochs of training.

Original: In our plots, later layers are denoted by darker shades.
Revised: In our plots, later layers are denoted by darker shades.

Original: In deeper layers, there was a decrease in usable color information, corresponding to more minimal representations.
Revised: In deeper layers, there was a decrease in usable color information, corresponding to more minimal representations.

Original: After training, the asymptotic representation in the last layer contained zero usable color information and 1 bit of usable direction information.
Revised: After training, the asymptotic representation in the last layer contained zero usable color information and 1 bit of usable direction information.

Original: To visualize this minimal sufﬁcient representation, we plotted the activations of the 3 units in the last layer of the Small FC network for different inputs.
Revised: To visualize this minimal sufﬁcient representation, we plotted the activations of the 3 units in the last layer of the Small FC network for different inputs.

Original: These visualizations are labeled by the correct color (red and green) and direction (cross or circle).
Revised: These visualizations are labeled by the correct color (red and green) and direction (cross or circle).

Original: In the asymptotic representation, representation of the input color is overlapping (red and green), while the representation of the direction output is separable (crosses and circles), forming a minimal sufﬁcient representation. 
Revised: In the asymptotic representation, representation of the input color is overlapping (red and green), while the representation of the direction output is separable (crosses and circles), forming a minimal sufﬁcient representation. [Figure]

Original: To test if this observed minimality was a result of our simple task, we extended the CB task to a variant with n input checkerboard colors, with n corresponding output direction classes.
Revised: To test if this observed minimality was a result of our simple task we extended the CB task to a variant with n input checkerboard colors, with n corresponding output direction classes.

Original: We trained networks using a larger architecture (Medium FC).
Revised: We trained networks using a larger architecture (Medium FC).

Original: We show results for n = 10 and n = 20 classes in Fig 3b,c. We observed similar phenomena to the n = 2 case: there was decreasing usable color information in deeper layers, and nearly zero color information in the last layer’s representation.
Revised: We show results for n = 10 and n = 20 classes in Fig 2b,c. We observed similar phenomena to the n = 2 case: there was decreasing usable color information in deeper layers, and nearly zero color information in the last layer’s representation.

Original: In contrast, there was signiﬁcant usable direction information across all layers in the asymptotic representation, with usable direction increasing for deeper layers.
Revised: In contrast, there was signiﬁcant usable direction information across all layers in the asymptotic representation, with usable direction increasing for deeper layers.

Original: We validated our results using different random initializations (Figures 6, 7, 8).
Revised: We validated our results using different random initializations (Figures 6, 7, 8).

Original: These results show that, for a simple task with SGD and random initialization, minimal sufﬁcient representations emerge through training.
Revised: These results show that, for a simple task with SGD and random initialization, minimal sufﬁcient representations emerge through training.

Original: Asymptotic representations were sufﬁcient to perform the task, but contained less usable color information in deeper layers, approaching zero color information in the last layer.
Revised: Asymptotic representations were sufﬁcient to perform the task, but contained less usable color information in deeper layers, approaching zero color information in the last layer.

Original: Schwartz-Ziv and Tishby suggested a “compression” phase in learning, where the network discards information about the inputs that do not need to be represented to solve the task.

Original: We did not always observe such compression in usable color information in later layers, which would imply an increase and then decrease in usable color information over training.

Original: Rather, we observed it was possible for the network to solve the task with nearly zero usable color information in its last layer across all training (Fig 3b,c).
Revised: We observed it was possible for the network to solve the task with nearly zero usable color information in its last layer across all training (Fig 2b,c).

Original: Additional runs are shown in Figures 9, 10, and 11.
Revised: Additional runs are shown in Figures 9, 10, and 11.

Original:  3.2 SGD with non-random initializations may not form minimal representations 
Revised: [Figure] 3.2 SGD with non-random initializations may not form minimal representations in the CB task

Original: Implicit regularization in SGD is hypothesized to result in minimal representations through compression of irrelevant input information, also called a “forgetting” phase (Shwartz-Ziv & Tishby, 2017;
Revised: Implicit regularization in SGD is hypothesized to result in minimal representations through compression of irrelevant input information, also called a “forgetting” phase (Shwartz-Ziv & Tishby, 2017;

Original: Achille & Soatto, 2018; Achille et al., 2019).
Revised: Achille & Soatto, 2018; Achille et al., 2019).

Original: We tested this hypothesis by initializing networks with signiﬁcant color information, and subsequently performing SGD on the CB task.
Revised: We tested this hypothesis by initializing networks with signiﬁcant color information, and subsequently performing SGD on the CB task.

Original: We then evaluated whether SGD resulted in networks with minimal color representations.
Revised: We then evaluated whether SGD resulted in networks with minimal color representations.

Original: We initialized networks by pretraining the network to output the color decision for 20 epochs, which required the network to represent color information.
Revised: We initialized networks by pretraining the network to output the color decision for 20 epochs, which required the network to represent color information.

Original: After 20 epochs, we reverted to training the CB task, where only the direction decision was reported.
Revised: After 20 epochs, we reverted to training the CB task, where only the direction decision was reported.

Original: Since the learning rate was kept constant, the pretrained weights can be viewed as a different initialization in parameter space for the modiﬁed task.
Revised: Since the learning rate was kept constant, the pretrained weights can be viewed as a different initialization in parameter space for the modiﬁed task.

Original: Strikingly, we found that the resulting representations were not minimal for the n = 2 checkerboard case (Fig 4a).
Revised: We found that the resulting representations were not minimal for the n = 2 checkerboard case (Fig 3a).

Original: This result also held for the CB task with n = 10 and n = 20 (Fig 2b,c).
Revised: This result also held for the CB task with n = 10 and n = 20 (Fig 4b,c).

Original: While we observed some compression of usable color information through training, asymptotic representations had signiﬁcantly greater than zero color information.
Revised: While we observed some compression of usable color information through training, asymptotic representations had signiﬁcantly greater than zero color information.

Original: In Fig 2b, we observed all layers had more usable color information than the direction information in the ﬁrst layer.
Revised: In Fig 4b, we observed all layers had more usable color information than the direction information in the ﬁrst layer.

Original: The network therefore solved the task using an alternative representation that was not minimal.
Revised: The network therefore solved the task using an alternative representation that was not minimal.

Original: We visualized the activations corresponding to the asymptotic non-minimal representations of Small FC in Fig 4d.
Revised: We visualized the activations corresponding to the asymptotic non-minimal representations of Small FC in Fig 3d.

Original: In the early epochs the red and green points converge (both crosses and dots) as a result of successful pretraining.
Revised: In the early epochs the red and green points converge (both crosses and dots) as a result of successful pretraining.

Original: However, when we trained the CB task starting at epoch 20, the representations changed.
Revised: However, when we trained the CB task starting at epoch 20, the representations changed.

Original: While the dot clusters for red and green checkerboards are overlapping, the cross clusters are not.
Revised: While the dot clusters for red and green checkerboards are overlapping, the cross clusters are not.

Original: This representation is not minimal as color information can be decoded above chance.
Revised: This representation is not minimal as color information can be decoded above chance.

Original: These results show that the initialization affects the asymptotic representation of neural networks.
Revised: These results show that the initialization affects the asymptotic representation of neural networks.

Original: SGD, under particular initializations, may not lead to minimal representations of task inputs.
Revised: SGD, under particular initializations, may not lead to minimal representations of task inputs.

Original: This suggests there is a trade-off between minimal representations and using existing representations present in the initial weights.
Revised: This suggests there is a trade-off between minimal representations and using existing representations present in the initial weights.

Original: Initial structure in the network representations from pretraining, such as the separation of the red and green crosses in the last layer representation, was maintained even when performing SGD to train a different task.
Revised: Initial structure in the network representations from pretraining, such as the separation of the red and green crosses in the last layer representation, was maintained even when performing SGD to train a different task.

Original: Together, these results suggest that while SGD compresses representations towards minimality, it ﬁnds a solution that is functionally related to the initial representation.
Revised: Together, these results suggest that while SGD compresses representations towards minimality, it ﬁnds a solution that is functionally related to the initial representation.

Original: This may correspond to a optima in the neighborhood of the initialization.
Revised: This may correspond to a optima in the neighborhood of the initialization.

Revised: Further, we found pretraining on the color choice led to worse generalization performance (Fig 4).

Revised:  Discussion

Revised: We introduced a notion of the usable information in the representation, which reﬂects the amount of information that can be extracted by a learned decoder.

Original: 3.3 Relationship between pretraining, minimality, and generalization
Revised: This deﬁnition is appealing, in part, due to its ﬂexibility. For instance, if it is important to understand how accessible the information is to a linear decoder, it sufﬁces to apply our formulation of usable information with a linear decoder trained with cross-entropy loss. In contrast, if the goal is extract all the information present in a representation, regardless of how accessible this information is, one can train a high capacity nonlinear decoder. Since neural networks are powerful function approximators, as the function approximation improves, the decoder will approach the optimal decoder.

Revised: In this case, the usable information approaches

Revised: Shannon mutual information, as the lower bound becomes tight (Section B.1).

Revised: Future theoretical and empirical work should investigate the tightness of this bound and its dependence on training parameters.

Revised: In our case, we used a relatively small nonlinear neural network as the decoder, which provided insight into the evolution of optimal representations through training on simple tasks inspired by neuroscience literature.

Revised: These tasks allowed us to show that random initializations and the implicit regularization from SGD play an important role in learning minimal sufﬁcient representations.

Revised: Notably, we found that a non-random initialization, corresponding to pretraining on a related but different task, led to solutions that were less likely to form optimal representations and had worse generalization performance.

Revised: Future work should evaluate the learning dynamics of more realistic network architectures on more challenging tasks.

Revised: A Additional experiments A.1 Relationship between pretraining, minimality, and generalization in the CB task

Revised: [Figure]

Original: Our results show that the minimality of network representations, and therefore solutions, depend on initialization.
Revised: Our results show that the minimality of network representations, and therefore solutions, depend on initialization.

Original: All trained networks (for n larger than 2 ), however, achieved zero training error.
Revised: All trained networks (for n larger than 2 ), however, achieved zero training error.

Original: A natural question to ask is how do the resulting representations affect generalization performance?
Revised: A natural question to ask is how do the resulting representations affect generalization performance?

Original: To answer this, we varied the number of epochs that we pretrained the CB tasks of n = 2 , n = 10 , and n = 20 classes, and quantiﬁed the usable color and direction information, as well as the trained network’s test accuracy to understand how the network generalizes (Fig. 2).
Revised: To answer this, we varied the number of epochs that we pretrained the CB tasks of n = 2 , n = 10 , and n = 20 classes, and quantiﬁed the usable color and direction information, as well as the trained network’s test accuracy to understand how the network generalizes (Fig. 4).

Original: We found a positive correlation with the minimality of the representation and generalization performance: networks with less usable color information achieved higher test set accuracy.
Revised: We found a positive correlation with the minimality of the representation and generalization performance: networks with less usable color information achieved higher test set accuracy.

Original: This was true regardless of the number of classes, but the effect was more pronounced (in terms of absolute difference in accuracy) when the network did not solve the task perfectly without pretraining.
Revised: This was true regardless of the number of classes, but the effect was more pronounced (in terms of absolute difference in accuracy) when the network did not solve the task perfectly without pretraining.

Original: We note that regardless of how long the networks were pretrained for, the networks were subsequently trained for the same number of epochs ( 80 ), with the same learning rate throughout training.
Revised: We note that regardless of how long the networks were pretrained for, the networks were subsequently trained for the same number of epochs ( 80 ), with the same learning rate throughout training.

Original: One interpretation is that when using existing structure to solve the task, the network learned a suboptimal solution to solving the task, increasing the chance of overﬁtting.
Revised: One interpretation is that when using existing structure to solve the task, the network learned a suboptimal solution to solving the task, increasing the chance of overﬁtting.

Original: 3.4 Binary MNIST classiﬁcation
Revised: A.2 Binary MNIST classiﬁcation

Original: Our earlier analyses use relatively simple tasks where it is straightforward to characterize relevant and irrelevant representations.
Revised: Our earlier analyses used relatively simple tasks where it is straightforward to characterize relevant and irrelevant representations.

Original: But do our ﬁndings that the resulting representations found by SGD changed based upon the initialization extend to a more realistic and complex task?
Revised: But do our ﬁndings that the resulting representations found by SGD changed based upon the initialization extend to a more realistic and complex task?

Original: To this end, we trained a network to predict whether digits from MNIST were either even or odd.
Revised: To this end, we trained a network to predict whether digits from MNIST were either even or odd.

Revised: One solution

Revised: [Figure]

Original: One solution the network could ﬁnd is to group features corresponding to even and odd digits, without explicitly representing the digits.
Revised: [Figure] [Figure] [Figure] the network could ﬁnd is to group features corresponding to even and odd digits, without explicitly representing the digits.

Original: This minimal solution would have 1 bit of digit information (corresponding to whether a digit is even or odd, but no other information).
Revised: This minimal solution would have 1 bit of digit information (corresponding to whether a digit is even or odd, but no other information).

Original: An alternative solution is to represent each digit and learn a classiﬁer that can group the digits into even or odd.
Revised: An alternative solution is to represent each digit and learn a classiﬁer that can group the digits into even or odd.

Original: This representation is not minimal, and would have closer to the maximal 3 .
Revised: This representation is not minimal, and would have closer to the maximal 3 .

Original: 32 bits of usable digit information.
Revised: 32 bits of usable digit information.

Original: When we trained a Large FC architecture to predict whether digits were even or odd, we found that the resulting representation was nearly minimal (Fig 5a).
Revised: When we trained a Large FC architecture to predict whether digits were even or odd, we found that the resulting representation was nearly minimal (Fig 5a).

Original: We are not claiming that the output layer has no additional input related information, but rather that the digit cannot be decoded from the representation.
Revised: We are not claiming that the output layer has no additional input related information, but rather that the digit cannot be decoded from the representation.

Original: We then changed the task, so that the network was ﬁrst asked to output the correct digit and then, the task was switched so that the network was only asked to output whether the digit was even or odd.
Revised: We then changed the task, so that the network was ﬁrst asked to output the correct digit and then, the task was switched so that the network was only asked to output whether the digit was even or odd.

Original: We pretrained the network for 20 epochs to output the correct digit, resulting in nearly 3 bits of usable digit information (Fig 5b).
Revised: We pretrained the network for 20 epochs to output the correct digit, resulting in nearly 3 bits of usable digit information (Fig 5b).

Original: After pretraining, we subsequently trained the network to only perform the even/odd classiﬁcation task.
Revised: After pretraining, we subsequently trained the network to only perform the even/odd classiﬁcation task.

Original: We found the asymptotic representation had little to no compression of digit information.
Revised: We found the asymptotic representation had little to no compression of digit information.

Original: Instead, it solved the task, with approximately the same amount of digit information as in the initialization.
Revised: Instead, it solved the task, with approximately the same amount of digit information as in the initialization.

Original: This suggests that SGD reused features present in the representation and arrived at an alternative task solution.Discussion
Revised: This suggests that SGD reused features present in the representation and arrived at an alternative task solution.

Original: We introduced the notion of the usable information in the representation, which reﬂects the amount of information that can be extracted by a learned decoder.

Original: This deﬁnition is appealing, in part, due to its ﬂexibility.

Original: For instance, if it is important to understand how accessible the information is to a linear decoder, it sufﬁces to apply our formulation of usable information with a linear decoder trained with cross-entropy loss.

Original: In contrast, if the goal is extract all the information present in a representation, regardless of how accessible this information is, one can train a high capacity nonlinear decoder.

Original: Since neural networks are powerful function approximators, as the function approximation improves, the decoder will approach the optimal decoder.

Original: In this case, the usable information approaches

Original: Shannon mutual information, as the lower bound becomes tight (Section A.1).

Original: Future theoretical and empirical work should investigate the tightness of this bound and its dependence on training parameters.

Original: In our case, we used a relatively small nonlinear neural network as the decoder, which provided insight into the evolution of optimal representations through training on simple tasks inspired by neuroscience literature.

Original: These tasks allowed us to show that random initializations and the implicit regularization from SGD play an important role in learning minimal sufﬁcient representations.

Original: Notably, we found that a non-random initialization, corresponding to pretraining on a related but different task, led to solutions that were less likely to form optimal representations and had worse generalization performance.

Original: Transfer learning relies on the idea that initialization from a related task will be useful for a new task, often one for which there is less data.

Original: Here, we ﬁnd that using a network pretrained on a related but different task as an initialization may not lead to optimal representations of the input data.

Original: In fact, it can decrease generalization performance (Fig 2), even when the network is subsequently trained for an equivalent number of training epochs.

Original: This is in line with recent literature showing the importance of the initial epochs of training (Achille et al., 2019), which alternatively can be viewed as different initializations.

Original: We believe our results contribute towards understanding the properties of representations and the settings in which they can be successfully ﬁne-tuned to different tasks.

Original: A Appendix A.1 Usable information lower bounds the mutual information
Revised: B Proofs B.1 Usable information lower bounds the mutual information

Original: The entropy of a distribution is deﬁned as
Revised: The entropy of a distribution is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The mutual information, I ( X ; Y ) , can be written in terms on an entropy term and as conditional entropy term:
Revised: The mutual information, I ( X ; Y ) , can be written in terms on an entropy term and as conditional entropy term:

Original: [Equation]
Revised: [Equation]

Original: We want to show that:
Revised: We want to show that:

Original: [Equation]
Revised: [Equation]

Original: It sufﬁces to show that:
Revised: It sufﬁces to show that:

Original: [Equation]
Revised: [Equation]

Original: where L CE is the cross-entropy loss on the test set.
Revised: where L CE is the cross-entropy loss on the test set.

Original: For our study, H ( Y ) represented the known distribution of output classes [Equation] which in our case were equiprobable.
Revised: For our study, H ( Y ) represented the known distribution of output classes, which in our case were equiprobable.

Original: [Equation]
Revised: [Equation]

Original: To approximate H ( Y | Z ) [Equation] we ﬁrst trained a neural network with cross-entropy loss to predict the output, Y , given the hidden activations, Z , learning a distribution q ( y | z ) .
Revised: To approximate H ( Y | Z ) , we ﬁrst trained a neural network with cross-entropy loss to predict the output, Y , given the hidden activations, Z , learning a distribution q ( y | z ) .

Original: The KL denotes the
Revised: The KL denotes the

Original: Kullback-Liebler divergence.
Revised: Kullback-Liebler divergence.

Original: We multiplied (and divided) by an arbitrary variational distribution, q ( y | z ) , in the logarithm of equation 6, leading to equation 7.
Revised: We multiplied (and divided) by an arbitrary variational distribution, q ( y | z ) , in the logarithm of equation 6, leading to equation 7.

Original: The ﬁrst term in equation 7 is the cross-entropy loss commonly used for training neural networks.
Revised: The ﬁrst term in equation 7 is the crossentropy loss commonly used for training neural networks.

Original: The second term is a KL divergence, and is therefore non-negative.
Revised: The second term is a KL divergence, and is therefore non-negative.

Original: In our approximator, the distribution, q ( y | x ) , is parametrized by a neural network.
Revised: In our approximator, the distribution, q ( y | x ) , is parametrized by a neural network.

Original: When the distribution q ( y | z ) = p ( y | z ) , our variational approximation of H ( Y | Z ) , and hence approximation of I ( Z ; Y ) is exact (Barber & Agakov, 2003; Poole et al., 2019).
Revised: When the distribution q ( y | z ) = p ( y | z ) , our variational approximation of H ( Y | Z ) , and hence approximation of I ( Z ; Y ) is exact (Barber & Agakov, 2003; Poole et al., 2019).

Revised: Recently, such variational approximations to mutual information have been viewed as a meaningful characterization of representations in deep networks, and the theoretical underpinnings of this approach is beginning to be investigated (Xu et al., 2020; Dubois et al., 2020).

Original: A.2 Details of neural network for usable information
Revised: C Experimental details C.1 Details of neural network for usable information

Original: To estimate usable information, we computed the cross-entropy loss of a decoder q(y|z) that predicts
Revised: To estimate usable information, we computed the cross-entropy loss of a decoder q(y|z) that predicts

Original: Y from Z .
Revised: Y from Z .

Original: The decoder was a three layer neural network, with 128, 64, and 32 units per layer, with leakyRelu activations (slope = 0.2), batchnorm and dropout ( p = 0 . 7 ).
Revised: The decoder was a three layer neural network, with 128, 64, and 32 units per layer, with leakyRelu activations (slope = 0.2), batchnorm and dropout ( p = 0 . 7 ).

Original: At each epoch, 1250 training samples were generated and supplied to the decoder, along with either the corresponding correct direction or color choice.
Revised: At each epoch, 1250 training samples were generated and supplied to the decoder, along with either the corresponding correct direction or color choice.

Original: We evaluated the cross-entropy loss on 3750 test samples to minimize overﬁtting.
Revised: We evaluated the cross-entropy loss on 3750 test samples to minimize overﬁtting.

Original: We trained the network for 100 epochs using a learning rate of 0 . 5 for ‘Medium FC’ and 0 .
Revised: We trained the network for 100 epochs using a learning rate of 0 . 5 for ‘Medium FC’ and 0 .

Original: for ‘Small FC.’
Revised: for ‘Small FC.’

Original: For the ‘Large FC’ used in MNIST experiments, we used a learning rate of 0 .
Revised: For the ‘Large FC’ used in MNIST experiments, we used a learning rate of 0 .

Original: 005 for 1000 epochs.
Revised: 005 for 1000 epochs.

Original: A.3 Checkerboard Task description
Revised: C.2 Checkerboard Task description

Original: Following the conventions of Kleinman et al.
Revised: Following the conventions of Kleinman et al.

Original: (2019), we modeled the CB task (Fig 1a), inputting the checkerboard color and target conﬁguration to a neural network that outputted the direction choice (Fig 1b).
Revised: (2019), we modeled the CB task (Fig 1a), inputting the checkerboard color and target conﬁguration to a neural network that outputted the direction choice (Fig 1b).

Original: We minimized the cross-entropy loss of the network output and the ground truth output.
Revised: We minimized the cross-entropy loss of the network output and the ground truth output.

Original: We extended the checkerboard task to the n checkerboard task by increasing the number of checkerboards.
Revised: We extended the checkerboard task to the n checkerboard task by increasing the number of checkerboards.

Original: Each target was 1 out of the n colors, with the targets forming an ‘n-polygon’.
Revised: Each target was 1 out of the n colors, with the targets forming an ‘n-polygon’.

Original: The correct direction corresponded to the direction of the target corresponding to the color of the checkerboard.
Revised: The correct direction corresponded to the direction of the target corresponding to the color of the checkerboard.

Original: We speciﬁed the color of each target as a one hot encoding, and the color of the checkerboard as a one hot encoding.
Revised: We speciﬁed the color of each target as a one hot encoding, and the color of the checkerboard as a one hot encoding.

Original: Noise with mean 0 and standard deviation of 0 .
Revised: Noise with mean 0 and standard deviation of 0 .

Original: 1 was added to the checkerboard inputs.
Revised: 1 was added to the checkerboard inputs.

Original: The targets and checkerboard color inputs were concatenated to form an input vector.
Revised: The targets and checkerboard color inputs were concatenated to form an input vector.

Original: The correct direction of the target was the output.
Revised: The correct direction of the target was the output.

Original: A.4 Details of experiments
Revised: C.3 Details of experiments

Original: The following are the hyperparameters used in our experiments.
Revised: The following are the hyperparameters used in our experiments.

Original: We trained three different network architectures, ‘Small FC’: 5 layers, with 10 −7 − 5 − 4 − 3 units in each layer, ‘Medium FC’: 100 − 20 − 20 − 20 , and ‘Large FC’: 1000 − 20 − 20 − 20 . We trained networks using SGD with a constant learning rate throughout training.
Revised: We trained three different network architectures, ‘Small FC’: 5 layers, with 10 −

Original: FC Small, n = 2 :
Revised: FC Small, n = 2 :

Original: • batch size: 32, learning rate: 0.05, number of data samples: 10000 ( 90% train, 10% validation) Medium FC, n = 10 : • batch size: 64, learning rate: 0.5, number of data samples: 25000 ( 90% train, 10% validation) Medium FC, n = 20 : • batch size: 128, learning rate: 0.5, number of data samples: 50000 ( 90% train, 10% validation) Medium FC, n = 25 : • batch size: 128, learning rate: 0.5, number of data samples: 75000 ( 90% train, 10% validation) Large FC, (MNIST): • batch size: 128, learning rate: 0.
Revised: • batch size: 32, learning rate: 0.05, number of data samples: 10000 ( 90% train, 10% validation) Medium FC, n = 10 : • batch size: 64, learning rate: 0.5, number of data samples: 25000 ( 90% train, 10% validation) Medium FC, n = 20 : • batch size: 128, learning rate: 0.5, number of data samples: 50000 ( 90% train, 10% validation) Medium FC, n = 25 : • batch size: 128, learning rate: 0.5, number of data samples: 75000 ( 90% train, 10% validation) Large FC, (MNIST): • batch size: 128, learning rate: 0.

Original: A.5 Deﬁnition of relevant and irrelevant information
Revised: C.4 Deﬁnition of relevant and irrelevant information

Original: In the CB task, the color of the checkerboard and target conﬁguration (inputs) are necessary to determine the correct direction to reach (output).
Revised: In the CB task, the color of the checkerboard and target conﬁguration (inputs) are necessary to determine the correct direction to reach (output).

Original: While both a color and direction decision are made, after the direction is determined, the color decision no longer needs to be represented: the network can generate the correct output with only the direction representation.
Revised: While both a color and direction decision are made, after the direction is determined, the color decision no longer needs to be represented: the network can generate the correct output with only the direction representation.

Original: Formally, the output y is conditionally independent of the color representation, Z c , given the direction representation
Revised: Formally, the output y is conditionally independent of the color representation, Z c , given the direction representation

Original: Z d (i.e., y ⊥⊥ ( Z c , Z t ) | Z d , as illustrated by the graph in Fig 1b).
Revised: Z d (i.e., y ⊥⊥ ( Z c , Z t ) | Z d , as illustrated by the graph in Fig 1b).

Original: Hence, given a representation of the direction choice, the color choice (and target conﬁguration) no longer needs to be represented.
Revised: Hence, given a representation of the direction choice, the color choice (and target conﬁguration) no longer needs to be represented.

Original: We emphasize that, in general, the output is not independent of the color representation and target conﬁguration representation Z t , i.e., y (cid:54)⊥⊥ ( Z c , Z t ) , hence information about the dominant color of the checkerboard is necessary to compute y .
Revised: We emphasize that, in general, the output is not independent of the color representation and target conﬁguration representation Z t , i.e., y (cid:54)⊥⊥ ( Z c , Z t ) , hence information about the dominant color of the checkerboard is necessary to compute y .

Original: When this conditional independence holds, we call the conditionally independent variable “irrelevant.”
Revised: When this conditional independence holds, we call the conditionally independent variable “irrelevant.”

Original: We therefore refer to color choice as “irrelevant" and the direction choice as “relevant.
Revised: We therefore refer to color choice as “irrelevant" and the direction choice as “relevant.

Original: " We study how these components evolve together throughout training.
Revised: " We study how these components evolve together throughout training.

Original: A.6 Binary MNIST task description
Revised: C.5 Binary MNIST task description

Original: We trained the FC Mnist architecture to output whether the digit was even or odd.
Revised: We trained the FC Mnist architecture to output whether the digit was even or odd.

Original: Accordingly, a minimal representation should only encode whether the digit was even or odd, and not the particular digit.
Revised: Accordingly, a minimal representation should only encode whether the digit was even or odd, and not the particular digit.

Original: B Additional Figures
Revised: D

Revised: Additional Figures

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: U s ab l e i n f o r m a t
Revised: U s ab l e i n f o r m a t

Original: i on ( b i t s )
Revised: i on ( b i t s )

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i n f
Revised: i n f

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: a t i
Revised: a t i

Original: [Figure]
Revised: [Figure]