Original: Spectral Bias in Practice: the Role of Function Frequency in Generalization
Revised: Spectral Bias in Practice: the Role of Function Frequency in Generalization

Revised: Raphael Gontijo

Revised: Lopes

Revised: Google Brain iraphael@google com

Revised: Rebecca Roelofs

Revised: Google Brain rofls@google com

Original: Abstract
Revised: Abstract

Original: Despite their ability to represent highly expressive functions, deep learning models seem to ﬁnd simple solutions that generalize surprisingly well.
Revised: Despite their ability to represent highly expressive functions, deep learning models seem to ﬁnd simple solutions that generalize surprisingly well.

Original: Spectral bias – the tendency of neural networks to prioritize learning low frequency functions – is one possible explanation for this phenomenon, but so far spectral bias has primarily been observed in theoretical models and simpliﬁed experiments.
Revised: Spectral bias – the tendency of neural networks to prioritize learning low frequency functions – is one possible explanation for this phenomenon, but so far spectral bias has primarily been observed in theoretical models and simpliﬁed experiments.

Original: In this work, we propose methodologies for measuring spectral bias in modern image classiﬁcation networks on CIFAR-10 and ImageNet.
Revised: In this work, we propose methodologies for measuring spectral bias in modern image classiﬁcation networks on CIFAR-10 and ImageNet.

Original: We ﬁnd that these networks indeed exhibit spectral bias, and that interventions that improve test accuracy on
Revised: We ﬁnd that these networks indeed exhibit spectral bias, and that interventions that improve test accuracy on

Original: CIFAR-10 tend to produce learned functions that have higher frequencies overall but lower frequencies in the vicinity of examples from each class.
Revised: CIFAR-10 tend to produce learned functions that have higher frequencies overall but lower frequencies in the vicinity of examples from each class.

Original: This trend holds across variation in training time, model architecture, number of training examples, data augmentation, and self-distillation.
Revised: This trend holds across variation in training time, model architecture, number of training examples, data augmentation, and self-distillation.

Original: We also explore the connections between function frequency and image frequency and ﬁnd that spectral bias is sensitive to the low frequencies prevalent in natural images.
Revised: We also explore the connections between function frequency and image frequency and ﬁnd that spectral bias is sensitive to the low frequencies prevalent in natural images.

Original: On ImageNet, we ﬁnd that learned function frequency also varies with internal class diversity, with higher frequencies on more diverse classes.
Revised: On ImageNet, we ﬁnd that learned function frequency also varies with internal class diversity, with higher frequencies on more diverse classes.

Original: Our work enables measuring and ultimately inﬂuencing the spectral behavior of neural networks used for image classiﬁcation, and is a step towards understanding why deep models generalize well.
Revised: Our work enables measuring and ultimately inﬂuencing the spectral behavior of neural networks used for image classiﬁcation, and is a step towards understanding why deep models generalize well.

Original:  Introduction
Revised:  Introduction

Original: Two fundamental questions in machine learning are why overparameterized models generalize and how to make them more robust to distribution shift and adversarial examples.
Revised: Two fundamental questions in machine learning are why overparameterized models generalize and how to make them more robust to distribution shift and adversarial examples.

Original: Resolving both of these questions requires understanding how complex our models should be.
Revised: Resolving both of these questions requires understanding how complex our models should be.

Original: For instance, it is thought that overparameterized models generalize well because there are implicit regularizers that constrain the complexity of the learned functions.
Revised: For instance, it is thought that overparameterized models generalize well because there are implicit regularizers that constrain the complexity of the learned functions.

Original: However, the precise nature of these implicit regularizers and their importance in practice remains unclear.
Revised: However, the precise nature of these implicit regularizers and their importance in practice remains unclear.

Original: As for how to achieve robustness, no consensus has emerged regarding function complexity.
Revised: As for how to achieve robustness, no consensus has emerged regarding function complexity.

Original: On the one hand, Cranko et al.
Revised: On the one hand, Cranko et al.

Original: [4] and Leino et al.
Revised: [4] and Leino et al.

Original: [24] argue that Lipschitz smoothness of the learned function offers a guarantee of robustness; on the other, Shah et al.
Revised: [24] argue that Lipschitz smoothness of the learned function offers a guarantee of robustness; on the other, Shah et al.

Original: [37] and Madry et al.
Revised: [37] and Madry et al.

Original: [25] argue that simplicity can be harmful and a robust model must actually be more complex than its non-robust counterpart.
Revised: [25] argue that simplicity can be harmful and a robust model must actually be more complex than its non-robust counterpart.

Original: One window into function complexity is spectral bias – the tendency of neural networks to learn low frequency (simple and smooth) functions ﬁrst, then gradually increase the frequency (complexity) of the learned function as training proceeds.
Revised: One window into function complexity is spectral bias – the tendency of neural networks to learn low frequency (simple and smooth) functions ﬁrst, then gradually increase the frequency (complexity) of the learned function as training proceeds.

Original: Foundational work in this area has shown theoretical evidence of spectral bias by analyzing convergence rates of neural networks towards functions of different frequencies [1, 35].
Revised: Foundational work in this area has shown theoretical evidence of spectral bias by analyzing convergence rates of neural networks towards functions of different frequencies [1, 35].

Original: In practice, spectral bias is difﬁcult to measure: the most direct method involves taking a Fourier transform with respect to the input, which is impractical to compute due to the high dimensionality of images.
Revised: In practice, spectral bias is difﬁcult to measure: the most direct method involves taking a Fourier transform with respect to the input, which is impractical to compute due to the high dimensionality of images.

Original: Early experimental work focused on low-dimensional synthetic data [1, 2] or used proxy measurements of spectral bias [44], for instance by inserting label noise of various frequencies during training [35].
Revised: Early experimental work focused on low-dimensional synthetic data [1, 2] or used proxy measurements of spectral bias [44], for instance by inserting label noise of various frequencies during training [35].

Original: However, the label noise method is limited to binary classiﬁcation and involves modifying the training data, making it difﬁcult to disentangle how other changes to the training data (such as augmentation) affect the spectral content of the learned function.
Revised: However, the label noise method is limited to binary classiﬁcation and involves modifying the training data, making it difﬁcult to disentangle how other changes to the training data (such as augmentation) affect the spectral content of the learned function.

Original: It remains an open question whether modern neural networks exhibit spectral bias and what role it plays in generalization.
Revised: It remains an open question whether modern neural networks exhibit spectral bias and what role it plays in generalization.

Original: In this work, we investigate model complexity through the lens of spectral bias, by introducing experimental methods to study the frequency decomposition of the functions learned by modern image classiﬁcation networks.
Revised: In this work, we investigate model complexity through the lens of spectral bias, by introducing experimental methods to study the frequency decomposition of the functions learned by modern image classiﬁcation networks.

Original: We ﬁnd that high-accuracy models learn a function that is highfrequency in regions between different image classes but low-frequency within each class, validating the intuition that a “good” model should have sharp decision boundaries to delineate different classes, but smooth behavior within each class.
Revised: We ﬁnd that high-accuracy models learn a function that is highfrequency in regions between different image classes but low-frequency within each class, validating the intuition that a “good” model should have sharp decision boundaries to delineate different classes, but smooth behavior within each class.

Original: Contributions.
Revised: Contributions.

Original: We extend the label noise procedure of Rahaman et al.
Revised: We extend the label noise procedure of Rahaman et al.

Original: [35] to enable measuring spectral bias in multi-class classiﬁcation via label smoothing, and apply this technique to understand the function frequencies present in high-accuracy models on CIFAR-10 [23].
Revised: [35] to enable measuring spectral bias in multi-class classiﬁcation via label smoothing, and apply this technique to understand the function frequencies present in high-accuracy models on CIFAR-10 [23].

Original: We also introduce a second method for measuring the smoothness of a learned function via linear interpolation between test examples, and apply this method to both CIFAR-10 and ImageNet [8] models.
Revised: We also introduce a second method for measuring the smoothness of a learned function via linear interpolation between test examples, and apply this method to both CIFAR-10 and ImageNet [8] models.

Original: We use this method to probe the effects of the training data on the learned function frequencies, and to distinguish the spectral content in paths between images from the same class (“within-class paths”) and paths between images from different classes (“between-class paths”).
Revised: We use this method to probe the effects of the training data on the learned function frequencies, and to distinguish the spectral content in paths between images from the same class (“within-class paths”) and paths between images from different classes (“between-class paths”).

Original: We ﬁnd that higher-test-accuracy models demonstrate greater separation in frequency content between these path types, with lower frequencies within-class and higher frequencies between-class.
Revised: We ﬁnd that higher-test-accuracy models demonstrate greater separation in frequency content between these path types, with lower frequencies within-class and higher frequencies between-class.

Original: On CIFAR-10, this trend holds as we improve the model architecture, train longer, increase the size of the training dataset, improve the data augmentation strategy, and apply self-distillation.
Revised: On CIFAR-10, this trend holds as we improve the model architecture, train longer, increase the size of the training dataset, improve the data augmentation strategy, and apply self-distillation.

Original: On
Revised: On

Original: ImageNet, this trend appears again as we consider models with more diverse architectures.
Revised: ImageNet, this trend appears again as we consider models with more diverse architectures.

Original: We also explore the relationship between image frequency and function frequency.
Revised: We also explore the relationship between image frequency and function frequency.

Original: By further extending the label noise methodology of Rahaman et al.
Revised: By further extending the label noise methodology of Rahaman et al.

Original: [35] to study spectral bias in directions of interest through the input space, we ﬁnd that CIFAR-10 models most readily learn functions of the low image frequencies common in natural images.
Revised: [35] to study spectral bias in directions of interest through the input space, we ﬁnd that CIFAR-10 models most readily learn functions of the low image frequencies common in natural images.

Original: Via linear interpolation within ImageNet classes, we ﬁnd that many models are higher-frequency in more internally diverse classes and lower-frequency in more internally consistent classes.
Revised: Via linear interpolation within ImageNet classes, we ﬁnd that many models are higher-frequency in more internally diverse classes and lower-frequency in more internally consistent classes.

Original:  Related Work
Revised:  Related Work

Original: Implicit bias.
Revised: Implicit bias.

Original: A common belief is that some form of implicit bias imposed by the training procedure or optimization algorithm may account for the generalization ability of overparameterized neural networks, and accordingly, much research has been directed towards understanding these implicit biases [38, 48, 22, 43, 14, 13, 15, 17, 3, 31, 29, 30, 26, 14, 20].
Revised: A common belief is that some form of implicit bias imposed by the training procedure or optimization algorithm may account for the generalization ability of overparameterized neural networks, and accordingly, much research has been directed towards understanding these implicit biases [38, 48, 22, 43, 14, 13, 15, 17, 3, 31, 29, 30, 26, 14, 20].

Original: Spectral bias.
Revised: Spectral bias.

Original: Spectral bias, also called the frequency principle, is a form of implicit bias with much recent attention including theoretical and experimental approaches [34, 44, 35, 1, 50].
Revised: Spectral bias, also called the frequency principle, is a form of implicit bias with much recent attention including theoretical and experimental approaches [34, 44, 35, 1, 50].

Original: Basri et al.
Revised: Basri et al.

Original: [1] studied spectral bias using a linear model of SGD training dynamics to show that models learn low frequency (simple) functions faster, assuming training data that is distributed uniformly on the hypersphere.
Revised: [1] studied spectral bias using a linear model of SGD training dynamics to show that models learn low frequency (simple) functions faster, assuming training data that is distributed uniformly on the hypersphere.

Original: Basri et al.
Revised: Basri et al.

Original: [2] extended the analysis to consider nonuniformly spaced training data, and found that learning is faster where samples are denser; if sampling is nonuniform, then during training, the learned function will be higher frequency in regions with denser samples.
Revised: [2] extended the analysis to consider nonuniformly spaced training data, and found that learning is faster where samples are denser; if sampling is nonuniform, then during training, the learned function will be higher frequency in regions with denser samples.

Original: Rahaman et al.
Revised: Rahaman et al.

Original: [35] used a more direct analysis to show the same spectral bias toward low frequency functions and also posited that high frequency components of the learned function are most sensitive to perturbations in the model parameters, connecting back to the idea of ﬂat optimization minima [30] or smooth loss landscape [27].
Revised: [35] used a more direct analysis to show the same spectral bias toward low frequency functions and also posited that high frequency components of the learned function are most sensitive to perturbations in the model parameters, connecting back to the idea of ﬂat optimization minima [30] or smooth loss landscape [27].

Original: They proposed experimental methods to study spectral bias in image classiﬁcation, but focused on a binary subset of the relatively simple MNIST dataset [9], with mean square error loss.
Revised: They proposed experimental methods to study spectral bias in image classiﬁcation, but focused on a binary subset of the relatively simple MNIST dataset [9], with mean square error loss.

Original: Our label smoothing experiments are a direct extension of Rahaman et al.
Revised: Our label smoothing experiments are a direct extension of Rahaman et al.

Original: [35] to multiclass classiﬁcation with the more common cross-entropy loss.
Revised: [35] to multiclass classiﬁcation with the more common cross-entropy loss.

Original: Our linear interpolation experiments are somewhat similar to the recently-proposed experimental methods in Zhang et al.
Revised: Our linear interpolation experiments are somewhat similar to the recently-proposed experimental methods in Zhang et al.

Original: [50], except that we sample along paths between images rather than in regions surrounding each image; this allows for a more global measurement of spectral bias that we apply to a broad and distinct set of training procedures beyond the double descent regime studient in Zhang et al.
Revised: [50], except that we sample along paths between images rather than in regions surrounding each image; this allows for a more global measurement of spectral bias that we apply to a broad and distinct set of training procedures beyond the double descent regime studient in Zhang et al.

Original: Model sensitivity to image frequency.
Revised: Model sensitivity to image frequency.

Original: While we focus on function frequency , prior research aimed to understand model sensitivity to image frequencies.
Revised: While we focus on function frequency , prior research aimed to understand model sensitivity to image frequencies.

Original: Jo and Bengio [19] found that CNNs are sensitive to Fourier statistics of the training data, even those irrelevant to human viewers.
Revised: Jo and Bengio [19] found that CNNs are sensitive to Fourier statistics of the training data, even those irrelevant to human viewers.

Original: Ortiz-Jiménez et al.
Revised: Ortiz-Jiménez et al.

Original: [33] studied the image frequency bias induced by using a convolutional architecture, and OrtizJiménez et al.
Revised: [33] studied the image frequency bias induced by using a convolutional architecture, and OrtizJiménez et al.

Original: [32] argued that models are sensitive primarily to discriminative Fourier directions in the training data.
Revised: [32] argued that models are sensitive primarily to discriminative Fourier directions in the training data.

Original: Yin et al.
Revised: Yin et al.

Original: [46] introduced a procedure to measure the sensitivity of trained models to
Revised: [46] introduced a procedure to measure the sensitivity of trained models to

Original: Fourier image perturbations, and applied it to study the effects of adversarial training and data augmentation.
Revised: Fourier image perturbations, and applied it to study the effects of adversarial training and data augmentation.

Original: Our work adds to this line of research a study of the interplay between two notions of frequency: the function frequency involved in spectral bias, and the image frequencies present in the data.
Revised: Our work adds to this line of research a study of the interplay between two notions of frequency: the function frequency involved in spectral bias, and the image frequencies present in the data.

Original:  Methodology
Revised:  Methodology

Original: The goal of our work is to measure the complexity of neural network functions through the lens of frequency.
Revised: The goal of our work is to measure the complexity of neural network functions through the lens of frequency.

Original: In low dimensions, measuring the frequency decomposition of a function is straightforward and tractable: evaluate the function at dense, uniform sampling positions and compute its discrete
Revised: In low dimensions, measuring the frequency decomposition of a function is straightforward and tractable: evaluate the function at dense, uniform sampling positions and compute its discrete

Original: Fourier transform.
Revised: Fourier transform.

Original: However, the function of interest in image classiﬁcation is unavoidably highdimensional, mapping images (with thousands of pixel values) to object classes.
Revised: However, the function of interest in image classiﬁcation is unavoidably highdimensional, mapping images (with thousands of pixel values) to object classes.

Original: It would be intractable even to collect sufﬁcient samples of this function to compute a discrete Fourier transform, let alone compute the transform itself.
Revised: It would be intractable even to collect sufﬁcient samples of this function to compute a discrete Fourier transform, let alone compute the transform itself.

Original: Instead, we employ two complementary approaches to measure informative proxies of this frequency decomposition.
Revised: Instead, we employ two complementary approaches to measure informative proxies of this frequency decomposition.

Original: 3.1 Label Smoothing
Revised: 3.1 Label Smoothing

Original: The core idea for measuring function frequency introduced by Rahaman et al.
Revised: The core idea for measuring function frequency introduced by Rahaman et al.

Original: [35] is to construct a sinusoid over the space of images, and to use that sinusoid as a form of label noise during training.
Revised: [35] is to construct a sinusoid over the space of images, and to use that sinusoid as a form of label noise during training.

Original: Let D =
Revised: Let D =

Original: ( X i , y i ) } n train i
Revised: ( X i , y i ) } n train i

Original: = be the training examples and D valid = { ( X j , y j ) } n val j =1 be the validation examples, with X i an image and y i a one-hot class encoding, where n train is the number of training examples, n val is the number of validation examples, d is the side length of an image (assumed to be square for simplicity), c is the number of color channels, and M is the number of classes, so
Revised: = be the training examples and D valid = { ( X j , y j ) } n val j =1 be the validation examples, with X i an image and y i a one-hot class encoding, where n train is the number of training examples, n val is the number of validation examples, d is the side length of an image (assumed to be square for simplicity), c is the number of color channels, and M is the number of classes, so

Original: X i ∈ R d × d × c and y i ∈ R M .
Revised: X i 2 R d ⇥ d ⇥ c and y i 2 R M .

Original: To extend this procedure to the multi-class setting, we add noise of various frequencies to an
Revised: To extend this procedure to the multi-class setting, we add noise of various frequencies to an

Original: M -dimensional label vector via label smoothing, originally introduced as a regularization approach in Szegedy et al.
Revised: M -dimensional label vector via label smoothing, originally introduced as a regularization approach in Szegedy et al.

Original: Intuitively, label smoothing removes some probability from the correct class and redistributes it equally among the remaining classes.
Revised: Intuitively, label smoothing removes some probability from the correct class and redistributes it equally among the remaining classes.

Original: Let S : R d × d × c → [0 , 1] be a noise function that maps an input image
Revised: Let S : R d ⇥ d ⇥ c ! [0 , 1] be a noise function that maps an input image

Original: X
Revised: X

Original: i to a scalar value between 0 and 1.
Revised: i to a scalar value between 0 and 1.

Original: We apply label smoothing to each label y i , mapping it to ¯ y i
Revised: We apply label smoothing to each label y i , mapping it to ¯ y i

Original: =
Revised: =

Original: y i (1 − S ( X i ))
Revised: y i (1  S ( X i ))

Original: + 1 M S
Revised: + 1 M S

Original: ( X i ) to retain a valid probability distribution, as in
Revised: ( X i ) to retain a valid probability distribution, as in

Original: Szegedy et al.
Revised: Szegedy et al.

Original: [39] but with different functions S .
Revised: [39] but with different functions S .

Original: Rather than using label smoothing as a regularizer to improve performance, we use it as a form of variable frequency label noise to study spectral bias.
Revised: Rather than using label smoothing as a regularizer to improve performance, we use it as a form of variable frequency label noise to study spectral bias.

Original: We train from scratch using the original examples X i and their smoothed labels ¯ y i .
Revised: We train from scratch using the original examples X i and their smoothed labels ¯ y i .

Original: We evaluate on the validation images X j , comparing to both their original one-hot labels y j and smoothed labels ¯ y j = y j (1 − S ( X j ))
Revised: We evaluate on the validation images X j , comparing to both their original one-hot labels y j and smoothed labels ¯ y j = y j (1  S ( X j ))

Original: + 1 M S ( X j ) .
Revised: + 1 M S ( X j ) .

Original: Our label smoothing experiments use the CIFAR-10 dataset [23], where n train = 50000 , n val =
Revised: Our label smoothing experiments use the CIFAR-10 dataset [23], where n train = 50000 , n val =

Original: 10000 , d = 32 , c
Revised: 10000 , d = 32 , c

Original: = 3 , and M = 10 .
Revised: = 3 , and M = 10 .

Original: This method is introduced in Figure 1.
Revised: This method is introduced in Figure 1.

Original: As training proceeds, we see both the training loss (between predictions and smoothed labels ¯ y i ) and noisy validation loss (between predictions and smoothed labels ¯ y j ) decrease.
Revised: As training proceeds, we see both the training loss (between predictions and smoothed labels ¯ y i ) and noisy validation loss (between predictions and smoothed labels ¯ y j ) decrease.

Original: The clean validation loss (between predictions and one-hot labels y j ), however, initially decreases but at some point in training plateaus or begins to increase.
Revised: The clean validation loss (between predictions and one-hot labels y j ), however, initially decreases but at some point in training plateaus or begins to increase.

Original: At this point, the model has begun to learn the noise function S .
Revised: At this point, the model has begun to learn the noise function S .

Original: Accordingly, we introduce noise ﬁtting , deﬁned as the difference between the validation loss on one-hot labels and the validation loss on labels smoothed with the same function S that was applied to the training labels: noise ﬁtting = clean validation loss − noisy validation loss .
Revised: Accordingly, we introduce noise ﬁtting , deﬁned as the difference between the validation loss on one-hot labels and the validation loss on labels smoothed with the same function S that was applied to the training labels: noise ﬁtting = clean validation loss  noisy validation loss .

Original: We compare different models by computing the minimum noise ﬁtting achieved throughout training.
Revised: We compare different models by computing the minimum noise ﬁtting achieved throughout training.

Original: A model with higher min noise ﬁtting more readily ﬁts the noise function S than a model with lower min noise ﬁtting.
Revised: A model with higher min noise ﬁtting more readily ﬁts the noise function S than a model with lower min noise ﬁtting.

Original: By choosing different functions S , we can probe nuances of spectral bias.
Revised: By choosing different functions S , we can probe nuances of spectral bias.

Original: Typically (inspired by
Revised: Typically (inspired by

Original: Rahaman et al.
Revised: Rahaman et al.

Original: [35]) we choose a radial wave: S ( X ) =[-α (1 + sin(
-]
Revised: [35]) we choose a radial wave: S ( X ) =[-α (1 + sin(
-]

Original: πf ( (cid:107) X (cid:107) − E D (cid:107) X (cid:107) ))) , where (cid:107) X (cid:107) denotes the Euclidean norm of the vectorized image, and α ∈ [0 , 0 . 5] to ensure that S ( X ) ∈ [0 , 1] .
Revised: ↵ (1 + sin(2 ⇡ f ( k X k  E D k X k ))) , where k X k denotes the Euclidean norm of the vectorized image, and ↵ 2 [0 , 0 . 5] to ensure that S ( X ) 2 [0 , 1] .

Original: We can vary the frequency f to understand spectral bias at this global scale.
Revised: We can vary the frequency f to understand spectral bias at this global scale.

Original: We can also choose more targeted functions S ; for instance, if V is a direction of interest through the space of images ( i.e. V is an image-shaped vector of unit norm), we can construct S ( X ) =
Revised: We can also choose more targeted functions S ; for instance, if V is a direction of interest through the space of images ( i.e. V is an image-shaped vector of unit norm), we can construct S ( X ) =

Original: α (1 + [-sin(
 πf (cid:104)-] X , V (cid:105) )) .
Revised: ↵ (1 + [-sin(
 πf (cid:104)-]sin(2 ⇡ f h X , V i )) .

Original: This allows us to vary both the frequency and direction of the noise function, to understand models’ learnability along different directions through image space.
Revised: This allows us to vary both the frequency and direction of the noise function, to understand models’ learnability along different directions through image space.

Original: However, label smoothing has a few limitations.
Revised: However, label smoothing has a few limitations.

Original: Because it involves perturbing the training labels, it cannot directly study the interaction between the training dataset and spectral bias, such as the effects of data augmentation.
Revised: Because it involves perturbing the training labels, it cannot directly study the interaction between the training dataset and spectral bias, such as the effects of data augmentation.

Original: It also requires retraining each model from scratch, with substantial investment of time and computational resources.
Revised: It also requires retraining each model from scratch, with substantial investment of time and computational resources.

Original: We address these limitations with our linear interpolation technique in the next section.
Revised: We address these limitations with our linear interpolation technique in the next section.

Original: 3.2 Linear Interpolation
Revised: 3.2 Linear Interpolation

Original: To complement the label smoothing approach, we also propose a methodology based on linear interpolation between validation images.
Revised: To complement the label smoothing approach, we also propose a methodology based on linear interpolation between validation images.

Original: Although we cannot take dense, regularly-spaced samples that cover the entire input (image) space, we can sample along speciﬁc paths and thereby glean glimpses into the spectral content of the learned function.
Revised: Although we cannot take dense, regularly-spaced samples that cover the entire input (image) space, we can sample along speciﬁc paths and thereby glean glimpses into the spectral content of the learned function.

Original: We consider two types of paths: those between images from the same class, and those between images from different classes.
Revised: We consider two types of paths: those between images from the same class, and those between images from different classes.

Original: When considering paths between CIFAR-10 images in the same class, we choose 200 random, distinct pairs of validation images from each of the 10 CIFAR-10 classes and average over the paths deﬁned by these pairs.
Revised: When considering paths between CIFAR-10 images in the same class, we choose 200 random, distinct pairs of validation images from each of the 10 CIFAR-10 classes and average over the paths deﬁned by these pairs.

Original: We do the same for ImageNet, choosing 50 random, distinct paths for each of the 1000 ImageNet classes.
Revised: We do the same for ImageNet, choosing 50 random, distinct paths for each of the 1000 ImageNet classes.

Original: When interpolating between CIFAR-10 classes, we choose 200 random, distinct validation image pairs from each of the 45 (10 choose 2) pairs of distinct classes, and average over the resulting paths; for ImageNet we choose 1000 random, distinct between-class paths, ensuring that each class is represented in exactly two paths.
Revised: When interpolating between CIFAR-10 classes, we choose 200 random, distinct validation image pairs from each of the 45 (10 choose 2) pairs of distinct classes, and average over the resulting paths; for ImageNet we choose 1000 random, distinct between-class paths, ensuring that each class is represented in exactly two paths.

Original: For each pair of images ( X 0 , X 1 ) , we vary λ ∈ [0 , 1] to trace out a path, where each image in the path is given by X λ = λ X [-
-] + (1 − λ ) 
Revised: For each pair of images ( X 0 , X 1 ) , we vary  2 [0 , 1] to trace out a path, where each image in the path is given by X  =  X [-
-]1 + (1   ) X 0 .

Original: X 0 . We choose λ s so that the distance between adjacent images on the path is constant; each path may produce a different number of samples depending on the total distance between X 0 and X 1 .
Revised: We choose  s so that the distance between adjacent images on the path is constant; each path may produce a different number of samples depending on the total distance between X 0 and X 1 .

Original: Figure 2 (Bottom Left) shows an illustrative example of two CIFARimages and a corresponding interpolation path between them.
Revised: Figure 2 (Bottom Left) shows an illustrative example of two CIFARimages and a corresponding interpolation path between them.

Original: For our experiments, we typically have 50 to 100 images along the interpolation path.
Revised: For our experiments, we typically have 50 to 100 images along the interpolation path.

Original: Relationship between interpolation and label smoothing experiments.
Revised: Relationship between interpolation and label smoothing experiments.

Original: To validate our interpolation methodology, we consider the oracle function that maps directly from the interpolated image X λ to the noised label used in our label smoothing experiments.
Revised: To validate our interpolation methodology, we consider the oracle function that maps directly from the interpolated image X  to the noised label used in our label smoothing experiments.

Original: We only consider within-class paths for this experiment, so that the original one-hot label (before smoothing) is ﬁxed across the entire path, and the only change in the oracle function along the path is due to the noise function used in label smoothing.
Revised: We only consider within-class paths for this experiment, so that the original one-hot label (before smoothing) is ﬁxed across the entire path, and the only change in the oracle function along the path is due to the noise function used in label smoothing.

Original: By increasing the frequency of the noise function, we create oracle functions with higher frequencies.
Revised: By increasing the frequency of the noise function, we create oracle functions with higher frequencies.

Original: We then verify that our interpolation measurement indeed measures higher frequencies on the higher frequency oracle functions.
Revised: We then verify that our interpolation measurement indeed measures higher frequencies on the higher frequency oracle functions.

Original: In Figure 2 (Left) , we show an example where we vary the label smoothing frequency from f =to f = 0 .
Revised: In Figure 2 (Left) , we show an example where we vary the label smoothing frequency from f =to f = 0 .

Original: We plot the value of λ along the interpolating path on the x axis, and the norm of the difference between the model output (softmax probabilities) on the interpolated image ˆ y λ and the model output on the original image ˆ y 0 on the y axis.
Revised: We plot the value of  along the interpolating path on the x axis, and the norm of the difference between the model output (softmax probabilities) on the interpolated image ˆ y  and the model output on the original image ˆ y 0 on the y axis.

Original: This difference norm always starts at zero (although we plot it by averaging many paths over buckets in λ , so the curves do not actually touch the axis), and tends to increase with λ as the model predictions change along the interpolating path.
Revised: This difference norm always starts at zero (although we plot it by averaging many paths over buckets in  , so the curves do not actually touch the axis), and tends to increase with  as the model predictions change along the interpolating path.

Original: The norm of the prediction difference allows us to visualize how smooth the function is along the interpolating path; smoother prediction difference norm indicates a lower frequency function.
Revised: The norm of the prediction difference allows us to visualize how smooth the function is along the interpolating path; smoother prediction difference norm indicates a lower frequency function.

Original: To quantify the frequency of the oracle function along the interpolating path, we compute the per-class discrete Fourier transform (DFT).
Revised: To quantify the frequency of the oracle function along the interpolating path, we compute the per-class discrete Fourier transform (DFT).

Original: Letting { X λ t } Tt =1 denote the set of interpolated images, the DFT coefﬁcient for frequency f is given by ˆ Y f 
Revised: Letting { X  t } Tt =1 denote the set of interpolated images, the DFT coefﬁcient for frequency f is given by ˆ Y f [ m ] = P

Original:  [ m ]= (cid:80) Tt
Revised: Tt [Equation] [ m ]

Original: =1 ˆ y λ t [ m ] exp( − i 2 πft ) , where i is the imaginary number and m ∈ { 1 , . ..
Revised: [Equation] [Equation] ft ) , where i is the imaginary number and m 2 { 1 , . .

Original: , M } denotes the class.
Revised: , M } denotes the class.

Original: To summarize how the magnitudes of the DFT coefﬁcients ˆ Y f are distributed between low and high frequencies, we average the coefﬁcient magnitudes for each of the classes and compute the fraction of the total average coefﬁcient magnitude that is allocated to frequencies above a threshold (0.05).
Revised: To summarize how the magnitudes of the DFT coefﬁcients ˆ Y f are distributed between low and high frequencies, we average the coefﬁcient magnitudes for each of the classes and compute the fraction of the total average coefﬁcient magnitude that is allocated to frequencies above a threshold (0.05).

Original: In Figure 2 (Right) we show this summary metric averaged over all within-class interpolating paths for the oracle functions, along with its standard error.
Revised: In Figure 2 (Right) we show this summary metric averaged over all within-class interpolating paths for the oracle functions, along with its standard error.

Original: In Figure 2 (Left) , the higher frequency oracle functions have less smooth prediction norm differences, and in Figure 2 (Right) , they have a higher fraction of DFT coefﬁcient magnitudes in the high frequencies.
Revised: In Figure 2 (Left) , the higher frequency oracle functions have less smooth prediction norm differences, and in Figure 2 (Right) , they have a higher fraction of DFT coefﬁcient magnitudes in the high frequencies.

Original: 3.3 Data and Models
Revised: 3.3 Data and Models

Original: We use the CIFAR-10 [23] dataset of low-resolution (32 × 32) natural images from ten animal and object classes and the ImageNet [8] dataset of higher-resolution (224 × 224) natural images from 1000 classes.
Revised: We use the CIFAR-10 [23] dataset of low-resolution (32 ⇥ 32) natural images from ten animal and object classes and the ImageNet [8] dataset of higher-resolution (224 ⇥ 224) natural images from 1000 classes.

Original: We consider a diversity of model architectures on CIFAR-10 (6 total) and ImageNet ( total).
Revised: We consider a diversity of model architectures on CIFAR-10 (6 total) and ImageNet ( total).

Original: See Section A.1 for more details on the data processing, models, and their accuracies.
Revised: See Section A.1 for more details on the data processing, models, and their accuracies.

Original:  Results on CIFAR- 
Revised:  Results on CIFAR-

Original: Throughout this section we show representative examples from our CIFAR-10 experiments; full results (on all six models we tested) are included in Section A.3.
Revised: Throughout this section we show representative examples from our CIFAR-10 experiments; full results (on all six models we tested) are included in Section A.3.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4.1 Spectral Bias
Revised: 4.1 Spectral Bias

Original: We begin by applying our label smoothing methodology to ﬁnd that modern image classiﬁcation
Revised: We begin by applying our label smoothing methodology to ﬁnd that modern image classiﬁcation

Original: CNNs exhibit spectral bias, learning low frequency functions early in training and learning higher frequency functions as training proceeds.
Revised: CNNs exhibit spectral bias, learning low frequency functions early in training and learning higher frequency functions as training proceeds.

Original: Note that Figure 3 (Left) shows spectral bias over a small but illustrative range of frequencies (the full range of frequencies we tested was 0 to 0.1); noise functions of sufﬁciently low frequency are ﬁt almost immediately and noise functions of sufﬁciently high frequency are never learned during the 250 epochs of training we tested.
Revised: Note that Figure 3 (Left) shows spectral bias over a small but illustrative range of frequencies (the full range of frequencies we tested was 0 to 0.1); noise functions of sufﬁciently low frequency are ﬁt almost immediately and noise functions of sufﬁciently high frequency are never learned during the 250 epochs of training we tested.

Original: Using our linear interpolation methodology and distinguishing between within-class paths (in which both endpoint images are of the same class) and between-class paths (where the path must traverse a class boundary) in Figure 3 (Right) we ﬁnd that this spectral bias is highly variable over the domain of images.
Revised: Using our linear interpolation methodology and distinguishing between within-class paths (in which both endpoint images are of the same class) and between-class paths (where the path must traverse a class boundary) in Figure 3 (Right) we ﬁnd that this spectral bias is highly variable over the domain of images.

Original: In particular, while the learned function does increase in frequency for between-class paths, it decreases in frequency for within-class paths, indicating that the model is learning to cluster and separate the different classes.
Revised: In particular, while the learned function does increase in frequency for between-class paths, it decreases in frequency for within-class paths, indicating that the model is learning to cluster and separate the different classes.

Original: Throughout our CIFAR-10 experiments, we ﬁnd that higher-accuracy models have greater separation between the frequencies of their within-class and between-class paths, with lower-frequency behavior inside each class and higher-frequency behavior in between.
Revised: Throughout our CIFAR-10 experiments, we ﬁnd that higher-accuracy models have greater separation between the frequencies of their within-class and between-class paths, with lower-frequency behavior inside each class and higher-frequency behavior in between.

Original: 4.2 Spectral Bias and Model Architecture
Revised: 4.2 Spectral Bias and Model Architecture

Original: Although all models we tested exhibit spectral bias, we found that the precise nature of the bias depends on the choice of model.
Revised: Although all models we tested exhibit spectral bias, we found that the precise nature of the bias depends on the choice of model.

Original: For example, with all else ﬁxed, increasing the width of a model decreases its spectral bias, enabling it to more readily ﬁt noise functions of a given frequency.
Revised: For example, with all else ﬁxed, increasing the width of a model decreases its spectral bias, enabling it to more readily ﬁt noise functions of a given frequency.

Original: This trend is evident in Figure 4 for the Wide-ResNet family (Left) and the Shake-Shake family (Center) .
Revised: This trend is evident in Figure 4 for the Wide-ResNet family (Left) and the Shake-Shake family (Center) .

Original: Although larger models learn high frequencies faster, we found that higher-accuracy models are not uniformly higher-frequency.
Revised: Although larger models learn high frequencies faster, we found that higher-accuracy models are not uniformly higher-frequency.

Original: Instead, using linear interpolation in Figure 4 (Right) we ﬁnd that higher- accuracy models (ordered left to right by increasing test accuracy) are higher-frequency along betweenclass paths but lower-frequency along within-class paths, perhaps evident of a better class clustering.
Revised: Instead, using linear interpolation in Figure 4 (Right) we ﬁnd that higher- accuracy models (ordered left to right by increasing test accuracy) are higher-frequency along betweenclass paths but lower-frequency along within-class paths, perhaps evident of a better class clustering.

Original: 4.3 Sensitivity to Natural Image Directions
Revised: 4.3 Sensitivity to Natural Image Directions

Original: Label smoothing with a radial wave offers a convenient global picture of spectral bias, by replacing the radial wave with other noise functions we can use the same methodology to test more localized aspects of spectral bias.
Revised: Label smoothing with a radial wave offers a convenient global picture of spectral bias, by replacing the radial wave with other noise functions we can use the same methodology to test more localized aspects of spectral bias.

Original: We take a step in this direction by considering the family of noise functions
Revised: We take a step in this direction by considering the family of noise functions

Original: S ( X ; k ) =
Revised: S ( X ; k ) =

Original: α (1 + sin(2 πf (cid:104) X , F k (cid:105) )) , where F k is a diagonal Fourier basis image with frequency k and the same dimensions as X , visualized in Figure 5 (Bottom) .
Revised: ↵ (1 + sin(2 ⇡ f h X , F k i )) , where F k is a diagonal Fourier basis image with frequency k and the same dimensions as X , visualized in Figure 5 (Bottom) .

Original: We consider (a subset of)
Revised: We consider (a subset of)

Original: Fourier basis images because the Fourier spectra statistics of natural images are well studied (see, e.g. Tolhurst et al.
Revised: Fourier basis images because the Fourier spectra statistics of natural images are well studied (see, e.g. Tolhurst et al.

Original: [41]): natural images tend to be composed of Fourier basis images with amplitude proportional to their inverse spatial frequency.
Revised: [41]): natural images tend to be composed of Fourier basis images with amplitude proportional to their inverse spatial frequency.

Original: Indeed, in Figure 5 we ﬁnd that
Revised: Indeed, in Figure 5 we ﬁnd that

Original: ShakeShake96 is more sensitive to label smoothing in low image frequency directions.
Revised: ShakeShake96 is more sensitive to label smoothing in low image frequency directions.

Original: This ﬁnding is consistent with theoretical predictions of Basri et al.
Revised: This ﬁnding is consistent with theoretical predictions of Basri et al.

Original: [2] that models learn faster in regions of higher density of training examples: since low image frequencies are more common in natural images, the effective sampling density enjoyed by a noise function is higher in these directions.
Revised: [2] that models learn faster in regions of higher density of training examples: since low image frequencies are more common in natural images, the effective sampling density enjoyed by a noise function is higher in these directions.

Original: It is also possible that this bias is a byproduct of the lower Fourier magnitudes of high frequency components in natural images [36], or is inherent to the convolutional model architecture regardless of the data distribution [33], or that a combination of multiple effects is at play.
Revised: It is also possible that this bias is a byproduct of the lower Fourier magnitudes of high frequency components in natural images [36], or is inherent to the convolutional model architecture regardless of the data distribution [33], or that a combination of multiple effects is at play.

Original: Determining the precise cause of this image frequency ﬁnding is an interesting direction for further study.
Revised: Determining the precise cause of this image frequency ﬁnding is an interesting direction for further study.

Original: 4.4 Spectral Bias and Training Data
Revised: 4.4 Spectral Bias and Training Data

Original: In our CIFAR-10 interpolation experiments, we consistently observed that variations in training that improve accuracy create greater disparity between the frequencies of within-class and between-class paths, with lower frequencies within-class and higher frequencies between-class.
Revised: In our CIFAR-10 interpolation experiments, we consistently observed that variations in training that improve accuracy create greater disparity between the frequencies of within-class and between-class paths, with lower frequencies within-class and higher frequencies between-class.

Original: This trend is evident in Figure 3 (Right) as a model trains for more epochs and in Figure 4 (Right) as the model architecture improves.
Revised: This trend is evident in Figure 3 (Right) as a model trains for more epochs and in Figure 4 (Right) as the model architecture improves.

Original: In Figure 6 we ﬁnd that the same trend also holds as we train on more data (Left) , apply more effective data augmentation (Center) , or apply self-distillation (Right) .
Revised: In Figure 6 we ﬁnd that the same trend also holds as we train on more data (Left) , apply more effective data augmentation (Center) , or apply self-distillation (Right) .

Original: Training Set Size.
Revised: Training Set Size.

Original: As shown in Figure 6 (Left) on WideResNet160 , we ﬁnd that increasing the number of training examples increases the difference in frequency content between within-class interpolating paths and between-class interpolating paths.
Revised: As shown in Figure 6 (Left) on WideResNet160 , we ﬁnd that increasing the number of training examples increases the difference in frequency content between within-class interpolating paths and between-class interpolating paths.

Original: Training with more data makes our CIFAR-10 models behave lower-frequency within each class but higher-frequency between classes, perhaps indicating that the models are learning a better clustering of the classes as they are provided with additional data.
Revised: Training with more data makes our CIFAR-10 models behave lower-frequency within each class but higher-frequency between classes, perhaps indicating that the models are learning a better clustering of the classes as they are provided with additional data.

Original: Data Augmentation.
Revised: Data Augmentation.

Original: Data augmentation is a common strategy to increase the effective training set size without the expense of actually collecting additional examples.
Revised: Data augmentation is a common strategy to increase the effective training set size without the expense of actually collecting additional examples.

Original: In Figure 6 (Center) we consider the effects of Mixup (with strength 0.1) [49], AutoAugment [6], and RandAugment [5], each of which improves the ﬁnal accuracy of the trained model (with RandAugment producing the most substantial beneﬁts in our experiments, followed by AutoAugment and then Mixup).
Revised: In Figure 6 (Center) we consider the effects of Mixup (with strength 0.1) [49], AutoAugment [6], and RandAugment [5], each of which improves the ﬁnal accuracy of the trained model (with RandAugment producing the most substantial beneﬁts in our experiments, followed by AutoAugment and then Mixup).

Original: RandAugment and AutoAugment generate new images by applying geometric and lighting transformations, whereas
Revised: RandAugment and AutoAugment generate new images by applying geometric and lighting transformations, whereas

Original: Mixup linearly interpolates between pairs of existing images.
Revised: Mixup linearly interpolates between pairs of existing images.

Original: Much like our experiments with training set size, we ﬁnd that augmentations that improve accuracy more also produce greater frequency separation between within-class paths and between-class paths.
Revised: Much like our experiments with training set size, we ﬁnd that augmentations that improve accuracy more also produce greater frequency separation between within-class paths and between-class paths.

Original: Mixup Strength.
Revised: Mixup Strength.

Original: Mixup augmentation [49] perturbs each batch of training data by randomly pairing the examples and perturbing each example towards its partner by an interpolation amount λ drawn from a symmetric beta distribution (with the same λ used for all examples in the batch).
Revised: Mixup augmentation [49] perturbs each batch of training data by randomly pairing the examples and perturbing each example towards its partner by an interpolation amount  drawn from a symmetric beta distribution (with the same  used for all examples in the batch).

Original: We refer to the parameter of this beta distribution as the Mixup strength, as it controls the degree to which the augmented images tend to lie close to an original training image or close to the average of two training images.
Revised: We refer to the parameter of this beta distribution as the Mixup strength, as it controls the degree to which the augmented images tend to lie close to an original training image or close to the average of two training images.

Original: A parameter of 0 corresponds to no augmentation ( λ = 0 or λ = 1 , always using the original images), a parameter of 1 corresponds to the uniform distribution over λ ∈ [0 , 1] , and a parameter of ∞ corresponds to λ = 0 .
Revised: A parameter of 0 corresponds to no augmentation (  = 0 or  = 1 , always using the original images), a parameter of 1 corresponds to the uniform distribution over  2 [0 , 1] , and a parameter of 1 corresponds to  = 0 .

Original: 5 , the exact midpoint between a pair of training images.
Revised: 5 , the exact midpoint between a pair of training images.

Original: Figure 6 (Center) compares a range of Mixup strengths and ﬁnds that relatively low values that only slightly perturb the original images tend to produce greater frequency separation (and higher test accuracy), whereas perturbing the images too much with stronger Mixup has the opposite effect on both frequency separation and accuracy.
Revised: Figure 6 (Center) compares a range of Mixup strengths and ﬁnds that relatively low values that only slightly perturb the original images tend to produce greater frequency separation (and higher test accuracy), whereas perturbing the images too much with stronger Mixup has the opposite effect on both frequency separation and accuracy.

Original: Of note, the range of Mixup strengths that increase or do not affect frequency separation in our experiments are well aligned with those recommended by the original authors [49]: 0.1 to 0.4.
Revised: Of note, the range of Mixup strengths that increase or do not affect frequency separation in our experiments are well aligned with those recommended by the original authors [49]: 0.1 to 0.4.

Original: Self-Distillation.
Revised: Self-Distillation.

Original: Finally, we consider the effect of self-distillation on the frequency content of the learned function.
Revised: Finally, we consider the effect of self-distillation on the frequency content of the learned function.

Original: In self-distillation, a “teacher” model is trained with some form of strong regularization, in our case a combination of weight decay and early stopping based on training loss.
Revised: In self-distillation, a “teacher” model is trained with some form of strong regularization, in our case a combination of weight decay and early stopping based on training loss.

Original: A “student” model with the same architecture is then trained from scratch to ﬁt the pseudolabels produced by the teacher Prior research [11] found that this procedure can train student models that outperform both their teachers and a baseline model trained as normal.
Revised: A “student” model with the same architecture is then trained from scratch to ﬁt the pseudolabels produced by the teacher Prior research [11] found that this procedure can train student models that outperform both their teachers and a baseline model trained as normal.

Original: Prior research has also sought to understand the mechanism behind self-distillation.
Revised: Prior research has also sought to understand the mechanism behind self-distillation.

Original: Mobahi et al.
Revised: Mobahi et al.

Original: [28] ﬁnds that self-distillation acts as a regularizer by limiting the basis functions available to the student to learn.
Revised: [28] ﬁnds that self-distillation acts as a regularizer by limiting the basis functions available to the student to learn.

Original: We complement their theoretical work with our interpolation experimental methodology in
Revised: We complement their theoretical work with our interpolation experimental methodology in

Original: Figure 6 (Right) , where indeed we ﬁnd that the student model learns a greater frequency separation between within-class and between-class paths than its teacher, while also achieving higher test accuracy.
Revised: Figure 6 (Right) , where indeed we ﬁnd that the student model learns a greater frequency separation between within-class and between-class paths than its teacher, while also achieving higher test accuracy.

Original: We conjecture that this effect is analogous to low-pass preﬁltering common in digital signal processing: a high-frequency noise function that we cannot adequately sample is ﬁrst smoothed (in this case by being approximated by a regularized teacher) and then it can be modeled via samples without further loss in ﬁdelity.
Revised: We conjecture that this effect is analogous to low-pass preﬁltering common in digital signal processing: a high-frequency noise function that we cannot adequately sample is ﬁrst smoothed (in this case by being approximated by a regularized teacher) and then it can be modeled via samples without further loss in ﬁdelity.

Original: Without this preﬁltering, our samples are inadequate to capture the complexity of the noise function, so we reconstruct an imperfect version corrupted by aliasing.
Revised: Without this preﬁltering, our samples are inadequate to capture the complexity of the noise function, so we reconstruct an imperfect version corrupted by aliasing.

Original:  Results on ImageNet
Revised:  Results on ImageNet

Original: Finally, we apply our linear interpolation methodology to study the frequency content of a range of pretrained models on ImageNet [8].
Revised: Finally, we apply our linear interpolation methodology to study the frequency content of a range of pretrained models on ImageNet [8].

Original: Here we show results on ResNet50 ; results on all ten models we tested on ImageNet are included in Section A.4.
Revised: Here we show results on ResNet50 ; results on all ten models we tested on ImageNet are included in Section A.4.

Original: 5.1 Training Time
Revised: 5.1 Training Time

Original: As we observed for CIFAR-10, our ImageNet models tend to increase the frequency gap between the two types of interpolating paths as training proceeds, with lower-frequencies within-class and higherfrequencies between-class.
Revised: As we observed for CIFAR-10, our ImageNet models tend to increase the frequency gap between the two types of interpolating paths as training proceeds, with lower-frequencies within-class and higherfrequencies between-class.

Original: We visualize this trend for ResNet50 in Figure 7 (Left) .
Revised: We visualize this trend for ResNet50 in Figure 7 (Left) .

Original: For our CIFARmodels, this increase in frequency difference between path types was achieved by a combination of increasing the average high frequency content on between-class paths and decreasing the average high frequency content on within-class paths.
Revised: For our CIFARmodels, this increase in frequency difference between path types was achieved by a combination of increasing the average high frequency content on between-class paths and decreasing the average high frequency content on within-class paths.

Original: On ImageNet, we again ﬁnd that average frequency content on between-class paths increases during training, but that within-class frequencies sometimes decrease during training ( e.g. for ResNets) but sometimes increase (slower than between-class paths) during training ( e.g. for CoATNets).
Revised: On ImageNet, we again ﬁnd that average frequency content on between-class paths increases during training, but that within-class frequencies sometimes decrease during training ( e.g. for ResNets) but sometimes increase (slower than between-class paths) during training ( e.g. for CoATNets).

Original: We speculate that the trend of decreasing within-class frequency as training time progresses is less consistent on ImageNet than CIFAR-10 because, as we explore in the next section, ImageNet classes are more internally diverse than CIFAR-10 classes.
Revised: We speculate that the trend of decreasing within-class frequency as training time progresses is less consistent on ImageNet than CIFAR-10 because, as we explore in the next section, ImageNet classes are more internally diverse than CIFAR-10 classes.

Original: 5.2 Class Coherence
Revised: 5.2 Class Coherence

Original: In addition to containing 100 × as many classes as CIFAR-10, ImageNet classes can also be more internally diverse.
Revised: In addition to containing 100 ⇥ as many classes as CIFAR-10, ImageNet classes can also be more internally diverse.

Original: This notion of within-class “consistency” is captured by C-Scores [18], which assign to each training example a value between 0 and 1 denoting how typical that example is.
Revised: This notion of within-class “consistency” is captured by C-Scores [18], which assign to each training example a value between 0 and 1 denoting how typical that example is.

Original: One way of capturing the internal consistency or diversity of a class is by studying the distribution of its
Revised: One way of capturing the internal consistency or diversity of a class is by studying the distribution of its

Original: C-Scores.
Revised: C-Scores.

Original: In Figure 7 (Right) we plot the mean and standard deviation of these scores for each of the 1000 ImageNet classes in gray and highlight the 10 classes whose ResNet50 interpolating paths between validation images are most high-frequency (red stars) and least high-frequency (blue squares).
Revised: In Figure 7 (Right) we plot the mean and standard deviation of these scores for each of the 1000 ImageNet classes in gray and highlight the 10 classes whose ResNet50 interpolating paths between validation images are most high-frequency (red stars) and least high-frequency (blue squares).

Original: We ﬁnd that ResNet50 is lower-frequency on more internally consistent classes and higher-frequency on more diverse classes, supporting the intuition that more diverse classes require more complex functions.
Revised: We ﬁnd that ResNet50 is lower-frequency on more internally consistent classes and higher-frequency on more diverse classes, supporting the intuition that more diverse classes require more complex functions.

Original: Figure
Revised: Figure

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we introduced two methods to measure spectral bias in modern image classiﬁcation neural networks, and applied these methods towards the central question:
Revised: In this paper, we introduced two methods to measure spectral bias in modern image classiﬁcation neural networks, and applied these methods towards the central question:

Original: What kinds of function frequencies are needed for modern neural networks to generalize?
Revised: What kinds of function frequencies are needed for modern neural networks to generalize?

Original: Speciﬁcally, we applied these methods to examine the impact of a variety of training choices on the learned frequencies.
Revised: Speciﬁcally, we applied these methods to examine the impact of a variety of training choices on the learned frequencies.

Original: On CIFAR-10, we found that higher-accuracy models typically have greater frequency separation between within-class and between-class paths, with lower frequencies withinclass and higher frequencies between-class.
Revised: On CIFAR-10, we found that higher-accuracy models typically have greater frequency separation between within-class and between-class paths, with lower frequencies withinclass and higher frequencies between-class.

Original: This trend holds regardless of whether high accuracy is achieved through longer training, choice of model architecture, increase in dataset size, data augmentation, or self-distillation.
Revised: This trend holds regardless of whether high accuracy is achieved through longer training, choice of model architecture, increase in dataset size, data augmentation, or self-distillation.

Original: On ImageNet, we again found that this frequency separation increases during training, and noted that models are higher-frequency within more internally diverse classes.
Revised: On ImageNet, we again found that this frequency separation increases during training, and noted that models are higher-frequency within more internally diverse classes.

Original: Our experimental methods offer a window onto the frequency structure of neural networks for image classiﬁcation, and its relationship to model performance.
Revised: Our experimental methods offer a window onto the frequency structure of neural networks for image classiﬁcation, and its relationship to model performance.

Original: In particular, although our primary goal is to improve our understanding of how neural networks generalize, our work is likely to have practical implications for improving generalization as well as robustness to image perturbations and distribution shifts.
Revised: In particular, although our primary goal is to improve our understanding of how neural networks generalize, our work is likely to have practical implications for improving generalization as well as robustness to image perturbations and distribution shifts.

Original: We hope that future work realizes these implications by applying our methods to study the spectral bias of robust models, and to induce model robustness directly, perhaps by incorporating our metrics as regularizers during training.
Revised: We hope that future work realizes these implications by applying our methods to study the spectral bias of robust models, and to induce model robustness directly, perhaps by incorporating our metrics as regularizers during training.

Original: Future work may also beneﬁt from extending our methodology to interpolating paths that adhere more closely to the hypothesized manifold of natural images, such as those produced by generative networks [21, 40].
Revised: Future work may also beneﬁt from extending our methodology to interpolating paths that adhere more closely to the hypothesized manifold of natural images, such as those produced by generative networks [21, 40].

Original: Reproducibility Statement
Revised: Reproducibility Statement

Original: Our
Revised: Our

Original: CIFARcode is available at https://github.com/google-research/ google-research/tree/master/
Revised: CIFARcode is available at https://github.com/google-research/ google-research/tree/master/

Original: spectral_bias ; for ImageNet we apply the same interpolation method to pre-cropped images and pretrained model checkpoints.
Revised: spectral_bias ; for ImageNet we apply the same interpolation method to pre-cropped images and pretrained model checkpoints.

Original: Our work uses the publicly-available CIFAR-10 [23] and ImageNet [8] datasets.
Revised: Our work uses the publicly-available CIFAR-10 [23] and ImageNet [8] datasets.

Original: Acknowledgments
Revised: Acknowledgments

Original: Many thanks to Yasaman Bahri, Ekin Dogus Cubuk, Hossein Mobahi, Michael Mozer, Maithra
Revised: Many thanks to Yasaman Bahri, Ekin Dogus Cubuk, Hossein Mobahi, Michael Mozer, Maithra

Original: Raghu, Samuel Schoenholz, Jonathon Shlens, and Piotr Teterwak for productive conversations and helpful pointers.
Revised: Raghu, Samuel Schoenholz, Jonathon Shlens, and Piotr Teterwak for productive conversations and helpful pointers.

Original: SFK is also funded by NSF GRFP.
Revised: SFK is also funded by NSF GRFP.

Original: A Appendix A.1 Data, Models, and Model Accuracies

Original: Images are pixel-wise normalized by the mean and standard deviation of the training images for each dataset, and for ImageNet all images are center cropped and resized to 224 × 224; this preprocessing is done before any interpolating paths are constructed.

Original: For CIFAR-10, we consider six different convolutional neural networks: two Wide-ResNets [47] of different sizes ( WideResNet32 and WideResNet160 ), three Shake-Shake regularized networks [12] of different sizes ( ShakeShake32 , ShakeShake96 , and ShakeShake112 ), and PyramidNet , which uses the even stronger Shake-Drop regularization [45].

Original: Our implementations are based on Cubuk et al.

Original: [6]; we use the same optimizer (stochastic gradient descent with momentum) and cosine learning rate schedule.

Original: We train without data augmentation (to ensure all models are trained on exactly the same examples), except for experiments that explicitly vary data augmentation.

Original: Without data augmentation, the test accuracies of our models are shown in

Original: Table 1.

Original: [Table]

Original: For ImageNet, we consider ten different pretrained models: three ResNets [16] of different sizes ( ResNet50 , ResNet101 , and ResNet152 ), two Vision Transformers [10] of different sizes ( ViT and ViTSmall ), three distilled transformers [42] ( DeiT , DeiTSmall , and DeiTClsToken ), and two CoAtNets [7] ( CoAtNet and CoAtNet0BF16 ).

Original: The top-1 classiﬁcation accuracies of these models are presented in Table 2.

Original: [Table]

Original: A.2 Linear Interpolation: Methodological Details

Original: For each sampled path, we compute the discrete Fourier transform (DFT) of the prediction function along the path separately for each of the M class predictions, take the (real) magnitude of the resulting (complex) DFT coefﬁcients, and average them among the M classes.

Original: We then compute the fraction of this averaged DFT magnitude that is allocated to the high frequency components, and aggregate over the many paths by calculating the mean high frequency fraction and its standard error.

Original: We use a simple threshold frequency of 0.05 for both CIFAR10 and ImageNet, although our paths are sampled with a distance of 1 for CIFAR10 and 7 for ImageNet (since ImageNet images are larger and thus farther apart).

Original: This precise threshold is not critical to our qualitative results, and different tradeoffs of frequency resolution and computational expense are possible.

Original: Limitations of linear interpolation.

Original: Like all proxy measurements of spectral bias, linear interpolation is a coarse metric.

Original: We only measure the Fourier content of the learned function along speciﬁc paths, and aggregate results across many paths, across all classes, and only summarize frequencies into coarse "low" and "high" bins.

Original: An additional limitation is that the paths we deﬁne interpolate in pixel space, which means that there are images along our interpolating paths that are off the hypothesized manifold of natural images.

Original: A.3 CIFAR-10 full results A.3.1 Label Smoothing

Original: Figure 8 shows the same results as Figure 3 (Left) on all six CIFAR-10 models we tested.

Original: [Figure]

Original: A.3.2 Sensitivity to Natural Image Directions

Original: Figure 9 shows the same results as Figure 5 on all six CIFAR-10 models we tested.

Original: A.3.3 Agreement Between Label Smoothing and Linear Interpolation

Original: Varying the frequency of the radial wave label smoothing When the frequency of the radial wave used for label smoothing increases, models take more time to ﬁt the smoothing noise.

Original: Figure(Left) and Figure 8 show this using label smoothing; the corresponding interpolation experiment is shown in Figure 10.

Original: The target frequencies between 0.035 and 0.04 are close enough that, were the model to ﬁt each perfectly, the interpolation curves would be nearly visually indistinguishable, as we can tell from Figure 2 (Left) .

Original: However, the model is actually smoother when trying to ﬁt higher frequency label smoothing noise, because it ﬁts this noise less well (in addition to ﬁtting it later in training).

Original: We can see this effect, for instance, by noting that WideResNet32 , WideResNet160 , and

Original: ShakeShake32 fail to ﬁt frequency 0.04, both in Figure 8 and Figure 10.

Original: It is also worth noting that, in Figure 10 and repeatedly across our interpolation experiments, the learned function is smoother (lower-frequency) within-class and less smooth (higher-frequency) between-class.

Original: This is to be expected since the model must change predictions somewhere along the path between examples from different classes.

Original: Learning low frequencies ﬁrst The label smoothing experiment presented in Figure 3 (Left) and

Original: Figure 8 shows that low frequency target functions are learned earlier than higher frequency targets; we can also conﬁrm this ﬁnding using interpolation with model checkpoints saved at different epochs throughout training (without any label smoothing) in Figure 11.

Original: We see that spectral bias in image classiﬁcation is more nuanced than simply increasing in frequency over time: as training proceeds, models become higher-frequency along between-class paths but lower-frequency along within-class

Original: [Figure]

Original: [Figure]

Original: paths.

Original: This may indicate that the model is learning an appropriate clustering of the data into its classes, and reducing the functional variation within each cluster.

Original: A.3.4 Weight Decay

Original: Figure 12 uses the label smoothing methodology on our six CIFAR-10 models to show that weight decay delays the learning of high frequencies.

Original: A.3.5 Training Set Size

Original: Figure 6 (Left) uses linear interpolation to show the within-class regularization (frequency reduction) effect of increasing dataset size for WideResNet160 .

Original: Figure 13 shows the same effect on all six CIFAR10 models we tested.

Original: A.3.6 Data Augmentation

Original: Figure 6 (Center) uses linear interpolation to study the effect of common data augmentation procedures on the learned frequencies for WideResNet160 .

Original: Figure 14 shows the same experiment on all six models we tested.

Original: Mixup Strength Figure 6 (Center) uses linear interpolation to show that training with Mixup data augmentation [49] causes WideResNet160 to learn a within-class lower-frequency function, but too much Mixup can produce a higher-frequency function within-class.

Original: Figure 15 repeats the experiment on all six CIFAR-10 models we tested.

Original: A.3.7 Self-Distillation

Original: Figure 6 (Right) uses linear interpolation to show that self-distillation with ShakeShake96 produces a student model that is lower-frequency within-class (and slightly higher-frequency between-class) than its teacher, and has higher validation accuracy than the teacher.

Original: Figure 16 shows the same result across all six CIFAR-10 models we tested.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: A.4 ImageNet Full Results A.4.1 Training Time

Original: Figure 7 (Left) shows that ResNet50 increases in between-class frequency as training proceeds, while within-class frequencies remain constant or slightly decrease, causing increasing frequency separation between these two types of paths.

Original: In Figure 17 we show the same experiment for all ten ImageNet models we tested.

Original: We ﬁnd that the overall frequency trends vary between models, but all models increase in frequency separation between path types during training.

Original: A.4.2 Class Coherence

Original: Figure 7 (Right) shows that ResNet50 has higher frequencies along paths within more internally diverse classes.

Original: In Figure 18 we show the same experiment for all ten ImageNet models we tested.

Original: Interestingly, the two CoATNets exhibit no clear correlation between class diversity and function frequency, and also dramatically increase in function frequency during training (Figure 17), unlike other models.

Original: [Figure]

Original: [Figure]