Original: Spectral Bias in Practice: the Role of Function Frequency in Generalization
Revised: Spectral Bias in Practice: the Role of Function Frequency in Generalization

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Despite their ability to represent highly expressive functions, deep learning modelsseem to ﬁnd simple solutions that generalize surprisingly well.
Revised: Despite their ability to represent highly expressive functions, deep learning modelsseem to ﬁnd simple solutions that generalize surprisingly well.

Original: Spectral bias –the tendency of neural networks to prioritize learning low frequency functions– is one possible explanation for this phenomenon, but so far spectral bias hasprimarily been observed in theoretical models and simpliﬁed experiments.
Revised: Spectral bias –the tendency of neural networks to prioritize learning low frequency functions– is one possible explanation for this phenomenon, but so far spectral bias hasprimarily been observed in theoretical models and simpliﬁed experiments.

Original: In thiswork, we propose methodologies for measuring spectral bias in modern imageclassiﬁcation networks on CIFAR-10 and ImageNet.
Revised: In thiswork, we propose methodologies for measuring spectral bias in modern imageclassiﬁcation networks on CIFAR-10 and ImageNet.

Original: We ﬁnd that these networksindeed exhibit spectral bias, and that interventions that improve test accuracy on
Revised: We ﬁnd that these networksindeed exhibit spectral bias, and that interventions that improve test accuracy on

Original: CIFAR-10 tend to produce learned functions that have higher frequencies overallbut lower frequencies in the vicinity of examples from each class.
Revised: CIFAR-10 tend to produce learned functions that have higher frequencies overallbut lower frequencies in the vicinity of examples from each class.

Original: This trend holdsacross variation in training time, model architecture, number of training examples,data augmentation, and self-distillation.
Revised: This trend holdsacross variation in training time, model architecture, number of training examples,data augmentation, and self-distillation.

Original: We also explore the connections betweenfunction frequency and image frequency and ﬁnd that spectral bias is sensitive tothe low frequencies prevalent in natural images.
Revised: We also explore the connections betweenfunction frequency and image frequency and ﬁnd that spectral bias is sensitive tothe low frequencies prevalent in natural images.

Original: On ImageNet, we ﬁnd that learnedfunction frequency also varies with internal class diversity, with higher frequencieson more diverse classes.
Revised: On ImageNet, we ﬁnd that learnedfunction frequency also varies with internal class diversity, with higher frequencieson more diverse classes.

Original: Our work enables measuring and ultimately inﬂuencingthe spectral behavior of neural networks used for image classiﬁcation, and is a steptowards understanding why deep models generalize well.
Revised: Our work enables measuring and ultimately inﬂuencingthe spectral behavior of neural networks used for image classiﬁcation, and is a steptowards understanding why deep models generalize well.

Original:  Introduction
Revised:  Introduction

Original: Two fundamental questions in machine learning are why overparameterized models generalize andhow to make them more robust to distribution shift and adversarial examples.
Revised: Two fundamental questions in machine learning are why overparameterized models generalize andhow to make them more robust to distribution shift and adversarial examples.

Original: Resolving both of thesequestions requires understanding how complex our models should be.
Revised: Resolving both of thesequestions requires understanding how complex our models should be.

Original: For instance, it is thought that overparameterized models generalize well because there are implicitregularizers that constrain the complexity of the learned functions.
Revised: For instance, it is thought that overparameterized models generalize well because there are implicitregularizers that constrain the complexity of the learned functions.

Original: However, the precise nature ofthese implicit regularizers and their importance in practice, remains unclear.
Revised: However, the precise nature ofthese implicit regularizers and their importance in practice, remains unclear.

Original: As for how to achieverobustness, no consensus has emerged regarding function complexity.
Revised: As for how to achieverobustness, no consensus has emerged regarding function complexity.

Original: On the one hand, Cranko et al.[4] and Leino et al.
Revised: On the one hand, Cranko et al.[4] and Leino et al.

Original: [22] argue that Lipschitz smoothness of the learned function offers a guarantee ofrobustness; on the other, Shah et al.
Revised: [23] argue that Lipschitz smoothness of the learned function offers a guarantee ofrobustness; on the other, Shah et al.

Original: [35] and Madry et al.
Revised: [36] and Madry et al.

Original: [23] argue that simplicity can be harmfuland a robust model must actually be more complex than its non-robust counterpart.
Revised: [24] argue that simplicity can be harmfuland a robust model must actually be more complex than its non-robust counterpart.

Original: One window into function complexity is spectral bias – the tendency of neural networks to learn lowfrequency (simple and smooth) functions ﬁrst, then gradually increase the frequency (complexity)of the learned function as training proceeds.
Revised: One window into function complexity is spectral bias – the tendency of neural networks to learn lowfrequency (simple and smooth) functions ﬁrst, then gradually increase the frequency (complexity)of the learned function as training proceeds.

Original: Foundational work in this area has shown theoreticalevidence of spectral bias by analyzing convergence rates of neural networks towards functions ofdifferent frequencies [1, 34].
Revised: Foundational work in this area has shown theoreticalevidence of spectral bias by analyzing convergence rates of neural networks towards functions ofdifferent frequencies [1, 34].

Original: In practice, spectral bias is difﬁcult to measure: the most direct method involves taking a Fouriertransform with respect to the input, which is impractical to compute due to the high dimensionalityof images.
Revised: In practice, spectral bias is difﬁcult to measure: the most direct method involves taking a Fouriertransform with respect to the input, which is impractical to compute due to the high dimensionalityof images.

Original: Early experimental work focused on low-dimensional synthetic data [1, 2] or used proxymeasurements of spectral bias [41], for instance by inserting label noise of various frequenciesduring training [34].
Revised: Early experimental work focused on low-dimensional synthetic data [1, 2] or used proxymeasurements of spectral bias [42], for instance by inserting label noise of various frequenciesduring training [34].

Original: However, the label noise method is limited to binary classiﬁcation and involvesmodifying the training data, making it difﬁcult to disentangle how other changes to the training data(such as augmentation) affect the spectral content of the learned function.
Revised: However, the label noise method is limited to binary classiﬁcation and involvesmodifying the training data, making it difﬁcult to disentangle how other changes to the training data(such as augmentation) affect the spectral content of the learned function.

Original: It remains an open questionwhether modern neural networks exhibit spectral bias and what role it plays in generalization.
Revised: It remains an open questionwhether modern neural networks exhibit spectral bias and what role it plays in generalization.

Original: In this work, we investigate model complexity through the lens of spectral bias, by introducingexperimental methods to study the frequency decomposition of the functions learned by modernimage classiﬁcation networks.
Revised: In this work, we investigate model complexity through the lens of spectral bias, by introducingexperimental methods to study the frequency decomposition of the functions learned by modernimage classiﬁcation networks.

Original: We ﬁnd that high-accuracy models learn a function that is highfrequency in regions between different image classes but low-frequency within each  class.
Revised: We ﬁnd that high-accuracy models learn a function that is highfrequency in regions between different image classes but low-frequency within each class, validatingthe intuition that a “good” model should have sharp decision boundaries to delineate different classes,but smooth behavior within each class.

Original: Contributions.
Revised: Contributions.

Original: We extend the label noise procedure of Rahaman et al.
Revised: We extend the label noise procedure of Rahaman et al.

Original: [34] to enable measuringspectral bias in multi-class classiﬁcation via label smoothing, and apply this technique to understandthe function frequencies present in high-accuracy models on CIFAR-10 [21].
Revised: [34] to enable measuringspectral bias in multi-class classiﬁcation via label smoothing, and apply this technique to understandthe function frequencies present in high-accuracy models on CIFAR-10 [22].

Original: We also introduce asecond method for measuring the smoothness of a learned function via linear interpolation betweentest examples, and apply this method to both CIFAR-10 and ImageNet [8] models.
Revised: We also introduce asecond method for measuring the smoothness of a learned function via linear interpolation betweentest examples, and apply this method to both CIFAR-10 and ImageNet [8] models.

Original: We use thismethod to probe the effects of the training data on the learned function frequencies, and to distinguishthe spectral content in paths between images from the same class (“within-class paths”) and pathsbetween images from different classes (“between-class paths”).
Revised: We use thismethod to probe the effects of the training data on the learned function frequencies, and to distinguishthe spectral content in paths between images from the same class (“within-class paths”) and pathsbetween images from different classes (“between-class paths”).

Original: Using our label smoothing methodology, we ﬁnd that, as expected, models are capable of learninghigher frequency target functions the longer they train.

Original: However, using our linear interpolationmethodology we ﬁnd that throughout training the learned function becomes lower-frequency onwithin-class paths even as it becomes higher-frequency on between-class paths.

Original: Indeed, we ﬁnd thathigher-accuracy modelsconsistently demonstrate greater separation in frequency content betweenwithin-class and between-class paths, with lower frequencies within-class and higher-frequenciesbetween-class.
Revised: We ﬁnd that higher-test-accuracy models demonstrate greater separation in frequency contentbetween these path types, with lower frequencies within-class and higher frequencies between-class.

Original: On CIFAR-10, this trend holds as we improve the model architecture,  increase thesize of the training dataset, improve the data augmentation strategy, and apply self-distillation.
Revised: On CIFAR-10, this trend holds as we improve the model architecture, train longer, increase thesize of the training dataset, improve the data augmentation strategy, and apply self-distillation.

Original: On
Revised: On

Original: ImageNet, this trend appears again as we consider models with more diverse architectures.
Revised: ImageNet, this trend appears again as we consider models with more diverse architectures.

Original: We also explore the relationship between image frequency and function frequency.
Revised: We also explore the relationship between image frequency and function frequency.

Original: By furtherextending the label noise methodology of Rahaman et al.
Revised: By furtherextending the label noise methodology of Rahaman et al.

Original: [34] to study spectral bias in directions ofinterest through the input space, we ﬁnd that CIFAR-10 models most readily learn functions of thelow image frequencies common in natural images.
Revised: [34] to study spectral bias in directions ofinterest through the input space, we ﬁnd that CIFAR-10 models most readily learn functions of thelow image frequencies common in natural images.

Original: Via linear interpolation within ImageNet classes, we ﬁnd that many models are higher-frequencyin more internally diverse classes and lower-frequency in more internally consistent classes.
Revised: Via linear interpolation within ImageNet classes, we ﬁnd that many models are higher-frequencyin more internally diverse classes and lower-frequency in more internally consistent classes.

Original:  Related Work
Revised:  Related Work

Original: Implicit bias.
Revised: Implicit bias.

Original: A common belief is that some form of implicit bias imposed by the training proceduremay account for the generalization ability of overparameterized neural networks, and accordingly,much research has been directed towards understanding these implicit biases [36, 45, 20, 13, 14, 15,17, 3, 30, 28, 29, 24, 13, 27]. One of the most well-known forms of implicit bias comes from theoptimization algorithm itself, which can guide networks to solutions that generalize well [36, 20, 40].
Revised: A common belief is that some form of implicit bias imposed by the training procedureor optimization algorithm may account for the generalization ability of overparameterized neuralnetworks, and accordingly, much research has been directed towards understanding these implicitbiases [37, 46, 21, 41, 14, 13, 15, 17, 3, 30, 28, 29, 25, 14, 20].

Original: Spectral bias.
Revised: Spectral bias.

Original: Spectral bias, also called the frequency principle, is a form of implicit bias with muchrecent attention including theoretical and experimental approaches [33, 41, 34, 1, 47].
Revised: Spectral bias, also called the frequency principle, is a form of implicit bias with muchrecent attention including theoretical and experimental approaches [33, 42, 34, 1, 48].

Original: Basri et al.[1] studied spectral bias using a linear model of SGD training dynamics to show that models learnlow frequency (simple) functions faster, assuming training data that is distributed uniformly on thehypersphere.
Revised: Basri et al.[1] studied spectral bias using a linear model of SGD training dynamics to show that models learnlow frequency (simple) functions faster, assuming training data that is distributed uniformly on thehypersphere.

Original: Basri et al.
Revised: Basri et al.

Original: [2] extended the analysis to consider nonuniformly spaced training data,and found that learning is faster where samples are denser; if sampling is nonuniform, then duringtraining, the learned function will be higher frequency in regions with denser samples.
Revised: [2] extended the analysis to consider nonuniformly spaced training data, andfound that learning is faster where samples are denser; if sampling is nonuniform, then during training,the learned function will be higher frequency in regions with denser samples.

Original: Rahaman et al.
Revised: Rahaman et al.

Original: [34] used a more direct analysis to show the same spectral bias toward low frequencyfunctions and also suggested that when training data lie on a manifold (as natural images are believedto), increasing the complexity of the manifold makes it easier for models to ﬁt high frequencyfunctions. Additionally, Rahaman et al. [34] posited that high frequency components of the learned function are most sensitive to perturbations in the model parameters, connecting back to the idea ofﬂat optimization minima [29] or smooth loss landscape [25].
Revised: [34]used a more direct analysis to show the same spectral bias toward low frequency functions and alsoposited that high frequency components of the learned function are most sensitive to perturbationsin the model parameters, connecting back to the idea of ﬂat optimization minima [29] or smooth losslandscape [26].

Original: They proposed experimental methodsto study spectral bias in image classiﬁcation, but focused on a binary subset of the relatively simple MNIST dataset [9], with mean square error loss.
Revised: They proposed experimental methods to study spectral bias in image classiﬁcation,but focused on a binary subset of the relatively simple MNIST dataset [9], with mean square error loss.

Original: Our label smoothing experiments are a direct extension of Rahaman et al.
Revised: Our label smoothing experiments are a direct extension of Rahaman et al.

Original: [34] to multiclass classiﬁcation with the more common cross-entropy loss.
Revised: [34] to multiclass classiﬁcation with the more common cross-entropy loss.

Original: Our linear interpolation experiments are somewhatsimilar to the recently-proposed experimental methods in Zhang et al.
Revised: Our linear interpolation experiments are somewhat similar to the recently-proposed experimental methods in Zhang et al.

Original: [47], except that we samplealong paths between images rather than in regions surrounding each image; this allows for a moreglobal measurement of spectral bias that we apply to a broad and distinct set of training proceduresbeyond the double descent regime studient in Zhang et al.
Revised: [48], except that we samplealong paths between images rather than in regions surrounding each image; this allows for a moreglobal measurement of spectral bias that we apply to a broad and distinct set of training proceduresbeyond the double descent regime studient in Zhang et al.

Original: Model sensitivity to image frequency.
Revised: Model sensitivity to image frequency.

Original: While we focus on function frequency in this work, priorresearch aimed to understand model sensitivity to image frequencies.
Revised: While we focus on function frequency , prior research aimedto understand model sensitivity to image frequencies.

Original: Jo and Bengio [19] foundthat CNNs are sensitive to Fourier statistics of the training data, even those irrelevant to humanviewers.
Revised: Jo and Bengio [19] found that CNNs are sensitive to Fourier statistics of the training data, even those irrelevant to human viewers.

Original: Ortiz-Jimenez et al.
Revised: Ortiz-Jiménezet al.

Original: [32] studied the image frequency bias induced by using a convolutionalarchitecture, and Ortiz-Jimenez et al.
Revised: [32] studied the image frequency bias induced by using a convolutional architecture, and OrtizJiménez et al.

Original: [31] argued that models are sensitive primarily to discriminative Fourier directions in the training data.
Revised: [31] argued that models are sensitive primarily to discriminative Fourier directions inthe training data.

Original: They posit that adversarial training induces large marginsin orthogonal directions, thereby improving robustness.

Original: Yin et al.
Revised: Yin et al.

Original: [43] introduced a procedure tomeasure the sensitivity of trained models to Fourier image perturbations, and showed that adversarialtraining andGaussian data augmentation increase robustness to high frequency perturbations butincrease sensitivity to low frequency perturbations.
Revised: [44] introduced a procedure to measure the sensitivity of trained models to Fourier image perturbations, and applied it to study the effects of adversarial training and data augmentation.

Original: Our work adds to this line of research a study ofthe interplay between two notions of frequency: the function frequency involved in spectral bias, andthe image frequencies present in the data.
Revised: Our work adds to this line of research a study of the interplay between two notions of frequency:the function frequency involved in spectral bias, and the image frequencies present in the data.

Original:  Methodology
Revised:  Methodology

Original: The goal of our work is to measure the complexity of neural network functions through the lens offrequency.
Revised: The goal of our work is to measure the complexity of neural network functions through the lens offrequency.

Original: In low dimensions, measuring the frequency decomposition of a function is straightforwardand tractable: evaluate the function at dense, uniform sampling positions and compute its discrete
Revised: In low dimensions, measuring the frequency decomposition of a function is straightforwardand tractable: evaluate the function at dense, uniform sampling positions and compute its discrete

Original: Fourier transform.
Revised: Fourier transform.

Original: However, the function of interest in image classiﬁcation is unavoidably highdimensional, mapping images (with thousands of pixel values) to object classes.
Revised: However, the function of interest in image classiﬁcation is unavoidably highdimensional, mapping images (with thousands of pixel values) to object classes.

Original: It would be intractableeven to collect sufﬁcient samples of this function to compute a discrete Fourier transform, letalone compute the transform itself.
Revised: It would be intractableeven to collect sufﬁcient samples of this function to compute a discrete Fourier transform, letalone compute the transform itself.

Original: Instead, we employ two complementary approaches to measureinformative proxies of this frequency decomposition.
Revised: Instead, we employ two complementary approaches to measureinformative proxies of this frequency decomposition.

Original: 3.1 Label Smoothing
Revised: 3.1 Label Smoothing

Original: The core idea for measuring function frequency introduced by Rahaman et al.
Revised: The core idea for measuring function frequency introduced by Rahaman et al.

Original: [34] is to construct asinusoid over the space of images, and to use that sinusoid as a form of label noise during training.
Revised: [34] is to construct asinusoid over the space of images, and to use that sinusoid as a form of label noise during training.

Original: Let D = 
Revised: Let D = { ( X

Original: ( X i, y i ) } n train i
Revised: i y i ) } n train i

Original: = be the training examples and D valid = { ( X j , y j ) } n val j =1 be the validationexamples, with X i an image and y i a one-hot class encoding, where n train is the number of trainingexamples, n val is the number of validation examples, d is the side length of an image (assumed tobe square for simplicity), c is the number of color channels, and M is the number of classes, so
Revised: = be the training examples and D valid = { ( X j , y j ) } n val j =1 be the validationexamples, with X i an image and y i a one-hot class encoding, where n train is the number of trainingexamples, n val is the number of validation examples, d is the side length of an image (assumed tobe square for simplicity), c is the number of color channels, and M is the number of classes, so

Original: X i ∈ R
Revised: X i ∈ R

Original: d × d × c and y i ∈ R M .
Revised: d × d × c and y i ∈ R M .

Original: To extend this procedure to the multi-class setting, we add noise of various frequencies to an Mdimensional label vector via label smoothing [37].
Revised: To extend this procedure to the multi-class setting, we add noise of various frequencies to an M -dimensional label vector via label smoothing, originally introduced as a regularization approachin Szegedy et al.

Revised: Intuitively, label smoothing removes some probability from the correct classand redistributes it equally among the remaining classes.

Original: Let S : R d × d × c → [0 , 1] be a target function thatmaps an input image
Revised: Let S : R d × d × c → [0 , 1] be a noise functionthat maps an input image

Original: X
Revised: X

Original: i to a scalar value between 0 and 1.
Revised: i to a scalar value between 0 and 1.

Original: We apply label smoothing to each labely i , mapping it to ¯ y i =
Revised: We apply label smoothing to eachlabel y i , mapping it to ¯ y i =

Original: y i (1 − S ( X i ))
Revised: y i (1 − S ( X i ))

Original: + 1 M S
Revised: + 1 M S

Original: ( X i ) to retain a valid probability distribution.
Revised: ( X i ) to retain a valid probability distribution, as in

Revised: Szegedy et al.

Revised: [38] but with different functions S .

Revised: Rather than using label smoothing as a regularizerto improve performance, we use it as a form of variable frequency label noise to study spectral bias.

Original: We thentrain from scratch using the original examples X i and their smoothed labels ¯ y i .
Revised: We train from scratch using the original examples X i and their smoothed labels ¯ y i .

Original: Finally, we evaluateon the validation images X j , comparing to both their original one-hot labels y j and smoothed labels¯ y j = y j (1 − S ( X j ))
Revised: We evaluate onthe validation images X j , comparing to both their original one-hot labels y j and smoothed labels¯ y j = y j (1 − S ( X j ))

Original: + 1 M S ( X j ) .
Revised: + 1 M S ( X j ) .

Original: Our label smoothing experiments use the CIFAR-10 dataset [21],where n train = 50000 , n val = 10000 , d = 32 , c = 3 , and M = 10 ; however, our experimentalsetup is sufﬁciently general as to transfer easily to any image classiﬁcation task.
Revised: Our label smoothing experiments use the CIFAR-10 dataset [22],where n train = 50000 , n val = 10000 , d = 32 , c = 3 , and M = 10 .

Original: This method is introduced in Figure 1.
Revised: This method is introduced in Figure 1.

Original: As training proceeds, we see both the training loss (between predictions and smoothed labels ¯ y i ) and noisy validation loss (between predictions andsmoothed labels ¯ y j ) decrease.
Revised: As training proceeds, we see both the training loss (between predictions and smoothed labels ¯ y i ) and noisy validation loss (between predictions andsmoothed labels ¯ y j ) decrease.

Original: The clean validation loss (between predictions and one-hot labelsy j ), however, initially decreases but at some point in training plateaus or begins to increase.
Revised: The clean validation loss (between predictions and one-hot labelsy j ), however, initially decreases but at some point in training plateaus or begins to increase.

Original: At
Revised: Atthis point, the model has begun to learn the noise function S . Accordingly, we introduce noiseﬁtting , deﬁned as the difference between the validation loss on one-hot labels and the validation loss on labels smoothed with the same function S that was applied to the training labels:noise ﬁtting = clean validation loss − noisy validation loss . We compare different models by computing the minimum noise ﬁtting achieved throughout training. A model with higher min noiseﬁtting more readily ﬁts the noise function S than a model with lower min noise ﬁtting.

Original: By choosing different functions S , we can probe nuances of spectral bias.
Revised: By choosing different functions S , we can probe nuances of spectral bias.

Original: Typically (inspired by
Revised: Typically (inspired by

Original: Rahaman et al.
Revised: Rahaman et al.

Original: [34]) we choose a radial wave: S ( X ) =  sin(

Revised: [34]) we choose a radial wave: S ( X ) = α (1 + sin(

Original: πf ( (cid:107) X (cid:107)− E D (cid:107) X (cid:107) )) , where (cid:107) X (cid:107) denotesthe Euclidean norm of the vectorized image.
Revised: πf ( (cid:107) X (cid:107) − E D (cid:107) X (cid:107) ))) , where (cid:107) X (cid:107)denotes the Euclidean norm of the vectorized image, and α ∈ [0 , 0 . 5] to ensure that S ( X ) ∈ [

Original: We can vary the frequency f to understand spectral biasat this global scale.
Revised: We can vary the frequency f to understand spectral bias at this global scale.

Original: We can also choose more targeted functions S ; for instance, if V is a directionof interest through the space of images ( i.e. V is an image-shaped vector of unit norm), we canconstruct S ( X ) =
Revised: We can also choosemore targeted functions S ; for instance, if V is a direction of interest through the space of images( i.e. V is an image-shaped vector of unit norm), we can construct S ( X ) =

Original:  sin(

Revised: α (1 + sin(

Original: πf (cid:104) X , V (cid:105) ) .
Revised: πf (cid:104) X , V (cid:105) )) .

Original: This allows us to vary both the frequency and direction of thetarget sinusoid, to understand models’ learnability along different directions through image space.
Revised: This allows us to vary both the frequency and direction of the noise function, to understand models’learnability along different directions through image space.

Original: However, label smoothing has a few limitations.
Revised: However, label smoothing has a few limitations.

Original: Because it involves perturbing the training labels, itcannot directly study the interaction between the training dataset and spectral bias, such as the effectsof data augmentation.
Revised: Because it involves perturbing the training labels, itcannot directly study the interaction between the training dataset and spectral bias, such as the effectsof data augmentation.

Original: It also requires retraining each model from scratch, with substantial investmentof time and computational resources.
Revised: It also requires retraining each model from scratch, with substantial investmentof time and computational resources.

Original: We address these limitations with our linear interpolationtechnique in the next section.
Revised: We address these limitations with our linear interpolationtechnique in the next section.

Original: 3.2 Linear Interpolation
Revised: 3.2 Linear Interpolation

Original: To complement the label smoothing approach, we also propose a methodology based on linearinterpolation between validation images.
Revised: To complement the label smoothing approach, we also propose a methodology based on linearinterpolation between validation images.

Original: Although we cannot take dense, regularly-spaced samplesthat cover the entire input (image) space, we can sample along speciﬁc paths and thereby gleanglimpses into the spectral content of the learned function.
Revised: Although we cannot take dense, regularly-spaced samplesthat cover the entire input (image) space, we can sample along speciﬁc paths and thereby gleanglimpses into the spectral content of the learned function.

Original: We consider two types of paths: those between images from the same class, and those between imagesfrom different classes.
Revised: We consider two types of paths: those between images from the same class, and those between imagesfrom different classes.

Original: When considering paths between CIFAR-10 images in the same class, wechoose 200 random, distinct pairs of validation images from each of the 10 CIFAR-10 classes andaverage over the paths deﬁned by these pairs.
Revised: When considering paths between CIFAR-10 images in the same class, wechoose 200 random, distinct pairs of validation images from each of the 10 CIFAR-10 classes andaverage over the paths deﬁned by these pairs.

Original: We do the same for ImageNet, choosing 50 random,distinct paths for each of the 1000 ImageNet classes.
Revised: We do the same for ImageNet, choosing 50 random,distinct paths for each of the 1000 ImageNet classes.

Original: When interpolating between CIFAR-10 classes,we choose 200 random, distinct validation image pairs from each of the 45 (10 choose 2) pairs ofdistinct classes, and average over the resulting paths; for ImageNet we choose 1000 random, distinctbetween-class paths, ensuring that each class is represented in exactly two paths.
Revised: When interpolating between CIFAR-10 classes,we choose 200 random, distinct validation image pairs from each of the 45 (10 choose 2) pairs ofdistinct classes, and average over the resulting paths; for ImageNet we choose 1000 random, distinctbetween-class paths, ensuring that each class is represented in exactly two paths.

Original: For each pair of images ( X 0 , X 1 ) , we vary λ ∈ [0 , 1] to trace out a path, where each image in the pathis given by X λ
Revised: For each pair of images ( X 0 , X 1 ) , we vary λ ∈ [0 , 1] to trace out a path, where each image in the pathis given by X λ

Original: = λ
Revised: = λ

Original: X 

Revised: X

Original: + (1 − λ )
Revised: + (1 − λ )

Original: X 0 .
Revised: X 0 .

Original: We choose λ s so that the distance between adjacent images onthe path is constant; each path may produce a different number of samples depending on the totaldistance between X 0 and X 1 .
Revised: We choose λ s so that the distance between adjacent images onthe path is constant; each path may produce a different number of samples depending on the totaldistance between X 0 and X 1 .

Original: Figure 2 (Bottom Left) shows an illustrative example of two CIFAR-10images and a corresponding interpolation path between them.
Revised: Figure 2 (Bottom Left) shows an illustrative example of two CIFAR-10images and a corresponding interpolation path between them.

Original: For our experiments, we typically have50 to 100 images along the interpolation path.
Revised: For our experiments, we typically have50 to 100 images along the interpolation path.

Original: Relationship between interpolation and label smoothing experiments.
Revised: Relationship between interpolation and label smoothing experiments.

Original: In order to validate ourinterpolation methodology, we consider the oracle function that maps directly from the interpolated image X λ to the target label used in our label smoothing experiments.
Revised: To validate ourinterpolation methodology, we consider the oracle function that maps directly from the interpolatedimage X λ to the noised label used in our label smoothing experiments.

Revised: We only consider within-classpaths for this experiment, so that the original one-hot label (before smoothing) is ﬁxed across theentire path, and the only change in the oracle function along the path is due to the noise function usedin label smoothing.

Original: By increasing the frequency of thelabel smoothing noise, we create oracle functions with higher frequencies.
Revised: By increasing the frequency of the noise function, we create oracle functionswith higher frequencies.

Original: We then verify that our interpolation measurement indeed measures higher frequencies on the higher frequency oracle functions.
Revised: We then verify that our interpolation measurement indeed measures higherfrequencies on the higher frequency oracle functions.

Original: In Figure 2 (Left) , we show an example where we vary the label smoothing frequency from f = 0to f = 0 .
Revised: In Figure 2 (Left) , we show an example where we vary the label smoothing frequency from f = 0to f = 0 .

Original: We plot the value of λ along the interpolating path on the x axis, and the norm ofthe difference between the model output  on the interpolated image ˆ y λ and the model output on theoriginal image ˆ y 0 on the y axis.
Revised: We plot the value of λ along the interpolating path on the x axis, and the norm of thedifference between the model output (softmax probabilities) on the interpolated image ˆ y λ and themodel output on the original image ˆ y 0 on the y axis.

Revised: This difference norm always starts at zero(although we plot it by averaging many paths over buckets in λ , so the curves do not actually touchthe axis), and tends to increase with λ as the model predictions change along the interpolating path.

Original: The norm of the prediction difference allows us to visualize howsmooth the function is along the interpolating path; smoother prediction difference norm indicates alower frequency function.
Revised: The norm of the prediction difference allows us to visualize how smooth the function is along theinterpolating path; smoother prediction difference norm indicates a lower frequency function.

Original: To quantify the frequency of the  function along the interpolating path, we compute the per-classdiscrete Fourier transform (DFT).
Revised: To quantify the frequency of the oracle function along the interpolating path, we compute the per-classdiscrete Fourier transform (DFT).

Original: Letting { X λ t } Tt =1 denote the set of interpolated images, the DFT coefﬁcient for frequency f is given by ˆ Y f
Revised: Letting { X λ t } Tt =1 denote the set of interpolated images, the DFT coefﬁcient for frequency f is given by ˆ Y f

Original: [ m ] = (cid:80) Tt
Revised: [ m ] = (cid:80) Tt

Original: =1 ˆ y λ t [ m ] exp( − i 2 πft ) , where i is theimaginary number and m ∈ { 1 , . . .
Revised: =1 ˆ y λ t [ m ] exp( − i 2 πft ) , where i is the imaginarynumber and m ∈ { 1 , . . .

Original: , M } denotes the class.
Revised: , M } denotes the class.

Original: To summarize how the magnitudes of the DFT coefﬁcients ˆ Y f are distributed between low and high frequencies, we average the coefﬁcientmagnitudes for each of the classes and compute the fraction of the total average coefﬁcient magnitudesabove a threshold.
Revised: To summarize how the magnitudes of the DFT coefﬁcients ˆ Y f are distributed between low and high frequencies, we average the coefﬁcient magnitudes foreach of the classes and compute the fraction of the total average coefﬁcient magnitude that is allocatedto frequencies above a threshold (0.05).

Original: In Figure 2 (Right) we show this summary metric averaged over all within-classinterpolating paths for the oracle functions, along with its standard error.
Revised: In Figure 2 (Right) we show this summary metric averagedover all within-class interpolating paths for the oracle functions, along with its standard error.

Original: In Figure 2 (Left) , the higherfrequency oracle functions have less smooth prediction norm differences, and in Figure 2 (Right) ,they have a higher fraction of DFT coefﬁcient magnitudes in the high frequencies.
Revised: In Figure 2 (Left) , the higher frequency oracle functions have less smooth prediction norm differences, andin Figure 2 (Right) , they have a higher fraction of DFT coefﬁcient magnitudes in the high frequencies.

Original: 3.3 Data and Models
Revised: 3.3 Data and Models

Original: Our experiments use the CIFAR-10 [21] dataset of low-resolution (32 × 32) natural images fromten animal  andobject classes, as well as the ImageNet [8] dataset of higher-resolution (224 × 224)natural images from 1000 animal and object classes.
Revised: We use the CIFAR-10 [22] dataset of low-resolution (32 × 32) natural images from ten animal andobject classes and the ImageNet [8] dataset of higher-resolution (224 × 224) natural images from1000 classes.

Original: We consider a diversity of model architectures on CIFAR-10 (6 total) and ImageNet (10 total).
Revised: We consider a diversity of model architectures on CIFAR-10 (6 total) and ImageNet (10total).

Original: See Section A.1 for more details on the data processing,models, and their accuracies.
Revised: See Section A.1 for more details on the data processing, models, and their accuracies.

Original:  Results on CIFAR-10
Revised:  Results on CIFAR-10

Original: Throughout this section we show representative examples from our CIFAR-10 experiments; fullresults (on all six models we tested) are included in Section A.3.
Revised: Throughout this section we show representative examples from our CIFAR-10 experiments; fullresults (on all six models we tested) are included in Section A.3.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4.1 Spectral Bias
Revised: 4.1 Spectral Bias

Original: We begin by applying our label smoothing methodology to ﬁnd that modern image classiﬁcation
Revised: We begin by applying our label smoothing methodology to ﬁnd that modern image classiﬁcation

Original: CNNs exhibit spectral bias, learning low frequencytarget functions early in training and learninghigher frequency functions as training proceeds.
Revised: CNNs exhibit spectral bias, learning low frequency functions early in training and learning higherfrequency functions as training proceeds.

Original: Note that Figure 3 (Left) shows spectral bias over asmall but illustrative range of frequencies (the full range of frequencies we tested was 0 to 0.1); targetfunctions of sufﬁciently low frequency are ﬁt almost immediately and target functions of sufﬁcientlyhigh frequency are never learned during the 250 epochs of training we tested.
Revised: Note that Figure 3 (Left) shows spectral bias over a smallbut illustrative range of frequencies (the full range of frequencies we tested was 0 to 0.1); noisefunctions of sufﬁciently low frequency are ﬁt almost immediately and noise functions of sufﬁcientlyhigh frequency are never learned during the 250 epochs of training we tested.

Original: Using our linear interpolation methodology and distinguishing between within-class paths (in whichboth endpoint images are of the same class) and between-class paths (where the path must traverse aclass boundary) in Figure 3 (Right) , we ﬁnd that this spectral bias is highly variable over the domainof images.
Revised: Using our linear interpolation methodology and distinguishing between within-class paths (in whichboth endpoint images are of the same class) and between-class paths (where the path must traverse aclass boundary) in Figure 3 (Right) , we ﬁnd that this spectral bias is highly variable over the domainof images.

Original: In particular, while the learned function does increase in frequency for between-class paths,it decreases in frequency for within-class paths, indicating that the model is learning to cluster andseparate the different classes.
Revised: In particular, while the learned function does increase in frequency for between-class paths,it decreases in frequency for within-class paths, indicating that the model is learning to cluster andseparate the different classes.

Original: Throughout our CIFAR-10 experiments, we ﬁnd that higher-accuracymodels have greater separation between the frequencies of their within-class and between-class paths,with lower-frequency behavior inside each class and higher-frequency behavior in between.
Revised: Throughout our CIFAR-10 experiments, we ﬁnd that higher-accuracymodels have greater separation between the frequencies of their within-class and between-class paths,with lower-frequency behavior inside each class and higher-frequency behavior in between.

Original: 4.2 Spectral Bias and Model Architecture
Revised: 4.2 Spectral Bias and Model Architecture

Original: Although all models we tested exhibit spectral bias, we found that the precise nature of the biasdepends on the choice of model.
Revised: Although all models we tested exhibit spectral bias, we found that the precise nature of the biasdepends on the choice of model.

Original: For example, with all else ﬁxed, increasing the width of a modeldecreases its spectral bias, enabling it to more readily ﬁt target frequency functions.
Revised: For example, with all else ﬁxed, increasing the width of a modeldecreases its spectral bias, enabling it to more readily ﬁt noise functions of a given frequency.

Original: This trend isevident in Figure 4 for the Wide-ResNet family (Left) and the Shake-Shake family (Center) .
Revised: Thistrend is evident in Figure 4 for the Wide-ResNet family (Left) and the Shake-Shake family (Center) .

Original: Although larger models learn high frequencies faster, we found that higher-accuracy models are notuniformly higher-frequency.
Revised: Although larger models learn high frequencies faster, we found that higher-accuracy models are notuniformly higher-frequency.

Original: Instead, using linear interpolation in Figure 4 (Right) we ﬁnd that higher- accuracy models (ordered left to right by increasing test accuracy) are higher-frequency along betweenclass paths but lower-frequency along within-class paths, perhaps evident of a better class clustering.
Revised: Instead, using linear interpolation in Figure 4 (Right) we ﬁnd that higher- accuracy models (ordered left to right by increasing test accuracy) are higher-frequency along betweenclass paths but lower-frequency along within-class paths, perhaps evident of a better class clustering.

Original: 4.3 Sensitivity to Natural Image Directions
Revised: 4.3 Sensitivity to Natural Image Directions

Original: Label smoothing with a radial wave offers a convenient global picture of spectral bias, by replacingthe radial wave with other target functions we can use the same methodology to test more localizedaspects of spectral bias.
Revised: Label smoothing with a radial wave offers a convenient global picture of spectral bias, by replacingthe radial wave with other noise functions we can use the same methodology to test more localizedaspects of spectral bias.

Original: We take a step in this direction by considering the family of target labelsmoothing functions S ( X ; k ) =[-sin(
-]
Revised: We take a step in this direction by considering the family of noise functions S ( X ; k ) =[-sin(
-]

Original:  πf (cid:104) X , F k (cid:105) ) , where F k is a diagonal Fourier basis image withfrequency k and the same dimensions as X , visualized in Figure 5 (Bottom) .
Revised: α (1 + sin(2 πf (cid:104) X , F k (cid:105) )) , where F k is a diagonal Fourier basis image with frequency kand the same dimensions as X , visualized in Figure 5 (Bottom) .

Original: We consider (a subset of)
Revised: We consider (a subset of)

Original: Fourier basis images because the Fourier spectra statistics of natural imagesare well studied (see, e.g. Tolhurst et al.
Revised: Fourier basis images because the Fourier spectra statistics of natural imagesare well studied (see, e.g. Tolhurst et al.

Original: [38]): natural images tend to be composed of Fourier basisimages with amplitude proportional to their inverse spatial frequency.
Revised: [39]): natural images tend to be composed of Fourier basisimages with amplitude proportional to their inverse spatial frequency.

Original: Indeed, in Figure 5 we ﬁnd that
Revised: Indeed, in Figure 5 we ﬁnd that

Original: ShakeShake96 is more sensitive to label smoothing in low image frequency directions.
Revised: ShakeShake96 is more sensitive to label smoothing in low image frequency directions.

Original: This ﬁnding is consistent with theoretical predictions of Basri et al.
Revised: This ﬁnding is consistent with theoretical predictions of Basri et al.

Original: [2] that models learn fasterin regions of higher density of training examples: since low image frequencies are more commonin natural images, the effective sampling density enjoyed by a label smoothing target is higher inthese directions.
Revised: [2] that models learn faster inregions of higher density of training examples: since low image frequencies are more common innatural images, the effective sampling density enjoyed by a noise function is higher in these directions.

Original: It is also possible that this bias is  inherent to the convolutional model architectureregardless of the data distribution, as discussed in Ortiz-Jimenez et al. [32], or that a combination ofboth effects is at play.
Revised: It is also possible that this bias is a byproduct of the lower Fourier magnitudes of high frequencycomponents in natural images [35], or is inherent to the convolutional model architecture regardlessof the data distribution [32], or that a combination of multiple effects is at play.

Original: Determining the precise cause of this image frequency ﬁnding is an interestingdirection for further study.
Revised: Determining theprecise cause of this image frequency ﬁnding is an interesting direction for further study.

Original: 4.4 Spectral Bias and Training Data
Revised: 4.4 Spectral Bias and Training Data

Original: In our CIFAR-10 interpolation experiments, we consistently observed that variations in training thatimprove accuracy create greater disparity between the frequencies of within-class and between-classpaths, with lower frequencies within-class and higher frequencies between-class.
Revised: In our CIFAR-10 interpolation experiments, we consistently observed that variations in training thatimprove accuracy create greater disparity between the frequencies of within-class and between-classpaths, with lower frequencies within-class and higher frequencies between-class.

Original: This trend is evidentin Figure 3 (Right) as a model trains for more epochs and in Figure 4 (Right) as the model architectureimproves.
Revised: This trend is evidentin Figure 3 (Right) as a model trains for more epochs and in Figure 4 (Right) as the model architectureimproves.

Original: In Figure 6 we ﬁnd that the same trend also holds as we train on more data (Left) , applymore effective data augmentation (Center) , or apply self-distillation (Right) .
Revised: In Figure 6 we ﬁnd that the same trend also holds as we train on more data (Left) , applymore effective data augmentation (Center) , or apply self-distillation (Right) .

Original: Training Set Size.
Revised: Training Set Size.

Original: As shown in Figure 6 (Left) on WideResNet160 , we ﬁnd that increasing the numberof training examples increases the difference in frequency content between within-class interpolatingpaths and between-class interpolating paths.
Revised: As shown in Figure 6 (Left) on WideResNet160 , we ﬁnd that increasing the numberof training examples increases the difference in frequency content between within-class interpolatingpaths and between-class interpolating paths.

Original: Training with more data makes our CIFAR-10 modelsbehave lower-frequency within each class but higher-frequency between classes, perhaps indicatingthat the models are learning a better clustering of the classes as they are provided with additional data.
Revised: Training with more data makes our CIFAR-10 modelsbehave lower-frequency within each class but higher-frequency between classes, perhaps indicatingthat the models are learning a better clustering of the classes as they are provided with additional data.

Original: Data Augmentation.
Revised: Data Augmentation.

Original: Data augmentation is a common strategy to increase the effective training setsize without the expense of actually collecting additional examples.
Revised: Data augmentation is a common strategy to increase the effective training setsize without the expense of actually collecting additional examples.

Original: In Figure 6 (Center) we considerthe effects of Mixup (with strength 0.1)
Revised: In Figure 6 (Center) we considerthe effects of Mixup (with strength 0.1)

Original: [46], AutoAugment [5], and RandAugment [6], each ofwhich improves the ﬁnal accuracy of the trained model (with RandAugment producing the mostsubstantial beneﬁts in our experiments, followed by AutoAugment and then Mixup).
Revised: [47], AutoAugment [6], and RandAugment [5], each ofwhich improves the ﬁnal accuracy of the trained model (with RandAugment producing the mostsubstantial beneﬁts in our experiments, followed by AutoAugment and then Mixup).

Original: RandAugmentand AutoAugment generate new images by applying geometric and lighting transformations, whereas
Revised: RandAugmentand AutoAugment generate new images by applying geometric and lighting transformations, whereas

Original: Mixup linearly interpolates between pairs of existing images.
Revised: Mixup linearly interpolates between pairs of existing images.

Original: Much like our experiments with trainingset size, we ﬁnd that augmentations that improve accuracy more also produce greater frequencyseparation between within-class paths and between-class paths.
Revised: Much like our experiments with trainingset size, we ﬁnd that augmentations that improve accuracy more also produce greater frequencyseparation between within-class paths and between-class paths.

Original: Mixup Strength.
Revised: Mixup Strength.

Original: Mixup augmentation [46] perturbs each batch of training data as follows: each example is matched with a partner based on a random permutation of the batch, and  each exampleis thenperturbed towards its partner by an interpolation amount λ drawn from a symmetric beta distribution(with the same λ used for all examples in the batch).
Revised: Mixup augmentation [47] perturbs each batch of training data by randomly pairing the examples and perturbing each example towards its partner by an interpolation amount λ drawnfrom a symmetric beta distribution (with the same λ used for all examples in the batch).

Original: We refer to the parameter of this beta distributionas the Mixup strength, as it controls the degree to which the augmented images tend to lie close to anoriginal training image or close to the average of two training images.
Revised: We referto the parameter of this beta distribution as the Mixup strength, as it controls the degree to which theaugmented images tend to lie close to an original training image or close to the average of two trainingimages.

Original: A parameter of 0 corresponds tono augmentation ( λ = 0 or λ = 1 , always using the original images), a parameter of 1 corresponds tothe uniform distribution over λ ∈ [0 , 1] , and a parameter of ∞ corresponds to λ = 0 .
Revised: A parameter of 0 corresponds to no augmentation ( λ = 0 or λ = 1 , always using the originalimages), a parameter of 1 corresponds to the uniform distribution over λ ∈ [0 , 1] , and a parameter of∞ corresponds to λ = 0 .

Original: 5 , the exact midpoint between a pair of training images.
Revised: 5 , the exact midpoint between a pair of training images.

Original: Figure 6 (Center) compares a range of Mixup strengths andﬁnds that relatively low values that only slightly perturb the original images tend to produce greaterfrequency separation (and higher  accuracy), whereas perturbing the images too much with stronger Mixup decreases the  frequency separation between within-class and between-class paths (and hurts ﬁnal accuracy).
Revised: Figure 6 (Center)compares a range of Mixup strengths and ﬁnds that relatively low values that only slightly perturb theoriginal images tend to produce greater frequency separation (and higher test accuracy), whereas perturbing the images too much with stronger Mixup has the opposite effect on both frequency separationand accuracy.

Original: Of note, the range of Mixup strengths that increase or do not affect frequency separationin our experiments are well aligned with those recommended by the original authors [46]: 0.1 to 0.4.
Revised: Of note, the range of Mixup strengths that increase or do not affect frequency separationin our experiments are well aligned with those recommended by the original authors [47]: 0.1 to 0.4.

Original: Self-Distillation.
Revised: Self-Distillation.

Original: Finally, we consider the effect of self-distillation on the frequency content ofthe learned function.
Revised: Finally, we consider the effect of self-distillation on the frequency content ofthe learned function.

Original: In self-distillation, a “teacher” model is trained with some form of strongregularization, in our case a combination of weight decay and early stopping based on training loss.
Revised: In self-distillation, a “teacher” model is trained with some form of strongregularization, in our case a combination of weight decay and early stopping based on training loss.

Original: A “student” model with the same architecture is then trained from scratch to ﬁt the pseudolabelsproduced by the teacher, instead of the original one-hot training labels (alternatively, an interpolationbetween the pseudolabels and the original labels may be used). Prior research [11] found that thisprocedure can train student models that outperform both their teachers and a baseline (the samearchitecture trained to completion on the one-hot labels).
Revised: A “student” model with the same architecture is then trained from scratch to ﬁt the pseudolabelsproduced by the teacher Prior research [11] found that this procedure can train student models thatoutperform both their teachers and a baseline model trained as normal.

Original: Prior research has also sought to understand the mechanism behind knowledge distillation andself-distillation in particular.
Revised: Prior research has also sought to understand the mechanism behind self-distillation.

Original: Mobahi et al.
Revised: Mobahi et al.

Original: [26] ﬁnds that self-distillation acts as a regularizer bylimiting the basis functions available to the student to learn.
Revised: [27]ﬁnds that self-distillation acts as a regularizer by limiting the basis functions available to the studentto learn.

Original: We complement their theoretical workwith our interpolation experimental methodology in Figure 6 (Right) , where indeed we ﬁnd thatthe student model learns a greater frequency separation between within-class and between-classpaths than its teacher, while also achieving higher test accuracy.
Revised: We complement their theoretical work with our interpolation experimental methodology in Figure 6 (Right) , where indeed we ﬁnd that the student model learns a greater frequency separationbetween within-class and between-class paths than its teacher, while also achieving higher testaccuracy.

Original: We conjecture that this effect isanalogous to low-pass preﬁltering common in digital signal processing: a high-frequency targetfunction that we cannot adequately sample is ﬁrst smoothed (in this case by being approximated by aregularized teacher) and then it can be modeled via samples without further loss in ﬁdelity.
Revised: We conjecture that this effect is analogous to low-pass preﬁltering common in digital signalprocessing: a high-frequency noise function that we cannot adequately sample is ﬁrst smoothed (inthis case by being approximated by a regularized teacher) and then it can be modeled via sampleswithout further loss in ﬁdelity.

Original: Withoutthis preﬁltering, our samples are inadequate to capture the complexity of the target function, so wereconstruct an imperfect version corrupted by aliasing.
Revised: Without this preﬁltering, our samples are inadequate to capture thecomplexity of the noise function, so we reconstruct an imperfect version corrupted by aliasing.

Original:  Results on ImageNet
Revised:  Results on ImageNet

Original: Finally, we apply our linear interpolation methodology to study the frequency content of a range ofpretrained models on ImageNet [8].
Revised: Finally, we apply our linear interpolation methodology to study the frequency content of a range ofpretrained models on ImageNet [8].

Original: Here we show results on ResNet50 ; results on all ten models wetested on ImageNet are included in Section A.4.
Revised: Here we show results on ResNet50 ; results on all ten models wetested on ImageNet are included in Section A.4.

Original: 5.1 Training Time
Revised: 5.1 Training Time

Original: As we observed for CIFAR-10, our ImageNet models tend to increase the frequency gap betweenthe two types of interpolating paths(within-class and between-class) as training proceeds, withlower-frequencies within-class and higher-frequencies between-class.
Revised: As we observed for CIFAR-10, our ImageNet models tend to increase the frequency gap between thetwo types of interpolating paths as training proceeds, with lower-frequencies within-class and higher- frequencies between-class.

Original: We visualize this trend for ResNet50 in Figure 7 (Left) .
Revised: We visualize this trend for ResNet50 in Figure 7 (Left) .

Original: For our CIFAR-10 models, this increase in frequency difference betweenpath types was achieved by a combination of increasing the average high frequency content onbetween-class paths and decreasing the average high frequency content on within-class paths.
Revised: For our CIFAR-10models, this increase in frequency difference between path types was achieved by a combination ofincreasing the average high frequency content on between-class paths and decreasing the averagehigh frequency content on within-class paths.

Original: On

Original:  ImageNet, we again ﬁnd that average frequency content on between-class paths increases duringtraining, but that within-class frequencies sometimes decrease during training ( e.g. for ResNets)but sometimes increase (slower than between-class paths) during training ( e.g. for CoATNets).
Revised: On ImageNet, we again ﬁnd that average frequencycontent on between-class paths increases during training, but that within-class frequencies sometimesdecrease during training ( e.g. for ResNets) but sometimes increase (slower than between-class paths)during training ( e.g. for CoATNets).

Original: Wespeculate that the trend of decreasing within-class frequency as training time progresses is lessconsistent on ImageNet than CIFAR-10 because, as we explore in the next section, ImageNet classesare more internally diverse than CIFAR-10 classes.
Revised: We speculate that the trend of decreasing within-class frequencyas training time progresses is less consistent on ImageNet than CIFAR-10 because, as we explore inthe next section, ImageNet classes are more internally diverse than CIFAR-10 classes.

Original: 5.2 Class Coherence
Revised: 5.2 Class Coherence

Original: In addition to containing 100 × as many classes as CIFAR-10, ImageNet classes can also be moreinternally diverse.
Revised: In addition to containing 100 × as many classes as CIFAR-10, ImageNet classes can also be moreinternally diverse.

Original: This notion of within-class “consistency” is captured by C-Scores [18], whichassign to each training example a value between 0 and 1 denoting how typical that example is.
Revised: This notion of within-class “consistency” is captured by C-Scores [18], whichassign to each training example a value between 0 and 1 denoting how typical that example is.

Original: Oneway of capturing the internal consistency or diversity of a class is by studying the distribution of its
Revised: Oneway of capturing the internal consistency or diversity of a class is by studying the distribution of its

Original: C-Scores.
Revised: C-Scores.

Original: In Figure 7 (Right) we plot the mean and standard deviation of these scores for each of the1000 ImageNet classes in gray and highlight the 10 classes whose ResNet50 interpolating paths betweenvalidation images are most high-frequency (red stars) and least high-frequency (blue squares).
Revised: In Figure 7 (Right) we plot the mean and standard deviation of these scores for each of the1000 ImageNet classes in gray and highlight the 10 classes whose ResNet50 interpolating paths betweenvalidation images are most high-frequency (red stars) and least high-frequency (blue squares).

Original: We ﬁndthat ResNet50 is lower-frequency on more internally consistent classes and higher-frequency on morediverse classes, supporting the intuition that more diverse classes require more complex functions.
Revised: We ﬁndthat ResNet50 is lower-frequency on more internally consistent classes and higher-frequency on morediverse classes, supporting the intuition that more diverse classes require more complex functions.

Revised: distribution of C-Scores among all 1000 ImageNet classes.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we introduced two methods to measure spectral bias in modern image classiﬁcationneural networks, and applied these methods towards the central question:
Revised: In this paper, we introduced two methods to measure spectral bias in modern image classiﬁcationneural networks, and applied these methods towards the central question:

Original: What kinds of function frequencies are needed for modern neural networks to generalize?
Revised: What kinds of function frequencies are needed for modern neural networks to generalize?

Original: Speciﬁcally, we applied these methods to examine the impact of a variety of training choices on thelearned frequencies.
Revised: Speciﬁcally, we applied these methods to examine the impact of a variety of training choices on thelearned frequencies.

Original: On CIFAR-10, we found that higher-accuracy models typically have greaterfrequency separation between within-class and between-class paths, with lower frequencies withinclass and higher frequencies between-class.
Revised: On CIFAR-10, we found that higher-accuracy models typically have greaterfrequency separation between within-class and between-class paths, with lower frequencies withinclass and higher frequencies between-class.

Original: This trend holds regardless of whether high accuracyis achieved through longer training, choice of model architecture, increase in dataset size, dataaugmentation, or self-distillation.
Revised: This trend holds regardless of whether high accuracyis achieved through longer training, choice of model architecture, increase in dataset size, dataaugmentation, or self-distillation.

Original: On ImageNet, we again found that this frequency separationincreases during training, and noted that models are higher-frequency within more internally diverseclasses.
Revised: On ImageNet, we again found that this frequency separationincreases during training, and noted that models are higher-frequency within more internally diverseclasses.

Original: Our experimental methods offer a window onto the frequency structure of neural networksfor image classiﬁcation, and its relationship to model performance.
Revised: Our experimental methods offer a window onto the frequency structure of neural networks forimage classiﬁcation, and its relationship to model performance.

Revised: In particular, although our primarygoal is to improve our understanding of how neural networks generalize, our work is likely to havepractical implications for improving generalization as well as robustness to image perturbations anddistribution shifts.

Revised: We hope that future work realizes these implications by applying our methodsto study the spectral bias of robust models, and to induce model robustness directly, perhaps byincorporating our metrics as regularizers during training.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Limitations of the label smoothing
Revised: [Yes] Limitations of the label smoothing

Original: methodology are discussed at the end of Section 3.1.
Revised: methodology are discussed at the end of Section 3.1.

Original: Limitations of the interpolation
Revised: Limitations of the interpolation

Original: methodology are discussed in Section A.2.
Revised: methodology are discussed in Section A.2.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?

Original: [Yes] Code isprovided in the supplement.
Revised: [Yes] Code is provided in the supplement.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Some of these details are provided in the appendix, and the restare evident from the code, provided in the supplement.
Revised: [Yes] Some of these details are provided in the appendix, and the rest are evident from the code, provided in the supplement.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running exper- iments multiple times)?

Original: [Yes] Standard errors are reported for linear interpolation
Revised: [Yes] Standard errors are reported for linear interpolation

Original: experiments.
Revised: experiments.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,

Original: type of GPUs, internal cluster, or cloud provider)?
Revised: type of GPUs, internal cluster, or cloud provider)?

Original: [No] These results did require
Revised: [No] These results did require

Original: substantial GPU compute, but we do not quantify the exact amount.
Revised: substantial GPU compute, but we do not quantify the exact amount.

Original: Each point in a
Revised: Each point in a

Revised: label smoothing

Revised: ﬁgure

Original: label smoothing ﬁgure required training a model from scratch, and each value in aninterpolation ﬁgure required evaluating a pretrained model on hundreds of thousands
Revised: required training a model from scratch, and each value in an interpolation ﬁgure required evaluating a pretrained model on hundreds of thousands

Original: of images.
Revised: of images

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite the creators
Revised: [Yes] We cite the creators

Original: of CIFAR-

Revised: of CIFAR-

Original: [21] and ImageNet [8], the image datasets we use, as well as the creators
Revised: [22] and ImageNet [8], the image datasets we use, as well as the creators

Original: of the models we use [44, 12, 42, 5, 16, 10, 39, 7].
Revised: of the models we use [45, 12, 43, 6, 16, 10, 40, 7].

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] The licenses of each dataset and model we use are available on their websites. 
Revised: [No] The licenses of each dataset and modelwe use are available on their websites. [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original:  (d) Did you discuss whether and how consent was obtained from people whose data you’re 
Revised: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original:  (e) Did you discuss whether the data you are using/curating contains personally identiﬁable 
Revised: [Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: A AppendixA.1 Data, Models, and Model Accuracies
Revised: A AppendixA.1 Data, Models, and Model Accuracies

Original: Images are pixel-wise normalized by the mean and standard deviation of the training images for eachdataset, and for ImageNet all images are center cropped and resized to 224 × 224; this preprocessingis done before any interpolating paths are constructed.
Revised: Images are pixel-wise normalized by the mean and standard deviation of the training images for eachdataset, and for ImageNet all images are center cropped and resized to 224 × 224; this preprocessingis done before any interpolating paths are constructed.

Original: For CIFAR-10, we consider six differentconvolutional neural networks: two Wide-ResNets [44] of different sizes ( WideResNet32 and WideResNet160 ), three Shake-Shake regularized networks [12] of different sizes ( ShakeShake32 , ShakeShake96 ,and ShakeShake112 ), and PyramidNet , which uses the even stronger Shake-Drop regularization [42].
Revised: For CIFAR-10, we consider six differentconvolutional neural networks: two Wide-ResNets [45] of different sizes ( WideResNet32 and WideResNet160 ), three Shake-Shake regularized networks [12] of different sizes ( ShakeShake32 , ShakeShake96 ,and ShakeShake112 ), and PyramidNet , which uses the even stronger Shake-Drop regularization [43].

Original: Our implementations are based on Cubuk et al.
Revised: Our implementations are based on Cubuk et al.

Original: [5]; we use the same optimizer (stochastic gradientdescent with momentum) and cosine learning rate schedule.
Revised: [6]; we use the same optimizer (stochastic gradientdescent with momentum) and cosine learning rate schedule.

Original: We train without data augmentation (toensure all models are trained on exactly the same examples), except for experiments that explicitlyvary data augmentation.
Revised: We train without data augmentation (toensure all models are trained on exactly the same examples), except for experiments that explicitlyvary data augmentation.

Original: Without data augmentation, the test accuracies of our models are shown in
Revised: Without data augmentation, the test accuracies of our models are shown in

Original: Table 1.
Revised: Table 1.

Original: [Table]
Revised: [Table]

Original: For ImageNet, we consider ten different pretrained models: three ResNets [16] of different sizes( ResNet50 , ResNet101 , and ResNet152 ), two Vision Transformers [10] of different sizes ( ViT and ViTSmall ),three distilled transformers [39] ( DeiT , DeiTSmall , and DeiTClsToken ), and two CoAtNets [7] ( CoAtNet0and CoAtNet0BF16 ).
Revised: For ImageNet, we consider ten different pretrained models: three ResNets [16] of different sizes( ResNet50 , ResNet101 , and ResNet152 ), two Vision Transformers [10] of different sizes ( ViT and ViTSmall ),three distilled transformers [40] ( DeiT , DeiTSmall , and DeiTClsToken ), and two CoAtNets [7] ( CoAtNet0and CoAtNet0BF16 ).

Original: The top-1 classiﬁcation accuracies of these models are presented in Table 2.
Revised: The top-1 classiﬁcation accuracies of these models are presented in Table 2.

Original: [Table]
Revised: [Table]

Original: A.2 Linear Interpolation: Methodological Details
Revised: A.2 Linear Interpolation: Methodological Details

Original: For each sampled path, we compute the discrete Fourier transform (DFT) of the prediction functionalong the path separately for each of the M class predictions, take the (real) magnitude of the resulting(complex) DFT coefﬁcients, and average them among the M classes.
Revised: For each sampled path, we compute the discrete Fourier transform (DFT) of the prediction functionalong the path separately for each of the M class predictions, take the (real) magnitude of the resulting(complex) DFT coefﬁcients, and average them among the M classes.

Original: We then compute the fractionof this averaged DFT magnitude that is allocated to the high frequency components, and aggregateover the many paths by calculating the mean high frequency fraction and its standard error.
Revised: We then compute the fractionof this averaged DFT magnitude that is allocated to the high frequency components, and aggregateover the many paths by calculating the mean high frequency fraction and its standard error.

Original: We use asimple threshold frequency of 0.05 for both CIFAR10 and ImageNet, although our paths are sampledwith a distance of 1 for CIFAR10 and 7 for ImageNet (since ImageNet images are larger and thusfarther apart).
Revised: We use asimple threshold frequency of 0.05 for both CIFAR10 and ImageNet, although our paths are sampledwith a distance of 1 for CIFAR10 and 7 for ImageNet (since ImageNet images are larger and thusfarther apart).

Original: This precise threshold is not critical to our qualitative results, and different tradeoffs offrequency resolution and computational expense are possible.
Revised: This precise threshold is not critical to our qualitative results, and different tradeoffs offrequency resolution and computational expense are possible.

Original: Limitations of linear interpolation.
Revised: Limitations of linear interpolation.

Original: Like all proxy measurements of spectral bias, linear interpolation is a coarse metric.
Revised: Like all proxy measurements of spectral bias, linear interpolation is a coarse metric.

Original: We only measure the Fourier content of the learned function along speciﬁcpaths, and aggregate results across many paths, across all classes, and only summarize frequenciesinto coarse "low" and "high" bins.
Revised: We only measure the Fourier content of the learned function along speciﬁcpaths, and aggregate results across many paths, across all classes, and only summarize frequenciesinto coarse "low" and "high" bins.

Original: An additional limitation is that the paths we deﬁne interpolatein pixel space, which means that there are images along our interpolating paths that are off thehypothesized manifold of natural images. 
Revised: An additional limitation is that the paths we deﬁne interpolatein pixel space, which means that there are images along our interpolating paths that are off thehypothesized manifold of natural images. [Figure]

Original: A.3 CIFAR-10 full results 
Revised: 3 CIFAR-

Original: [Figure]
Revised: [Figure]

Original: A.3.1 Label Smoothing
Revised: A.3.1 Label Smoothing

Original: [Figure]
Revised: [Figure]

Original: Figure 8 shows the same results as Figure 3 (Left) on all six CIFAR-10 models we tested.
Revised: Figure 8 shows the same results as Figure 3 (Left) on all six CIFAR-10 models we tested.

Original: [Figure]
Revised: [Figure]

Original: A.3.2 Sensitivity to Natural Image Directions
Revised: A.3.2 Sensitivity to Natural Image Directions

Original: Figure 9 shows the same results as Figure 5 on all six CIFAR-10 models we tested.
Revised: Figure 9 shows the same results as Figure 5 on all six CIFAR-10 models we tested.

Original: A.3.3 Agreement Between Label Smoothing and Linear Interpolation
Revised: A.3.3 Agreement Between Label Smoothing and Linear Interpolation

Original: Varying the frequency of the radial wave label smoothing When the frequency of the radialwave used for label smoothing increases, models take more time to ﬁt the smoothing noise.
Revised: Varying the frequency of the radial wave label smoothing When the frequency of the radialwave used for label smoothing increases, models take more time to ﬁt the smoothing noise.

Original: Figure 3(Left) and Figure 8 show this using label smoothing; the corresponding interpolation experiment is shown in Figure 10.
Revised: Figure 3(Left) and Figure 8 show this using label smoothing; the corresponding interpolation experiment is shown in Figure 10.

Original: The target frequencies between 0.035 and 0.04 are close enough that, were themodel to ﬁt each perfectly, the interpolation curves would be nearly visually indistinguishable, aswe can tell from Figure 2 (Left) .
Revised: The target frequencies between 0.035 and 0.04 are close enough that, were themodel to ﬁt each perfectly, the interpolation curves would be nearly visually indistinguishable, aswe can tell from Figure 2 (Left) .

Original: However, the model is actually smoother when trying to ﬁt higherfrequency label smoothing noise, because it ﬁts this noise less well (in addition to ﬁtting it laterin training).
Revised: However, the model is actually smoother when trying to ﬁt higherfrequency label smoothing noise, because it ﬁts this noise less well (in addition to ﬁtting it laterin training).

Original: We can see this effect, for instance, by noting that WideResNet32 , WideResNet160 , and
Revised: We can see this effect, for instance, by noting that WideResNet32 , WideResNet160 , and

Original: ShakeShake32 fail to ﬁt frequency 0.04, both in Figure 8 and Figure 10.
Revised: ShakeShake32 fail to ﬁt frequency 0.04, both in Figure 8 and Figure 10.

Original: It is also worth noting that, in Figure 10 and repeatedly across our interpolation experiments, thelearned function is smoother (lower-frequency) within-class and less smooth (higher-frequency)between-class.
Revised: It is also worth noting that, in Figure 10 and repeatedly across our interpolation experiments, thelearned function is smoother (lower-frequency) within-class and less smooth (higher-frequency)between-class.

Original: This is to be expected since the model must change predictions somewhere along thepath between examples from different classes.
Revised: This is to be expected since the model must change predictions somewhere along thepath between examples from different classes.

Original: Learning low frequencies ﬁrst The label smoothing experiment presented in Figure 3 (Left) and
Revised: Learning low frequencies ﬁrst The label smoothing experiment presented in Figure 3 (Left) and

Original: Figure 8 shows that low frequency target functions are learned earlier than higher frequency targets;we can also conﬁrm this ﬁnding using interpolation with model checkpoints saved at different epochsthroughout training (without any label smoothing) in Figure 11.
Revised: Figure 8 shows that low frequency target functions are learned earlier than higher frequency targets;we can also conﬁrm this ﬁnding using interpolation with model checkpoints saved at different epochsthroughout training (without any label smoothing) in Figure 11.

Original: We see that spectral bias in imageclassiﬁcation is more nuanced than simply increasing in frequency over time: as training proceeds,models become higher-frequency along between-class paths but lower-frequency along within-class
Revised: We see that spectral bias in imageclassiﬁcation is more nuanced than simply increasing in frequency over time: as training proceeds,models become higher-frequency along between-class paths but lower-frequency along within-class

Original: [Figure]
Revised: [Figure]

Original: frequency, which are dominant in natural images.
Revised: , which are dominant in natural images.

Original: Here we show noise ﬁtting when training each model with label smoothing of frequency0.038 in various unit norm direction corresponding to Fourier basis images indexed by frequency k .
Revised: Here we show noise ﬁtting when training each model with label smoothing of frequency in various unit norm direction corresponding to Fourier basis images indexed by frequency k .

Original: [Figure]
Revised: [Figure]

Original: paths.
Revised: paths.

Original: This may indicate that the model is learning an appropriate clustering of the data into its classes,and reducing the functional variation within each cluster.[Figure]
Revised: This may indicate that the model is learning an appropriate clustering of the data into its classes,and reducing the functional variation within each cluster.

Original: A.3.4 Weight Decay
Revised: A.3.4 Weight Decay

Original: [Figure]
Revised: [Figure]

Original: Figure 12 uses the label smoothing methodology on our six CIFAR-10 models to show that weightdecay delays the learning of high frequencies.[Figure]
Revised: Figure 12 uses the label smoothing methodology on our six CIFAR-10 models to show that weightdecay delays the learning of high frequencies.

Original: A.3.5 Training Set Size
Revised: A.3.5 Training Set Size

Original: [Figure]
Revised: [Figure]

Original: Figure 6 (Left) uses linear interpolation to show the within-class regularization (frequency reduction)effect of increasing dataset size for WideResNet160 .
Revised: Figure 6 (Left) uses linear interpolation to show the within-class regularization (frequency reduction)effect of increasing dataset size for WideResNet160 .

Original: Figure 13 shows the same effect on all six CIFAR10 models we tested.[Figure]
Revised: Figure 13 shows the same effect on all six CIFAR10 models we tested.

Original: A.3.6 Data Augmentation
Revised: A.3.6 Data Augmentation

Original: [Figure]
Revised: [Figure]

Original: Figure 6 (Center) uses linear interpolation to study the effect of common data augmentation procedures on the learned frequencies for WideResNet160 .
Revised: Figure 6 (Center) uses linear interpolation to study the effect of common data augmentation procedures on the learned frequencies for WideResNet160 .

Original: Figure 14 shows the same experiment on all sixmodels we tested.
Revised: Figure 14 shows the same experiment on all sixmodels we tested.

Original: Mixup Strength Figure 6 (Center) uses linear interpolation to show that training with Mixup dataaugmentation [46] causes WideResNet160 to learn a within-class lower-frequency function, but toomuch Mixup can produce a higher-frequency function within-class.
Revised: Mixup Strength Figure 6 (Center) uses linear interpolation to show that training with Mixup dataaugmentation [47] causes WideResNet160 to learn a within-class lower-frequency function, but toomuch Mixup can produce a higher-frequency function within-class.

Original: Figure 15 repeats the experimenton all six CIFAR-10 models we tested.
Revised: Figure 15 repeats the experimenton all six CIFAR-10 models we tested.

Original: A.3.7 Self-Distillation
Revised: A.3.7 Self-Distillation

Original: [Figure] Figure 6 (Right) uses linear interpolation to show that self-distillation with ShakeShake96 producesa student model that is lower-frequency within-class (and slightly higher-frequency between-class)than its teacher, and has higher validation accuracy than the teacher.
Revised: Figure 6 (Right) uses linear interpolation to show that self-distillation with ShakeShake96 producesa student model that is lower-frequency within-class (and slightly higher-frequency between-class)than its teacher, and has higher validation accuracy than the teacher.

Original: Figure 16 shows the same resultacross all six CIFAR-10 models we tested.
Revised: Figure 16 shows the same resultacross all six CIFAR-10 models we tested.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A.4 ImageNet FullResults
Revised: ImageNet Full

Original: [Figure]
Revised: [Figure]

Original: [Equation] [Figure] Figure 7 (Left) shows that ResNet50 increases in between-class frequency as training proceeds, whilewithin-class frequencies remain constant or slightly decrease, causing increasing frequency separationbetween these two types of paths.
Revised: Figure 7 (Left) shows that ResNet50 increases in between-class frequency as training proceeds, whilewithin-class frequencies remain constant or slightly decrease, causing increasing frequency separationbetween these two types of paths.

Original: In Figure 17 we show the same experiment for all ten ImageNetmodels we tested.
Revised: In Figure 17 we show the same experiment for all ten ImageNetmodels we tested.

Original: We ﬁnd that the overall frequency trends vary between models, but all modelsincrease in frequency separation between path types during training.
Revised: We ﬁnd that the overall frequency trends vary between models, but all modelsincrease in frequency separation between path types during training.

Original: A.4.2 Class Coherence
Revised: A.4.2 Class Coherence

Original: Figure 7 (Right) shows that ResNet50 has higher frequencies along paths within more internallydiverse classes.
Revised: Figure 7 (Right) shows that ResNet50 has higher frequencies along paths within more internallydiverse classes.

Original: In Figure 18 we show the same experiment for all ten ImageNet models we tested.
Revised: In Figure 18 we show the same experiment for all ten ImageNet models we tested.

Original: Interestingly, the two CoATNets exhibit no clear correlation between class diversity and functionfrequency, and also dramatically increase in function frequency during training (Figure 17), unlikeother models.
Revised: Interestingly, the two CoATNets exhibit no clear correlation between class diversity and functionfrequency, and also dramatically increase in function frequency during training (Figure 17), unlikeother models.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]