Original: S HORTCUT L EARNING T HROUGH THE L ENS OF E ARLY T RAINING D YNAMICS
Revised: S HORTCUT L EARNING T HROUGH THE L ENS OF E ARLY T RAINING D YNAMICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Neural Networks (DNNs) are prone to learn shortcut patterns that damage the generalization of the DNN during deployment.
Revised: Deep Neural Networks (DNNs) are prone to learn shortcut patterns that damage the generalization of the DNN during deployment.

Original: Shortcut Learning is concerning, particularly when the DNNs are applied to safety-critical domains.
Revised: Shortcut Learning is concerning, particularly when the DNNs are applied to safety-critical domains.

Original: This paper aims to better understand shortcut learning through the lens of the learning dynamics of the internal neurons during the training process.
Revised: This paper aims to better understand shortcut learning through the lens of the learning dynamics of the internal neurons during the training process.

Original: More specifically, we make the following observations: (1) While previous works treat shortcuts as synonymous with spurious correlations, we emphasize that not all spurious correlations are shortcuts.
Revised: More specifically, we make the following observations: (1) While previous works treat shortcuts as synonymous with spurious correlations, we emphasize that not all spurious correlations are shortcuts.

Original: We show that shortcuts are only those spurious features that are “easier” than the core features.
Revised: We show that shortcuts are only those spurious features that are “easier” than the core features.

Original: (2) We build upon this premise and use instance difficulty methods (like Prediction Depth (Baldock et al., 2021)) to quantify “easy” and to identify this behavior during the training phase.
Revised: (2) We build upon this premise and use instance difficulty methods (like Prediction Depth (Baldock et al., 2021)) to quantify “easy” and to identify this behavior during the training phase.

Original: (3) We empirically show that shortcut learning can be detected by observing the learning dynamics of the
Revised: (3) We empirically show that shortcut learning can be detected by observing the learning dynamics of the

Original: DNN’s early layers , irrespective of the network architecture used.
Revised: DNN’s early layers , irrespective of the network architecture used.

Original: In other words, easy features learned by the initial layers of a DNN early during the training are potential shortcuts.
Revised: In other words, easy features learned by the initial layers of a DNN early during the training are potential shortcuts.

Original: We verify our claims on simulated and real medical imaging data and justify the empirical success of our hypothesis by showing the theoretical connections between Prediction Depth and information-theoretic concepts like
Revised: We verify our claims on simulated and real medical imaging data and justify the empirical success of our hypothesis by showing the theoretical connections between Prediction Depth and information-theoretic concepts like

Original: V -usable information (Ethayarajh et al., 2021).
Revised: V -usable information (Ethayarajh et al., 2021).

Original: Lastly, our experiments show the insufficiency of monitoring only accuracy plots during training (as is common in machine learning pipelines), and we highlight the need for monitoring early training dynamics using example difficulty metrics.
Revised: Lastly, our experiments show the insufficiency of monitoring only accuracy plots during training (as is common in machine learning pipelines), and we highlight the need for monitoring early training dynamics using example difficulty metrics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Shortcuts are spurious features that perform well on standard benchmarks but fail to generalize to real-world settings (Geirhos et al., 2020).
Revised: Shortcuts are spurious features that perform well on standard benchmarks but fail to generalize to real-world settings (Geirhos et al., 2020).

Original: Deep neural networks (DNNs) tend to rely on shortcuts even in the presence of core features that generalize well, which poses serious problems when deploying them in safety-critical applications such as finance, healthcare, and autonomous driving (Geirhos et al., 2020; Oakden-Rayner et al., 2020; DeGrave et al., 2021).
Revised: Deep neural networks (DNNs) tend to rely on shortcuts even in the presence of core features that generalize well, which poses serious problems when deploying them in safety-critical applications such as finance, healthcare, and autonomous driving (Geirhos et al., 2020; Oakden-Rayner et al., 2020; DeGrave et al., 2021).

Original: Previous works view shortcut learning as a distribution shift problem (Kirichenko et al., 2022;
Revised: Previous works view shortcut learning as a distribution shift problem (Kirichenko et al., 2022;

Original: Wiles et al., 2021; Bellamy et al., 2022; Adnan et al., 2022).
Revised: Wiles et al., 2021; Bellamy et al., 2022; Adnan et al., 2022).

Original: However, we show that not all spurious correlations are shortcuts.
Revised: However, we show that not all spurious correlations are shortcuts.

Original: Models suffer from shortcut learning only when the spurious features are much easier to learn than signals that generalize well.
Revised: Models suffer from shortcut learning only when the spurious features are much easier to learn than signals that generalize well.

Original: We show how monitoring example difficulty metrics like Prediction
Revised: We show how monitoring example difficulty metrics like Prediction

Original: Depth (PD) (Baldock et al., 2021) can reveal valuable insights into shortcut learning quite early during the training process.
Revised: Depth (PD) (Baldock et al., 2021) can reveal valuable insights into shortcut learning quite early during the training process.

Original: Early detection of shortcut learning is useful as it can help develop intervention schemes to fix the shortcut early.
Revised: Early detection of shortcut learning is useful as it can help develop intervention schemes to fix the shortcut early.

Original: To the best of our knowledge, we are the first to detect shortcut learning by monitoring the training dynamics of the model.
Revised: To the best of our knowledge, we are the first to detect shortcut learning by monitoring the training dynamics of the model.

Original: Geirhos et al.
Revised: Geirhos et al.

Original: (2020) define shortcuts as spurious correlations that exist in standard benchmarks but fail to hold in more challenging test conditions, like real-world settings.
Revised: (2020) define shortcuts as spurious correlations that exist in standard benchmarks but fail to hold in more challenging test conditions, like real-world settings.

Original: The emphasis on shortcuts being synonymous with spurious correlations has led to widespread adoption of viewing shortcut learning as a distribution shift problem (Bellamy et al., 2022;
Revised: The emphasis on shortcuts being synonymous with spurious correlations has led to widespread adoption of viewing shortcut learning as a distribution shift problem (Bellamy et al., 2022;

Original: Wiles et al., 2021; Adnan et al., 2022;
Revised: Wiles et al., 2021; Adnan et al., 2022;

Original: Kirichenko et al., 2022).
Revised: Kirichenko et al., 2022).

Original: While the distribution shift explains part of the story, we emphasize that what is equally important for shortcut learning is the difficulty of the spurious features themselves (see Fig-1).
Revised: While the distribution shift explains part of the story, we emphasize that what is equally important for shortcut learning is the difficulty of the spurious features themselves (see Fig-1).

Original: Previous works like Shah et al.
Revised: Previous works like Shah et al.

Original: ; Scimeca et al.
Revised: ; Scimeca et al.

Original: (2021) hint at this.
Revised: (2021) hint at this.

Original: But we take this line of thought further by viewing shortcut learning as a phenomenon that impacts the dataset difficulty, which can be captured by monitoring early training dynamics.
Revised: But we take this line of thought further by viewing shortcut learning as a phenomenon that impacts the dataset difficulty, which can be captured by monitoring early training dynamics.

Original: [Figure]
Revised: [Figure]

Original: Whereas classifying prime numbers, as shown in scenario 1, is challenging.
Revised: Whereas classifying prime numbers, as shown in scenario 1, is challenging.

Original: So the model ignores such spurious features.
Revised: So the model ignores such spurious features.

Original: This shows not all spurious correlations are shortcuts.
Revised: This shows not all spurious correlations are shortcuts.

Original: The premises that support our hypothesis are as follows:
Revised: The premises that support our hypothesis are as follows:

Original: (P1) Shortcuts are only those spurious features that are “easier” to learn than the core features (see Fig-1).
Revised: (P1) Shortcuts are only those spurious features that are “easier” to learn than the core features (see Fig-1).

Original: (P2)
Revised: (P2)

Original: Initial layers of a DNN tend to learn easy features, whereas the later layers tend to learn the harder ones (Zeiler & Fergus, 2014; Baldock et al., 2021).
Revised: Initial layers of a DNN tend to learn easy features, whereas the later layers tend to learn the harder ones (Zeiler & Fergus, 2014; Baldock et al., 2021).

Original: (P3) Easy features are learned much earlier than the harder ones during training (Mangalam & Prabhu, 2019; Rahaman et al., 2019).
Revised: (P3) Easy features are learned much earlier than the harder ones during training (Mangalam & Prabhu, 2019; Rahaman et al., 2019).

Original: Premises (P1-3) lead us to conjecture that: “Easy features learned by the initial layers of a DNN early during the training are potential shortcuts.”
Revised: Premises (P1-3) lead us to conjecture that: “Easy features learned by the initial layers of a DNN early during the training are potential shortcuts.”

Original: We empirically show that our hypothesis works well on simulated and real medical imaging data (section-4.2) regardless of the DNN architecture used.
Revised: We empirically show that our hypothesis works well on simulated and real medical imaging data (section-4.2) regardless of the DNN architecture used.

Original: We justify this empirical success by theoretically connecting prediction depth with information-theoretic concepts like V -usable information (Ethayarajh et al., 2021) (section-3 and appendix-A.1).
Revised: We justify this empirical success by theoretically connecting prediction depth with information-theoretic concepts like V -usable information (Ethayarajh et al., 2021) (section-3 and appendix-A.1).

Original: Lastly, our experiments highlight that monitoring only accuracy during training, as is common in machine learning pipelines, is insufficient.
Revised: Lastly, our experiments highlight that monitoring only accuracy during training, as is common in machine learning pipelines, is insufficient.

Original: Rather we need to monitor the learning dynamics of the model using suitable metrics to detect shortcut learning (section-4.3).
Revised: Rather we need to monitor the learning dynamics of the model using suitable metrics to detect shortcut learning (section-4.3).

Original: This could potentially save a lot of time and computational costs and help develop reliable models that do not rely on spurious features.
Revised: This could potentially save a lot of time and computational costs and help develop reliable models that do not rely on spurious features.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Not all spurious correlations are shortcuts: Geirhos et al.
Revised: Not all spurious correlations are shortcuts: Geirhos et al.

Original: (2020) define shortcuts as spurious correlations that exist in standard benchmarks but fail to hold in more challenging test conditions.
Revised: (2020) define shortcuts as spurious correlations that exist in standard benchmarks but fail to hold in more challenging test conditions.

Original: Wiles et al.
Revised: Wiles et al.

Original: (2021) view shortcut learning as a distribution shift problem where two or more attributes are correlated during training but are independent in the test data.
Revised: (2021) view shortcut learning as a distribution shift problem where two or more attributes are correlated during training but are independent in the test data.

Original: Bellamy et al.
Revised: Bellamy et al.

Original: (2022) use causal diagrams to explain shortcuts as spurious correlations that hold during training but not during deployment.
Revised: (2022) use causal diagrams to explain shortcuts as spurious correlations that hold during training but not during deployment.

Original: All these papers characterize shortcuts purely as a consequence of distribution shift; methods exist to build models robust to such shifts (Arjovsky et al., 2019; Krueger et al., 2021; Puli et al., 2022).
Revised: All these papers characterize shortcuts purely as a consequence of distribution shift; methods exist to build models robust to such shifts (Arjovsky et al., 2019; Krueger et al., 2021; Puli et al., 2022).

Original: In contrast, we stress that not all spurious correlations are shortcuts.
Revised: In contrast, we stress that not all spurious correlations are shortcuts.

Original: Rather only those spurious features that are easier than the core features are potential shortcuts (see Fig-1).
Revised: Rather only those spurious features that are easier than the core features are potential shortcuts (see Fig-1).

Original: Previous works like Shah et al.
Revised: Previous works like Shah et al.

Original: ; Scimeca et al.
Revised: ; Scimeca et al.

Original: (2021) hint at this by saying that DNNs are biased towards simple solutions, and Dagaev et al.
Revised: (2021) hint at this by saying that DNNs are biased towards simple solutions, and Dagaev et al.

Original: (2021) use the “too-good-to-be-true” prior to emphasize that simple solutions are unlikely to be valid across contexts.
Revised: (2021) use the “too-good-to-be-true” prior to emphasize that simple solutions are unlikely to be valid across contexts.

Original: Veitch et al.
Revised: Veitch et al.

Original: (2021) distinguish various model features using tools from causality and stress test the models for counterfactual invariance.
Revised: (2021) distinguish various model features using tools from causality and stress test the models for counterfactual invariance.

Original: Other works in natural language inference, visual question answering, and action recognition, also assume that simple solutions could be potential shortcuts (Sanh et al., 2020; Li & Vasconcelos, 2019;
Revised: Other works in natural language inference, visual question answering, and action recognition, also assume that simple solutions could be potential shortcuts (Sanh et al., 2020; Li & Vasconcelos, 2019;

Original: Clark et al., 2019; Cadene et al., 2019; He et al., 2019).
Revised: Clark et al., 2019; Cadene et al., 2019; He et al., 2019).

Original: We take this line of thought further by viewing shortcuts as simple solutions or, more explicitly, easy features, which affect the early training dynamics of the model.
Revised: We take this line of thought further by viewing shortcuts as simple solutions or, more explicitly, easy features, which affect the early training dynamics of the model.

Original: We suggest using suitable example difficulty metrics to measure this effect.
Revised: We suggest using suitable example difficulty metrics to measure this effect.

Original: Estimating Example Difficulty: There are different metrics in the literature for measuring instance-specific difficulty (Agarwal et al., 2022;
Revised: Estimating Example Difficulty: There are different metrics in the literature for measuring instance-specific difficulty (Agarwal et al., 2022;

Original: Hooker et al., 2019; Lalor et al., 2018).
Revised: Hooker et al., 2019; Lalor et al., 2018).

Original: Jiang et al.
Revised: Jiang et al.

Original: (2020) train many models on data subsets of varying sizes to estimate a consistency score that captures the probability of predicting the true label for a particular example.
Revised: (2020) train many models on data subsets of varying sizes to estimate a consistency score that captures the probability of predicting the true label for a particular example.

Original: Toneva et al.
Revised: Toneva et al.

Original: (2018) define example difficulty as the minimum number of iterations needed for a particular example to be predicted correctly in all subsequent iterations.
Revised: (2018) define example difficulty as the minimum number of iterations needed for a particular example to be predicted correctly in all subsequent iterations.

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2022) propose a VoG (variance-ofgradients) score which captures example difficulty by averaging the pre-softmax activation gradients across training checkpoints and image pixels.
Revised: (2022) propose a VoG (variance-ofgradients) score which captures example difficulty by averaging the pre-softmax activation gradients across training checkpoints and image pixels.

Original: Feldman & Zhang (2020) use a statistical viewpoint of measuring example difficulty and develop influence functions to estimate the actual leave-one-out influences for various examples.
Revised: Feldman & Zhang (2020) use a statistical viewpoint of measuring example difficulty and develop influence functions to estimate the actual leave-one-out influences for various examples.

Original: Ethayarajh et al.
Revised: Ethayarajh et al.

Original: (2021) use an information-theoretic approach to propose a metric called pointwise V -usable information (PVI) to compute example difficulty.
Revised: (2021) use an information-theoretic approach to propose a metric called pointwise V -usable information (PVI) to compute example difficulty.

Original: Baldock et al.
Revised: Baldock et al.

Original: (2021) define prediction depth (PD) as the minimum number of layers required by the
Revised: (2021) define prediction depth (PD) as the minimum number of layers required by the

Original: DNN to classify a given input and use this to compute instance difficulty.
Revised: DNN to classify a given input and use this to compute instance difficulty.

Original: In our experiments, we use the PD metric to provide a proof of concept for our hypothesis.
Revised: In our experiments, we use the PD metric to provide a proof of concept for our hypothesis.

Original: Monitoring Training Dynamics: Other works that monitor training dynamics have a significantly different goal than ours.
Revised: Monitoring Training Dynamics: Other works that monitor training dynamics have a significantly different goal than ours.

Original: While we monitor training dynamics to detect shortcut learning, Rabanser et al.
Revised: While we monitor training dynamics to detect shortcut learning, Rabanser et al.

Original: (2022) use neural network training dynamics for selective classification.
Revised: (2022) use neural network training dynamics for selective classification.

Original: They use the disagreement between the ground truth label and the intermediate model predictions to reject examples and obtain a favorable accuracy/coverage trade-off.
Revised: They use the disagreement between the ground truth label and the intermediate model predictions to reject examples and obtain a favorable accuracy/coverage trade-off.

Original: Feng & Tu (2021) use a statistical physics-based approach to study the training dynamics of stochastic gradient descent (SGD).
Revised: Feng & Tu (2021) use a statistical physics-based approach to study the training dynamics of stochastic gradient descent (SGD).

Original: While they study the effect of mislabeled data on SGD training dynamics, we study the effect of shortcut learning on the early-time learning dynamics of DNNs.
Revised: While they study the effect of mislabeled data on SGD training dynamics, we study the effect of shortcut learning on the early-time learning dynamics of DNNs.

Original: Hu et al.
Revised: Hu et al.

Original: (2020) use the early training dynamics of neural networks to show that a simple linear model can often mimic the learning of a two-layer fully connected neural network.
Revised: (2020) use the early training dynamics of neural networks to show that a simple linear model can often mimic the learning of a two-layer fully connected neural network.

Original: Adnan et al.
Revised: Adnan et al.

Original: (2022) have a similar goal as ours and use mutual information to monitor shortcut learning.
Revised: (2022) have a similar goal as ours and use mutual information to monitor shortcut learning.

Original: However, computing mutual information is intractable for high-dimensional data, and hence their work is only limited to infinite-width neural networks that offer tractable bounds for this computation.
Revised: However, computing mutual information is intractable for high-dimensional data, and hence their work is only limited to infinite-width neural networks that offer tractable bounds for this computation.

Original: Our work, on the contrary, is more general and holds for different neural network architectures and datasets.
Revised: Our work, on the contrary, is more general and holds for different neural network architectures and datasets.

Original:  B ACKGROUND AND M ETHODOLOGY
Revised:  B ACKGROUND AND M ETHODOLOGY

Original: In this section, we formalize the notion of spurious features, shortcuts, and task difficulty.
Revised: In this section, we formalize the notion of spurious features, shortcuts, and task difficulty.

Original: Let P tr and P te be the training and test distributions defined over the random variables X (input), y (label), and s ( latent spurious feature).
Revised: Let P tr and P te be the training and test distributions defined over the random variables X (input), y (label), and s ( latent spurious feature).

Original: Definition-1 (Spurious Feature s ):
Revised: Definition-1 (Spurious Feature s ):

Original: A latent feature s is called spurious if it is correlated with label y in the training data but not in the test data.
Revised: A latent feature s is called spurious if it is correlated with label y in the training data but not in the test data.

Original: Specifically, the joint probability distributions
Revised: Specifically, the joint probability distributions

Original: P tr and P te can be factorized as follows.
Revised: P tr and P te can be factorized as follows.

Original: [Equation]
Revised: [Equation]

Original: The variable s appears to be causally related to y but is not.
Revised: The variable s appears to be causally related to y but is not.

Original: This is shown in Fig-1.
Revised: This is shown in Fig-1.

Original: We also need a notion of task difficulty.
Revised: We also need a notion of task difficulty.

Original: The difficulty of a task depends on the model and data distribution ( X , y ).
Revised: The difficulty of a task depends on the model and data distribution ( X , y ).

Original: Definition-2 (Task Difficulty Ψ ): Let Ψ P M
Revised: Definition-2 (Task Difficulty Ψ ): Let Ψ P M

Original: ( X → y ) indicates the difficulty of predicting X → y for a model M , where X , y ∼ P .
Revised: ( X → y ) indicates the difficulty of predicting X → y for a model M , where X , y ∼ P .

Original: Consider a joint distribution ( X , y , t ) ∼ P for two tasks, t , and y .
Revised: Consider a joint distribution ( X , y , t ) ∼ P for two tasks, t , and y .

Original: Then, Ψ P M ( X → y ) >
Revised: Then, Ψ P M ( X → y ) >

Original: Ψ P M ( X → t ) indicates that the task X → y is harder than X → t for a given model M .
Revised: Ψ P M ( X → t ) indicates that the task X → y is harder than X → t for a given model M .

Original: Definition-3 (Shortcut): The spurious feature s is a potential shortcut for model M iff Ψ P
Revised: Definition-3 (Shortcut): The spurious feature s is a potential shortcut for model M iff Ψ P

Original: tr M ( X → y ) >
Revised: tr M ( X → y ) >

Original: Ψ P tr M ( X → s ) .
Revised: Ψ P tr M ( X → s ) .

Original: In other words, given the input X , predicting spurious feature s is easier for
Revised: In other words, given the input X , predicting spurious feature s is easier for

Original: M than predicting the true label y .
Revised: M than predicting the true label y .

Original: Note that we use the term “potential shortcut” because, in the presence of multiple spurious features,
Revised: Note that we use the term “potential shortcut” because, in the presence of multiple spurious features,

Original: Scimeca et al.
Revised: Scimeca et al.

Original: (2021) empirically show that the model selectively favors the easiest spurious feature.
Revised: (2021) empirically show that the model selectively favors the easiest spurious feature.

Original: We now explain two metrics (Prediction Depth and V -Usable Information) to measure Ψ P M .
Revised: We now explain two metrics (Prediction Depth and V -Usable Information) to measure Ψ P M .

Original: Baldock et al.
Revised: Baldock et al.

Original: (2021) proposed the notion of Prediction Depth (PD) to estimate example difficulty.
Revised: (2021) proposed the notion of Prediction Depth (PD) to estimate example difficulty.

Original: The PD of input is defined as the minimum number of layers required by the model to classify the input.
Revised: The PD of input is defined as the minimum number of layers required by the model to classify the input.

Original: Our work relies on monitoring early training dynamics using example difficulty metrics, and we use PD estimation for this purpose.
Revised: Our work relies on monitoring early training dynamics using example difficulty metrics, and we use PD estimation for this purpose.

Original: The PD metric is also well-suited for our work as it is defined for easy inputs even when the DNN is not fully trained.
Revised: The PD metric is also well-suited for our work as it is defined for easy inputs even when the DNN is not fully trained.

Original: We use a binary classification setting to explain the concepts used in this section.
Revised: We use a binary classification setting to explain the concepts used in this section.

Original: Notion of Prediction Depth: The PD is defined by building k -NN classifiers on the embedding layers of the model.
Revised: Notion of Prediction Depth: The PD is defined by building k -NN classifiers on the embedding layers of the model.

Original: The PD is simply the earliest layer after which all subsequent k -NN predictions remain the same (0 or 1):
Revised: The PD is simply the earliest layer after which all subsequent k -NN predictions remain the same (0 or 1):

Original: [Equation]
Revised: [Equation]

Original: f knn is a k -NN classifier that outputs 0 or 1 based on a given threshold (see Appendix-A.4 for details), ϕ i is the feature embedding for the given input at
Revised: f knn is a k -NN classifier that outputs 0 or 1 based on a given threshold (see Appendix-A.4 for details), ϕ i is the feature embedding for the given input at

Original: layer-
Revised: layer-

Original: i , and N is the index of the final layer of the model.
Revised: i , and N is the index of the final layer of the model.

Original: The lower the PD of input, the easier it is to classify.
Revised: The lower the PD of input, the easier it is to classify.

Original: We also use the notion of undefined PD to work with models that are not fully trained.
Revised: We also use the notion of undefined PD to work with models that are not fully trained.

Original: We treat k -NN predictions close to 0.
Revised: We treat k -NN predictions close to 0.

Original: (for a binary classification setting) as invalid.
Revised: (for a binary classification setting) as invalid.

Original: If the k -NN predictions for the last three layers (for a given input to the model) are invalid, we treat the PD of the input as undefined (see Appendix-A.4).
Revised: If the k -NN predictions for the last three layers (for a given input to the model) are invalid, we treat the PD of the input as undefined (see Appendix-A.4).

Original: Figure-2 illustrates how to read the PD plots used in our experiment.
Revised: Figure-2 illustrates how to read the PD plots used in our experiment.

Original: [Figure]
Revised: [Figure]

Original: with undefined PD, and the dotted vertical line indicates the mean PD of the plot.
Revised: with undefined PD, and the dotted vertical line indicates the mean PD of the plot.

Original: Notice that the undefined samples (shown in red) slowly accumulate in layer 88 as training progresses.
Revised: Notice that the undefined samples (shown in red) slowly accumulate in layer 88 as training progresses.

Original: This is because the model needs more time to learn the challenging samples which accumulate at higher prediction depth, i.e., later layers.
Revised: This is because the model needs more time to learn the challenging samples which accumulate at higher prediction depth, i.e., later layers.

Original: Notion of V -Usable Information: The Mutual Information between input and output, I ( X ; Y ) , is invariant with respect to lossless encryption of the input, i.e., I ( τ ( X ); Y )
Revised: Notion of V -Usable Information: The Mutual Information between input and output, I ( X ; Y ) , is invariant with respect to lossless encryption of the input, i.e., I ( τ ( X ); Y )

Original: =
Revised: =

Original: I ( X ; Y ) .
Revised: I ( X ; Y ) .

Original: Such a definition assumes unbounded computation and is counter-intuitive to define task difficulty as heavy encryption of X does not change the task difficulty.
Revised: Such a definition assumes unbounded computation and is counter-intuitive to define task difficulty as heavy encryption of X does not change the task difficulty.

Original: The notion of “Usable Information” introduced by Xu et al.
Revised: The notion of “Usable Information” introduced by Xu et al.

Original: (2020) assumes bounded computation based on the model family V under consideration.
Revised: (2020) assumes bounded computation based on the model family V under consideration.

Original: Usable information is measured under a framework called predictive V -information (Xu et al., 2020).
Revised: Usable information is measured under a framework called predictive V -information (Xu et al., 2020).

Original: Ethayarajh et al.
Revised: Ethayarajh et al.

Original: (2021) introduce pointwise V -information (PVI) for measuring example difficulty.
Revised: (2021) introduce pointwise V -information (PVI) for measuring example difficulty.

Original: [Equation]
Revised: [Equation]

Original: The function g is trained on ( ϕ, y ) input-label pairs, where ϕ is a null input that provides no information about the label y .
Revised: The function g is trained on ( ϕ, y ) input-label pairs, where ϕ is a null input that provides no information about the label y .

Original: g ′ is trained on ( x, y ) pairs from the training data.
Revised: g ′ is trained on ( x, y ) pairs from the training data.

Original: Lower PVI instances are harder for V and vice-versa.
Revised: Lower PVI instances are harder for V and vice-versa.

Original: Since the first term in Eq-4 corresponding to g is independent of the input x , we only consider the second term having g ′ in our experiments.
Revised: Since the first term in Eq-4 corresponding to g is independent of the input x , we only consider the second term having g ′ in our experiments.

Original: In what follows, we relate the notions of PD and V -usable information.
Revised: In what follows, we relate the notions of PD and V -usable information.

Original: We use V cnn (of finite depth and width) in our proof as our experiments mainly use CNN architectures.
Revised: We use V cnn (of finite depth and width) in our proof as our experiments mainly use CNN architectures.

Original: Proposition 1: (Informal) Consider two datasets: D s ∼ P tr ( X , y ) with shortcuts and D i ∼
Revised: Proposition 1: (Informal) Consider two datasets: D s ∼ P tr ( X , y ) with shortcuts and D i ∼

Original: P te ( X , y ) without shortcuts.
Revised: P te ( X , y ) without shortcuts.

Original: For some mild assumptions on PD (see Appendix-A.1), if the mean
Revised: For some mild assumptions on PD (see Appendix-A.1), if the mean

Original: PD of D s is less than the mean PD of D i , then the V cnn -usable-information for D s is larger than the
Revised: PD of D s is less than the mean PD of D i , then the V cnn -usable-information for D s is larger than the

Original: V cnn -usable-information for D
Revised: V cnn -usable-information for D

Original: i : I D s V cnn
Revised: i : I D s V cnn

Original: ( X → Y ) >
Revised: ( X → Y ) >

Original: I D i V cnn ( X → Y ) .
Revised: I D i V cnn ( X → Y ) .

Original: See proof in Appendix-A.1.
Revised: See proof in Appendix-A.1.

Original: The proposition intuitively implies that a sufficient gap between the mean PD of shortcuts and core features can result in shortcuts having more V cnn -usable information than core features.
Revised: The proposition intuitively implies that a sufficient gap between the mean PD of shortcuts and core features can result in shortcuts having more V cnn -usable information than core features.

Original: In such a scenario, the model will be more inclined to learn shortcuts over core features.
Revised: In such a scenario, the model will be more inclined to learn shortcuts over core features.

Original: This proposition justifies using the PD metric to detect shortcut learning, as demonstrated in the following experiments.
Revised: This proposition justifies using the PD metric to detect shortcut learning, as demonstrated in the following experiments.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We set up four experiments to evaluate our hypothesis.
Revised: We set up four experiments to evaluate our hypothesis.

Original: First , we consider two kinds of datasets, one where the spurious feature is easier than the core feature and another where the spurious feature is harder.
Revised: First , we consider two kinds of datasets, one where the spurious feature is easier than the core feature and another where the spurious feature is harder.

Original: We train a classifier on each dataset and observe that the model can learn the easy spurious feature but not the harder one.
Revised: We train a classifier on each dataset and observe that the model can learn the easy spurious feature but not the harder one.

Original: This experiment demonstrates that not all spurious correlations are shortcuts, but only those spurious features that are easier than the core features are potential shortcuts.
Revised: This experiment demonstrates that not all spurious correlations are shortcuts, but only those spurious features that are easier than the core features are potential shortcuts.

Original: Second , we use toy and multiple medical datasets to demonstrate how monitoring the learning dynamics of the initial layers can reveal suspicious shortcut learning activity.
Revised: Second , we use toy and multiple medical datasets to demonstrate how monitoring the learning dynamics of the initial layers can reveal suspicious shortcut learning activity.

Original: We show that an early peak in the PD plot indicates potential shortcuts, and visualization techniques like grad-CAM can provide intuition about the feature used at that layer.
Revised: We show that an early peak in the PD plot indicates potential shortcuts, and visualization techniques like grad-CAM can provide intuition about the feature used at that layer.

Original: Third , we show how shortcuts can often be detected relatively early during training.
Revised: Third , we show how shortcuts can often be detected relatively early during training.

Original: This is because initial layers which learn the shortcuts converge very early during the training.
Revised: This is because initial layers which learn the shortcuts converge very early during the training.

Original: We observe this by monitoring PD plots across training epochs.
Revised: We observe this by monitoring PD plots across training epochs.

Original: In all of our experiments, the shortcut is revealed by the PD plot within two epochs of training.
Revised: In all of our experiments, the shortcut is revealed by the PD plot within two epochs of training.

Original: Fourth , we show that datasets with shortcuts have more “usable information” (Ethayarajh et al., 2021) compared to their counterparts without shortcuts.
Revised: Fourth , we show that datasets with shortcuts have more “usable information” (Ethayarajh et al., 2021) compared to their counterparts without shortcuts.

Original: Due to higher usable information, the model requires fewer layers to classify the images with shortcuts.
Revised: Due to higher usable information, the model requires fewer layers to classify the images with shortcuts.

Original: We use this experiment to empirically justify Proposition-1 outlined in Appendix-A.1.
Revised: We use this experiment to empirically justify Proposition-1 outlined in Appendix-A.1.

Original: 4. N OT ALL SPURIOUS CORRELATIONS ARE SHORTCUTS
Revised: 4. N OT ALL SPURIOUS CORRELATIONS ARE SHORTCUTS

Original: We use the Dominoes binary classification dataset (formed by concatenating two datasets vertically; see Fig-3) similar to the setup of Kirichenko et al.
Revised: We use the Dominoes binary classification dataset (formed by concatenating two datasets vertically; see Fig-4.1) similar to the setup of Kirichenko et al.

Original: The bottom (top) image acts as the core (spurious) feature.
Revised: The bottom (top) image acts as the core (spurious) feature.

Original: Images are of size 64 × 32 .
Revised: Images are of size 64 × 32 .

Original: We construct three pairs of domino datasets such that each pair has both a hard and an easy spurious feature with respect to the common core feature (see Table-1).
Revised: We construct three pairs of domino datasets such that each pair has both a hard and an easy spurious feature with respect to the common core feature (see Table-1).

Original: We use classes { 0,1 } for MNIST and SVHN, { coat,dress } for FMNIST, and { airplane, automobile } for CIFAR10.
Revised: We use classes { 0,1 } for MNIST and SVHN, { coat,dress } for FMNIST, and { airplane, automobile } for CIFAR10.

Original: We also include two classes from Kuzushiji-MNIST (or
Revised: We also include two classes from Kuzushiji-MNIST (or

Original: KMNIST)
Revised: KMNIST)

Original: and construct a modification of this dataset called KMNpatch, which has a spurious patch feature (5x5 white patch on the top-left corner) for one of the two classes of KMNIST.
Revised: and construct a modification of this dataset called KMNpatch, which has a spurious patch feature (5x5 white patch on the top-left corner) for one of the two classes of KMNIST.

Original: The spurious features are perfectly correlated with the target.
Revised: The spurious features are perfectly correlated with the target.

Original: The order of dataset difficulty based on the meanPD is as follows: KMNpatch(1.1) <
Revised: The order of dataset difficulty based on the meanPD is as follows: KMNpatch(1.1) <

Original: MNIST(2.2) < FMNIST(3.9) < KMNIST(5)
Revised: MNIST(2.2) < FMNIST(3.9) < KMNIST(5)

Original: <
Revised: <

Original: SVHN(5.9) < CIFAR10(6.8).
Revised: SVHN(5.9) < CIFAR10(6.8).

Original: We use a ResNet18 model and measure the test and core-only accuracies.
Revised: We use a ResNet18 model and measure the test and core-only accuracies.

Original: The test accuracy is measured on a held-out dataset sampled from the same distribution.
Revised: The test accuracy is measured on a held-out dataset sampled from the same distribution.

Original: For the coreonly accuracy, we blank out the spurious feature (top-half image) by replacing it with zeroes (same as Kirichenko et al.
Revised: For the coreonly accuracy, we blank out the spurious feature (top-half image) by replacing it with zeroes (same as Kirichenko et al.

Original: The higher the core-only accuracy, the lesser the model’s reliance on the spurious feature.
Revised: The higher the core-only accuracy, the lesser the model’s reliance on the spurious feature.

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: From Table-1, we observe that all datasets achieve a high test accuracy as expected.
Revised: From Table-1, we observe that all datasets achieve a high test accuracy as expected.

Original: The core-only accuracy stays high ( > 98 %) for datasets where the spurious feature is harder to learn than the core feature, indicating the model’s high reliance on core features.
Revised: The core-only accuracy stays high ( > 98 %) for datasets where the spurious feature is harder to learn than the core feature, indicating the model’s high reliance on core features.

Original: When the spurious feature is easier than the core, the model learns to leverage them, and hence the core-only accuracy drops to nearly random chance ( 50%).
Revised: When the spurious feature is easier than the core, the model learns to leverage them, and hence the core-only accuracy drops to nearly random chance ( 50%).

Original: This experiment demonstrates that a spurious feature that is harder than the core feature fails to act as a shortcut that the model can leverage to “cheat”.
Revised: This experiment demonstrates that a spurious feature that is harder than the core feature fails to act as a shortcut that the model can leverage to “cheat”.

Original: 4. M ONITORING I NITIAL L AYERS C AN R EVEAL S USPICIOUS S HORTCUT L EARNING A CTIVITY
Revised: 4. M ONITORING I NITIAL L AYERS C AN R EVEAL S USPICIOUS S HORTCUT L EARNING A CTIVITY

Original: Synthetic Shortcut on Toy Dataset: To provide a proof of concept, we demonstrate our method on the Kuzushiji-MNIST (KMNIST)
Revised: Synthetic Shortcut on Toy Dataset: To provide a proof of concept, we demonstrate our method on the Kuzushiji-MNIST (KMNIST)

Original: (Clanuwat et al., 2018) dataset comprising Japanese Hiragana characters.
Revised: (Clanuwat et al., 2018) dataset comprising Japanese Hiragana characters.

Original: The dataset has ten classes and images of size 28 × 28 , similar to MNIST.
Revised: The dataset has ten classes and images of size 28 × 28 , similar to MNIST.

Original: We insert a white patch (shortcut) at a particular location for each of the ten classes.
Revised: We insert a white patch (shortcut) at a particular location for each of the ten classes.

Original: The location of the patch is class-specific.
Revised: The location of the patch is class-specific.

Original: We train two VGG16 models, one on the KMNIST with a patch shortcut ( M sh ) and another on the original KMNIST without the patch ( M orig ).
Revised: We train two VGG16 models, one on the KMNIST with a patch shortcut ( M sh ) and another on the original KMNIST without the patch ( M orig ).

Original: Fig-4 shows the prediction depth (PD) plots for this experiment.
Revised: Fig-4 shows the prediction depth (PD) plots for this experiment.

Original: The vertical dashed lines show the mean PD for each plot.
Revised: The vertical dashed lines show the mean PD for each plot.

Original: Intuitively KMNIST should be harder than MNIST but easier than CIFAR10.
Revised: Intuitively KMNIST should be harder than MNIST but easier than CIFAR10.

Original: But introducing the white patch makes KMNIST easier than even MNIST for M sh (see Fig - 4A & 4D).
Revised: But introducing the white patch makes KMNIST easier than even MNIST for M sh (see Fig - 4A & 4D).

Original: The white patch is a very easy feature, and hence the model only needs a single layer to detect it.
Revised: The white patch is a very easy feature, and hence the model only needs a single layer to detect it.

Original: The Grad-CAM maps for the layer-1 show that M sh focuses mainly on the patch (see Fig-4D), and hence the test accuracy on the original KMNIST images is very low ( 8%).
Revised: The Grad-CAM maps for the layer-1 show that M sh focuses mainly on the patch (see Fig-4D), and hence the test accuracy on the original KMNIST images is very low ( 8%).

Original: The PD plot for
Revised: The PD plot for

Original: M orig (see Fig-4E) is not as skewed toward lower depth as the plot for M sh .
Revised: M orig (see Fig-4E) is not as skewed toward lower depth as the plot for M sh .

Original: This is expected as
Revised: This is expected as

Original: M orig is not looking at the shortcut and therefore utilizes more layers to make the prediction.
Revised: M orig is not looking at the shortcut and therefore utilizes more layers to make the prediction.

Original: The mean PD for M orig suggests that the original KMNIST is harder than Fashion-MNIST but easier than CIFAR10, which is intuitive.
Revised: The mean PD for M orig suggests that the original KMNIST is harder than Fashion-MNIST but easier than CIFAR10, which is intuitive.

Original: M orig also achieves a higher test accuracy ( 98%).
Revised: M orig also achieves a higher test accuracy ( 98%).

Original: This experiment demonstrates how models that learn shortcuts ( M sh ) exhibit PD plots that are suspiciously skewed towards the initial layers.
Revised: This experiment demonstrates how models that learn shortcuts ( M sh ) exhibit PD plots that are suspiciously skewed towards the initial layers.

Original: If the dataset is sufficiently difficult, the skewed PD plot should raise concerns.
Revised: If the dataset is sufficiently difficult, the skewed PD plot should raise concerns.

Original: Visualization techniques like Grad-CAM can further aid our intuition about what shortcut the model is utilizing at any given layer.
Revised: Visualization techniques like Grad-CAM can further aid our intuition about what shortcut the model is utilizing at any given layer.

Original: Semi-Synthetic Shortcut on Medical Datasets: We follow the procedure by DeGrave et al.
Revised: Semi-Synthetic Shortcut on Medical Datasets: We follow the procedure by DeGrave et al.

Original: (2021) to create the ChestX-ray14/GitHub-COVID dataset.
Revised: (2021) to create the ChestX-ray14/GitHub-COVID dataset.

Original: This dataset comprises Covid19 positive images from Github Covid repositories and negative images from ChestX-ray14 dataset (Wang et al., 2017b).
Revised: This dataset comprises Covid19 positive images from Github Covid repositories and negative images from ChestX-ray14 dataset (Wang et al., 2017b).

Original: In addition, we also create the Chex-MIMIC dataset following the procedure by Puli et al.
Revised: In addition, we also create the Chex-MIMIC dataset following the procedure by Puli et al.

Original: This dataset comprises 90% images of Pneumonia from Chexpert (Irvin et al., 2019) and 90% healthy images from MIMIC-CXR (Johnson et al., 2019).
Revised: This dataset comprises 90% images of Pneumonia from Chexpert (Irvin et al., 2019) and 90% healthy images from MIMIC-CXR (Johnson et al., 2019).

Original: We train two DenseNet121 models, M covid on the ChestX-ray14/GitHub-COVID dataset, and M chex on the Chex-MIMIC dataset.
Revised: We train two DenseNet121 models, M covid on the ChestX-ray14/GitHub-COVID dataset, and M chex on the Chex-MIMIC dataset.

Original: We use DenseNet121, a common and standard architecture for medical image analysis.
Revised: We use DenseNet121, a common and standard architecture for medical image analysis.

Original: Images are resized to 512 × 512 .
Revised: Images are resized to 512 × 512 .

Original: [Figure]
Revised: [Figure]

Original: Fig-5 shows the PD plots for M chex and M covid .
Revised: Fig-5 shows the PD plots for M chex and M covid .

Original: Both the plots are highly skewed towards initial layers, similar to the KMNIST with patch shortcut in Fig-4D.
Revised: Both the plots are highly skewed towards initial layers, similar to the KMNIST with patch shortcut in Fig-4D.

Original: This again indicates that the models are using very easy features to make the predictions, which is counterintuitive as the two tasks (pneumonia and covid19 detection) are hard tasks even for humans.
Revised: This again indicates that the models are using very easy features to make the predictions, which is counterintuitive as the two tasks (pneumonia and covid19 detection) are hard tasks even for humans.

Original: Examining the Grad-CAM maps at layer-4 reveals that these models focus on irrelevant spurious features outside the lung region.
Revised: Examining the Grad-CAM maps at layer-4 reveals that these models focus on irrelevant spurious features outside the lung region.

Original: This raises concern because both diseases are known to affect mainly the lungs.
Revised: This raises concern because both diseases are known to affect mainly the lungs.

Original: The reason for this suspicious behavior is that, in both these datasets, the healthy and diseased samples have been acquired from two different sources.
Revised: The reason for this suspicious behavior is that, in both these datasets, the healthy and diseased samples have been acquired from two different sources.

Original: This creates a shortcut because source-specific attributes or tokens are predictive of the disease and can be easily learned, as pointed out by DeGrave et al.
Revised: This creates a shortcut because source-specific attributes or tokens are predictive of the disease and can be easily learned, as pointed out by DeGrave et al.

Original: Real Shortcut on Medical Dataset: For this experiment, we use the NIH dataset (Wang et al., 2017a) which has the popular chest drain shortcut (for Pneumothorax detection)
Revised: Real Shortcut on Medical Dataset: For this experiment, we use the NIH dataset (Wang et al., 2017a) which has the popular chest drain shortcut (for Pneumothorax detection)

Original: (Oakden-Rayner et al., 2020).
Revised: (Oakden-Rayner et al., 2020).

Original: Chest drains are used to treat positive Pneumothorax cases.
Revised: Chest drains are used to treat positive Pneumothorax cases.

Original: The presence of a chest drain in the lung is, therefore, positively correlated with the presence of Pneumothorax and can be used by the deep learning model (Oakden-Rayner et al., 2020).
Revised: The presence of a chest drain in the lung is, therefore, positively correlated with the presence of Pneumothorax and can be used by the deep learning model (Oakden-Rayner et al., 2020).

Original: Appendix-A.3 outlines the procedure we use to obtain chest drain annotations for the NIH dataset.
Revised: Appendix-A.3 outlines the procedure we use to obtain chest drain annotations for the NIH dataset.

Original: We train a DenseNet121 model ( M nih ) for Pneumothorax detection on NIH images of size 128 × 128 .
Revised: We train a DenseNet121 model ( M nih ) for Pneumothorax detection on NIH images of size 128 × 128 .

Original: [Figure]
Revised: [Figure]

Original: Fig-6A shows the PD plot for M nih .
Revised: Fig-6A shows the PD plot for M nih .

Original: We observe that the distribution is not as skewed as the plots in the previous experiments.
Revised: We observe that the distribution is not as skewed as the plots in the previous experiments.

Original: This is because all the images come from a single dataset (NIH).
Revised: This is because all the images come from a single dataset (NIH).

Original: But we see two suspicious peaks at layers - 4 & 40.
Revised: But we see two suspicious peaks at layers - 4 & 40.

Original: Pneumothorax classification is challenging even for radiologists, and hence peaks at initial layers raise suspicion.
Revised: Pneumothorax classification is challenging even for radiologists, and hence peaks at initial layers raise suspicion.

Original: The Grad-CAM maps in Figs -6B &
Revised: The Grad-CAM maps in Figs -6B &

Original: [Figure]
Revised: [Figure]

Original: 6C reveal that layer-4 looks at irrelevant artifacts in the chest X-ray image, whereas layer-40 looks at chest drain in the image.
Revised: 6C reveal that layer-4 looks at irrelevant artifacts in the chest X-ray image, whereas layer-40 looks at chest drain in the image.

Original: The chest drain shortcut is much harder and is therefore detected at layer 40.
Revised: The chest drain shortcut is much harder and is therefore detected at layer 40.

Original: To verify that layer-40 is looking for chest drains, we ran a tube detector on the images at layer-40 and found that most diseased patients had a chest drain inserted into their lungs, while most healthy patients did not have the chest drain (see Fig-6D).
Revised: To verify that layer-40 is looking for chest drains, we ran a tube detector on the images at layer-40 and found that most diseased patients had a chest drain inserted into their lungs, while most healthy patients did not have the chest drain (see Fig-6D).

Original: This provides evidence that layercategorizes patients based on the chest drain shortcut.
Revised: This provides evidence that layercategorizes patients based on the chest drain shortcut.

Original: Appendix-A.3 provides details on how we train a tube detector.
Revised: Appendix-A.3 provides details on how we train a tube detector.

Original: Fig-6E shows that the AUC performance is 0.94 when the diseased patients have a chest drain and 0.74 when they don’t.
Revised: Fig-6E shows that the AUC performance is 0.94 when the diseased patients have a chest drain and 0.74 when they don’t.

Original: In both cases, the set of healthy patients remains the same.
Revised: In both cases, the set of healthy patients remains the same.

Original: This observation is consistent with the findings of Oakden-Rayner et al.
Revised: This observation is consistent with the findings of Oakden-Rayner et al.

Original: and indicates that the model looks at chest drains to classify positive Pneumothorax cases.
Revised: and indicates that the model looks at chest drains to classify positive Pneumothorax cases.

Original: The above experiments demonstrate how a peak located in the initial layers of the PD plot should raise suspicion, especially when the classification task is challenging.
Revised: The above experiments demonstrate how a peak located in the initial layers of the PD plot should raise suspicion, especially when the classification task is challenging.

Original: Visualization techniques like
Revised: Visualization techniques like

Original: Grad-CAM can further aid our intuition and help identify the shortcuts being learned by the model.
Revised: Grad-CAM can further aid our intuition and help identify the shortcuts being learned by the model.

Original: This approach works well even for realistic scenarios and challenging shortcuts (like chest drain for
Revised: This approach works well even for realistic scenarios and challenging shortcuts (like chest drain for

Original: Pneumothorax classification), as shown above.
Revised: Pneumothorax classification), as shown above.

Original: 4. D ETECTING S HORTCUTS E ARLY
Revised: 4. D ETECTING S HORTCUTS E ARLY

Original: Fig-7 shows the evolution of the PD plot across epochs for M nih (which is the model used in Fig-6).
Revised: Fig-7 shows the evolution of the PD plot across epochs for M nih (which is the model used in Fig-6).

Original: This visualization helps us observe the training dynamics of the various layers.
Revised: This visualization helps us observe the training dynamics of the various layers.

Original: The red bar in the
Revised: The red bar in the

Original: PD plots shows the samples with undefined prediction depths.
Revised: PD plots shows the samples with undefined prediction depths.

Original: These plots reveal several useful insights into the learning dynamics of the model.
Revised: These plots reveal several useful insights into the learning dynamics of the model.

Original: Firstly, we see three prominent peaks in epoch-1 at layers-4,40,88 (see Fig-7A).
Revised: Firstly, we see three prominent peaks in epoch-1 at layers-4,40,88 (see Fig-7A).

Original: The magnitude of the initial peaks (like layers-4&40) remains nearly constant throughout the training.
Revised: The magnitude of the initial peaks (like layers-4&40) remains nearly constant throughout the training.

Original: The peaks at layers-4&40 correspond to shortcuts, as discussed in the previous section.
Revised: The peaks at layers-4&40 correspond to shortcuts, as discussed in the previous section.

Original: This indicates that easy shortcuts can often be identified early (epoch-1 in this case).
Revised: This indicates that easy shortcuts can often be identified early (epoch-1 in this case).

Original: Fig-8 shows the PD plots at epoch-2 for other datasets with shortcuts.
Revised: Fig-8 shows the PD plots at epoch-2 for other datasets with shortcuts.

Original: It is clear from Fig-8 that the suspiciously high peak at the initial layer is visible in the second epoch itself.
Revised: It is clear from Fig-8 that the suspiciously high peak at the initial layer is visible in the second epoch itself.

Original: The Grad-CAM maps reveal that this layer looks at irrelevant artifacts in the dataset.
Revised: The Grad-CAM maps reveal that this layer looks at irrelevant artifacts in the dataset.

Original: This behavior is seen in all datasets shown in Fig-8.
Revised: This behavior is seen in all datasets shown in Fig-8.

Original: Secondly, we also see that accuracy or AUC plots do not reveal shortcut learning patterns, and we need to monitor the training dynamics using suitable metrics (like PD) to detect this behavior.
Revised: Secondly, we also see that accuracy or AUC plots do not reveal shortcut learning patterns, and we need to monitor the training dynamics using suitable metrics (like PD) to detect this behavior.

Original: Thirdly, the red peak (undefined samples) decreases in magnitude with time, and we see a proportional increase in the layer-88 peak.
Revised: Thirdly, the red peak (undefined samples) decreases in magnitude with time, and we see a proportional increase in the layer-88 peak.

Original: This corroborates well with the observation that later layers take more time to converge (Rahaman et al., 2019; Mangalam & Prabhu, 2019; Zeiler & Fergus, 2014).
Revised: This corroborates well with the observation that later layers take more time to converge (Rahaman et al., 2019; Mangalam & Prabhu, 2019; Zeiler & Fergus, 2014).

Original: Therefore, samples with higher PD are initially undefined and do not appear in the PD plot.
Revised: Therefore, samples with higher PD are initially undefined and do not appear in the PD plot.

Original: Nonetheless, samples with lower PD show up very early during the training, and this helps us in the
Revised: Nonetheless, samples with lower PD show up very early during the training, and this helps us in the

Original: [Figure]
Revised: [Figure]

Original: early detection of shortcuts.
Revised: early detection of shortcuts.

Original: Early detection can consequently help develop intervention schemes that fix the shortcut early.
Revised: Early detection can consequently help develop intervention schemes that fix the shortcut early.

Original: 4. P REDICTION D EPTH ≈ V -U SABLE I NFORMATION
Revised: 4. P REDICTION D EPTH ≈ V -U SABLE I NFORMATION

Original: Table-2 measures the influence of shortcuts on NIH and KMNIST using PD and PVI metrics.
Revised: Table-2 measures the influence of shortcuts on NIH and KMNIST using PD and PVI metrics.

Original: All diseased patients in the “NIH w/ Shortcut” dataset have a chest drain, whereas all diseased patients in the “NIH w/o Shortcut” dataset have no chest drain.
Revised: All diseased patients in the “NIH w/ Shortcut” dataset have a chest drain, whereas all diseased patients in the “NIH w/o Shortcut” dataset have no chest drain.

Original: The set of healthy patients is common for the two datasets.
Revised: The set of healthy patients is common for the two datasets.

Original: The KMNIST datasets are the same as those used in Section-4.2.
Revised: The KMNIST datasets are the same as those used in Section-4.2.

Original: We use VGG for KMNIST and DenseNet121 for NIH.
Revised: We use VGG for KMNIST and DenseNet121 for NIH.

Original: Other training details remain the same as in Section-4.2.
Revised: Other training details remain the same as in Section-4.2.

Original: [Figure]
Revised: [Figure]

Original: Table-2 shows that datasets with shortcuts ( D s ) have smaller mean PD values than their counterparts without shortcuts ( D i ).
Revised: Table-2 shows that datasets with shortcuts ( D s ) have smaller mean PD values than their counterparts without shortcuts ( D i ).

Original: Proposition-1 (see Section-3, Appendix-A.1) shows that a sufficient gap between the mean PDs of D s and D i causes the V -Information of D s to be greater than D i .
Revised: Proposition-1 (see Section-3, Appendix-A.1) shows that a sufficient gap between the mean PDs of D s and D i causes the V -Information of D s to be greater than D i .

Original: Tableconfirms this in a medical-imaging dataset with a real shortcut, and we see that the mean “usableinformation” increases when there is a shortcut.
Revised: Tableconfirms this in a medical-imaging dataset with a real shortcut, and we see that the mean “usableinformation” increases when there is a shortcut.

Original: This implies that the model learns shortcut features as they have more usable-information than the core features.
Revised: This implies that the model learns shortcut features as they have more usable-information than the core features.

Original: Ethayarajh et al.
Revised: Ethayarajh et al.

Original: (2021) also show that V -information is positively correlated with test accuracy.
Revised: (2021) also show that V -information is positively correlated with test accuracy.

Original: This explains the significant change in AUC observed in Fig-6E. Proposition 1 bridges the gap between the notions of PD and V -usable information.
Revised: This explains the significant change in AUC observed in Fig-6E. Proposition 1 bridges the gap between the notions of PD and V -usable information.

Original: This connection between V -information and PD indicates that monitoring early training dynamics using PD not only helps detect shortcut learning but also bears insights into the dataset’s difficulty (in information-theoretic terms) for a given model class.
Revised: This connection between V -information and PD indicates that monitoring early training dynamics using PD not only helps detect shortcut learning but also bears insights into the dataset’s difficulty (in information-theoretic terms) for a given model class.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we study shortcut learning by monitoring the early training dynamics of DNNs.
Revised: In this paper, we study shortcut learning by monitoring the early training dynamics of DNNs.

Original: We emphasize that not all spurious correlations are shortcuts and empirically show that models suffer from shortcut learning only when the spurious features are easier than the core features.
Revised: We emphasize that not all spurious correlations are shortcuts and empirically show that models suffer from shortcut learning only when the spurious features are easier than the core features.

Original: We hypothesize that “Potential shortcuts can be found by monitoring the easy features learned by the initial layers of a DNN early during the training.” and validate this hypothesis on real medical imaging data.
Revised: We hypothesize that “Potential shortcuts can be found by monitoring the easy features learned by the initial layers of a DNN early during the training.” and validate this hypothesis on real medical imaging data.

Original: We empirically demonstrate that shortcuts are learned quite early during the training and that one can detect shortcut learning by monitoring the early training of DNNs using suitable instance difficulty metrics like Prediction Depth (PD).
Revised: We empirically demonstrate that shortcuts are learned quite early during the training and that one can detect shortcut learning by monitoring the early training of DNNs using suitable instance difficulty metrics like Prediction Depth (PD).

Original: Further, we show a theoretical connection between PD and V -information to support our empirical results.
Revised: Further, we show a theoretical connection between PD and V -information to support our empirical results.

Original: Datasets with shortcuts have more
Revised: Datasets with shortcuts have more

Original: V -information causing the model to learn the shortcut.
Revised: V -information causing the model to learn the shortcut.

Original: To conclude, relying only on accuracy plots is insufficient, and we recommend monitoring the DNN training dynamics using additional metrics like PD for the early detection of shortcuts.
Revised: To conclude, relying only on accuracy plots is insufficient, and we recommend monitoring the DNN training dynamics using additional metrics like PD for the early detection of shortcuts.

Original: A A PPENDIX A. P ROOF OF P ROPOSITION - 
Revised: A A PPENDIX A. P ROOF OF P ROPOSITION -

Original: Proposition 1.
Revised: Proposition 1.

Original: Given two datasets, D s with shortcuts and D i without shortcuts, we assume the following:
Revised: Given two datasets, D s with shortcuts and D i without shortcuts, we assume the following:

Original: (Well-Trained Model Assumption)
Revised: (Well-Trained Model Assumption)

Original: The part of the network from any representation to the label is one of the functions that compute V -information.
Revised: The part of the network from any representation to the label is one of the functions that compute V -information.

Original: (Function Class Complexity Assumption) Assume that there exists a K ∈ { 1 , N } such that V cnn of depth N − K is deep enough to be a strictly larger function class than V knn with a fixed neighbor size ( 29 in this paper).
Revised: (Function Class Complexity Assumption) Assume that there exists a K ∈ { 1 , N } such that V cnn of depth N − K is deep enough to be a strictly larger function class than V knn with a fixed neighbor size ( 29 in this paper).

Original: Assume that this V knn is a larger function class than a linear function.
Revised: Assume that this V knn is a larger function class than a linear function.

Original: (Controlled Confidence Growth Assumption)
Revised: (Controlled Confidence Growth Assumption)

Original: For both datasets D ∈ { D s , D i } , assume that the for all k ∈ { 1 , · · · , N } ,
Revised: For both datasets D ∈ { D s , D i } , assume that the for all k ∈ { 1 , · · · , N } ,

Original: [Equation]
Revised: [Equation]

Original: Prediction Depth Separation Assumption) Let L be an integer such that, L ≤ K and
Revised: Prediction Depth Separation Assumption) Let L be an integer such that, L ≤ K and

Original: [Equation]
Revised: [Equation]

Original: Let there exist a gap in prediction depths of samples in D s and D i : ψ ∈ (0 , 0 . 5) such that 1 − ψ fraction of D s has prediction depth ≤ L and 1 − ψ fraction of D i has prediction depth > K .
Revised: Let there exist a gap in prediction depths of samples in D s and D i : ψ ∈ (0 , 0 . 5) such that 1 − ψ fraction of D s has prediction depth ≤ L and 1 − ψ fraction of D i has prediction depth > K .

Original: Then, for a model class of N -layer CNNs, we show that the V cnn -information for D s is greater than
Revised: Then, for a model class of N -layer CNNs, we show that the V cnn -information for D s is greater than

Original: V cnn -information for D i :
Revised: V cnn -information for D i :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We proceed in two parts: first, we lower bound V cnn -information for D s , and then we upper bound V cnn for D i .
Revised: We proceed in two parts: first, we lower bound V cnn -information for D s , and then we upper bound V cnn for D i .

Original: Assumpition 3 implies:
Revised: Assumpition 3 implies:

Original: [Equation]
Revised: [Equation]

Original: PD - PVI connection.
Revised: PD - PVI connection.

Original: Note that by definition, when the prediction depth is k for a sample X , then
Revised: Note that by definition, when the prediction depth is k for a sample X , then

Original: PV I knn ( ϕ k ( X )) ≥
Revised: PV I knn ( ϕ k ( X )) ≥

Original: δ but PV I knn ( ϕ k − 1 ( X )) <
Revised: δ but PV I knn ( ϕ k − 1 ( X )) <

Original: δ .
Revised: δ .

Original: This follows from how we compute PD (see
Revised: This follows from how we compute PD (see

Original: Section-3 in main paper, and Appendix-A.4).
Revised: Section-3 in main paper, and Appendix-A.4).

Original: Lower bounding I D
Revised: Lower bounding I D

Original: s V cnn
Revised: s V cnn

Original: [Equation]
Revised: [Equation]

Original: { Telescoping Sum }
Revised: { Telescoping Sum }

Original: [Equation]
Revised: [Equation]

Original: { Assumption-3 }
Revised: { Assumption-3 }

Original: [Equation]
Revised: [Equation]

Original: { Prediction Depth Separation }
Revised: { Prediction Depth Separation }

Original: [Equation]
Revised: [Equation]

Original: { Prediction Depth Separation }
Revised: { Prediction Depth Separation }

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Sum
Revised: Sum

Original: [Equation]
Revised: [Equation]

Original: Prediction Depth Separation }
Revised: Prediction Depth Separation }

Original: [Equation]
Revised: [Equation]

Original: The proof follows by comparing the lower bound on I D
Revised: The proof follows by comparing the lower bound on I D

Original: s V cnn and the upper bound on I
Revised: s V cnn and the upper bound on I

Original: D i V cnn .
Revised: D i V cnn .

Original: Intuitively what this means is that when there is a sufficiently large gap in the mean PD between D s and D i , then the V -information of D s exceeds the V -information of D i , which is why the model prefers learning the shortcut rather than using the core features for the task.
Revised: Intuitively what this means is that when there is a sufficiently large gap in the mean PD between D s and D i , then the V -information of D s exceeds the V -information of D i , which is why the model prefers learning the shortcut rather than using the core features for the task.

Original: A. G RAD -CAM V ISUALIZATION
Revised: A. G RAD -CAM V ISUALIZATION

Original: PD plots help us understand the layers of the model which are actively used for classifying different images.
Revised: PD plots help us understand the layers of the model which are actively used for classifying different images.

Original: To further aid our intuition, we visualize the Grad-CAM outputs for an arbitrary layer k of the model by attaching a soft-KNN head.
Revised: To further aid our intuition, we visualize the Grad-CAM outputs for an arbitrary layer k of the model by attaching a soft-KNN head.

Original: Let g knn denote the soft and differentiable version of k-NN.
Revised: Let g knn denote the soft and differentiable version of k-NN.

Original: We compute g knn as follows:
Revised: We compute g knn as follows:

Original: This is a sample text in g knn ( ϕ kq ; ϕ ki ∈{ 1 , 2 ,...m } ) =
Revised: This is a sample text in g knn ( ϕ kq ; ϕ ki ∈{ 1 , 2 ,...m } ) =

Original: [Table]
Revised: [Table]

Original: This function makes the KNN differentiable and can be used to compute Grad-CAM (Selvaraju et al., 2017).
Revised: This function makes the KNN differentiable and can be used to compute Grad-CAM (Selvaraju et al., 2017).

Original: We use the L 1 norm for all distance computations.
Revised: We use the L 1 norm for all distance computations.

Original: ϕ kq corresponds to feature at layer- k for query image x q .
Revised: ϕ kq corresponds to feature at layer- k for query image x q .

Original: Let ϕ ki ∈{ 1 , 2 ,...m } be the training data for KNN.
Revised: Let ϕ ki ∈{ 1 , 2 ,...m } be the training data for KNN.

Original: Let N denote the neighborhood function.
Revised: Let N denote the neighborhood function.

Original: N ( ϕ kq , :) returns the indices of K-nearest neighbors for ϕ kq .
Revised: N ( ϕ kq , :) returns the indices of K-nearest neighbors for ϕ kq .

Original: N ( ϕ kq , 1) returns indices of images with positive label ( y = 1 ) from the set of K-nearest neighbors for ϕ kq .
Revised: N ( ϕ kq , 1) returns indices of images with positive label ( y = 1 ) from the set of K-nearest neighbors for ϕ kq .

Original: s is the median for the set of L 1 norms {∥ ϕ kq − ϕ kj ∥} for j ∈ N ( ϕ kq , :) .
Revised: s is the median for the set of L 1 norms {∥ ϕ kq − ϕ kj ∥} for j ∈ N ( ϕ kq , :) .

Original: A. C HEST D RAIN A NNOTATIONS FOR NIH D ATASET
Revised: A. C HEST D RAIN A NNOTATIONS FOR NIH D ATASET

Original: To reproduce the results by Oakden-Rayner et al.
Revised: To reproduce the results by Oakden-Rayner et al.

Original: , we need chest drain annotations for the
Revised: , we need chest drain annotations for the

Original: NIH dataset (Wang et al., 2017a), which is not natively provided.
Revised: NIH dataset (Wang et al., 2017a), which is not natively provided.

Original: To do this, we use the MIMICCXR dataset (Johnson et al., 2019), which has rich meta-data information available in the form of radiology reports.
Revised: To do this, we use the MIMICCXR dataset (Johnson et al., 2019), which has rich meta-data information available in the form of radiology reports.

Original: We collaborate with radiologists to identify terms related to Pneumothorax from the MIMIC-CXR reports.
Revised: We collaborate with radiologists to identify terms related to Pneumothorax from the MIMIC-CXR reports.

Original: These include pigtail catheters, pleural tubes, chest tubes, thoracostomy tubes, etc.
Revised: These include pigtail catheters, pleural tubes, chest tubes, thoracostomy tubes, etc.

Original: We collect chest drain annotations for MIMIC-CXR by parsing the reports for these terms using the RadGraph NLP pipeline (Jain et al., 2021).
Revised: We collect chest drain annotations for MIMIC-CXR by parsing the reports for these terms using the RadGraph NLP pipeline (Jain et al., 2021).

Original: We train a DenseNet121 model to detect chest drains relevant to Pneumothorax using these annotations.
Revised: We train a DenseNet121 model to detect chest drains relevant to Pneumothorax using these annotations.

Original: Finally, we run this trained model on the
Revised: Finally, we run this trained model on the

Original: NIH dataset to obtain the needed chest drain annotations.
Revised: NIH dataset to obtain the needed chest drain annotations.

Original: We use these annotations to obtain the results shown in Figs - 6D and 6E. Fig-6E closely reproduces the results obtained by Oakden-Rayner et al.
Revised: We use these annotations to obtain the results shown in Figs - 6D and 6E. Fig-6E closely reproduces the results obtained by Oakden-Rayner et al.

Original: A. N OTION OF U NDEFINED P REDICTION D EPTH
Revised: A. N OTION OF U NDEFINED P REDICTION D EPTH

Original: Section-3 shows how we compute PD in our experiments.
Revised: Section-3 shows how we compute PD in our experiments.

Original: While fully trained models give valid PD values, our application requires working with arbitrary deep-learning models that are not necessarily fully trained.
Revised: While fully trained models give valid PD values, our application requires working with arbitrary deep-learning models that are not necessarily fully trained.

Original: We, therefore, introduce the notion of undefined PD by treating k -NN predictions close to 0.5 (for a binary classification setting) as invalid.
Revised: We, therefore, introduce the notion of undefined PD by treating k -NN predictions close to 0.5 (for a binary classification setting) as invalid.

Original: We define a δ such that | f knn ( x ) − 0 .
Revised: We define a δ such that | f knn ( x ) − 0 .

Original: 5 | < δ implies an invalid k -NN output.
Revised: 5 | < δ implies an invalid k -NN output.

Original: We use δ = 0 .
Revised: We use δ = 0 .

Original: 1 and k = 29 in our experiments.
Revised: 1 and k = 29 in our experiments.

Original: If any k -NN predictions for the last three layers are invalid, we treat the PD of the input image to be undefined.
Revised: If any k -NN predictions for the last three layers are invalid, we treat the PD of the input image to be undefined.

Original: To work with high-resolution images (like 512 × 512 ) [Figure] we downsample the spatial resolution of all training embeddings to 8 × 8 before using the k -NN classifiers on the intermediate layers.
Revised: To work with high-resolution images (like 512 × 512 ), we downsample the spatial resolution of all training embeddings to 8 × 8 before using the k -NN classifiers on the intermediate layers.

Original: We empirically see that our results are insensitive to k in the range [5 , 30] [Figure]
Revised: We empirically see that our results are insensitive to k in the range [5 , 30] .

Original: A. PD P LOTS AT E POCH - 
Revised: A. PD P LOTS AT E POCH -

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We show epoch-0 PD plots for various tasks on the NIH dataset (see Figures-9&10).
Revised: We show epoch-0 PD plots for various tasks on the NIH dataset (see Figures-9&10).

Original: We expect most samples to be undefined at epoch 0 (shown by the red bar), and we see some additional noisy patterns in the plot.
Revised: We expect most samples to be undefined at epoch 0 (shown by the red bar), and we see some additional noisy patterns in the plot.

Original: The model is not trained during initialization, and hence the weights and features are random vectors carrying little information.
Revised: The model is not trained during initialization, and hence the weights and features are random vectors carrying little information.

Original: Most of the instance difficulty metrics are ill-defined until the model is fully trained, and therefore we find this uninterpretable behavior in PD plots at epoch-0.
Revised: Most of the instance difficulty metrics are ill-defined until the model is fully trained, and therefore we find this uninterpretable behavior in PD plots at epoch-0.

Original: Although most of the instance difficulty metrics fail to give meaningful information at the start of training Ethayarajh et al.
Revised: Although most of the instance difficulty metrics fail to give meaningful information at the start of training Ethayarajh et al.

Original: (2021); Agarwal et al.
Revised: (2021); Agarwal et al.

Original: (2022), we empirically observe that PD plots can capture the distribution of the relatively easy samples in the training dataset even during early stages of training.
Revised: (2022), we empirically observe that PD plots can capture the distribution of the relatively easy samples in the training dataset even during early stages of training.

Original: This favors our hypothesis, which relies on monitoring easy samples in early layers to detect shortcut learning.
Revised: This favors our hypothesis, which relies on monitoring easy samples in early layers to detect shortcut learning.

Original: We, therefore, use PD over other metrics to analyze the early training dynamics of the model.
Revised: We, therefore, use PD over other metrics to analyze the early training dynamics of the model.

Original: A. C ODE R EPRODUCIBILITY
Revised: A. C ODE R EPRODUCIBILITY

Original: Please find the code here: https://github.com/anonymCloud/ICLR2023_Shortcut_
Revised: Please find the code here: https://github.com/anonymCloud/ICLR Shortcut

Original:  Learning_Through_Training_Dynamics 
Revised: _ Learning_Through_Training_Dynamics A. V ISION E XPERIMENTS

Revised: We use the NICO++ (Non-I.I.D. Image dataset with Contexts) dataset Zhang et al.

Revised: (2022) to create multiple spurious datasets (Cow vs. Bird; Dog vs. Lizard; Flower vs. Lizard) such that the context/background is spuriously correlated with the target.

Revised: NICO++ is a Non-I.I.D image dataset that uses context to differentiate between the test and train distributions.

Revised: This forms an ideal setup to investigate what spurious correlations the model learns during training.

Revised: We follow the procedure outlined by Puli et al.

Revised: (2022) to create datasets with spurious correlations (90% prevalence) in the training data, and the test data has the relationship between spurious attributes and the true labels flipped.

Revised: This is similar to the Chex-MIMIC dataset illustrated in section-4.2.

Revised: We test our hypothesis using ResNet-18 and VGG16.

Revised: We train our models for 30 epochs using an Adam optimizer and a base learning rate of 0.01.

Revised: We choose the best checkpoint using early stopping.

Revised: [Figure]

Revised: [Figure]

Revised: test accuracy of 80.2% (as compared to 59.9% in Fig-11D for the spurious dataset).

Revised: Figures-11,13,14 show PD plots and train/test accuracies for models that learn the spurious background feature present in the NICO++ dataset.

Revised: While all models achieve > 85 % training accuracy, they have poor accuracies ( 50%) on the test data where the spurious correlation is flipped.

Revised: This can be seen simply by observing the PD plots for the model on the training data.

Revised: The plots are skewed towards the initial layers indicating that the model relies heavily on very simple (potentially spurious) features for the task.

Revised: GradCAM maps also confirm that the model often focuses on the background context rather than the foreground object of interest.

Revised: We further observe in Fig-12 that balancing the training data (to remove the spurious correlation) results in a model with improved test accuracy (80.2%) as expected.

Revised: This is also reflected in the

Revised: PD plot (Fig-12B), where we see that the distribution of the peaks, as well as the mean PD, shift

Revised: [Figure]

Revised: 18 reveals a

Revised: [Figure]

Revised: The spurious background reveals the target class

Revised: The model achieves 87.87% training accuracy

Revised: PD plot for

Revised: 18 reveals a spurious peak at layer plots

Revised: 1 confirm that the model uses the rock texture in the background to detect lizards, whereas it uses simple (nonspurious) color features to detect the flowers in the image

Revised: Due to partial reliance on spurious features, the model obtains low test accuracy (45.16%) on the test data with the spurious correlation flipped (i.e., images contain flowers on rock and lizards on grass

Revised: ) proportionately towards the later layers, indicating that the model is now using a combination of both easy and complex features for the prediction task.

Revised: By monitoring PD plots during training and using suitable visualization techniques, we show that one can obtain useful insights about the spurious correlations that the model may be learning.

Revised: This can also help the user make an educated guess about the generalization behavior of the model on various test datasets with different distributions.