Original: On Understanding the Hardnessof Samples in Neural Networks
Revised: Angular Visual Hardness

Original: Abstract
Revised: Abstract

Original: The mechanisms used by the human visual system and artiﬁcial convolutional neural networks (CNN) to understand images are vastly different.
Revised: The mechanisms used by the human visual system and artiﬁcial convolutional neural networks (CNN) to understand images are vastly different.

Original: The two systems have different notions of hardness, meaning the set of images which appear to be ambiguous and hard to classify are different.
Revised: The two systems have different notions of hardness, meaning the set of images which appear to be ambiguous and hard to classify are different.

Original: In this paper, we answer the following question: are there measures we can compute in the trained
Revised: In this paper, we answer the following question: are there measures we can compute in the trained

Original: CNN models that correspond closely to human visual hardness?
Revised: CNN models that correspond closely to human visual hardness?

Original: We employ human selection frequency, the frequency with which human annotators label a given image, as a surrogate for human visual hardness.
Revised: We employ human selection frequency, the frequency with which human annotators label a given image, as a surrogate for human visual hardness.

Original: This information is recently made available on the ImageNet validation set (16).
Revised: This information is recently made available on the ImageNet validation set (16).

Original: The CNN model conﬁdence does not correlate well with this human visual hardness score, and it is not surprising given that there are calibration issues in the models.
Revised: The CNN model conﬁdence does not correlate well with this human visual hardness score, and it is not surprising given that there are calibration issues in the models.

Original: We propose a novel measure known as angular visual hardness (AVH).
Revised: We propose a novel measure known as angular visual hardness (AVH).

Original: It is the normalized angular distance between the image feature embedding and the weights of the target category.
Revised: It is the normalized angular distance between the image feature embedding and the weights of the target category.

Original: We demonstrate that AVH is strongly correlated with human visual hardness across a broad range of CNN architectures.
Revised: We demonstrate that AVH is strongly correlated with human visual hardness across a broad range of CNN architectures.

Original: We conduct an in-depth scientiﬁc study and test multiple hypotheses to draw this conclusion.
Revised: We conduct an in-depth scientiﬁc study and test multiple hypotheses to draw this conclusion.

Original: We observe that CNN models with the highest validation accuracy also have the best AVH scores.
Revised: We observe that CNN models with the highest validation accuracy also have the best AVH scores.

Original: This agrees with the earlier ﬁnding that the stateof-art (SOTA) models are improving classiﬁcation of harder examples.
Revised: This agrees with the earlier ﬁnding that the stateof-art (SOTA) models are improving classiﬁcation of harder examples.

Original: We also observe that during the training of CNNs, AVH reaches a plateau in early stages even as the training loss keeps improving.
Revised: We also observe that during the training of CNNs, AVH reaches a plateau in early stages even as the training loss keeps improving.

Original: We conjecture the different causes for such plateau of easy and hard examples, which suggests the need to design better loss functionsCorrespondence to: that can target harder examples more effectively and improve SOTA accuracy.
Revised: We conjecture the different causes for such plateau of easy and hard examples, which suggests the need to design better loss functions that can target harder examples more effectively and improve SOTA accuracy.

Original: 1. Introduction
Revised: 1. Introduction

Original: Convolutional Neural Networks (CNN) have achieved rapid progress on many computer vision tasks such as image classiﬁcation (8), face recognition, and scene analysis.
Revised: Convolutional Neural Networks (CNN) have achieved rapid progress on many computer vision tasks such as image classiﬁcation (8), face recognition, and scene analysis.

Original: On large benchmark datasets such as ImageNet (3) they have even surpassed human-level accuracy.
Revised: On large benchmark datasets such as ImageNet (3) they have even surpassed human-level accuracy.

Original: Despite this, these models are no match to the human visual system when it comes to other measures such as robustness and few-shot learning (1; 21; 15).
Revised: Despite this, these models are no match to the human visual system when it comes to other measures such as robustness and few-shot learning (1; 21; 15).

Original: This is not surprising given that they have entirely different processing mechanisms.
Revised: This is not surprising given that they have entirely different processing mechanisms.

Original: Due to the blackbox nature of CNNs and our limited understanding of the human brain, it is challenging to map out these differences precisely.
Revised: Due to the blackbox nature of CNNs and our limited understanding of the human brain, it is challenging to map out these differences precisely.

Original: In this paper, we focus on one key aspect, viz.
Revised: In this paper, we focus on one key aspect, viz.

Original: , how different are the hard examples for these two systems?
Revised: , how different are the hard examples for these two systems?

Original: By hard examples, we mean the the set of images that appear ambiguous and are error prone.
Revised: By hard examples, we mean the the set of images that appear ambiguous and are error prone.

Original: Much of current deep learning research focuses on measuring the hardness of an image sample for deep models rather than for a human.
Revised: Much of current deep learning research focuses on measuring the hardness of an image sample for deep models rather than for a human.

Original: For example, hardness for models can be deﬁned using the loss value (18), relative Euclidean distance (17; 20) and gradient norm (11).
Revised: For example, hardness for models can be deﬁned using the loss value (18), relative Euclidean distance (17; 20) and gradient norm (11).

Original: On the other hand, there is a rich history in cognitive and neuroscience communities to understand human visual perception (6; 2).
Revised: On the other hand, there is a rich history in cognitive and neuroscience communities to understand human visual perception (6; 2).

Original: Many works focus on mechanisms used by the human brain to translate visual information into mental representations.
Revised: Many works focus on mechanisms used by the human brain to translate visual information into mental representations.

Original: These representations are subject to many correspondence differences and errors, and thereby are not isomorphic to the real world (13).
Revised: These representations are subject to many correspondence differences and errors, and thereby are not isomorphic to the real world (13).

Original: They can be affected by the ambiguity of different semantics (10) such as occlusion, distortion, motion blur, and inherent similarity among objects.
Revised: They can be affected by the ambiguity of different semantics (10) such as occlusion, distortion, motion blur, and inherent similarity among objects.

Original: However, such detailed semantic information is typically not present in large-scale image benchmarks used to train the CNN models.
Revised: However, such detailed semantic information is typically not present in large-scale image benchmarks used to train the CNN models.

Original: A natural approach to measure human visual hardness is the human selection frequency, i.e. the rate with which human annotators select a speciﬁc image as belonging to a certain category.
Revised: A natural approach to measure human visual hardness is the human selection frequency, i.e. the rate with which human annotators select a speciﬁc image as belonging to a certain category.

Original: Unfortunately, most publicly available benchmarks do not have this information.
Revised: Unfortunately, most publicly available benchmarks do not have this information.

Original: But thanks to the recent efforts of (16), we now have this information on the
Revised: But thanks to the recent efforts of (16), we now have this information on the

Original: ImageNet test set.
Revised: ImageNet test set.

Original: We employ this new dataset in this paper and as a result, come up with many novel insights.
Revised: We employ this new dataset in this paper and as a result, come up with many novel insights.

Original: In addition, we propose a novel measure that closely aligns with the human selection frequency, and hence, can be employed in other datasets where such information is not available.
Revised: In addition, we propose a novel measure that closely aligns with the human selection frequency, and hence, can be employed in other datasets where such information is not available.

Original: Our Contributions: We conduct an in-depth exploratory study on the ImageNet testset with newly available human selection frequency information.
Revised: Our Contributions: We conduct an in-depth exploratory study on the ImageNet testset with newly available human selection frequency information.

Original: We employ the scientiﬁc method and carefully test multiple hypotheses and present our ﬁndings.
Revised: We employ the scientiﬁc method and carefully test multiple hypotheses and present our ﬁndings.

Original: • We observe that the CNN model conﬁdence and human selection frequency are not strongly correlated.
Revised: • We observe that the CNN model conﬁdence and human selection frequency are not strongly correlated.

Original: The CNN model tends to be overconﬁdent and this is a well known calibration issue (7).
Revised: The CNN model tends to be overconﬁdent and this is a well known calibration issue (7).

Original: • We propose a new measure known as angular visual hardness ( AVH ) described below.
Revised: • We propose a new measure known as angular visual hardness ( AVH ) described below.

Original: • We observe that AVH is strongly correlated with human selection frequency across a wide range of CNN models.
Revised: • We observe that AVH is strongly correlated with human selection frequency across a wide range of CNN models.

Original: To the best of our knowledge, this is the ﬁrst model score that correlates strongly with human visual hardness.
Revised: To the best of our knowledge, this is the ﬁrst model score that correlates strongly with human visual hardness.

Original: Hence, it can serve as its proxy on datasets where such information is not available.
Revised: Hence, it can serve as its proxy on datasets where such information is not available.

Original: • We observe that the state-of-art (SOTA) models have the highest AVH score.
Revised: • We observe that the state-of-art (SOTA) models have the highest AVH score.

Original: This implies that improving SOTA accuracy will entail improving accuracy of hard examples.
Revised: This implies that improving SOTA accuracy will entail improving accuracy of hard examples.

Original: AVH serves as a good measure to mine such hard examples in any dataset.
Revised: AVH serves as a good measure to mine such hard examples in any dataset.

Original: • We observed the evolution of AVH score during training of CNN models.
Revised: • We observed the evolution of AVH score during training of CNN models.

Original: It plateaus early in training even as the training (cross-entropy) loss function keeps improving.
Revised: It plateaus early in training even as the training (cross-entropy) loss function keeps improving.

Original: This suggests the need to design better loss functions that can improve performance on hard examples.
Revised: This suggests the need to design better loss functions that can improve performance on hard examples.

Original: Angular visual hardness: We propose a new score for a given CNN model based on the normalized angular distance between the image feature embedding and the weights of the target category.
Revised: Angular visual hardness: We propose a new score for a given CNN model based on the normalized angular distance between the image feature embedding and the weights of the target category.

Original: The normalization takes into account the angular distances to other categories.
Revised: The normalization takes into account the angular distances to other categories.

Original: We argue that the semantic ambiguity that affects human visual hardness is strongly correlated with this score.
Revised: We argue that the semantic ambiguity that affects human visual hardness is strongly correlated with this score.

Original: This is inspired by the intuition in (14) that the angle between image feature embedding and the weights of the target class accounts for the inter-class semantic differences while the (cid:96) 2 norm of the feature embedding accounts for intra-class variation.
Revised: This is inspired by the intuition in (14) that the angle between image feature embedding and the weights of the target class accounts for the inter-class semantic differences while the (cid:96) 2 norm of the feature embedding accounts for intra-class variation.

Original: (14) used this insight to try to improve the generalization of the model.
Revised: (14) used this insight to try to improve the generalization of the model.

Original: On the other hand, we use it to study the correspondence with human visual hardness.
Revised: On the other hand, we use it to study the correspondence with human visual hardness.

Original: 2. A Discovery of the Bridge: Angular Visual Hardness
Revised: 2. A Discovery of the Bridge: Angular Visual Hardness

Original: In order to quantify Human Visual Hardness and Model
Revised: In order to quantify Human Visual Hardness and Model

Original: Predictions for convenience purposes in experiments, we use corresponding surrogates which are formally deﬁned as the following throughout the paper.
Revised: Predictions for convenience purposes in experiments, we use corresponding surrogates which are formally deﬁned as the following throughout the paper.

Original: Deﬁnition 1 (Model Conﬁdence) .
Revised: Deﬁnition 1 (Model Conﬁdence) .

Original: We deﬁne model conﬁdence on a single sample as the probability score of the true objective class output by the CNN models,
Revised: We deﬁne model conﬁdence on a single sample as the probability score of the true objective class output by the CNN models,

Original: [Equation]
Revised: [Equation]

Original: We employ the standard image benchmark ImageNet(4) in all following experiments.
Revised: We employ the standard image benchmark ImageNet(4) in all following experiments.

Original: Particularly, we take advantage of the Human Selection Frequency information for validation images provided by the recent paper (16).
Revised: Particularly, we take advantage of the Human Selection Frequency information for validation images provided by the recent paper (16).

Original: Recall that such information serves as a proxy for Human Visual Hardness.
Revised: Recall that such information serves as a proxy for Human Visual Hardness.

Original: Besides, in order to verify that the our experimental results hold consistently across models instead of a particular model, we use four popular ImageNet pre-trained models
Revised: Besides, in order to verify that the our experimental results hold consistently across models instead of a particular model, we use four popular ImageNet pre-trained models

Original: AlexNet (12), VGG19 (19), DenseNet121 (9), ResNet50 (8).
Revised: AlexNet (12), VGG19 (19), DenseNet121 (9), ResNet50 (8).

Original: We select ResNet50 as the representative model for some experiments.
Revised: We select ResNet50 as the representative model for some experiments.

Original: 2.1. Gap between Human Visual Hardness and Model Predictions
Revised: 2.1. Gap between Human Visual Hardness and Model Predictions

Original: Studying the precise connection or gap between human visual hardness and model predictions is not feasible because data collection involving human labelling or annotation requires large amount of work.
Revised: Studying the precise connection or gap between human visual hardness and model predictions is not feasible because data collection involving human labelling or annotation requires large amount of work.

Original: In addition, usually those human data is application or dataset speciﬁc, which makes the scalability of this study even worse.
Revised: In addition, usually those human data is application or dataset speciﬁc, which makes the scalability of this study even worse.

Original: Therefore, all the testing and experiments we design are at best effort given the limited resources.
Revised: Therefore, all the testing and experiments we design are at best effort given the limited resources.

Original: That is exactly another motivation for us to bridge the gap between Human and models because models predictions require minimum costs compared to human efforts.
Revised: That is exactly another motivation for us to bridge the gap between Human and models because models predictions require minimum costs compared to human efforts.

Original: An interesting observation in (16) shows that Human Selection Frequency has strong inﬂuence on the Model Conﬁdence.
Revised: An interesting observation in (16) shows that Human Selection Frequency has strong inﬂuence on the Model Conﬁdence.

Original: Speciﬁcally, examples with low Human Selection
Revised: Speciﬁcally, examples with low Human Selection

Original: Frequency tends to have relatively low Model Conﬁdence.
Revised: Frequency tends to have relatively low Model Conﬁdence.

Original: Naturally we examine if the correlation between Model Conﬁdence and Human Selection Frequency is strong.
Revised: Naturally we examine if the correlation between Model Conﬁdence and Human Selection Frequency is strong.

Original: Speciﬁcally, all ImageNet validation images are evaluated by the pre-trained models.
Revised: Speciﬁcally, all ImageNet validation images are evaluated by the pre-trained models.

Original: The corresponding output is simply the
Revised: The corresponding output is simply the

Original: Model Conﬁdence on each image.
Revised: Model Conﬁdence on each image.

Original: In addition, because each such image is provided with the frequency of being identiﬁed as the labeled class out of 50 workers who manually perform the labeling task, i.e. Human Selection Frequency.
Revised: In addition, because each such image is provided with the frequency of being identiﬁed as the labeled class out of 50 workers who manually perform the labeling task, i.e. Human Selection Frequency.

Original: The left plot in ﬁgure 1 presents a two-dimensional histogram for the correlation visualization.
Revised: The left plot in ﬁgure 1 presents a two-dimensional histogram for the correlation visualization.

Original: The x-axis represents Human Selection Frequency, and the y-axis represents
Revised: The x-axis represents Human Selection Frequency, and the y-axis represents

Original: Model Conﬁdence.
Revised: Model Conﬁdence.

Original: Each bin exhibits the number of images which lie in the corresponding range.
Revised: Each bin exhibits the number of images which lie in the corresponding range.

Original: We can observe the high density at the right corner, which means the majority of the images have both high human and model accuracy.
Revised: We can observe the high density at the right corner, which means the majority of the images have both high human and model accuracy.

Original: However, there is a considerable amount of density on the range of medium human accuracy but either extremely low
Revised: However, there is a considerable amount of density on the range of medium human accuracy but either extremely low

Original: 0.0 0.2 0.4 0.6 0.8 1.0 Human Selection Frequency 0.
Revised: 0.0 0.2 0.4 0.6 0.8 1.0 Human Selection Frequency 0.

Original: M o d e l C o n f i d e n c e 0 100300 400600 700 800 # o f S a m p l e s 0.0 0.2 0.4 0.

Revised: M o d e l C o n f i d e n c e 0 100300 400600 700 800 # o f S a m p l e s 0.0 0.2 0.4 0.

Original: 0.8 1.0 Human Selection Frequency 0.
Revised: 0.8 1.0 Human Selection Frequency 0.

Original: || x
Revised: || x

Original: || 

Revised: ||

Original: 2575 100150 175 200 # o f S a m p l e s 0.0 0.2 0.4 0.

Revised: 2575 100150 175 200 # o f S a m p l e s 0.0 0.2 0.4 0.

Original: 0.8 1.0 Human Selection Frequency 0.
Revised: 0.8 1.0 Human Selection Frequency 0.

Original: AV H ( x ) 0200400600 # o f S a m p l e s or high model accuracy.
Revised: AV H ( x ) 0200400600 # o f S a m p l e s or high model accuracy.

Original: Overall, Model Conﬁdence and
Revised: Overall, Model Conﬁdence and

Original: Human Selection Frequency are not in direct proportion and thereby not strongly correlated.
Revised: Human Selection Frequency are not in direct proportion and thereby not strongly correlated.

Original: 2.2. Bridging the Gap
Revised: 2.2. Bridging the Gap

Original: Followed by identifying the gap in last section, we naturally propose a hypothesis:
Revised: Followed by identifying the gap in last section, we naturally propose a hypothesis:

Original: Hypothesis 2.
Revised: Hypothesis 2.

Original: There exists some characteristic in CNN
Revised: There exists some characteristic in CNN

Original: Models strongly correlates with Human Selection Frequency to bridge the gap?
Revised: Models strongly correlates with Human Selection Frequency to bridge the gap?

Original: In this section, We ﬁrst provide two predictions and test them accordingly.
Revised: In this section, We ﬁrst provide two predictions and test them accordingly.

Original: Denote S n as the unit n -sphere, formally,
Revised: Denote S n as the unit n -sphere, formally,

Original: S n = { x ∈ R n +1 |(cid:107) x (cid:107) 2 = 1 } .
Revised: S n = { x ∈ R n +1 |(cid:107) x (cid:107) 2 = 1 } .

Original: Below by A ( · , · ) , we denote the angular distance between two points on S n , i.e. ,
Revised: Below by A ( · , · ) , we denote the angular distance between two points on S n , i.e. ,

Original: A ( u , v ) =
Revised: A ( u , v ) =

Original: arccos( (cid:104) u , v (cid:105) (cid:107) u (cid:107)(cid:107) v (cid:107) ) .
Revised: arccos( (cid:104) u , v (cid:105) (cid:107) u (cid:107)(cid:107) v (cid:107) ) .

Original: Let x be the feature embeddings input for the layer before the last one of the classiﬁer of the pretrained CNN models, eg.
Revised: Let x be the feature embeddings input for the layer before the last one of the classiﬁer of the pretrained CNN models, eg.

Original: FC2 for VGG19.
Revised: FC2 for VGG19.

Original: Let C be the number of classes for a classiﬁcation task.
Revised: Let C be the number of classes for a classiﬁcation task.

Original: Denote
Revised: Denote

Original: W = w i | 0 < i ≤ C} as the set of weights for all C classes in the ﬁnal layer of the classiﬁer.
Revised: W = w i | 0 < i ≤ C} as the set of weights for all C classes in the ﬁnal layer of the classiﬁer.

Original: Deﬁnition (Angular
Revised: Deﬁnition (Angular

Original: Visual
Revised: Visual

Original: Hardness (AVH)) .
Revised: Hardness (AVH)) .

Original: AVH, for any x , is deﬁned as,
Revised: AVH, for any x , is deﬁned as,

Original: AVH (x) =
Revised: AVH (x) =

Original: A(x,w y ) (cid:80) Ci =1 A ( x,w i ) ,which w y represents the weights of the target class.
Revised: A(x,w y ) (cid:80) Ci =1 A ( x,w i ) ,which w y represents the weights of the target class.

Original: Prediction 1: (cid:107) x (cid:107) has
Revised: Prediction 1: (cid:107) x (cid:107) has

Original: a strong correlation with Human Selection Frequency
Revised: a strong correlation with Human Selection Frequency

Original: (14) conjectures that (cid:107) x (cid:107) 2 accounts for intra-class Human/Model Conﬁdence.
Revised: (14) conjectures that (cid:107) x (cid:107) 2 accounts for intra-class Human/Model Conﬁdence.

Original: Particularly, if the norm is larger, the prediction from the model is also more conﬁdent, to some extent.
Revised: Particularly, if the norm is larger, the prediction from the model is also more conﬁdent, to some extent.

Original: Therefore, we conduct similar experiments like previous section to demonstrate the correlation between (cid:107) x (cid:107) 2 and Human Selection Frequency.
Revised: Therefore, we conduct similar experiments like previous section to demonstrate the correlation between (cid:107) x (cid:107) 2 and Human Selection Frequency.

Original: Initially, we compute the (cid:107) x (cid:107) 2 for every validation sample for all models.
Revised: Initially, we compute the (cid:107) x (cid:107) 2 for every validation sample for all models.

Original: Then we normalize (cid:107) x (cid:107) 2 within each class.
Revised: Then we normalize (cid:107) x (cid:107) 2 within each class.

Original: The middle plot in ﬁgure 1 uses a two-dimensional histogram to show the correlation for all the validation images.
Revised: The middle plot in ﬁgure 1 uses a two-dimensional histogram to show the correlation for all the validation images.

Original: Given that the norm has been normalized with each class, naturally, there is notable density when the norm is 0 or 1 .
Revised: Given that the norm has been normalized with each class, naturally, there is notable density when the norm is 0 or 1 .

Original: Except for that, there is no obvious correlation between (cid:107) x (cid:107) 2 and Human
Revised: Except for that, there is no obvious correlation between (cid:107) x (cid:107) 2 and Human

Original: Selection Frequency.
Revised: Selection Frequency.

Original: We further verify if presenting all samples across 1000 different classes affects the visualization of the correlation.
Revised: We further verify if presenting all samples across 1000 different classes affects the visualization of the correlation.

Original: According to WordNet (5) hierarchy, we map the original 1000 ﬁne-grained classes to 45 higher hierarchical classes.
Revised: According to WordNet (5) hierarchy, we map the original 1000 ﬁne-grained classes to 45 higher hierarchical classes.

Original: Figure 7 exhibits the relationship between Human Selection
Revised: Figure 7 exhibits the relationship between Human Selection

Original: Frequency and (cid:107) x (cid:107) 2 for three representative higher classes containing 58 , 7 , 1 ﬁne-grained classes respectively.
Revised: Frequency and (cid:107) x (cid:107) 2 for three representative higher classes containing 58 , 7 , 1 ﬁne-grained classes respectively.

Original: Noted that there is still not any visible direct proportion between these two variables across all plots.
Revised: Noted that there is still not any visible direct proportion between these two variables across all plots.

Original: Prediction 2: AVH ( x ) has a strong correlation with Human Selection Frequency
Revised: Prediction 2: AVH ( x ) has a strong correlation with Human Selection Frequency

Original: We test the correlation between AVH ( x ) and Human Selection Frequency.
Revised: We test the correlation between AVH ( x ) and Human Selection Frequency.

Original: Correspondingly, after evaluating each validation sample on pre-trained models, we extract feature embeddings x and also the class weights W to compute
Revised: Correspondingly, after evaluating each validation sample on pre-trained models, we extract feature embeddings x and also the class weights W to compute

Original: AVH ( x ) .
Revised: AVH ( x ) .

Original: Noted that we linear scale the range of AVH ( x ) to [0 , 1] .
Revised: Noted that we linear scale the range of AVH ( x ) to [0 , 1] .

Original: The plot on the right in Figure 1 shows strong correlation between AVH ( x ) and Human Selection Frequency for validation images.
Revised: The plot on the right in Figure 1 shows strong correlation between AVH ( x ) and Human Selection Frequency for validation images.

Original: One intuition behind this correlation is that the class weights W might corresponds to human semantic for each category and thereby AVH ( x ) corresponds to human semantic categorization of an image.
Revised: One intuition behind this correlation is that the class weights W might corresponds to human semantic for each category and thereby AVH ( x ) corresponds to human semantic categorization of an image.

Original: Embedding (cid:96)
Revised: Embedding (cid:96)

Original: Norm (cid:107) x (cid:107) 2 is on the other hand irrelevant.
Revised: Norm (cid:107) x (cid:107) 2 is on the other hand irrelevant.

Original: In order to test if the strong correlation holds for all models, we perform the same experiments on AlexNet, VGG19 and
Revised: In order to test if the strong correlation holds for all models, we perform the same experiments on AlexNet, VGG19 and

Original: DenseNet121.
Revised: DenseNet121.

Original: Figure 6 shows the strong correlation of
Revised: Figure 6 shows the strong correlation of

Original: AVH ( x ) and Human Selection Frequency consistently.
Revised: AVH ( x ) and Human Selection Frequency consistently.

Original: 3. Dynamics of AVH during Training
Revised: 3. Dynamics of AVH during Training

Original: After discovering the strong correlation of human visual hardness and AVH score, a natural question would be: What role does AVH play during the training process?
Revised: After discovering the strong correlation of human visual hardness and AVH score, a natural question would be: What role does AVH play during the training process?

Original: Optimization Algorithms are used to update weights and biases i.e. the internal parameters of a model to improve the training loss.
Revised: Optimization Algorithms are used to update weights and biases i.e. the internal parameters of a model to improve the training loss.

Original: Both the angles between feature embedding and classiﬁers and the L 2 norm of the embedding can inﬂuence the loss.
Revised: Both the angles between feature embedding and classiﬁers and the L 2 norm of the embedding can inﬂuence the loss.

Original: While it is well-known that the training loss or accuracy keeps improving but it is not obvious what would be the dynamics of the angles and norms separately during training.
Revised: While it is well-known that the training loss or accuracy keeps improving but it is not obvious what would be the dynamics of the angles and norms separately during training.

Original: we design the experiments to observe the training dynamics of various network measurements like (cid:107) x (cid:107) 2 and
Revised: we design the experiments to observe the training dynamics of various network measurements like (cid:107) x (cid:107) 2 and

Original: AVH(x).
Revised: AVH(x).

Original: Experiment Settings For datasets and models, we use exactly the same setting as the experiments in ?? .
Revised: Experiment Settings For datasets and models, we use exactly the same setting as the experiments in ?? .

Original: Nevertheless, observing training dynamics involves training models from scratch on ImageNet training set instead of directly using the pre-trained models.
Revised: Nevertheless, observing training dynamics involves training models from scratch on ImageNet training set instead of directly using the pre-trained models.

Original: Therefore, we follow the standard training process of AlexNet (12), VGG19 (19),
Revised: Therefore, we follow the standard training process of AlexNet (12), VGG19 (19),

Original: DenseNet121 (9), ResNet50 (8).
Revised: DenseNet121 (9), ResNet50 (8).

Original: For consistency, we train all four models for 90 epochs and decay the initial learning
Revised: For consistency, we train all four models for 90 epochs and decay the initial learning

Original: 20 40 60 80 Epoch30 40 5070 80|| x || 20 40 60 80 Epoch

Revised: 20 40 60 80 Epoch30 40 5070 80|| x || 20 40 60 80 Epoch

Original: || x
Revised: || x

Original: || 

Revised: ||

Original: 40 60 80 Epoch

Revised: 40 60 80 Epoch

Original: || x || 20 40 60 80 Epoch 7.75 8.
Revised: || x || 20 40 60 80 Epoch 7.75 8.

Original: AV H ( x ) 1e  20 40 60 80 Epoch 7. AV H ( x ) 1e  20 40 60 80 Epoch 6.
Revised: AV H ( x ) 1e  20 40 60 80 Epoch 7. AV H ( x ) 1e  20 40 60 80 Epoch 6.

Original: AV H ( x ) 1e  20 40 60 80 Epoch

Revised: AV H ( x ) 1e  20 40 60 80 Epoch

Original: M o d e l A cc u r a c y 20 40 60 80 Epoch 020 3050 

Revised: M o d e l A cc u r a c y 20 40 60 80 Epoch 020 3050

Original: M o d e l A cc u r a c y 20 40 60 80 Epoch30 

Revised: M o d e l A cc u r a c y 20 40 60 80 Epoch30

Original: M o d e l A cc u r a c y rate by a factor of 10 every 30 epochs.
Revised: M o d e l A cc u r a c y rate by a factor of 10 every 30 epochs.

Original: The initial learning rate for AlexNet and VGG19 is 0 .
Revised: The initial learning rate for AlexNet and VGG19 is 0 .

Original: 01 and for DensetNet and ResNet50 is 0 .
Revised: 01 and for DensetNet and ResNet50 is 0 .

Original: We split all the validation images into 5 bins based on their human selection frequency, respectively [0 . 0 , 0 . 2] , [0 . 2 , 0 . 4] , [0 . 4 , 0 . 6] , [0 . 6 , 0 . 8] , [0 . 8 , 1 . 0] for experiments on dynamics under different human selection frequency.
Revised: We split all the validation images into 5 bins based on their human selection frequency, respectively [0 . 0 , 0 . 2] , [0 . 2 , 0 . 4] , [0 . 4 , 0 . 6] , [0 . 6 , 0 . 8] , [0 . 8 , 1 . 0] for experiments on dynamics under different human selection frequency.

Original: Note that for all the ﬁgures in this section, Epoch starts from 1.
Revised: Note that for all the ﬁgures in this section, Epoch starts from 1.

Original: Observation 1: Dynamics of (cid:107) x (cid:107) 2 and model accuracy are similarly increasing Figure 2 presents the dynamics of the average (cid:107) x (cid:107) 2 and the dynamics of the accuracy for testing samples vary in 90 epochs during the training on four architectures.
Revised: Observation 1: Dynamics of (cid:107) x (cid:107) 2 and model accuracy are similarly increasing Figure 2 presents the dynamics of the average (cid:107) x (cid:107) 2 and the dynamics of the accuracy for testing samples vary in 90 epochs during the training on four architectures.

Original: Note that we are using the validation data for dynamics observation and thereby have never ﬁt them into the model.
Revised: Note that we are using the validation data for dynamics observation and thereby have never ﬁt them into the model.

Original: The average (cid:107) x (cid:107) 2 increases with a small initial slope but it suddenly climbs after 30 epochs when the ﬁrst learning rate decay happens.
Revised: The average (cid:107) x (cid:107) 2 increases with a small initial slope but it suddenly climbs after 30 epochs when the ﬁrst learning rate decay happens.

Original: The accuracy curve is very similar to that of the average (cid:107) x (cid:107) 2 .
Revised: The accuracy curve is very similar to that of the average (cid:107) x (cid:107) 2 .

Original: The above observations are consistent in all models.
Revised: The above observations are consistent in all models.

Original: Observation 2: AVH(x) hits a plateau very early even when the accuracy or loss is still improving Figure 2 exhibits the change of average AVH(x) for testing samples in 90 epochs of training on four models.
Revised: Observation 2: AVH(x) hits a plateau very early even when the accuracy or loss is still improving Figure 2 exhibits the change of average AVH(x) for testing samples in 90 epochs of training on four models.

Original: The average AVH(x) for AlexNet and VGG19 decreases sharply at the beginning and then starts to bounce back a little bit before converging.
Revised: The average AVH(x) for AlexNet and VGG19 decreases sharply at the beginning and then starts to bounce back a little bit before converging.

Original: However, the dynamics of the average AVH(x) for
Revised: However, the dynamics of the average AVH(x) for

Original: DenseNet121 and ResNet50 are different.
Revised: DenseNet121 and ResNet50 are different.

Original: They both decrease slightly and then quickly hits a plateau in all three learning rate decay stages.
Revised: They both decrease slightly and then quickly hits a plateau in all three learning rate decay stages.

Original: But the common observation is that they all stop improving even when (cid:107) x (cid:107) 2 and model accuracy are increasing.
Revised: But the common observation is that they all stop improving even when (cid:107) x (cid:107) 2 and model accuracy are increasing.

Original: However, AVH(x) is more impor20 40 60 80 Epoch30 40 5070 80||
Revised: However, AVH(x) is more impor20 40 60 80 Epoch30 40 5070 80||

Original: x ||
Revised: x ||

Original: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0]
Revised: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0]

Original: 20 40 60 80 Epoch3050|| x ||
Revised: 20 40 60 80 Epoch3050|| x ||

Original: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 200 1030 40|| x ||
Revised: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 200 1030 40|| x ||

Original: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 8.0 8.
Revised: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 8.0 8.

Original: AV H ( x ) 1e
Revised: AV H ( x ) 1e

Original: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 7.0 7.
Revised: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 7.0 7.

Original: AV H ( x ) 1e
Revised: AV H ( x ) 1e

Original: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 6.5 7.
Revised: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 6.5 7.

Original: AV H ( x ) 1e
Revised: AV H ( x ) 1e

Original: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 0.0 0.
Revised: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 0.0 0.

Original: M o d e l A cc u r a c y Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 0.0 0.1 0.
Revised: M o d e l A cc u r a c y Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 0.0 0.1 0.

Original: M o d e l A cc u r a c y
Revised: M o d e l A cc u r a c y

Original: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 0.1 0.
Revised: Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] 20 40 60 80 Epoch 0.1 0.

Original: M o d e l A cc u r a c y Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] tant than (cid:107) x (cid:107) 2 because during inference it is the key factor deciding which class the input sample is classiﬁed to The dynamics for (cid:107) x (cid:107) 2 should be monotonically decreasing so we raise the question that is the cross-entropy loss the best for CNNs?
Revised: M o d e l A cc u r a c y Freq=[0.0 ,0.2] Freq=[0.2 ,0.4] Freq=[0.4 ,0.6] Freq=[0.6 ,0.8] Freq=[0.8 ,1.0] tant than (cid:107) x (cid:107) 2 because during inference it is the key factor deciding which class the input sample is classiﬁed to The dynamics for (cid:107) x (cid:107) 2 should be monotonically decreasing so we raise the question that is the cross-entropy loss the best for CNNs?

Original: Observation 3: AVH(x)’s correlation with human selection frequency holds across models and throughout the training process.
Revised: Observation 3: AVH(x)’s correlation with human selection frequency holds across models and throughout the training process.

Original: Figure 3 demonstrates the change of (cid:107) x (cid:107)and AVH(x) similar to Figure 2, but average over testing samples in ﬁve human selection frequency bins separately.
Revised: Figure 3 demonstrates the change of (cid:107) x (cid:107)and AVH(x) similar to Figure 2, but average over testing samples in ﬁve human selection frequency bins separately.

Original: We can observe that for (cid:107) x (cid:107) 2 , the gaps between the samples with different human visual hardness are not obvious in
Revised: We can observe that for (cid:107) x (cid:107) 2 , the gaps between the samples with different human visual hardness are not obvious in

Original: ResNet50 and DenseNet121.
Revised: ResNet50 and DenseNet121.

Original: Besides and closed near convergence.
Revised: Besides and closed near convergence.

Original: However, for AVH(x), such gaps are signiﬁcant and consistent across every single model during the whole training process.
Revised: However, for AVH(x), such gaps are signiﬁcant and consistent across every single model during the whole training process.

Original: Observation 4: AVH(x) is an indicator of model’s generalization ability From Figure 2 and Figure 3, we observe that better models have better AVH(x) throughout the training process and also across samples under different human selection frequency.
Revised: Observation 4: AVH(x) is an indicator of model’s generalization ability From Figure 2 and Figure 3, we observe that better models have better AVH(x) throughout the training process and also across samples under different human selection frequency.

Original: For instance, Alexnet is the worst model and its overall AVH(x) or average AVH(x) on each of ﬁve bins are worse than those of the other three models.
Revised: For instance, Alexnet is the worst model and its overall AVH(x) or average AVH(x) on each of ﬁve bins are worse than those of the other three models.

Original: This observation aligned with the earlier observations of (16) that better models also generalize better on samples across different human selection frequencies.
Revised: This observation aligned with the earlier observations of (16) that better models also generalize better on samples across different human selection frequencies.

Original: A. Additional Experiments
Revised: A. Additional Experiments

Original: Figure 4 illustrates how the average norm of the feature embedding and angles between feature and class embedding for testing samples vary in 60 iterations during the training process.
Revised: Figure 4 illustrates how the average norm of the feature embedding and angles between feature and class embedding for testing samples vary in 60 iterations during the training process.

Original: The average norm increases with a large initial slope but it ﬂattens slightly after 10 iterations.
Revised: The average norm increases with a large initial slope but it ﬂattens slightly after 10 iterations.

Original: On the other hand, the average angle decreases sharply at the beginning and then becomes almost ﬂat after 10 iterations.
Revised: On the other hand, the average angle decreases sharply at the beginning and then becomes almost ﬂat after 10 iterations.

Original: Moreover, we explore the difference between norm and angle change for easy and hard human examples in more details.
Revised: Moreover, we explore the difference between norm and angle change for easy and hard human examples in more details.

Original: Figure 5 also plots the angle and norm changes for two examples, which are hard and easy for human visualization, in the training phase.
Revised: Figure 5 also plots the angle and norm changes for two examples, which are hard and easy for human visualization, in the training phase.

Original: Note that both examples are testing data and thereby have never ﬁt into the model.
Revised: Note that both examples are testing data and thereby have never ﬁt into the model.

Original: We can see that for the angle, both of them drop largely initially and then the angle for the easy one converges to a much lower value.
Revised: We can see that for the angle, both of them drop largely initially and then the angle for the easy one converges to a much lower value.

Original: For the norm, both of them are increasing drastically at an early stage but that for the harder example keeps climbing even when that for the easy one saturates.
Revised: For the norm, both of them are increasing drastically at an early stage but that for the harder example keeps climbing even when that for the easy one saturates.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: L 2 N o r m 0 26 812# o f S a m p l e s 0.0 0.2 0.4 0.

Revised: L 2 N o r m 0 26 812# o f S a m p l e s 0.0 0.2 0.4 0.

Original: 0.8 1.0 Human Accuracy 0.
Revised: 0.8 1.0 Human Accuracy 0.

Original: L 2 N o r m 0.0 0.
Revised: L 2 N o r m 0.0 0.

Original: # o f S a m p l e s 0.0 0.2 0.4 0.

Revised: # o f S a m p l e s 0.0 0.2 0.4 0.

Original: 0.8 1.0 Human Accuracy 0.
Revised: 0.8 1.0 Human Accuracy 0.

Original: L 2 N o r m 0.00 0.25 0.
Revised: L 2 N o r m 0.00 0.25 0.

Original: # o f S a m p l e s under different class granularity (according to WordNet hierarchy).
Revised: # o f S a m p l e s under different class granularity (according to WordNet hierarchy).

Original: From left to right, there are 58, 7, 1 classes respectively.
Revised: From left to right, there are 58, 7, 1 classes respectively.

Original: The human selection frequency is therefore computed based on the new class granularity.
Revised: The human selection frequency is therefore computed based on the new class granularity.