CVPR #*****
Representation Analysis of Neural Networks Trained on Biased and Out-Of-Distribution Data
Anonymous CVPR submission
Paper ID *****
Abstract
It is observed that neural networks trained on standard image classification datasets are less robust to distributional shifts and are pertaining to certain levels of biases in the representations.
The existing literature provides insights of implying distinct objective functions to improve the performance and representation quality of neural networks.
But, there is less research conducted describing the quality of representations acquired when neural networks are exposed to biased and out-of-distribution data.
In this work, we analyze the performance and the quality of representations acquired from standard neural networks exposed to biased and out-of-distribution data.
Specifically, we investigate the representation quality acquired for distinct objective functions (probabilistic and margin-based) and provide a detailed analysis of these distinct objective functions convolution-based neural networks.
Our analysis reports that representations acquired by ResNets using Softmax Cross-Entropy and Negative Log-Likelihood (NLL) are equally competent in providing superior performance and fine representations on OOD and biased data.
Subsequently, we comprehend that without refined representations the ResNets are less likely to be robust on cross-data generalization.
This work is publicly available at https://github.com/libra0901/lossanalysis-cka .
1. Introduction
The advent of deep learning has provided us with robust models and the finest neural architecture.
Even with tremendous performance, these neural networks lack in discriminating the representations because of training them in the controlled setting [24].
Hence, they tend to perform poorly with distributional shifts and biases in data.
To mitigate this issue various methods were implied with either optimization of neural networks or by following a paradigm to mitigate this issue [20] [2] [27] [9].
Further, there are fewer works addressing the representations acquired by neural networks on biased data.
Comprehending the representations acquired from biased data would aid researchers to provide a novel de-biasing neural network or a bias mitigation strategy.
Similarly, there are fewer works specifying the importance of refined representations for outof-distribution (OOD) data and their implications on crossdata.
The work by Simon et al.
[15] has provided insights regarding the transferability of representations when exposed to diverse loss functions and mainly focused on probabilistic objectives.
But, Janocha et al.
[11] provided theoretical justification for margin-based objectives.
Hence, these concerns motivate us to understand the behavior of neural networks on various datasets, mainly biased and OOD, both in terms of performance and representational strength.
2. Contributions
We address certain queries that were raised while com- prehending existing literature.
The contribution of our work is the analysis conducted with the raised queries.
Query.
Which objective function acquires good representations for data with the distributional shifts and biases?
Ans.
By experimenting on several standard data, including biased and OOD, we find Softmax CrossEntropy (SCE) and Negative Log-Likelihood (NLL) are equally competitive to produce higher performance and acquire reasonable representations and can be standardized for generic image classification settings.
Query.
Do the representations of neural networks vary by varying objective functions (probabilistic and margin-based losses) for OOD and biased data?
Ans.
Yes, they do vary.
The representations acquired by probabilistic losses were more qualitative and onthe-other-hand
margin-based representations saturate at initial layers.
Query.
How the performance and representations differ for data trained for another task (cross-data generalization)?
Ans
If the neural networks are poorly correlated with their layers then they are less likely to acquire good representations on unseen data (cross-data).
If the neural networks were pre-trained on larger sample sets, then we obtain greater performance with better representations.
3. Empirical Analysis
Terminology The input data is represented as X ∈ { x (1) , x (2) , . . .
, x ( n ) } where n is total number of samples.
The encoder Enc ( . ) is used to extract features from a given input X .
The features extracted from encoder are presented as, fv ← Enc ( X ) ; where fv ∈ { fv (1) , fv (2) , . .
. , fv ( n ) } .
The dimensions of the feature vector vary by changing the encoder.
As most of the experiments were conducted using a supervised framework, the datasets do have certain ground truth labels and these are represented as Y ∈ { y (1) , y (2) , . . .
, y ( n ) } .
The activation functions sigmoid and softmax are denoted as σ ( a i )
=
1 / (1 + e − a i ) and
S ( a i ) =
e a i / (cid:80)
j e a j respectively.
The loss (objective) function are denoted by L ( . ) and the suffix indicate its specified variant.
The norms ∥·∥ 1 , and ∥·∥ 2 indicates Manhattan and Euclidean norms 1 respectively.
Datasets As mentioned, we choose the data for solving three diverse problems.
First, we consider well studied and experimented datasets such as MNIST [18], CIFAR10 [17] and ImageNet-200.
These data sets are widely used for standard image classification tasks likewise, they do not have inherent biases and do not contain OOD samples.
Next, Colored MNIST (C-MNIST) and Corrupted CIFAR
(C-CIFAR)
[19] data sets are utilized for understanding biased representations.
For both the datasets, C-MNSIT and C-CIFAR we perform experiments according to Lee et al.
Lastly, we consider data with distributional shifts such as ImageNet Renditions (ImageNet-R) [7] and Modified MNIST
(MNIST-M) [6].
The collection of datasets that are considered for experimentation is tabulated in Table.
Models In this work, we use ResNet18 and ResNet50 to understand convolution-type representations.
ResNet18 is used for small-sized data which are illustrated in the tablesimilarly, for medium-sized data we imply ResNet50.
We have considered standard ResNets with global average pooling.
The fully connected layers for ResNet18 and ResNet are [512-c] and [2048-512-c] respectively.
Intermediate dropout layers are used with a drop rate of 40% and set constant for all datasets to have a fair evaluation. {+ni a 
+}
i [Table] Training and fine-tuning To train each model we utilized Adam [14] as optimizer with a standard learning rate of 10 − 3 and a weight decay of 10 − 5 .
For certain executions [Table] we utilized pre-trained ImageNet weights and finetuned the models on that specific dataset (if not particularly mentioned, we trained the model from scratch).
We fedsamples as a batch into the neural networks by varying objective functions.
The early stopping criterion is embedded with the patience of 12 epochs to ensure the model does not overfit with excessive training.
The results are reported in the Table.
2 are produced without considering any augmentations (except normalization) and without any pre-trained weights.
3.1. Performance Analysis of Objective Function
The choice of the objective function to train a deep neural network remains a question.
Janocha et al.
[11] provides theoretical justification and conducts experiments on MNIST pointing to the importance of L 1 , and L 2 not just as regularizers but as objective functions for better generalizations.
Hui et al.
[10] empirically prove that square loss with a little parametric tuning would produce significant outcomes for the majority of Natural Language Processing (NLP) and Automatic Speech Recognition (ASR) tasks.
They specifically mentioned that proposed square loss is not brittle to randomized initialization.
A very recent analysis by Simon et al.
[15] provides insights noting that representations acquired to classify certain tasks with more class separation leads to poor transferable features.
These studies imply diverse objective functions and observe the performance and quality of representations for the standard computer vision classification datasets.
There is sufficient literature explaining theoretical and empirical justifications for standard (generic) classification datasets in computer vision.
So we shed a light on these challenges and perform experimentation on datasets with biased and OOD samples.
It is worth mentioning that, Simon et al.
[15] has experimented on various loss functions for OOD data
[Table]
such as ImageNet-C, ImageNet-R [7] and ImageNet-A [8].
But there was less information regarding the relevance of representation learned by margin-based loss functions and did not perform detailed representation analysis on biased and OOD data.
Hence
[Table]
we provide analysis for two variants of objective functions to understand the learned representations (Table. 1).
calculate the error approximating underlying probabilities for representations acquired from an encoder.
: These objective functions calculate the error by discriminating the extracted representations from an encoder.
3.1.1 Probabilistic Objectives
Softmax Cross-Entropy (SCE)
This loss function has leveraged its utility for multi-class image classification.
This objective function is obtained by applying softmax at the final layer of the neural network and this feed is minimized by negative log-likelihood.
(NLL)
The equation is illustrated as,
[Equation]
Binary Cross-Entropy (BCE) Binary cross-entropy is obtained by applying sigmoid activation ( σ ( · ) ) at final layer of neural networks and this information is minimized by
NLL.
Mostly, this loss function is used for binary classification problem but, a recent work empirically proves that its implication on multi-class would lead to better performance [3] by applying one-vs-rest strategy.
BCE objective is noted as,
[Table]
[Equation]
Negative Log-Likelihood (NLL)
The likelihood provides the joint probability of the sample distribution and instead of maximizing the likelihood, it is preferred to minimize the negative logarithm of the obtained likelihood [4].
[Equation]
3.1.2 Margin-Based Objectives
Mean Absolute Error ( L 1 )
This objective function finds the Manhattan distance between the two representations.
We acquire the theoretical motivation from Janocha et al.
[11] that L 1 would reduce sparseness in the representations.
So before directly discriminating the representations at the last layer, we imply softmax to ensure appropriate learning without saturation of partial derivatives 2 .
[Equation]
Mean Squared Error ( L 2 )
This objective function finds the Euclidean distance between two representations.
The underlying theoretical justifications from Janocha et al.
[11] illustrates that, L 2 reduces the weights reaching the infinity.
Similar to L 1 , the final layer representations undergo a softmax activation to produce L 2 as,
[Equation]
Sum-of-Squares (SoS)
The sum of squares objective is widely used for tasks with continuous variables (preferably for regression task).
Recently, Hui et al.
[10] rescaled SoS to be more robust by providing two parameters α, β .
Inserting these parameters had resulted in decent performance for
NLP and ASR tasks but, poorly performed on Computer Vision tasks.
For experimentation we have picked α = β
=and this reduces to generic SoS.
The equation is illustrated as,
[Equation]
Now let under understand the empirical performance of these objective functions trained on all variants of data detailed in Table.
For generic data, in most of the cases,
L SCE and L NLL were able to achieve top accuracy scores.
But, L 2 obtained the highest accuracy for CIFAR-10 and competed closely with L NLL on MNIST.
Considering the case of biased data L NLL performed standalone; L SCE and
L SoS were able compete closely with L NLL .
Surprisingly,
L 1 turned out to have good performance for MNIST-M, and L SoS was closely on par.
But for ImageNet-R except L SCE and L NLL all other objectives failed to obtain at least 10% accuracy score.
Hence, aggregating these results we can strongly recommend using probabilistic loss functions ( L SCE and L NLL ) for decent accuracy scores on most of the data.
The empirical performance attained by these objective functions is well understood but, the question arises with the internal representations of the model trained on these data. CVPR #***** CVPR
4. Representation Analysis
In this section, we analyze the visual representations learned of neural networks using CKA [16].
First, we address representations learned by individual objective functions trained on ResNets (as per Table. 2).
As we get huge combinations so, we depict CKA for all the objective functions trained on three variants of data MNIST, C-MNIST, and MNIST-M.
Secondly, we compare and contrast representations learned by ResNets for cross-data generalization.
4.1. CKA
It is well established in literature that CKA [16] acquires qualitative representations compared to PwCCA [21] and
SVCCA [23].
CKA not only captures the correspondence between the representations of a neural network but also allows to compute the similarity between pairs of layers.
Given that, linear CKA is ought to compete with RBF CKA on a performance basis without any additional kernel parameters.
To reduce computational expense consumed by linear CKA, mini-batch CKA is applied by computing the mean of HISC scores over selected mini-batches ( N ).
[Equation]
Where, ˜ X i =
X i X Ti ; ˜ Y i
=
Y i
Y Ti .
These X i ∈ R n × dand Y i ∈ R n × d 2 are activation matrices for i th mini-batch of examples without replacement.
This strategy is implemented according to the Thao et al.
Now we try to analyse the representations using CKA for all the objective functions on aforementioned data.
5.2. Exploring Underlying Representations 5.2.1 Analysing Representations on Variant Objectives
The Figure 1 and Figure 2 visually illustrates the representations learned by ResNet18 for all the objective functions
CVPR #***** CVPR #*****
P r e t r a i n e d :
N on e
CMNIST
MNIST-M
CMNIST
MNIST-M CMNIST MNIST-M
P r e t r a i n e d : I m a g e N e t 1k
5. Representation Analysis
In this section, we analyze the visual representations learned of neural networks using CKA [16].
First, we address representations learned by individual objective functions trained on ResNets (as per Table. 2). As we get huge combinations so, we depict CKA for all the objective functions trained on three variants of data MNIST, C-MNIST, and MNIST-M. Secondly, we compare and contrast representations learned by ResNets for cross-data generalization. 5.1. CKA
It is well established in literature that CKA [16] acquires qualitative representations compared to PwCCA [21] and
SVCCA [23].
CKA not only captures the correspondence between the representations of a neural network but also allows to compute the similarity between pairs of layers.
Given that, linear CKA is ought to compete with RBF CKA on a performance basis without any additional kernel parameters.
To reduce computational expense consumed by linear CKA, mini-batch CKA is applied by computing the mean of HISC scores over selected mini-batches ( N ). (provided in Table 2) on MNIST, C-MNIST and MNIST-M.
First, let us try to understand the representations acquired by probabilistic objectives.
In the Figure 1, for MNIST data (first column), the representations of L SCE , L BCE and
L NLL at 44 th layer has least correlation with other layers.
Further, L SCE representations have a high correlation with all the layers.
Whereas for L BCE and L NLL the similarity slowly decreases with progressing layers.
Therefore, L SCE having high similarity with utmost layers indicates that most of the layers acquire similar patterns with redundant computations.
Next, considering the scenario of CMNIST and MNIST-M, it is very clear that most of the representations are slowly reducing the intensity of correlation and form a partial box-shaped structure of representations.
This indication obliges to comprehend that, performance saturates with 3/4 th of complete layers and rejects the use of end layers (after 44).
Now, from Figure 2, the visualizations of objectives Land L 2 are similar but, SoS seems to capture representations very differently.
For SoS on MNIST, we can reject half of the ResNet18 layers as they mostly provide exact representations for another half portion.
The CKA for L 1 and L 2 for
C-MNIST data seem to have uniformity in representations till 44 th layer and after that, the correlation declines.
The representation behavior of SoS on C-MNIST is not typical as the first half of the network has high similarity and for the next portion, the similarity declines continuously.
While considering the case of MNIST-M, the correlation acquired by L 1 and L 1 started decaying just after 20 th layer but, SoS forms a box kind of representation structure which indicates that the first 20 layers are sufficient to learn the invariances.
These indications are clear to note that, the objectives
L SCE and L NLL provide not only provide decent empirical performance but capture fine representations with ResNets.
Hence, the representations acquired by probabilistic objectives are comparatively fine to provide good generalizations for diverse datasets. CVPR #***** CVPR #***** REVIEW
(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)
[Table]
[Table] [Table] [Table] 5.2.2 Analysing Cross-Data Representations
The empirical performance obtained by ResNet18 for crossdata generalization, is depicted in Table 3.
The relevant
CKA visualizations for Table 4 and 5 are illustrated in the
Figure ?? .
First, while focusing on empirical performance, in Table 3 when data sets were trained and tested on the same model the generalization was good.
Whereas, for crossdata, the accuracy drop is observed.
To analyze this crossdata generalization issue, we have not used any pre-trained weights and also used ImageNet1k as pre-trained weights for ResNets.
For ResNets trained on C-MNIST without any weights the performance of it on MNIST-M is 11.45% and whereas, its performance on its test samples is nearly 95.34%.
Similarly, consider the case for ResNets trained on C-MNIST with ImageNet1k pre-trained weights, the accuracy score tested on the same data gives 95.69% and on MNIST-M gives 52.4%.
Similarly, for remaining all cases this tends to justify that, there is a huge impact on providing sufficient data to produce refined representations.
Now let us compare the visualizations from the Figure ??
, the representations acquired by ResNets trained and tested on C-MNIST (or) MNIST-M produces good representations.
Whereas trained and tested on other data, the correlation strength for most of the layers seems zero i.e, similarity reduces drastically.
In the presence of pre-trained weights, the models can sustain new distributions and provide a tiny increment in the performance (both empirical and representational).
Hence, it is understood that neural networks when trained on additional data comprising of various distributions then we observe greater performance with refined representations.
6. Related Work 6.1. Bias
Some works focus on developing a new objective function to tackle the bias problem.
Kim et al.
[13] utilizes the gradient reversal layer and proposes a novel regularization loss function based on mutual information obtained from InfoGANs [5] to unlearn the target bias i.e., the bias present in the data.
This eventually minimizes the detrimental effect of bias in the data.
Adeli et al.
[1] proposed a loss function for adversarial training with two objectives to learn features that have maximum discriminative power and minimal statistical dependencies with protected bias.
Jung et al.
[12] provided two loss functions BiasCon and SoftCon which aims to reduce bias in the representations through contrastive learning paradigm.
Certain works provided novel optimization methods for solving bias problem [20] [2].
6.2. OOD
StableNet by Zhang et al.
[27] reduces the statistical relation between the irrelevant and relevant features acquired from data.
To achieve this, they have implied a sample weighting technique and trained on a new objective which is modified on SCE.
Under different settings, StableNet reported superior performance for various OOD datasets including MNIST-M.
The work by Rui [9] presents a novel method name GradNorm which can understand OOD samples from the gradient space constructed with the data proving greater performance for standard OOD datasets.
Sunil et al.
[26] proposed a new method of solving the OOD problem using principle of abstention (which encourages to predict the samples classes which weren’t seen earlier by the model) to provide OOD generalization. CVPR #*****
7. Limitations
This work has certain limitations and the authors would hope to fill these limitations in their future work.
To our knowledge, the perceived limitations are,
We have not compared with kernel parameters for RBF-based CKA and did not provide any analysis.
This analysis is constricted to small and medium-sized datasets.
In the future, we see the potential requirement of datasets comprising of samples with distribution shifts with a greater sample size as of ImageNet1k.
Likewise, the representations acquired by the models are to be ensured with the least bias possible.
8. Broader Impact
Since scientists and analysts routinely conclude, albeit based on evidence-based insights, OOD and Biased data may be inherently misleading.
The impact of such misinterpretation may be corrected through exposure, experience, and expertise when humans are involved in the interpretation.
However, if the data itself is OOD or biased or only the interpretations of the data were presented by researchers, the danger of unconscious or even conscious biases cannot be ruled out entirely.
While Smith’s book [25] is an elaborate example of how research methodologies are inherently designed with ignorance towards the subjects of study, subtler cases of bias can be innate to the ways data are sourced, stored, pre-processed, analyzed, and interpreted.
The current work emphasizes the need to methodically improve the representation quality of OOD and biased data without proposing radical paradigm shifts in the current methodologies.
The work does not organically provide scope for misinterpretation of the data or biased decisions made through data analytics.
Moreover, the work facilitates a better and more uniform representation of data by reminding researchers to consciously consider biases and OOD aspects of data which may be rather inconspicuous.
This work will have more pronounced impacts once the limitations stated in Section 7 are also overcome systematically.
A stronger motivation arises as Artificial Intelligence and
Machine Learning continue to be used in various technology and social domains for diverse applications.