Representation Analysis of Neural Networks Trained on Biased and Out-Of-Distribution Data
{+BMVC 
+}
Submission #
Abstract
Neural networks trained on standard image classification data sets are observed to be less robust to distributional shifts and pertain to certain levels of bias in representations.
Thus, it is pertinent to identify the kind of objective function that could correspond to better performance for data with biases and distribution shifts, and how can that objective function be justified to be the apt choice.
There is, however, less literature that focuses on the choice of the objective function and its representational structure when trained on such data sets.
In this work, we analyse the performance and the internal representational structure of convolution-based neural networks ( e.g . ResNets) trained by varying objective functions on biased and out-of-distribution (OOD) data.
Specifically, we investigate similarities in representations (using CKA) acquired for distinct objective functions (probabilistic and margin-based) and provide a detailed analysis of the chosen ones.
Our analysis reports that representations acquired by ResNets using Softmax Cross-Entropy (SCE) and Negative Log-Likelihood (NLL) as objectives are equally competent in providing superior performance and fine representations on OOD and biased data.
Subsequently, we comprehend that the ResNets are less likely to be robust on cross-data generalisation without refined representational similarity.
Introduction
The advent of deep learning has provided us with robust models and the finest neural architectures.
Even with tremendous performance, most neural networks do not discriminate representations well as they are trained in a controlled setting [24].
They thus tend to perform poorly with distributional shifts and biases in the data.
Various methods with neural network optimisation or a new paradigm of solving were implied to mitigate this issue [2], [9], [20], [27].
So, we question thus: what could be an optimal objective function for data with biases and distributional shifts? Eventually, what kind of representations are learnt by the neural networks when exposed to such data? How are the internal representations of neural networks altered by varying the data samples?
Finally, how transferable are the representations that are trained with data biases and data with distributional shifts?
In the existing literature, the work of Simon et al .
[15] has provided insight on the transferability of representations when exposed to various loss functions, but has focused mainly [Table] [Table] on probabilistic objectives.
Kim et al . [12] utilised the gradient reversal layer and proposed a novel regularised loss function based on mutual information obtained from InfoGANs [6] to unlearn the target bias, that is, the bias present in the data. This eventually minimises the detrimental effects of bias in the data.
Adeli et al . [1] proposed a loss function for adversarial training with two objectives to learn features that have maximum discriminative power and minimal statistical dependencies with protected bias. Certain works provided novel optimisation methods to solve the bias problem [2], [20]. StableNet by Zhang et al .
[27] reduces the statistical relation between irrelevant and relevant features acquired from the data. To achieve this, they have implied a sample weighting technique and trained on a new objective which is modified on SCE. Under different settings, StableNet reported superior performance for various OOD datasets including MNIST-M. Sunil et al .
[26] proposed a new method of solving the OOD problem using the principle of abstention (which encourages to predict the sample classes which were unseen by the model) to provide OOD generalisation. The existing literature is less evident on the use of objective functions on data with biases and distributional shifts.
The representational characteristics of a given objective function are not specifically studied.
The transferability of the representations—produced by training neural networks on biased or OOD data—to standard classification data is not illustrated.
Hence, these concerns motivate us to understand the behaviour of neural networks on various data sets, mainly biased and OOD.
First, we study the empirical performance of the objective functions by dividing them into two variants: a) probabilistic, and b)margin-based.
Secondly, we analyse the importance of individual objective functions which provide representation structure with good generalisation and transferability.
Setup
Terminology The input data is represented as X ∈ { x ( 1 ) , x ( 2 ) ,..., x ( n ) } where n is the total number of samples.
The encoder Enc ( . ) is used to extract features from a given input X .
The features extracted from the encoder are presented as fv ← Enc ( X ) ; where fv ∈ { fv ( 1 ) , fv ( 2 ) ,..., fv ( n ) } .
The dimensions of the feature vector vary by changing the encoder.
As most of the experiments were carried out using a supervised framework, the data sets do have certain ground truth labels, and these are represented as Y ∈ { y ( 1 ) , y ( 2 ) ,..., y ( n ) } .
The activation functions, sigmoid and softmax, are indicated by σ ( a i )
=
1 / ( 1 + e − a i ) and
S ( a i ) =
e a i / ∑ j e a j , respectively.
The loss (objective) function, Objective Function
Equation [Table] Euclidean norms 1 respectively.
Datasets As mentioned, we choose the data for solving three diverse problems.
First, we consider well-studied and experimented datasets such as MNIST [18], CIFAR-10[17] and ImageNet-200.
These data sets are widely used for standard image classification tasks; also, they do not have inherent biases and do not contain OOD samples.
Next, Colored MNIST (C-MNIST) and Corrupted [-CIFAR
-]CIFAR-10 (C-CIFAR) [19] data sets are utilized for understanding biased representations.
For both the datasets, C-MNIST and C-CIFAR, we perform experiments according to Lee et al .
Lastly, we consider data with distributional shifts such as ImageNet Renditions (ImageNet-R) [8] and Modified MNIST (MNIST-M) [7].
The collection of data sets considered for experimentation are tabulated in Table 1.
Models In this work, we use ResNet18 and ResNet50 to understand convolution-type representations.
ResNet18 is used for small data, which are illustrated in Table 1, similarly, for medium-sized data, we imply ResNet50.
We have considered standard ResNets with global average pooling.
The fully connected layers for ResNet18 and ResNet50 are [512- c ] and [2048-512- c ] respectively (Where c denotes the number of classes).
Intermediate dropout layers are used with a drop rate of 40% and a set constant for all data sets to have a fair evaluation.
Training and fine-tuning To train each model we utilized Adam [13] as an optimizer with a standard learning rate of 10 − 3 and a weight decay of 10 − 5 .
For certain executions, we utilised pre-trained ImageNet weights and fine-tuned the models on that specific data set (and when not particularly mentioned, we trained the model from scratch).
We fedsamples in batches to neural networks by varying the objective functions.
The early stopping criterion is embedded with the patience of 12 epochs to ensure the model does not over-fit with excessive training.
The results reported in Tables 3 and 4 are produced without any augmentations (except normalisation) and pre-trained weights.
Empirical Analysis
The choice of the objective function to train a deep neural network, on specified data remains a question.
Janocha et al .
[11] provided a theoretical justification and conducted experiments on MNIST pointing out the importance of L 1 and L 2 not just as regularizers, but as objective functions for better generalisations.
Hui et al .
[10] empirically proves that square loss with a little parametric tuning would produce significant results for most tasks of natural language processing (NLP) and automatic speech recognition (ASR).
Hui et al .
[10] specifically mentioned that the proposed square loss is not brittle for randomised initialisation.
A recent analysis by Simon et al .
[15] provides insights noting that the representations acquired to classify certain tasks with more class separation lead to poor transferable features.
This work implies various objective functions to observe both the performance and quality of representations for the standard computer vision classification data sets.
The previous literature focuses on the training and transferability of features acquired by training standard neural architectures with varying objective functions.
But, there is sparse literature noting the relevance of both probabilistic and margin-based objective functions on data with biases and distributional shifts.
Hence, we provide empirical analysis for two variants of objective functions to understand the performance of each objective function on various data sets mentioned in Table 1.
Probabilistic objective functions calculate the error that approximates the underlying probabilities for representations acquired from an encoder (ResNet).
In this paper, we include three probabilistic objectives and they are detailed in Table 2.
First, the Softmax cross-entropy [5], a highly used objective, is obtained by applying softmax activation in the final layer of the neural network, and this feed is minimised by the negative log-likelihood (NLL).
Next, Binary cross-entropy (BCE) is obtained by applying sigmoid activation ( σ ( · ) ) at the final layer of neural networks and this information is minimised by NLL.
Primarily, the BCE loss function is used for binary classification problems but, a recent work empirically proves that its implication on multi-class would lead to better performance [3] by applying the one-vs-rest strategy.
Finally, the likelihood provides the joint probability of the sample distribution and minimises the negative logarithm of the obtained likelihood [4].
3. Margin-based Objectives
Margin-based objective functions calculate the error by discriminating the representations extracted from an encoder (ResNet).
Similarly to probabilistic objectives, we include three margin-based objectives, which are detailed in Table 2.
First, the mean absolute error ( L 1 objective function) finds the Manhattan distance between the two representations.
We acquire the theoretical motivation of Janocha et al .
[11] that L 1 would reduce the sparseness in the representations.
Rather than directly discriminating the representations in the final layer, we use softmax to ensure appropriate learning without saturation of partial derivatives 2 .
Similar to L 1 , we use L 2 to find find the Euclidean distance between two representations (final layer).
Lastly, Hui et al .
[10] rescaled SoS to be more robust by providing two parameters α , β .
Injection of these parameters resulted in a decent performance for the NLP and ASR tasks, but was poorly performed on the computer vision tasks.
For experimentation, we have chosen α , β = 1, and this reduces to standard SoS. [Table]
[Table] Objectives Variants
Generic [Table]
OOD [Table] 3. Evaluation Now, let us understand the empirical performance of these objective functions trained on all variants of the data detailed in Table 3.
For generic data, in most of the cases, L SCE and L NLL were able to achieve top accuracy scores.
But, L 2 obtained the highest accuracy for CIFAR-10 and competed closely with L NLL on MNIST.
Taking into account the case of biased data L NLL performed standalone; L SCE and L SoS were able to compete closely with L NLL .
Surprisingly, L 1 turned out to have good performance for MNIST-M, and L SoS was closely on par.
But for ImageNet-R except L SCE and L NLL all other objectives failed to obtain at least 10% accuracy score.
Further, the variance in the accuracy is higher for margin-based objectives compared to that of probabilistic.
Hence, aggregating these results, it is strongly recommended that using probabilistic loss functions ( L SCE and L NLL ) would obtain decent performance on most of the data.
The empirical performance attained by these objective functions is well-understood but, the question arises with the internal representations of the model trained on these data.
Representation Analysis
In systems neuroscience, the Representation Similarity Analysis (RSA) framework [16] was the most successful in understanding the representations acquired from various activity patterns in various regions of the brain.
In addition, it provides insights into the similarity between representations by constructing Representation Dissimilarity Matrices (RDMs). With this motivation a method Centered Kernel Alignment (CKA) was devised to understand the representation structure of artificial neural networks.
It is well established in the literature that CKA [14] acquires qualitative representations compared to PwCCA [21] and SVCCA [23].
CKA not only captures the correspondence between the representations of a neural network but also allows us to compute the similarity between pairs of layers.
Therefore, three major interpretations can be made by visualising CKA. Which are Which part of the layers, of a certain neural architecture, constitute similar representations? Comparing the representations obtained from two diverse neural architectures, and
S t anda r d M N I S T C - M N I S T ( B i a s ed ) M N I S T - M ( OO D ) determine which could be a robust choice for given data. Comparing the internal representations of a certain neural architecture by varying the data sets, we determine which architecture would be sensitive to a certain data.
To reduce the computational expense consumed by linear CKA, mini-batch CKA is applied by computing the mean of HSIC (Hilbert-Schmidt Independence Criterion) scores on selected mini-batches ( N ).
This strategy is implemented straightforwardly as Thao et al .
The mini-batch CKA is detailed as follows:
[Equation]
where, ˜ X i =
X i X T i ; ˜ Y i =
Y i Y T i .
These X i ∈ R n × d 1 and Y i ∈ R n × d 2 are activation matrices for i th mini-batch of examples without replacement.
We now try to analyse the representations using CKA for all the objective functions on the aforementioned data in two steps.
4. Step-I
The first step aims at addressing which layers correspond to similar representations in a specific neural network trained on a certain objective function.
In the following, we are going to understand which representations would lead to better outcomes and which do not.
For this, we intend to choose all the objective functions for representation analysis using
P r e t r a i n e d : N on e
CMNIST
MNIST-M
CMNIST
MNIST-M
CMNIST MNIST-M
P r e t r a i n e d : I m a g e N e t 1k
CKA.
As including all the data sets would be redundant, we have chosen MNIST variants i.e., choosing a similar type of data set from the variants mentioned in Table 1. In Figure 1, when considering MNIST data, the CKA representation matrix formed for L BCE , L NLL and L 2 seems to have similar characteristics. While considering the case for the C-MNIST data set, all the loss functions tend to form a small box-like structure at the ultimate layers (after 50 th layer).
But, L NLL , L 1 , and L SCE seem to have uniformly distributed representation with decreasing similarity with the depth of the neural network.
Finally, for MNIST-M data the refined representation similarity is obtained for L 1 , L SCE and
L NLL .
However, surprisingly, the performance for MNIST-M is higher for margin-based objectives.
Considering the case of L SoS objective, it is prone to have block structured representations on utmost all the data [22].
This structured block resembles the neural network as overparameterised model.
The underlying reason is either that the model has fewer data samples or a deeper network.
This can be surmounted by truncating the layers with identical representational similarity.
These indications are clear to note that, the objectives L SCE and L NLL not only provide decent empirical performance but capture fine representations with ResNets.
Hence, it is understood that the representations acquired by probabilistic objectives are comparatively better to provide good generalisations for diverse data sets.
4. Step-II
Following this
[Table]
a new question arises.
How similar are the representations that initialise the training with pre-trained weights ( e
[Table]
g . ImageNet1k) on the considered data sets?
How similar are the representations of neural networks trained on biased data versus OOD data?
Can we transfer neural architecture weights trained on biased data sets to solve the problem of distribution shifts and vice versa?
[Table] [Table]
[Table] [Table] [Table] [Table]
(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)(cid:80)
[Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Table]
[Figure] data were trained on the mentioned dataset and the accuracy scores were obtained on the test datasets. We consider both the cases of training ResNets with and without pre-trained weights and as a note, ImageNet1k weights are used as pre-trained weights.
To address these questions, we first perform an empirical analysis on objectives L SCE and L NLL . Now, we consider two combinations of data sets.
From Table 4 (a) and Table 4 (c) we infer that in the presence of ImageNet1k weights the ResNet trained on C-MNIST and tested on M-MNIST provides half transferable performance ( ≈ 50%) but, the accuracy drops if C-MNIST is trained from scratch. In the same line, the presence of ImageNet1k weights the ResNet trained on M-MNIST shows maximum transferability ( > 95%) and the accuracy diminishes if trained from scratch. But is counterintuitive for the combination of CIFAR10 and C-CIFAR.
It can be observed that, even with ImageNet1k weights, the transferable performance is minimal. Especially, when trained on CIFAR-10 and tested on C-CIFAR the transferable performance of L SCE is small-scale ( < 1%).
Hence, there is no guarantee to attain greater performance on biased data and data with distribution shifts for standard neural networks by initializing the training with pre-trained weights ( e.g . ImageNet1k). Now we compare and contrast the representational structure of the neural networks having initialised with pre-trained weights and those that are trained totally from scratch (without any augmentations).
In Figure 2 we examine that when the network is trained and tested on the same data (that is, ResNet18 trained on MNIST-M and tested on its test set), the representations are refined and similarity decreases as the network progress with depth. If you consider the case of cross-data generalisation, the neural network with a uniform progression of decrease in similarity is highly likely to perform well.
E.g .
Consider the case of ResNet trained on C-MNIST and evaluated on MNIST-M. The resulting performance was superior only when the C-MNIST training was initialised with ImageNet1k weights.
In this case, when ResNet is trained from scratch, it has poor performance, and this is reflected in the CKA representation matrix.
As most of the layers tend to be the same, the ResNet (trained on C-MNIST) could not generalize well for the given new set of samples (tested on
MNIST-M).
So, if the neural network generalises well or provides good transferability, its performance can be assessed by visualising CKA and comprehending that the concentration of representational similarity progressively decreases with the depth of the network.  Conclusion
By summarising the above, we infer that L SCE and L NLL objectives would be apt for most of the data sets (inclusive of Biased and OOD).
But while experimenting, it should be noted that the variance in accuracy must be minimalist to ensure robustness. Secondly, if neural networks are initialised with pre-trained weights there is no guarantee for superior performance on biased and OOD data.
Finally, if the neural networks are exposed to cross-data, gener- alisation is attained only when the layers of CKA matrices have a progressive dissimilarity with the depth of the network.
In the future, we see the potential requirement of data sets comprising samples with distribution shifts with a greater sample size as of ImageNet.
Likewise, the representations acquired by the models are to be ensured with the least bias possible.
We believe that, comprehending the representations acquired from biased data would aid researchers in providing a novel debiasing neural network or a bias mitigation strategy.
Broader Impact
Since scientists and analysts routinely conclude, albeit based on evidence-based insights, OOD and biased data can be inherently misleading.
The impact of such misinterpretation can be corrected through exposure, experience, and expertise when humans are involved in the interpretation.
However, if the data itself is OOD or biased or only the interpretations of the data were presented by researchers, the danger of unconscious or even conscious biases cannot be ruled out entirely.
While Smith’s book [25] is an elaborate example of how research methodologies are inherently designed with ignorance towards the subjects of study, subtler cases of bias can be innate to the ways data are sourced, stored, pre-processed, analyzed, and interpreted.
The current work emphasises the need to methodically improve the quality of OOD representation and biased data without proposing radical paradigm shifts in current methodologies.
This work does not organically provide scope for misinterpretation of data or biased decisions made through data analytics.
Furthermore, the work facilitates a better and more uniform representation of the data by reminding researchers to consciously consider the biases and OOD aspects of the data, which may be rather inconspicuous. A stronger motivation arises as Artificial Intelligence and Machine Learning continue to be used in various technology and social domains for diverse applications.  Data
A vivid description of the data sets utilized is provided in Table 1. But, for a clear intuition for the reader, we provide how the samples differ for different data variants. The datasets utilized for the task of generic image classification were MNIST[7], CIFAR-10 [6] and ImageNet200. Specifically, MNIST and CIFAR-10 data sets contain 10 classes each, and ImageNet200, which was introduced by the Tiny ImageNet visual recognition challenge consists of 200 classes.
The number of samples for each data set has been provided in Table 1.
As mentioned, for bias data classification, we have utilized the data set provided by Lee et al.
[8] where they aim to solve the bias problem by providing 3 different biased datasets:
Colored MNIST, Corrupted CIFAR, and Biased FFHQ.
From the provided datasets, we have analysed Colored MNIST and Corrupted CIFAR data sets, with 10 classes each.
Each data set consists of various diversity ratios in order to tackle the problem of bias.
In which, bias is reduced by providing diverse bias-conflicting samples i.e, align and conflict divisions in the data set.
The partition of data conflict is based on a percentage of diversity and we have used 5% diversity, among the varying diversity ratios.
The number of diverse samples in that particular bias-conflicting data set is more compared to that of 0.5% and 1% i.e., in colored
MNIST bias-conflicting samples are considered to have more images with differently colored digits whereas in 0.5% and 1% we have less diversity of bias-conflicting samples.
To analyse the out-of-distribution problem in data, MNIST-M[2] and ImageNet-R[3] have been utilized.
MNIST-M (Modified MNIST) is a data set formed by masking an overlay of the BSDS500[1] data set on the generic MNIST dataset.
Similarly, ImageNet-R contains renditions of 200 ImageNet classes resulting in 30,000 images.
This is considered as a testing data set in general; whereas we have also used it for analysing representations over individual loss functions by training and testing on the same.
The resulting representations have been provided in 3.
A detailed explanation regarding the obtained representations has been provided in Section 3.
L 1 and L 2 with (out) Logits
Here, we confirm whether the theoretical proposition given by Janocha et al . [4] regarding L 1 and L 2 that, without providing non-linearity could saturate gradients and halt the learning [Figure]
[Figure]
[Figure]
of neural networks.
We understand through the experiments that, L 1 without logits is able to perform well on small data sets but, either by increasing sample size or the number of classes the performance drops.
Similarly, L 2 without logits is equally competent with L 2 with logits for small data samples.
However, overall, L 2 without logits performs poorly.
Table 2 describes the performance of L 1 and L 2 without any nonlinear activation.
Hence, these concerns motivate us to understand the behavior of neural networks on various datasets, mainly biased and OOD, both in terms of performance and representational strength
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] ImageNet21K) for the training models. These experiments were carried out three times for each objective function for a fair evaluation.
The tabulated mean and standard deviation (mean ± std) in each cell depicts accuracy scores obtained after experimenting thrice on ’test’ data.
The bold represents the highest test accuracy scores for each particular dataset.
[Figure]
Representation Analysis
This section is an extension of Section 4 of the main paper.
Here, we analyse the visual representations learned from neural networks using CKA[5].
Firstly, we analyse the representations learned by the ImageNet-R data set on individual objective functions trained on ResNets.
Later, compare and contrast representations learned by ResNets for cross-data generalization over CIFAR-10 and C-CIFAR data sets.
P r e t r a i n e d : N on e
CIFAR
C
CIFAR-
P r e t r a i n e d : I m a g e N e t 1k
3. Analysing Representations on Variant Objective Functions
The samples in the ImageNet-R data set are provided with various renditions such as origami, sculpture, paintings, etc.
This data set consists of 200 classes which are a subset of ImageNet classes with sufficient distributional shifts.
To interpret the learning trend in this data set, we have analysed numerous objective functions with a train-test split of 80%-20%.
First, let us try to understand the representations learned for probabilistic objective functions.
In Figure 2, first column indicates the representations attained by probabilistic objective functions.
The representations of L SCE and L NLL seemed to have the highest correlation by maintaining a trend of uniform stable learning.
Whereas the representations captured by
L BCE have started to decrease for the progressing layers, post 75 th layer.
Hence, it is very clear that the representations learned by L SCE and L NLL are considered to be most useful because of their ability to acquire similar patterns in almost all the ResNet layers.
Now, for the set of margin-based objective functions i.e, L 1 , L 2 and L SoS , the representations acquired by L 2 have better correlation compared to that of L 1 and SoS; even though the learning is unstable.
L 1 had produced a box-like representation structure which eventually indicates overparametrisation of the model.
Similarly, L SoS seems to have a considerable correlation until 90 th layer (approx.), later the correlation gradually declined.
We have also tried to analyse this data set by experimenting with it as a test set by using
ImageNet-200 as a train set.
The representations acquired in this particular setting for each loss function seemed to have a very poor correlation, as depicted in Figure 1.
3. Analysing Cross-Data Representations
The empirical performance obtained by ResNet18 for cross-data generalisation is depicted in Table 3 in the main paper.
The relevant CKA visualisations for Tables 5 and 7 of the main article are illustrated in Figure 3.
Now, let us compare the visualizations produced for this specific task.
The representations for this task have been illustrated in Figure 3.
The representations acquired by ResNets when both trained and tested on CIFAR-10 (or) C-CIFAR were considered to be good with refined correlation.
Whereas when trained on a specific dataset and tested on other data, the correlation between the layers was very poor i.e, the similarity strength of each layer reduced drastically.
Also,the presence of pre-trained weights mostly does not gives the model to sustain new distributions and produce a tiny increment in performance but does not aid the learning process for biased and ODD-type data (as illustrated earlier in the main paper).