Representational Structure of Neural Networks Trained on Biased and Out-Of-Distribution Data
[-BMVC 
-]Anonymous Author(s)
AffiliationAddress email
Abstract
Neural networks trained on standard image classification data sets are observedto be less robust to distributional shifts and pertain to certain levels of bias inrepresentations.
Thus, it is pertinent to identify the kind of objective function thatcould correspond to better performance for data with biases and distribution shifts,and how can that objective function be justified to be the apt choice.
There is,however, less literature that focuses on the choice of the objective function and itsrepresentational structure when trained on such data sets.
In this work, we analyse the performance and the internal representational structureof convolution-based neural networks (eg. ResNets) trained by varying objectivefunctions on biased and out-of-distribution (OOD) data.
Specifically, we interpretsimilarities in representations (using CKA) acquired for distinct objective functions(probabilistic and margin-based) and provide a detailed analysis of the chosenones.
Our analysis reports that representations acquired by ResNets using Softmax Cross-Entropy ( L SCE ) and Negative Log-Likelihood ( L NLL ) as objectives areequally competent in providing superior performance and fine representations on OOD and biased data.
Subsequently, we interpret that the ResNets are less likely tobe robust on cross-data generalisation without refined representational similarity.
Introduction
The advent of deep learning has provided us with robust models and the finest neural architectures.
Even with tremendous performance, most neural networks do not discriminate representations wellas they are trained in a controlled setting [26].
They thus tend to perform poorly with distributionalshifts and biases in the data.
Various methods with neural network optimisation or a new paradigm ofsolving were implied to mitigate this issue [3], [9], [22], [29].
So, we question thus: what could be an optimal objective function for data with biases and distributional shifts?
Eventually, what kind of representations are learnt by the neural networks whenexposed to such data?
How are the internal representations of neural networks altered by varying thedata samples?
Finally, how transferable are the representations that are trained with data biases anddata with distributional shifts?
In the existing literature, the work of Simon et al.
[15] has provided insight on the transferabilityof representations when exposed to various loss functions, but has focused mainly on probabilisticobjectives.
Kim et al.
[13] utilised the gradient reversal layer and proposed a novel regularisedloss function based on mutual information obtained from InfoGANs [6] to unlearn the target bias,that is, the bias present in the data.
This eventually minimises the detrimental effects of bias inthe data.
Adeli et al.
[1] proposed a loss function for adversarial training with two objectives tolearn features that have maximum discriminative power and minimal statistical dependencies with [Table] protected bias.
Certain works provided novel optimisation methods to solve the bias problem [3],[22].
StableNet by Zhang et al.
[29] reduces the statistical relation between irrelevant and relevantfeatures acquired from the data.
To achieve this, they have implied a sample weighting technique andtrained on a new objective which is modified on SCE.
Under different settings, StableNet reportedsuperior performance for various OOD datasets including MNIST-M. Sunil et al.
[28] proposed anew method of solving the OOD problem using the principle of abstention (which encourages topredict the sample classes which were unseen by the model) to provide OOD generalisation.
The existing literature is less evident on the use of objective functions on data with biases anddistributional shifts.
The representational characteristics of a given objective function are notspecifically studied.
The transferability of the representations—produced by training neural networkson biased or OOD data—to standard classification data is not illustrated.
Hence, these concernsmotivate us to understand the behaviour of neural networks on various data sets, mainly biasedand OOD.
First, we study the empirical performance of the objective functions by dividing theminto two variants: a) probabilistic, and b)margin-based.
Secondly, we analyse the importance ofindividual objective functions which provide representation structure with good generalisation andtransferability.
Additionally, we see the potential of CKA as one of the criterion to measure the
Interpretability of the neural networks [12].
Setup
Terminology The input data is represented as X ∈ { x (1) , x (2) , , x ( n ) } where n is the total numberof samples.
The encoder Enc ( . ) is used to extract features from a given input X .
The featuresextracted from the encoder are presented as fv ← Enc ( X ) ; where fv ∈ { fv (1) , fv (2) , , fv ( n ) } .
The dimensions of the feature vector vary by changing the encoder.
As most of the experimentswere carried out using a supervised framework, the data sets do have certain ground truth labels,and these are represented as Y ∈ { y (1) , y (2) , , y ( n ) } .
The activation functions, sigmoid and softmax,are indicated by σ ( a i )
=
1 / (1 + e − a i ) and S ( a i ) =
e a i / (cid:80)
j e a j , respectively.
The loss (objective)function, is denoted by L ( . ) and the suffixes indicate its specified variant.
The norms ∥ · ∥ 1 and ∥ · ∥ 2indicate Manhattan and Euclidean norms 1 respectively.
Datasets As mentioned, we choose the data for solving three diverse problems.
First, we considerwell-studied and experimented datasets such as MNIST [19], CIFAR-10[18] and ImageNet-200.
These data sets are widely used for standard image classification tasks; also, they do not have inherentbiases and do not contain OOD samples.
Next, Colored MNIST (C-MNIST) and Corrupted CIFAR-10(C-CIFAR) [21] data sets are utilized for understanding biased representations.
For both the datasets, C-MNIST and C-CIFAR, we perform experiments according to Lee et al.
Lastly, we considerdata with distributional shifts such as ImageNet Renditions (ImageNet-R) [8] and Modified MNIST(MNIST-M) [7].
The collection of data sets considered for experimentation are tabulated in Table 1.
Models In this work, we use ResNet18 and ResNet50 to understand convolution-type representations.
ResNet18 is used for small data, which are illustrated in Table 1, similarly, for medium-sized
[Table]
Training and fine-tuning To train each model we utilized Adam [14] as an optimizer with astandard learning rate of 10 − 3 and a weight decay of 10 − 5 .
For certain executions, we utilisedpre-trained ImageNet weights and fine-tuned the models on that specific data set (and when notparticularly mentioned, we trained the model from scratch).
We fed 512 samples in batches to neuralnetworks by varying the objective functions.
The early stopping criterion is embedded with thepatience of 12 epochs to ensure the model does not over-fit with excessive training.
The resultsreported in Tables 3 and 4 are produced without any augmentations (except normalisation) andpre-trained weights.
Empirical Analysis
The choice of the objective function to train a deep neural network, on specified data remains aquestion.
Janocha et al.
[11] provided a theoretical justification and conducted experiments on MNIST pointing out the importance of L 1 and L 2 not just as regularizers, but as objective functionsfor better generalisations.
Hui et al.
[10] empirically proves that square loss with a little parametrictuning would produce significant results for most tasks of natural language processing (NLP) andautomatic speech recognition (ASR).
Hui et al.
[10] specifically mentioned that the proposed squareloss is not brittle for randomised initialisation.
A recent analysis by Simon et al.
[15] providesinsights noting that the representations acquired to classify certain tasks with more class separationlead to poor transferable features.
This work implies various objective functions to observe both theperformance and quality of representations for the standard computer vision classification data sets.
The previous literature focuses on the training and transferability of features acquired by trainingstandard neural architectures with varying objective functions.
But, there is sparse literature notingthe relevance of both probabilistic and margin-based objective functions on data with biases anddistributional shifts.
Hence, we provide empirical analysis for two variants of objective functions tounderstand the performance of each objective function on various data sets mentioned in Table 1.
3.1 Probabilistic Objectives
Probabilistic objective functions calculate the error that approximates the underlying probabilitiesfor representations acquired from an encoder (ResNet).
In this paper, we include three probabilisticobjectives and they are detailed in Table 2.
First, the Softmax cross-entropy [ ? ], a highly usedobjective, is obtained by applying softmax activation in the final layer of the neural network, and thisfeed is minimised by the negative log-likelihood (NLL).
Next, Binary cross-entropy (BCE) is obtainedby applying sigmoid activation ( σ ( · ) ) at the final layer of neural networks and this information isminimised by NLL.
Primarily, the BCE loss function is used for binary classification problems but, arecent work empirically proves that its implication on multi-class would lead to better performance[4] by applying the one-vs-rest strategy.
Finally, the likelihood provides the joint probability of thesample distribution and minimises the negative logarithm of the obtained likelihood [5]. [Table]
3.2 Margin-based Objectives
Margin-based objective functions calculate the error by discriminating the representations extractedfrom an encoder (ResNet).
Similarly to probabilistic objectives, we include three margin-basedobjectives, which are detailed in Table 2.
First, the mean absolute error ( L 1 objective function) findsthe Manhattan distance between the two representations.
We acquire the theoretical motivation of Janocha et al.
[11] that L 1 would reduce the sparseness in the representations.
Rather than directlydiscriminating the representations in the final layer, we use softmax to ensure appropriate learningwithout saturation of partial derivatives 2 .
Similar to L 1 , we use L 2 to find find the Euclidean distancebetween two representations (final layer).
Lastly, Hui et al.
[10] rescaled SoS to be more robust byproviding two parameters α , β .
Injection of these parameters resulted in a decent performance for the NLP and ASR tasks, but was poorly performed on the computer vision tasks.
For experimentation,we have chosen α , β = 1 , and this reduces to standard SoS.
3.3 Evaluation
Now, let us understand the empirical performance of these objective functions trained on all variantsof the data detailed in Table 3.
For generic data, in most of the cases, L SCE and L NLL wereable to achieve top accuracy scores.
But, L 2 obtained the highest accuracy for CIFAR-10 andcompeted closely with L NLL on MNIST.
Taking into account the case of biased data L NLL performedstandalone; L SCE and L SoS were able to compete closely with L NLL .
Surprisingly, L 1 turned outto have good performance for MNIST-M, and L SoS was closely on par.
But for ImageNet-R except L SCE and L NLL all other objectives failed to obtain at least 10% accuracy score.
Further, thevariance in the accuracy is higher for margin-based objectives compared to that of probabilistic.
Hence, aggregating these results, it is strongly recommended that using probabilistic loss functions( L SCE and L NLL ) would obtain decent performance on most of the data.
The empirical performanceattained by these objective functions is well-understood but, the question arises with the internalrepresentations of the model trained on these data.
Representation Analysis
In systems neuroscience, the Representation Similarity Analysis (RSA) framework [ ? ] was the mostsuccessful in understanding the representations acquired from various activity patterns in variousregions of the brain.
In addition, it provides insights into the similarity between representations byconstructing Representation Dissimilarity Matrices (RDMs).
With this motivation a method Centered Kernel Alignment (CKA) was devised to understand the representation structure of artificial neuralnetworks.
It is well established in the literature that CKA [16] acquires qualitative representationscompared to PwCCA [23] and SVCCA [25]. CKA not only captures the correspondence between S t anda r d M N I S T
C - M N I S T ( B i a s ed ) M N I S T - M ( OO D ) the representations of a neural network but also allows us to compute the similarity between pairs oflayers.
Therefore, three major interpretations can be made by visualising CKA.
Which are
Which part of the layers, of a certain neural architecture, constitute similar representations
Comparing the representations obtained from two diverse neural architectures, and determinewhich could be a robust choice for given data
Comparing the internal representations of a certain neural architecture by varying the data sets, we determine which architecture would be sensitive to a certain data.
To reduce the computational expense consumed by linear CKA, mini-batch CKA is applied bycomputing the mean of HSIC (Hilbert-Schmidt Independence Criterion) scores on selected minibatches ( N ).
This strategy is implemented straightforwardly as Thao et al.
The mini-batch CKA is detailed as follows:
[Equation]
where, X i =
X i X
Ti ; Y i =
Y i Y Ti .
These X i ∈ R n × d 1 and Y i ∈ R n × d 2 are activation matrices fori th mini-batch of examples without replacement.
We now try to analyse the representations using CKA for all the objective functions on the aforementioned data in two steps.
4.1 Step-I
The first step aims at addressing which layers correspond to similar representations in a specificneural network trained on a certain objective function.
In the following, we are going to understandwhich representations would lead to better outcomes and which do not.
For this, we intend to chooseall the objective functions for representation analysis using CKA.
As including all the data sets wouldbe redundant, we have chosen MNIST variants i.e., choosing a similar type of data set from thevariants mentioned in Table 1.
In Figure 1, when considering MNIST data, the CKA representation matrix formed for L BCE , L NLLand L 2 seems to have similar characteristics.
While considering the case for the C-MNIST data
P r e t r a i n e d : N on e
CMNIST
MNIST-M
CMNIST
MNIST-M
CMNIST MNIST-M
P r e t r a i n e d : I m a g e N e t 1k set, all the loss functions tend to form a small box-like structure at the ultimate layers (after 50 thlayer).
But, L NLL , L 1 , and L SCE seem to have uniformly distributed representation with decreasingsimilarity with the depth of the neural network.
Finally, for MNIST-M data the refined representationsimilarity is obtained for L 1 , L SCE and L NLL .
However, surprisingly, the performance for MNISTM is higher for margin-based objectives.
Considering the case of L SoS objective, it is prone to haveblock structured representations on utmost all the data [24].
This structured block resembles theneural network as overparameterised model.
The underlying reason is either that the model has fewerdata samples or a deeper network.
This can be surmounted by truncating the layers with identicalrepresentational similarity.
These indications are clear to note that, the objectives L SCE and L NLL not only provide decentempirical performance but capture fine representations with ResNets.
Hence, it is understood thatthe representations acquired by probabilistic objectives are comparatively better to provide goodgeneralisations for diverse data sets.
4.2 Step-II
Following this, a new question arises.
How similar are the representations that initialise the trainingwith pre-trained weights (eg.
ImageNet1k) on the considered data sets?
How similar are therepresentations of neural networks trained on biased data versus OOD data?
Can we transfer neuralarchitecture weights trained on biased data sets to solve the problem of distribution shifts and viceversa?
To address these questions, we first perform an empirical analysis on objectives L SCE and L NLL .
Now, we consider two combinations of data sets.
From Table 4 (a) and Table 4 (c) we infer thatin the presence of ImageNet1k weights the ResNet trained on C-MNIST and tested on M-MNISTprovides half transferable performance ( ≈ 50% ) but, the accuracy drops if C-MNIST is trained fromscratch.
In the same line, the presence of ImageNet1k weights the ResNet trained on M-MNISTshows maximum transferability ( > 95% ) and the accuracy diminishes if trained from scratch.
But iscounterintuitive for the combination of CIFAR-10 and C-CIFAR.
It can be observed that, even with ImageNet1k weights, the transferable performance is minimal.
Especially, when trained on CIFAR-10and tested on C-CIFAR the transferable performance of L SCE is small-scale ( < 1% ).
Hence, thereis no guarantee to attain greater performance on biased data and data with distribution shifts forstandard neural networks by initializing the training with pre-trained weights (eg. ImageNet1k).
Now we compare and contrast the representational structure of the neural networks having initialisedwith pre-trained weights and those that are trained totally from scratch (without any augmentations).
In Figure 5 we examine that when the network is trained and tested on the same data (that is, ResNet18trained on MNIST-M and tested on its test set), the representations are refined and similarity decreasesas the network progress with depth.
If you consider the case of cross-data generalisation, the neuralnetwork with a uniform progression of decrease in similarity is highly likely to perform well.
Eg.
Consider the case of ResNet trained on C-MNIST and evaluated on MNIST-M.
The resultingperformance was superior only when the C-MNIST training was initialised with ImageNet1k weights.
In this case, when ResNet is trained from scratch, it has poor performance, and this is reflectedin the CKA representation matrix.
As most of the layers tend to be the same, the ResNet (trainedon C-MNIST) could not generalize well for the given new set of samples (tested on MNIST-M).
So, if the neural network generalises well or provides good transferability, its performance can beassessed by visualising CKA and comprehending that the concentration of representational similarityprogressively decreases with the depth of the network. [Table] [Table] [Table]
[Table]
[Table] [Table] [Table]  Conclusion
By summarising the above [Table] we infer that L SCE and L NLL objectives would be apt for most of thedata sets (inclusive of Biased and OOD).
But while experimenting, it should be noted that the variancein accuracy must be minimalist to ensure robustness.
Secondly, if neural networks are initialised withpre-trained weights there is no guarantee for superior performance on biased and OOD data.
Finally,if the neural networks are exposed to cross-data, generalisation is attained only when the layers of CKA matrices have a progressive dissimilarity with the depth of the network.
In the future, we see the potential requirement of data sets comprising samples with distribution shiftswith a greater sample size as of ImageNet.
Likewise, the representations acquired by the modelsare to be ensured with the least bias possible.
We believe that, comprehending the representationsacquired from biased data would aid researchers in providing a novel debiasing neural network ora bias mitigation strategy.
Also we believe that, this CKA framework can be extended to study thedetailed intrepretability of the neural networks [12]
Broader Impact
Since scientists and analysts routinely conclude, albeit based on evidence-based insights, OOD andbiased data can be inherently misleading.
The impact of such misinterpretation can be correctedthrough exposure, experience, and expertise when humans are involved in the interpretation.
However,if the data itself is OOD or biased or only the interpretations of the data were presented by researchers,the danger of unconscious or even conscious biases cannot be ruled out entirely.
While Smith’s book[27] is an elaborate example of how research methodologies are inherently designed with ignorancetowards the subjects of study, subtler cases of bias can be innate to the ways data are sourced, stored,pre-processed, analyzed, and interpreted.
The current work emphasises the need to methodically improve the quality of OOD representationand biased data without proposing radical paradigm shifts in current methodologies.
This work does not organically provide scope for misinterpretation of data or biased decisions made through dataanalytics.
Furthermore, the work facilitates a better and more uniform representation of the data byreminding researchers to consciously consider the biases and OOD aspects of the data, which may berather inconspicuous.
A stronger motivation arises as Artificial Intelligence and Machine Learningcontinue to be used in various technology and social domains for diverse applications.
A Data
A vivid description of the data sets utilized is provided in Table 1.
But, for a clear intuition for thereader, we provide how the samples differ for different data variants.
The datasets utilized for the taskof generic image classification were MNIST[20], CIFAR-10 [17] and ImageNet-200.
Specifically, MNIST and CIFAR-10 data sets contain 10 classes each, and ImageNet-200, which was introducedby the Tiny ImageNet visual recognition challenge consists of 200 classes.
The number of samplesfor each data set has been provided in Table 1.
As mentioned, for bias data classification, we have utilized the data set provided by Lee et al.
[21]where they aim to solve the bias problem by providing 3 different biased datasets: Colored MNIST,
Corrupted CIFAR, and Biased FFHQ.
From the provided datasets, we have analysed Colored MNISTand Corrupted CIFAR data sets, with 10 classes each.
Each data set consists of various diversity ratiosin order to tackle the problem of bias.
In which, bias is reduced by providing diverse bias-conflictingsamples i.e, align and conflict divisions in the data set.
The partition of data conflict is based ona percentage of diversity and we have used 5% diversity, among the varying diversity ratios.
Thenumber of diverse samples in that particular bias-conflicting data set is more compared to that of 0.5%and 1% i.e., in colored MNIST bias-conflicting samples are considered to have more images withdifferently colored digits whereas in 0.5% and 1% we have less diversity of bias-conflicting samples.
To analyse the out-of-distribution problem in data, MNIST-M[7] and ImageNet-R[8] have beenutilized.
MNIST-M (Modified MNIST) is a data set formed by masking an overlay of the BSDS500[2]data set on the generic MNIST dataset.
Similarly, ImageNet-R contains renditions of 200 ImageNetclasses resulting in 30,000 images.
This is considered as a testing data set in general; whereas wehave also used it for analysing representations over individual loss functions by training and testingon the same.
The resulting representations have been provided in 5.
A detailed explanation regardingthe obtained representations has been provided in Section 3.
[Table]
B L 1 and L 2 with (out) Logits
Here, we confirm whether the theoretical proposition given by Janocha et al.
[11] regarding L 1and L 2 that, without providing non-linearity could saturate gradients and halt the learning of neuralnetworks.
We understand through the experiments that, L 1 without logits is able to perform well on small datasets but, either by increasing sample size or the number of classes the performance drops.
Similarly, L 2 without logits is equally competent with L 2 with logits for small data samples.
However, overall, L 2 without logits performs poorly.
Table 5 describes the performance of L 1 and L 2 without anynonlinear activation.
Hence, these concerns motivate us to understand the behavior of neural networks on various datasets,mainly biased and OOD, both in terms of performance and representational strength.
[Figure]
C Representation Analysis
This section is an extension of Section 4 of the main paper.
Here, we analyse the visual representationslearned from neural networks using CKA[16].
Firstly, we analyse the representations learned bythe ImageNet-R data set on individual objective functions trained on ResNets.
Later, compareand contrast representations learned by ResNets for cross-data generalization over CIFAR-10 and
C-CIFAR data sets.
C.1 Analysing Representations on Variant Objective Functions
The samples in the ImageNet-R data set are provided with various renditions such as origami,sculpture, paintings, etc.
This data set consists of 200 classes which are a subset of ImageNet classeswith sufficient distributional shifts.
To interpret the learning trend in this data set, we have analysednumerous objective functions with a train-test split of 80%-20%.
First, let us try to understand the representations learned for probabilistic objective functions.
In
Figure 4, first column indicates the representations attained by probabilistic objective functions.
Therepresentations of L SCE and L NLL seemed to have the highest correlation by maintaining a trendof uniform stable learning.
Whereas the representations captured by L BCE have started to decreasefor the progressing layers, post 75 th layer.
Hence, it is very clear that the representations learned by L SCE and L NLL are considered to be most useful because of their ability to acquire similar patternsin almost all the ResNet layers.
Now, for the set of margin-based objective functions i.e, L 1 , L 2 and L SoS , the representations acquired by L 2 have better correlation compared to that of L 1 and SoS;even though the learning is unstable.
L 1 had produced a box-like representation structure whicheventually indicates overparametrisation of the model.
Similarly, L SoS seems to have a considerablecorrelation until 90 th layer (approx.), later the correlation gradually declined.
We have also tried to analyse this data set by experimenting with it as a test set by using ImageNet-200as a train set.
The representations acquired in this particular setting for each loss function seemed tohave a very poor correlation, as depicted in Figure 3. P r e t r a i n e d : N on e CIFARC-CIFAR CIFARC-CIFAR
CIFAR-10 C-CIFAR
P r e t r a i n e d : I m a g e N e t 1k
C.2 Analysing Cross-Data Representations
The empirical performance obtained by ResNet18 for cross-data generalisation is depicted in Table3 in the main paper.
The relevant CKA visualisations for Tables 5 and 7 of the main article areillustrated in Figure 5.
Now, let us compare the visualizations produced for this specific task.
The representations for thistask have been illustrated in Figure 5.
The representations acquired by ResNets when both trainedand tested on CIFAR-10 (or) C-CIFAR were considered to be good with refined correlation.
Whereaswhen trained on a specific dataset and tested on other data, the correlation between the layers wasvery poor i.e, the similarity strength of each layer reduced drastically.
Also, the presence of pre-trained weights mostly does not gives the model to sustain new distributionsand produce a tiny increment in performance but does not aid the learning process for biased and ODD-type data (as illustrated earlier in the main paper).