Deep Within-Class Covariance Analysis for Robust Audio Representation Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Deep Neural Networks (DNNs) are known for excellent performance in supervisedtasks such as classiﬁcation.
Convolutional Neural Networks (CNNs), in particular,can learn effective features and build high-level representations that can be used forclassiﬁcation, but also for querying and nearest neighbor search.
However, CNNshave also been shown to suffer from a performance drop when the distribution ofthe data changes from training to test data.
In this paper we analyze the internalrepresentations of CNNs and observe that the representations of unseen data ineach class, spread more (with higher variance) in the embedding space of the CNNcompared to representations of the training data.
More importantly, this differenceis more extreme if the unseen data comes from a shifted distribution.
Based onthis observation, we objectively evaluate the degree of representation’s variance ineach class by applying eigenvalue decomposition on the within-class covariance ofthe internal representations of CNNs and observe the same behaviour.
This can beproblematic as larger variances might lead to mis-classiﬁcation if the sample crossesthe decision boundary of its class.
We apply nearest neighbor classiﬁcation on therepresentations and empirically show that the embeddings with the high varianceactually have signiﬁcantly worse KNN classiﬁcation performances, although thiscould not be foreseen from their end-to-end classiﬁcation results.
To tackle thisproblem, we propose Deep Within-Class Covariance Analysis (DWCCA) , a deepneural network layer that signiﬁcantly reduces the within-class covariance of a
DNN’s representation, improving performance on unseen test data from a shifteddistribution.
We empirically evaluate DWCCA on two datasets for Acoustic
Scene Classiﬁcation (DCASE2016 and DCASE2017).
We demonstrate that notonly does DWCCA signiﬁcantly improve the network’s internal representation, italso increases the end-to-end classiﬁcation accuracy, especially when the test set
exhibits a slight distribution shift.
By adding DWCCA to a VGG neural network,we achieve around 6 percentage points improvement in the case of a distributionmismatch.
Introduction
Convolutional Neural Networks (CNNs) are the state of the art in many supervised learning taskssuch as classiﬁcation, and using the power of convolutional layers, CNNs can learn useful featuresthat are often superior to engineered features, and build internal representations that can achieve highclassiﬁcation performance.
It has been shown that CNNs have a surprising ability to ﬁt data, so much so that they can evenperfectly learn from data with random labels [33].
But of course, memorising the training data is not sufﬁcient: a model is expected to generalize to unseen data points.
Additionally, a robust model has to be able to not only deal with unseen data points that are similar to the training set, but alsocope with unseen data points that may come from a slightly different distribution than the trainingdata ( distribution mismatch ).
When there is a distribution shift between the training and test sets,robustness of the model’s representation becomes more important as it has to classify or embed datapoints that are quite different from the ones it has observed in the training set.
In this paper, we investigate this by using a well-known DNN architecture (VGG [29]) that is adaptedfor audio classiﬁcation [10] and is widely used among researchers.
We evaluate VGG on data with aswell as without distribution mismatch and observe that while VGG exhibits a reasonable performanceon the data without distribution mismatch, its performance signiﬁcantly drops when tested on datafrom a shifted distribution.
We start by analyzing the internal representations of the network by using visualisations.
As will beseen in the ﬁrst (a-c) and the 3rd rows (g-i) of Figure 2, the network’s internal representations in eachclass spread more in the embedding space for the unseen data (validation or test) compared to thetraining data.
This is even more extreme when the unseen data comes from a shifted distribution (i).
For an objective evaluation of the amount of the representation’s variance in each class, we computethe within-class covariance of the representations of the network for each class, and we applyeigenvalue decomposition to compute the eigenvalues of each class’s covariance matrix.
We thenreport the sorted eigenvalues of the within-class covariance of the representations in Figure 3.
Asthe blue curves show, the eigenvalues in unseen data of validation (b and e) and test (c and d) haveconsiderably higher ranges than train data (a and d) for all the datasets we used.
To better understand the effect of such high variance in the quality of generalisation in the representations of our network, we carry out K-nearest neighbor (KNN) experiments on the dataset without,and the dataset with distribution shift.
As the results in Figure 4 show, the performance degredationfrom validation (c) compared to test (d) in case of distribution mismatch is signiﬁcantly highercompared to the performance drop from validation (a) to test (b) when the test data comes from asimilar distribution.
This observation is also aligned with what we observed in the visualisationsfrom Figure 2 that showed the data is more spread than validation data, when coming from a shifteddistribution.
To tackle this problem, we propose Deep Within-Class Covariance Analysis (DWCCA) , a deepneural network layer that reformulates the conventional Within-Class Covariance Normalization(WCCN) [13] as a DNN-compatible version.
DWCCA is trained end-to-end using back-propagation,can be placed in any arbitrary position in a DNN, and is capable of signiﬁcantly reducing thewithin-class covariance of the internal representation in a DNN.
We empirically show that DWCCA signiﬁcantly reduces the within-class covariance of the DNN’srepresentations, in both cases.
Further, we evaluate the generalization quality of the DNN’s representations after applying DWCCA by performing nearest neighbor classiﬁcation on its representations.
Our results show that DWCCA signiﬁcantly improves the nearest neighbor classiﬁcation results
inboth cases, hence improving the generalization quality of the representations.
And ﬁnally we reportthe end-to-end classiﬁcation results of the trained models on an acoustic scene classiﬁcation task,using data from the annual IEEE Challenges on Detection and Classiﬁcation of Acoustic Scenesand Events (DCASE).
It turns out that the classiﬁcation results for the dataset with distribution shiftare signiﬁcantly improved by integrating the DWCCA layer, while the performance on the datasetwithout distribution mismatch stayed the same.
Related Work
The characteristics of the representations learned in CNNs can be inﬂuenced by many factors such asnetwork architecture and the training data.
The authors in [20] investigated how architecture topologies affect the robustness and generalization of a representation and showed which representations from different architectures better transfer to other datasets and tasks
While the topology of the architecture inﬂuences the generality of its representations, several authors proposed methods that can improve the internal representation of a DNN.
[9] proposed a loss which learns linearly separable latent representations on top of a DNN, by maximising their smallest Linear
Discriminant Analysis (LDA) [11] eigenvalues.
And in [1], the authors propose creating a maximally correlated representation from different modalities
It has been shown that stand-alone CNNs are not very successful at generalizing to unseen datapoints from shifted distributions in tasks such as Acoustic Scene Classiﬁcation (ASC), where suchdistributional shifts are common.
ASC is deﬁned as classifying environments from the sounds theyproduce [2], and often these environments (e.g, home, park, city center) may sound very differentin different cities or during different times of the year.
Although in [24, 31, 10] CNNs have shownpromising results in ASC when the unseen test data has a similar distribution to the training data,in [22, 34] similar CNNs that previously performed successfully in a matched distribution case,suffered signiﬁcantly from the lack of generalization to a shifted distribution in the test set.
To cope with this drawback of CNNs, [28] investigated CNN manifold data augmentation using
Generative Adversarial Networks [12], while in [16, 32, 28] the authors used an ensemble of CNNsas feature extractors and processed the deep features via Support Vector Machines (SVMs), followedby a late fusion of various models.
They showed that although CNNs do not generalize well inthe distribution shift case, they can learn useful features that can be incorporated for building newclassiﬁers with better generalization properties.
In this paper, we try to better understand these problems.
We focus our efforts on investigating thereasons for the performance drop in CNNs when the data distribution is shifted.
To this end, weanalyze the internal representations in CNNs and propose DWCCA to tackle this issue.
Deep Within-Class Covariance Analysis
We start by introducing a common notation which will be used throughout the paper.
We then ﬁrstdescribe Within-Class Covariance Normalization (WCCN) –a classic machine learning method–, andthen show how to cast it into a deep learning compatible version.
3.1 Conventional Within-Class Covariance Normalization
Let W denote a set of N d -dimensional observations (feature vectors) belonging to C differentclasses c ∈ { 1 , ..., C } .
The observations are in the present case either hand crafted features (e.g.i-vectors [7]) or any intermediate hidden representation of a deep neural network.
WCCN is a linear projection that provides an effective compensation for the within-class variabilityand has proven to be effectively used in combination with different scoring functions [6, 23].
The
WCCN projection scales the feature space in the opposite direction of its within-class covariancematrix, which has the advantage that ﬁnding decision boundaries on the WCCN-projected databecomes easier [14].
The within-class covariance S w is estimated as:
[Equation]
where W c = 1 N c (cid:80) N c
i =1 W ci is the mean feature vector of class c and W ci are the samples belongingto this class.
N c is the number of observations of class c in the training set.
We use the inverse ofmatrix S w to normalize the direction of the projected feature vectors.
The WCCN projection matrix
B can be estimated using the Cholesky decomposition as B = cholesky ( S − 1 w ) .
3.2 Deep Within-Class Covariance Analysis (DWCCA)
Based on the deﬁnitions above we propose Deep Within-Class Covariance Analysis (DWCCA),a DNN-compatible formulation of WCCN.
The parameters of our networks are optimized with
Stochastic Gradient Descent (SGD), and trained with mini-batches.
The deterministic version of
WCCN described above is usually estimated on the entire training set, which by the deﬁnition of
SGD is not available in the present case.
In the following we propose a DWCCA Layer which helpsto circumvent these problems.
Figure 1 shows a schematic sketch of how the DWCCA Layer can beincorporated in a neural network, and its gradient ﬂow.
Instead of computing the within-class covariance matrix S w on the entire training set W
we providean estimate ˆ S b on the observations W b of the respective mini-batches.
Given this estimate wecompute the corresponding mini-batch projection matrix ˆ B b and use it to maintain a moving averageprojection matrix ¯ B as ¯ B = (1 − α ) ¯ B + α ˆ B b .
This is done similarly when computing the mean and standard deviation for normalising the activationsin batch normalization [17].
The hyper parameter α controls the inﬂuence of the data in the currentbatch b on the ﬁnal DWCCA projection matrix.
The output of this processing step is the DWCCAprojected data W (cid:48)
b = W b ¯ B of the respective mini-batch.
The DWCCA Layer can be seen as a special dense layer with a predeﬁned weight matrix, theprojection matrix ¯ B , with the difference that the parameters are computed via the activations of theprevious layer, and not learned via SGD.
The proposed covariance normalization is applied directlyduring optimization.
For training our network with back-propagation, we need to establish gradientﬂow.
To this end, we implement the DWCCA Layer using the automatic differentiation framework
Theano [4] which already provides the derivatives of matrix inverses and the Cholesky decomposition.
We refer the interested reader to [30] for details on this derivative.
Empirical Results
In this section, we detail our experimental setup and present our empirical results.
4.1 Dataset
To evaluate the performance of CNNs in situations with and without distribution mismatch, we use the
TUT Acoustic Scenes 2016 (DCASE2016) [27] and TUT Acoustic Scenes
(DCASE2017)
[26]datasets.
The DCASE2016 dataset consists of 15 different acoustic scenes: lakeside beach, bus, cafe/restaurant,car, city center, forest path, grocery store, home, library, metro station, ofﬁce, urban park, residentialarea, train, and tram.
All audio material was collected as 3-5 minutes length recordings and thenwas cut into segments of 30 seconds length.
The DCASE2017 dataset consists of the same 15classes and uses the same recordings in its development set (training and validation) as used for
DCASE2016 development set.
But these recordings were split into only 10s long segments insteadof 30s, which makes the learning and classiﬁcation task harder.
A new evaluation set (unseen test)was recorded for DCASE2017.
The reason for choosing these two datasets is that DCASE2016 and
DCASE2017 are based on the same development (= training and validation) data, but have differentevaluations (= independent test) sets.
While the test data for DCASE2016 were collected jointlywith the development data, new test recordings were collected for DCASE2017, at a later time, indifferent places.
Hence, there is a distribution mismatch between development and test data in thecase of DCASE2017, as the many environments sound different from those in the developmentset (which will also show, indirectly, in our experimental results).
This is detailed in the report of
DCASE2017 [26].
In the following, we will use DCASE2016 to represent a data scenario without distribution shift,and DCASE2017 as the distribution shift case.
In all experiments, we use the ofﬁcial 4-fold crossvalidation splits provided with the datasets and report CV results on all folds separately, as well asthe performance of our method on the unseen test set.
[Table]
4.2 Baseline Systems and Experimental Setup
As explained before, we use a VGG-Style CNN proposed in [10] which uses audio spectrograms asinputs.
We will refer to this model as vanilla .
To evaluate the effect of our proposed method, we apply a DWCCA layer on the output of globalaverage pooling, as can be seen in Table 1.
We will refer to this model in our results as dwcca .
Theinternal representations we will analyze are computed by using the output of global average poolinglayer in vanilla , and the output of the DWCCA layer in dwcca .
We also provide additional baselines for comparison: We report the performance of the best end-toend CNN model [24] in the DCASE2016 challenge [25].
This method uses an ensemble of various
CNN models.
For the DCASE2017 experiments, we also choose the best-performing end-to-end
CNN from the DCASE2017 challenge [26] as an additional baseline to vanilla: [34] that uses anensemble fusion of various ResNets [15] using several channels of audio (left, right and difference).
We report classiﬁcation results of our end-to-end CNNs on all folds, and on the unseen test set.
Forthe unseen test, we average the probabilities produced by the 4 models trained on the individualtraining parts in CV folds from the development data.
Additionally, we report calibrated resultswhere we use a linear logistic regression model in each fold to regress the prediction probabilitiesof each model to the true labels.
This method is better known as late fusion and proved successfulin [10].
The probabilistic averaging for unseen test is done similarly for late fusion, as explainedabove.
The setup used in all of our experiments are as follows:
The initial learning rate is set to 0 . 0001 , andthe ADAM optimizer is used [19].
We choose a similar α (0.1) to the one used in batchnorm.
Amodel selection with max.
patience of 20 is applied and the learning rate is halved in each step ofmaximum patience.
The architecture of our models is speciﬁed in Table 1.
All models are trainedwith stratiﬁed batch iterators and batch size of 75.
In all our experiments, the spectrogram of thewhole audio segment (30 sec in DCASE2016 and 10 sec in DCASE2017) was fed to our models.
Allspectrograms in our experiments are extracted with the madmom library [5] using the mono signal,and the Lasagne library [8] was used for building neural network architectures.
The DWCCA Layerwas implemented as a Lasagne-compatible layer
4.3 Experiments Performed
We carried out various experiments to demonstrate the issue we encounter in ASC with CNNs.
First,we train our models on DCASE2016 and DCASE2017.
For a better understanding of the issues inthe representation learned by the CNN, we provide a visualization of the VGG representations afterbeing projected into 2D via Principal Component Analysis (PCA) [18].
The result can be found in
Figure 2.
For analysing the within-class covariance of the representations, we apply an eigenvalue decomposition on the covariance matrix of the representations from each class.
This technique is alsoused in other studies such as [21] to investigate the dynamics of learning in neural networks.
Theseeigenvalues are a good indicator of the covariance of the representations in each class.
If high, itmeans that the representations on that speciﬁc class have a high covariance, and if the differencebetween the highest eigenvalue and the lowest eigenvalue is high it also indicates that the variance ofthe representation in that class is not spread equally in all dimensions.
These results can be found in
Figure 3.
The shade in this plot represents the variance of eigenvalues over different classes.
To indirectly assess the structure of the internal representation spaces learned, we carry out a knearest-neighbor classiﬁcation experiment in these representation spaces; the K-nearest neighborresults on both datasets will be shown in Figure 4.
Finally, we report the end-to-end classiﬁcation results on both datasets, for various methods; these canbe found in Table 2.
For a deeper understanding of the proposed method, we additionally providedthe class-wise f-measure of DWCCA and the baseline VGG (vanilla) in Table 3.
4.4 Results and Discussions
In Figure 2, the network’s internal representations in each class are projected into 2D via PCA andeach class is represented by a different colour.
Looking at ﬁrst (a-c) and second (d-f) row, it can beseen that for the dataset without mismatched distribution the embeddings of unseen data (validationand test) are spread less after applying DWCCA.
Also comparing the unseen embeddings to thetraining embeddings (with lower opacity and in grey) it can be seen that the unseen embeddingsprojected closer to the training embeddings after applying DWCCA.
Comparing third
(g-i) and fourth(j-l) row, it can be seen that for the case of a distribution shift DWCCA also reduces the variance ofthe embeddings in each class, resulting in them being embedded closer to the training embeddings(grey).
This suggests that this property can improve the generalisation of the representations.
Wewill empirically evaluate this hypothesis later in this section by applying KNN classiﬁcation on therepresentations.
Looking at Figure 3, we can see that in all plots from dataset with, and dataset without distributionshift, DWCCA signiﬁcantly reduces the within-class variability.
This can be observed by lookingat the eigenvalues of the covariances of the representations.
An interesting observation is the rangeof eigenvalues in vanilla:
In both datasets, eigenvalues have signiﬁcantly larger range on unseendata (validation and test) compared to the training data.
The maximum eigenvalue in DCASE2016is around 0.7, while the maximum eigenvalue for unseen is around 7, about 10 times more.
Alsothe maximum eigenvalue of the train set of DCASE2017 is around 2, while the max.
eigenvalue onunseen data is around 20 (10 times larger).
By looking at the KNN results in Fig.
4 it can be seen that in both cases (mismatch / no mismatch),the KNN classiﬁcation accuracy increases by adding DWCCA.
Also, while the KNN performance isin a reasonable range on the validation set of both datasets, the test accuraty in the mismatch case(DCASE2017) drops signiﬁcantly compared to the validation set.
Additionally it can be seen thatapplying DWCCA signiﬁcantly improves the performance on the test set with shifted distribution,adding an improvement of about 6 percentage point, while the improvement on the test set withoutmismatch is around 2 percentage points.
Looking at the results of end-to-end classiﬁcations in Table 2,we see that the performance of vanilla on DCASE 2017 consistently and signiﬁcantly improves whenadding DWCCA, on all development folds as well as on the unseen test data.
We observe around 6percentage points improvement by adding DWCCA to VGG.
Looking at the results of the dataset without mismatch, we see that although the results on all foldswere improved by adding DWCCA, the results on the unseen test set do not signiﬁcantly change.
Thiscan be explained better by looking at Figure 2: the embeddings of validation (b) and test (c) indicate
[Figure]
that the test data is projected closer to the training set than the validation set.
This observation suggeststhat the unseen test in DCASE2016 might be similar (even more similar than the validation data) tothe training set.
This can also be conﬁrmed by looking at the results of the best CNN baseline, as wellas vanilla: the performances on the unseen test set are consistently higher than all the validation folds.
Hence, DWCCA could not help as there was not a large generalisation gap between training and test.
It is worth mentioning that both vanilla and DWCCA are single models, trained on mono singlechannel spectrograms and no ensemble or multi-channel features were used in these experiments. In [Figure]
other words, a single VGG model achieves comparable performances to an ensemble of multi-channel
Resnets.
We also provide class-wise f-measures on the unseen test set for both datasets in Table 3.
Whileon the dataset without distribution shift, the average f1 stays the same by adding DWCCA in bothcalibrated and non calibrated models, we can observe that there is a boost of 13 percentage points onthe "train" class which was the class with the lowest f1 (both calibrated and non calibrated).
It seemsthat DWCCA does not have a signiﬁcant impact on classes with high f1: "ofﬁce" and "beach" whichstay the highest correctly predicted classes and do not face signiﬁcant changes by DWCCA.
On the dataset with distribution shift, we can see a signiﬁcant improvement of 4 and 7 percentagepoints on average f1 for non-calibrated and calibrated models, respectively.
The worst class in
DCASE2017 was "beach" with 32%, which was boosted by 24 and 37 percentage points for noncalibrated and calibrated models, respectively.
On the other hand, the best performing class, "forestpath", drops by only 2 and 3 percentage points for non-calibrated and calibrated models, respectively.
From the experimental results, we may thus conclude that overall, reducing the within-class covarianceof representations using DWCCA results in more robust performance and, in case of a large gapbetween training and test, DWCCA can improve the generalisation.
Additionally, the networks tendto reach a more uniform performance across various classes by improving the performance on theworst classes while not signiﬁcantly degrading the best performing classes.
Conclusion
In this paper, we presented the DWCCA layer, a DNN compatible version of the classic WCCN which is used to normalize the within-class covariance of the DNN’s representation and improve the performance of CNNs on data-points with shifted distributions.
Using DWCCA, we improved the performance of the VGG network by around 6 percentage point when the test datapoints were from ashifted distribution.
We analysed the embedding’s generalisation by reporting KNN classiﬁcation accuracies and showed that DWCCA also improves the generalisation of DNN representations both for with and without distribution mismatch.
We also showed that large within-class covariance of representations can be a sign for bad generalisation and showed that DWCCA can signiﬁcantly reduce
WCC and improve generalisation of the representations
[Figure]
[Table]