From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper22/AnonReviewer4
title: Moving average PCA of deep features before softmax layer for Acoustic Scene Classification.
review: The article describes a reasonable proposal for an in-network moving-average approach to covariance-based normalization of the deep features / embeddings produced by a CNN, prior to the categorical softmax layer. Evaluation (classification results as well as feature visualization) is done using known tasks in Acoustic Scene Classification. The results suggest some effectiveness for the proposed method.

The main issue I have with the proposal is that the authors don't provide reasons why small within-class variance will necessarily help the categorical softmax layer. Training the overall vanilla structure with categorical CE should find something close to the optimal embeddings AND the optimal decision boundaries for the given task. It's not in itself a problem for classification for the embeddings (or deep features, the input to the proposed layer) to have high within-class variance. A problem would be if there was intrinsic , unavoidable overlap across the class boundaries, but that is not what is being claimed here. The results suggest the method has successfully reduce within-class variance, supported by feature visualization and eigenvalue analysis. Those findings are as expected. But are the better precision results really expected? I'm wondering if there is a mismatch between the training criterion CE and the F1 measure that is reported. 

However, overall I think this is a decent paper for the workshop. A number of more specific comments follow. 

It would be helpful if the authors were specific about what "shifted distribution" exactly means, e.g. early on in the introductory discussion about the lack of generalization of CNNs.

Can a theoretical justification be given for the "calibration" of probabilities?

The "deep clustering" work by John Hershey et al. may be of interests to the authors, e.g. http://www.merl.com/publications/docs/TR2016-003.pdf.

How sensitive are the results to "alpha", controlling the moving average?

Line 170: "CV": I assume this is short for "Cross-validation"?

Line 174: "vanilla": What is the relation of "vanilla" to Table 1, if any?

Line 193: "model selection with max. patience of 20", what is patience? Can you provide a reference?

"All models are trained 195 with stratified batch iterators and batch size of 75. " Can you provide an explanation/reference for "stratified batch iterators"?

line 224: "Also comparing the unseen embeddings to the"
line 225:  "training embeddings (with lower opacity and in grey) it can be seen that the unseen embeddings"
line 226: "projected closer to the training embeddings after applying DWCCA"

I'm not sure i agree. Can you be more specific?

line 234: " An interesting observation is the range of eigenvalues in vanilla"

If there is no normalization, the large range isn't surprising, is it?

line 236: "data (validation and test) compared to the training data. The maximum eigenvalue in DCASE2016"
line 237: "is around 0.7"

If the covariance-based normalization were perfect, we would expect the eigenvalues to all be close to 1.0, correct? Is 0.7 a good result, or a sign that the method only works in an approximate sense?

Re: the KNN results. with non-normalized data it's normal that KNN wouldn't do as well as with normalized data. In principle KNN should eventually do as well, but with a large sample size? KNN uses a distance metric treating all dimensions equally. But the key question is how does the categorical softmax layer perform with, versus without, the normalizing layer?

Also, please check spelling and grammar throughout the draft. E.g., the following:

"We evaluate VGG on data with as well as without distribution mismatch ... " --> insert commas for readability.

"As (line 55) the blue curves show, the eigenvalues in unseen data of validation (b and e) and test (c and d) have
(line 56) considerably higher ranges than train data (a and d) for all the datasets we used." --> rewrite sentence for readability & grammar.

line 58 ", we carry out K-nearest neighbor (KNN) experiments on" --> probably want past tense, "we carried out"
line 59: "the performance degredation" --> check spelling.

line 116: " and has proven to be effectively used in combination with different scoring functions" --> "effective" instead of "effectively used"?

line 191: "The setup used in all of our experiments are as follows" ---> "... is as follows ..."

Finally, I noticed that the paper stretches to 12 pages, when 10 was the maximum for submission.
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper22/AnonReviewer2
title: Approach to generalize better to unseen data by using a WCCN normalization layer in a CNN.
review: The authors presented an approach to modify a neural network to generalize better to unseen test data. The experimental setup is multi-class classification of acoustic scenes using CNNs. The novelty is to integrate a WCC layer into the CNN. Different experiments and baselines are shown to demonstrate mainly a classification improvement. The paper in general is well-written.I liked that the authors discussed (to some degree) the different effects of the variance in the data distributions. 

Suggested improvements:
In line 123, the Cholesky decomposition is introduced for the first time, but with no proper definition or justification or reference to related work. 
Could authors provide any intuition or comment on how overfitting is affected by the DWCCA approach? For instance, in DCASE16, the performance of the folds increased significantly, but then this didn’t hold in the test. Moreover, in DCASE17, although there is a great improvement in the test set when using the DWCCA approach, the performance in test does not really resembles the training set as DCASE16.
What are the internal representations in this paper, to which layer output do they correspond to?
Does adding the WCCA layer makes the CNN converge faster or slower?
Could you improve the explanation of the takeaway of Figure 3?
Not sure why “beach” had high scores in DCASE16, but not in DCASE17, could you explain that? 
What is the explanation of the lower performance of DCASE17-Tram when using DWCCA?
Out of curiosity, it would be interesting to hear what authors have to say about why WCC was considered to be more important as opposed to BCC (Between-class covariance).
I understand that the main purpose of the paper is to demonstrate that the technique works and that space is limited so feel free to avoid including comments on this paragraph. For this publication in a brief manner or for future work in a deeper manner, it would be helpful to understand better how variance affect the acoustic scenes, which will serve as a stronger justification of the approach. For example, what classes exhibit more or less variance, are there many of them, is it related to length or to other physical characteristic?; if the DWCCA not always helps to improve performance, then under what conditions it does; how does this approach might differ from Deep Canonical Covariance Analysis. 
rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper22/Authors
title: Authors response
comment: Dear reviewer, 
Thank you for your review and positive feedback, we appreciate it!
We will fix the issues you pointed out above, in our next revision.

From: NIPS.cc/2018/Workshop/IRASL/Paper22/AnonReviewer3
title: Well-written paper, clear contribution with good results, good fit for the robustness and interpretability workshop
review: In this paper, Deep Within-Class Covariance Analysis (DWCCA), a DNN layer version of the conventional WCCA is proposed. DWCCA is shown to improve the network's internal representation, as well as increase the end-to-end classification accuracy.  

The paper proposes a solution in order to have a model that is robust enough to handle training and test distribution mismatch. In such cases, CNN have shown drop in performance. It is suggested that this is due to larger class variance in test than training data when there is a shifted distribution in unseen test data. This is shown by visualizing the internal representations of the network in a 2D  space using PCA. In addition, an objective evaluation of the class variance is presented by computing eigenvalues of each class' covariance matrix, with a higher range of eigenvalues in unseen data is observed. Finally, the KNN performance is  evaluated and found to be worse for higher variance. 

After this interpretability part, the paper focuses on suggesting a solution to make the system more robust by using the DWCCA. This is implemented as a DNN layer that can be placed anywhere and that reduces that within-class covariance of DNN internal representations. Using such a layer improves both KNN performance and end-to-end classification of an acoustic score detection task with distribution shift.

The idea of the paper is presented in a clear fashion and the results support the claims of the authors. 

Some  minor comments:
-There are some repetitions of the main ideas/contributions of the paper that could be removed
-In figure 2, the "lower opacity and in grey" representation of training embeddings is not clear in my version of the paper
-Some tidy up of the References is needed, e.g. capitalize when needed.


rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper22/Authors
title: Authors response
comment: Dear reviewer, thank you for taking the time to review our paper! We appreciate your detailed review and positive feedback. Please find our response below:

- The point raised regarding the dimensionality of the representation and the size of the minibatch is a valid argument and we would like to respond with a couple of possible solutions:
First: Recent papers [1,2,3] have been reported that increasing the batchsize (to the scale of several thousands) is possible without sacrificing the performance. When the dimensionality is high, the batchsize can be increased accordingly to avoid the rank-deficiency of the covariances.
Second: Another possibility (which we preliminarily explored) is the use of what we call "sub-stratified" batch iterators and is described as follows. In each batch, a subset of C_s classes from C available classes are randomly chosen and then an equal number of samples from each of these C_s classes are selected for a batch. The reduction in the number of classes, increases the possibility of using more samples from each class in each batch to avoid the rank-deficiency of the covariances. Of course this implies that we maintain the running-average of the transformation matrix. We did not observe any negative effects in our preliminary results using sub-stratified batch iterators instead of stratified batch iterators. Though it still requires more investigations and experimentation which will be done in our future work. 

- We experimented with running-average of both within-class covariance and B (the transformation) and both resulted in similar performances, though B provided slightly better performances on the datasets we used hence is the one presented in the paper.
Our DWCCA layer implementation (to be released soon) provides options for choosing various running-averages including WCC and the transformation B.

- Regarding the position of the DWCCA layer, we agree that placing it closer to the output makes more sense and we agree with your argument. Yet depending on the application (transfer learning, distribution mismatch, etc) this position can be adjusted.  We will update the text with a better description in our next revision. 

- Authors of [4] report that, applying sliding-window on 30 seconds snippets of DCASE2016 (various 149 x 149 instead of one 149 x 938 input) improves the results (they used a similar VGG architecture) on the validation set, compared to using the whole 30 seconds. Hence, it can be seen that the long duration slightly affects the performance on the validation sets. Though comparing [4] performances to ours, we observe that the unseen test results are very close.
It can also be seen that VGG+DWCCA copes better with the long-duration effect and improves the results on the validation set of DCASE2016. This can be interpreted as longer duration causing more variance in the representation of VGG, which DWCCA can reduce. We omitted the sliding window approach since our focus was more on the representation learning part and evaluating the generalisation of the  learned representations and the sliding-window approach was limiting the rest of the evaluations we carried out (knn).

- Thank you for the interesting paper! We will look into this for our future work.

- And thank you for the proof-reading and suggestions for the table and the figure. We will update these in our next revision.

[1] "Measuring the Effects of Data Parallelism on Neural Network Training", C. j. Shallue et al.
[2] "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour", P. Goyal et al.
[3] "Train longer, generalize better: closing the generalization gap in large batch training of neural networks", E. Hoffer et al.
[4] "A hybrid approach with multi-channel i-vectors and convolutional neural networks for acoustic scene classification", H. Eghbal-zadeh et al.

From: NIPS.cc/2018/Workshop/IRASL/Paper22/AnonReviewer1
title: A simple, sensible idea that is clearly communicated, but with an implementation that may not be general enough
review: This paper proposes a version of within-class covariance analysis that is compatible with the the minibatch gradient-based approaches to training that are popular for deep learning models. The proposed "deep within-class covariance analysis" (DWCCA) layer performs a linear transformation of its input that is intended to minimize the within-class covariance of the output representation. In an off-line setting, this is accomplished by making the linear transformation the Cholesky decomposition of the inverse of the average within-class covariance matrix. The DWCCA layer computes the within-class covariance matrix over a minibatch, computes the Cholesky decomposition of the inverse of the minibatch within-class covariance, and maintains a running average of the Cholesky decompositions over minibatches as the linear projection. This on-line computation is similar in spirit to the computation done by a batch normalization layer. The implementation described in the paper relies on Theano to compute the necessary gradients, but the computation should also be possible in other deep learning frameworks such as PyTorch.  The empirical work focuses on the audio scene classification task, using the DCASE2016 data as an example of a dataset with no distribution shift between the training and test data and the DCASE2017 data as an example of a dataset with a distribution shift. A VGG architecture, adapted for audio scene classification, is used as the model in all experiments. 2d PCA visualizations of the representations learned by the models, the eigenvalue spectra of the within-class covariances of the learned representations, K-NN classification results, and audio scene classification performance results all indicate that addition of a DWCCA layer to the VGG model makes it more resilient in the face of distribution shift.

The basic idea in the paper makes sense, the experiments are well designed and well executed, and the topic is well matched to the theme of the workshop.

However, there is an important issue about estimation of the WCCA transform that is not addressed in the paper that I think may limit the usefulness of this implementation. Namely, if the dimensionality of the layer at which DWCCA is applied is large relative to the size of the minibatch used in training, there is a danger that S_w may be rank-deficient and therefore not invertible. Under such circumstances, one would need a different estimation framework for DWCCA to be usable. The specific conditions in the experiments in this paper seem to be designed to sidestep this issue: DWCCA is applied on a small (15-dimensional) layer, the minibatch size is 5 times that, and (it appears) stratified sampling is used to ensure that each class has the same number of samples in each minibatch.

On a related note, would it make more sense to maintain a running estimate of the average within-class covariance matrix during training instead of a running average of the transform?

While the introduction says that DWCCA can be placed "in any arbitrary position in a DNN," doesn't it make the most sense to place it as close to the output as possible? The class-conditional distributions of deeper representations are much more likely to be unimodal than representations closer to the input are.

Isn't it more accurate to describe DWCCA as implementing a "transformation" of its input instead of a "projection"? There does not appear to be any dimensionality reduction going on.

I find it surprising that the DCASE2017 validation results are better than the DCASE2016 validation results, given that DCASE2017 uses more challenging 10-sec audio segments. Can you explain that?

Another good paper on the challenges CNNs experience in the face of mode shift is J. Jo and Y. Bengio, "Measuring the tendency of CNNs to Learn Surface Statistical Regularities," arXiv:1711.11561 [cs.LG].

The first line in Table 1 is confusing.  Should it read "Input: 938 x 149 (DCASE2016) or 313 x 149 (DCASE2017)"?

Figures 3 and 4 would be clearer if all plots in the same row had the same y-axis limits.

Proofreading:

degree of representation’s variance => degree of a representation’s variance

and test (c and d) => and test (c and f)

performance degredation => performance degradation

collected as 3-5 minutes length recordings => collected as 3-5 minute long recordings

and then was cut into segments of 30 seconds length => that were subsequently cut into 30-second segments

were split into only 10s long segments => were split into 10s long segments

using the output of global average pooling => using the output of the global average pooling

the test accuraty in the mismatch case => the test accuracy in the mismatch case

of about 6 percentage point => of about 6 percentage points

by around 6 percentage point when => by around 6 percentage points when

rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct