From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper65/Authors
title: Author response
comment: Thank you for your constructive feedback on our paper.

> In reading the paper I was assuming that the paper was going to show that the grapheme representation was the desired output. But it turns out that there is a very weak signal. [...] And then the paper described how one worked and the other didn't.
We assume you refer to the indistinct clusters of graphemes when talking about "very weak signal".
This weak signal in layers 9 and 10 does not mean, that grapheme representation did not work.
Our model is predicting graphemes from speech.
The training objective forces grapheme representation in the output layer (Figure 3 bottom left).
However, intermediate phoneme representations are not guaranteed (which motivates our research question).
Therefore, we focus more on the phoneme representation.

> Why does the clustering get so much better between layers 9 and 10?
This is likely related to the network architecture.
Layers 9 and 10 are the layers with the largest number of features in the network. Both have 2048 output features, while other layers have 256 features at most. This comes with many trainable parameters.
Moreover, layer 10 has a kernel width of 1. This means that this layer only applies a feature transformation, which might emphasize the clusters of layer 9.

> And why did you pick these layers? I don't remember seeing it in the paper.
Thanks, you're right. We did not explain this explicitly and will add this in the camera-ready version.
We picked the layers, in which distinct clusters of phonemes emerge.
Lower layers did not show clusters that reflect phoneme similarities.
We will also include Figures for all layers as supplemental data.
In future work we will investigate, how to choose the layers with a quantifiable measure.

> Can you quantify the cleanness of the clusters with respect to layer?
We computed Silhouette scores for quantifying cleanness of clusters.
However, cluster cleanness is not telling, whether class similarities are reflected.
Take Figure 4 as example. For graphemes (left), the Silhouette score is higher in layer 9 than in layer 10 (0.16 and 0.1, respectively). This indicates better clustering for graphemes in layer 9, although no clusters of similar graphemes emerge.
As mentioned above, finding suitable quantifiable measures will be our future work.

> Should you reference CCA as an alternative approach for comparing responses? I saw a paper in the archives recently about this.
Thank you for this hint. We were not aware of this research and will reference it in the camera-ready version.

> I found the color scales hard to understand... black on white is simpler to understand
The extra contrast is needed in some layers, where clusters are less distinct.
We will test, whether black on white has enough contrast in Figures 3 and 4, as those clusters are more distinct.

> Figure 1E is hard to interpret.  Can you select a neuron that has excitations further from the lines?
Figure 1E shows each neuron's normalized activation (Figure 1D), sorted on the time axis.
For all neurons, there are few time points with strong excitation, mostly in the center of the time frame.
Sorting those activations always results in few values close to the lines.
We will optimize the Figure to make the activations in Fig 1E easier to see (e.g. using a lighter colored box).

Thank you for all your smaller suggestions, as well.
We will review them in the camera-ready version.

From: NIPS.cc/2018/Workshop/IRASL/Paper65/AnonReviewer3
title: Interesting paper and analysis, but results are overstated
review: I'm really glad the authors are pursuing this type of analysis.  Hurray.  Thank you.

But I'm disappointed in the results. Most importantly, stating that "phonemes are learned as intermediate representation for predicting graphemes.  The paper, particularly Figure 4, shows that phones are represented. But saying "are learned" implies causality, which I don't think you can say. 

Still I would like to see this paper presented, and I'd like to know more.

In reading the paper I was assuming that the paper was going to show that the grapheme representation was the desired output. But it turns out that there is a very weak signal.  I think the paper would be better if the phoneme and grapheme representations were treated more in parallel.  And then the paper described how one worked and the other didn't.

Why does the clustering get so much better between layers 9 and 10?  (And why did you pick these layers?  I don't remember seeing it in the paper.) Can you quantify the cleanness of the clusters with respect to layer?

Smaller issues:
Line 74: decision -> responses (since you don't actually see the decision boundaries)
Line 74: by projecting -> by back projecting
L124 and 127: Change "we" to "our model" to make it clearer that the two things are both modifications to the analysis (at the start of the paragraph).

Should you reference CCA as an alternative approach for comparing responses?  I saw a paper in the archives recently about this.

Figure 1: I found the color scales hard to understand... black on white is simpler to understand, and I'm not sure you need the extra contrast provided by a color scale.  Figure 1E is hard to interpret.  Can you select a neuron that has excitations further from the lines?
L168: I didn't understand the splitting.  Can you provide an example to make it clearer?
L180: euclidean -> Euclidean
L201: We hypothesized {better, worse} encoding...

L207: where did blue come from?


rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper65/Authors
title: Author response
comment: Thank you for your detailed review of our submission.
We would like to clarify as many of your comments as possible.

> Hierarchical clustering of neuron activation profiles for the interpretation of acoustic networks has previously been described in Nagamine 2014.
We agree and also cite the work of Nagamine et al.
We also state that our method is inspired by their research in the third paragraph of Section 3 (line 123 ff.).
There, we explain that our analysis is similar to theirs, but we needed to develop a novel method for determining neuron responses, which is not limited to multi-layer perceptrons.
We are sorry, if this relation was not clear. We will emphasize this in the camera-ready version.

> The question, does the grapheme classifier implicitly learn a representation of phonemes, could have been addressed more directly, by trying to linearly decode phonemes from intermediate representations.
We agree that this approach is more direct, but it is only little interpretable.
It can only tell something about the information content of the representation, but not whether similarity between classes is reflected in the representations.
In the fourth paragraph of Section 4 (line 265 ff.), we point out that we will include this method (linear classifier probes) in future work as a means of automatically detecting layers where certain classes are encoded.
To improve clarity, we will include a discussion why this basic analysis is not sufficient to answer our specific question in the camera-ready version.

> Additionally, the phonetic labels were produced from the predicted graphemes. Why were they not produced from the ground truth transcriptions?
We analyzed a model, which was trained on LibriSpeech [Panayotov et al. 2015].
This data set does not provide phoneme transcriptions as a ground truth.
In our opinion, an introspection method should not require using a particular data set, so we left the training data unchanged.
However, we agree that a data set with phoneme transcriptions would have provided a more controlled task.

regarding the cons:
* important details are missing about the network architecture, making it impossible to replicate the results
Our focus was on introspection rather than the speech recognition model training.
In Section 2.1 we refer to the paper of Kunze et al. 2017 for details about the architecture (which includes code in an online repository for reproducibility).

* writing not clear, many ungrammatical sentences
We would be grateful if you could point out specific examples such that we can improve the clarity of writing in the camera-ready version and future work.

* overuse of the word “complex”. Please be more specific.
We will review this for the camera-ready version.

* analysis feels incomplete. results from only a subset of layers are provided. decoding of phonemes from activation patterns at each layer is not performed.
Figures for all layers will be included as supplemental material in the camera-ready version of the paper. We omitted this for the review.
See above for explanation why we did not decode phonemes directly.

From: NIPS.cc/2018/Workshop/IRASL/Paper65/AnonReviewer2
title: Interesting and relevant work
review: The paper presents an interesting approach in interpreting the activations of a network, while performing grapheme prediction. The topic is relevant to the workshop and the analysis seems to be extend enough. The results seem to support the claims, but only to a small extend. 

Additionally, the paper could benefit from providing the ability for reproducing the results. There is not a clear presentation of the specifics of the method (in whole) or (as an alternative) a link to an online repository with the code. 


rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper65/AnonReviewer1
title: Interesting analysis but novelty overstated and claims weakened by lack of clarity and incompleteness
review: The authors demonstrate the use of time-independent Neuron Activation Profiles (NAPs) to interpret a convolutional speech recognition system. NAPs are averaged within phoneme or grapheme classes and then hierarchical clustering is applied to similarity matrices for each layer.  The authors claim that their analysis demonstrates that their speech to grapheme classifier implicitly learns an intermediate phonetic representation because phoneme NAPs at layer 9 cluster into meaningful phonetic categories, while meaningful clusters of graphemes does not occur until the following layer.

Hierarchical clustering of neuron activation profiles for the interpretation of acoustic networks has previously been described in Nagamine 2014. The work presented in this paper would be better described as extending this previous approach to a new different context, rather than a novel method, as it is presented. 

Upon first read of the paper, I was not convinced of the authors main claim, that the grapheme classifier implicitly learns a representation of phonemes. After further consideration, I see the strength of their analysis. Clarity could be improved greatly to require less effort on the part of the reader. 

The question, does the grapheme classifier implicitly learn a representation of phonemes, could have been addressed more directly, by trying to linearly decode phonemes from intermediate representations. The authors do not perform this basic analysis and do not discuss why not. They do not motivate why their analysis is well-suited to answer this type of question (I can imagine good reasons but they are not mentioned, e.g. taking into account within class structure, not merely class identity). I would have liked to see the basic analysis, decoding phonemes directly, as a precursor to the analysis shown here.  The authors are not explicit about what it means for a clustering to be “meaningful”, but this meaning is key to making their hypothesis falsifiable.  Additionally, the phonetic labels were produced from the predicted graphemes. Why were they not produced from the ground truth transcriptions?

The authors assume that neuron selectivity to a particular class is indicative of a functional role of that unit in representing that class. It is known that deep networks learn distributed representations where individual unit selectivity may have little to do with how different classes are represented. For example, Morcos et al showed that class-selectivity of individual units was unrelated to the effect of unit ablations on performance. This is part of why I would have liked to have seen the result of a decoding analysis which may find distributed patterns that code for phonemes. 

This work is important but I am not sure it is ready for presentation/publication.

Pros:
* Highly relevant to the topic of the workshop
* reasonable and simple extension of previously a previously described analysis technique to a new context

Cons:
* important details are missing about the network architecture, making it impossible to replicate the results
* writing not clear, many ungrammatical sentences
* overuse of the word “complex”. Please be more specific.
* analysis feels incomplete. results from only a subset of layers are provided. decoding of phonemes from activation patterns at each layer is not performed. 

References:

Nagamine, T., Seltzer, M. L., & Mesgarani, N. (2015). Exploring How Deep Neural Networks Form Phonemic Categories. Interspeech, 1912–1916.

Morcos, A. S., Barrett, D. G., Rabinowitz, N. C., & Botvinick, M. (2018). On the importance of single directions for generalization. ICLR
rating: 2: Marginally below acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature