From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper48/Authors
title: Regarding the obviousness of our results
comment: Thank you for your reviews. To clarify one point raised by both reviewer 3 and reviewer 2: We began with the obvious statement that early layers would be more task-general and hence transferable. The primary goal of this paper is to quantify and specify *exactly* how task-general are the representations in each layer. Without such an analysis, one might expect a linear increase in task-specifity as we ascend to deeper layers. This is not what we find. Instead, we find a sharp cutoff at the beginning of the fully connected layers, where preceding layers trained on the source language can be used exactly, without fine-tuning on the target language, with no effect on performance compared to baseline. It is the specific shape of this transition from task-general to task-specific that we evaluate and discuss in this paper.

From: NIPS.cc/2018/Workshop/IRASL/Paper48/AnonReviewer1
title: Both vision and acoustic models learn more general and then more specific features "bottom up"
review: The idea in this paper is to use layer weights from an acoustic model trained on one language (e.g. German) to initialise a second acoustic model intended for another language (e.g. English). 

As in visual object recognition (Yosinski et al. 2014), the authors found that networks used in acoustic models converge "bottom up" and learn more general and then more specific features. For example, the first two layers were entirely general (i.e. transferable between German and English), the next six layers were "highly transferable", and the final (fully connected) layers were mostly language specific (not at all surprising given that the output targets varied between language). 

Surprisingly, the best results were initially found in the control case of transferring weights between acoustic models for the same language. In order to understand this result, the authors were led to explore the use of "freeze-training" (Raghu et al. 2017), where layer weights are frozen from the bottom up in stages during the process of training. This in turn led to better overall performance, "especially among the transfer networks". 

I like this paper very much. In addition to investigating layers in acoustic models, and the generalisability of vision-based ideas from Yosinski et al. 2014, it promises a practical application for low-resource languages when weights can be pre-trained in another, better-documented language. 

It will be interesting in future work to see how this promise bears out between more distinct language pairs, given that German and English were selected for their similarity. Many low-resource languages will be less closely related to any high-resource "world language".

Although the paper was well written, I could not find some minor details that I would need to replicate the study. What was the cost function? If it was cross-entropy (my guess), was there a softmax layer after the final fully-connected layer? 

Also, out of interest, how big were the phonesets for each language? And how were the 9000 "target context-dependent phone classes" constructed? 

Minor comments:

* "CCNs" -> "CNNs" (lines 11 and 46).
* "in (ASR) systems" -> "in ASR systems" (line 37).
* There are a few mismatched quotation marks, e.g. around "bottom up" (line 12), "network surgeries" (line 68) and "selfer" (line 71).
* Please make explicit somewhere that the triplets on lines 59-60 represent filter sizes and number of feature maps.
* Likewise on line 60, for explicitness please state that the singletons represent number of hidden units per layer.
* Finally, unless the authors feel they should not be compared, could the plots in Figure 1 have the same ylims?
rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper48/AnonReviewer3
title: acoustic model transfer across languages
review: This problem is relevant for low resources languages and has been studied for quite some time.

‘CCN’ typo in abstract

Don’t understand this:
“Few studies have characterized the intermediate representations of networks trained on acoustic tasks [Lee et al., 2009, Golik et al., 2015], which are often not trained long enough to converge completely (test error still slowly decreasing at the end of training).”
Most model are trained to full convergence, no ?

Similar work has been done before, DARPA Babel program comes to mind.

Some of the conclusions seem obvious,  like “early layers need not be finetuned on the target language at all.”

rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper48/AnonReviewer2
title: How transferable are features in convolutional neural network acoustic models across languages?
review: 
The paper measures the transferability of features for each layer in CNN-based acoustic models across languages (English and German). According to the results of several experiments, the conclusions are:
1) Features from lower layers are more transferable than those from higher layers.
2) The AMs trained with “the feeze training” technique significantly outperformed all other transferred models. With this technique, even AM trained solely on one language was better than the one trained with naïve transfer approach.

- Pros:
It is a well-written paper. The experiments seem complete, and very reasonable rationale was provided for each result.

- Cons:
This is not the paper that proposes a novel solution. Some conclusions seem obvious, and “the freeze training” technique was already proposed by the previous work (although it was not in the speech domain).

- typos (minor):
1) CCN seems a typo for CNN (in line 11 and 46).
2) The languages should be written inversely in the legend of Figure 1 (b). For example, “GER->ENG” must be “ENG->GER”.
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct