Paper:
There are more than 7,000 languages in the world, but due to the lack of training sets, only a small number of them have speech recognition systems. Multilingual speech recognition provides a solution if at least some audio training data is available. Often, however, phoneme inventories differ between the training languages and the target language, making this approach infeasible. In this work, we address the problem of building an acoustic model for languages with zero audio resources. Our model is able to recognize unseen phonemes in the target language, if only a small text corpus is available. We adopt the idea of zero-shot learning, and decompose phonemes into corresponding phonetic attributes such as vowel and consonant. Instead of predicting phonemes directly, we first predict distributions over phonetic attributes, and then compute phoneme distributions with a customized acoustic model. We extensively evaluate our English-trained model on 20 unseen languages, and find that on average, it achieves 9.9% better phone error rate over a traditional CTC based acoustic model trained on English.

Review:
Overview:
This paper proposed an approach for zero-shot phoneme recognition, where it is possible to recognise phonemes in a target language which has never been seen before. Rather than just training a phoneme recogniser directly on background data and then applying it to unseen data, phonetic features are first predicted, allowing phonemes not in the source language set to be predicted.
Main strengths:
The paper's main strength lies in that this is a very unexplored area that could assist in the development of speech technology where it is currently not possible. The proposed model (Section 2) has also not been considered in prior work.
Main weaknesses:
The paper's main weakness is in some of its claims and that it misses some very relevant literature. Detailed comments together with a minimal list of references are given below (but I would encourage the authors to also read a bit more broadly). But in short I do not think it is that easy to claim that this is the first paper to do zero-shot learning on speech; many of the zero-resource studies where unlabelled audio is used could be seen as doing some for of zero-shot matching. Specifically [5] is able to predict unseen phoneme targets.  Multilingual bottleneck features can be applied to languages that have never been seen before [2], and the output of phoneme recognisers trained on one language have long been applied to get output on another unseen language. The first one-shot learning speech paper [4] (to my knowledge) is also not mentioned at all. The approach in the paper also still relies on some text data from the target language; if this then can be described as "zero-shot" learning, then I think many of these previous studies c also make this claim.
Overall feedback:
There is definitely value in this work, but it should be much better situated within the broader literature. Below I give some editorial suggestions and also outline some suggestions for further experiments.
Detailed comments, suggestions and questions:
- Abstract: It would be useful to have some details of the "baseline model" here already, especially since it is such a new task.
- Introduction: "... but they can hardly predict phones or words directly due to their unsupervised nature." This is a strong statement that maybe requires more justification. On the one hand, the statement is true, and the high word error rates in e.g. [3] can be cited. On the other hand, it has been shown that at the phone-distinction level, these models perform quite well and sometimes outperform supervised models [1]. Since this paper also considers phone error rate as a metric, I think care should be taken with such statements.
- Introduction: "While zero-shot learning has attracted a lot of attention in *the* computer vision community, this setup has hardly been studied in speech recognition research especially in acoustic modeling." Definitely look at some of the studies mentioned below, and also [4] specifically.
- "However, we note that our model can be combined with a well-resourced language model to recognize words." How would this be done, since I think this is actually quite a challenging task.
- Section 2: "... useful the original ESZSL architecture ..." -> "... useful in the original ESZSL architecture ..."
- Section 2.2: I assume the small text corpus is at the phone level (and not characters directly)? This should be clarified, and it could raise the question of whether this approach is truly "zero-shot".
- Section 3.2: "We used EESEN framework ..." -> "We used the EESEN framework ..."
- Section 4: You could look at the recent work in [2], which uses multilingual bottleneck features trained on 10 languages and applied to multiple unseen languages. It would be interesting to also train your approach on multiple languages instead of only English.
Missing references:
1. M. Heck, S. Sakti, and S. Nakamura, "Feature Optimized DPGMM Clustering for Unsupervised Subword Modeling: A Contribution to Zerospeech 2017," in Proc. ASRU, 2017.
2. E. Hermann and S. J. Goldwater, "Multilingual bottleneck features for subword modeling in zero-resource languages," in Proc. Interspeech, 2018.
3. H. Kamper, K. Livescu, and S. Goldwater, An embedded segmental k-means model for unsupervised segmentation and clustering of speech," in Proc. ASRU, 2017.
4. B. M. Lake, C.-Y. Lee, J. R. Glass, and J. B. Tenenbaum, "One-shot learning of generative speech concepts," in Proc. CogSci, 2014.
5. O. Scharenborg, F. Ciannella, S. Palaskar, A. Black, F. Metze, L. Ondel, and M. Hasegawa-Johnson, "Building an ASR system for a low-resource language through the adaptation of a high-resource language asr system: Preliminary results,"in Proc. ICNLSSP, 2017.
Edit: Based on the rebuttal I've changed my rating from 4 to 5.