Paper:
UNSUPERVISED SPEECH RECOGNITION VIA SEGMEN- TAL EMPIRICAL OUTPUT DISTRIBUTION MATCHING
We consider the problem of training speech recognition systems without using any labeled data, under the assumption that the learner can only access to the input utterances and a phoneme language model estimated from a non-overlapping corpus. We propose a fully unsupervised learning algorithm that alternates between solving two sub-problems: (i) learn a phoneme classifier for a given set of phoneme segmentation boundaries, and (ii) refining the phoneme boundaries based on a given classifier. To solve the first sub-problem, we introduce a novel unsupervised cost function named Segmental Empirical Output Distribution Matching, which generalizes the work in (Liu et al., 2017) to segmental structures. For the second sub-problem, we develop an approximate MAP approach to refining the boundaries obtained from Wang et al. (2017). Experimental results on TIMIT dataset demonstrate the success of this fully unsupervised phoneme recognition system, which achieves a phone error rate (PER) of 41.6%. Although it is still far away from the state-of-the-art supervised systems, we show that with oracle boundaries and matching language model, the PER could be improved to 32.5%. This performance approaches the supervised system of the same model architecture, demonstrating the great potential of the proposed method.

Review:
Overview:
This paper proposes a new approach to do unsupervised phoneme recognition by learning from unlabelled speech in combination with a trained phoneme language model. The proposed loss function is a combination of a term encouraging the language model of predicted phonemes to match the given language model distribution, and a term to encourage adjacent speech frames to be assigned to the same phoneme class. Phoneme boundaries are iteratively refined using a separate model. Experiments where a hidden Markov model is applied on top of the predicted phonemes are also performed.
Main strengths:
The paper is clear and addresses a very important research problem. The approach and losses proposed in Section 2 have also not been proposed before, and given that an external language model is available, are very natural choices.
Main weaknesses:
The main weakness of this paper is that it does not situate itself within the rich body of literature on this problem.  I give several references below, but I think the authors can include even more studies -- there are several studies around "zero-resource" speech processing, and I would encourage the authors to work through the review papers [1, 6].
Concretely, I do not think the authors can claim that "this is the first fully unsupervised speech recognition method that does not use any oracle segmentation or labels." I think it could be argued that the system of [3] is doing this, and there are even earlier studies. I also don't think this claim is actually necessary since the paper has enough merit to stand on its own, as long as the related work is discussed properly.
For instance, the proposed approach shares commonalities with several other approaches: [2] also used two separate steps for acoustic modelling and boundary segmentation; [4, 7, 8] builds towards the setting where non-matching text data is available (for language model training) together with untranscribed speech for model development; the approach of [5] uses a very similar refinement step to the one described in Section 3, where an HMM model is initialised and retrained on noisy predicted labels.
In the experiments (Section 4), it would also be useful to report more fine-grained metrics. [6] gives an overview of several of the standard metrics used in this area, but at a minimum phoneme boundary recall, precision and F-scores should be reported in order to allow comparisons to other studies.
Overall feedback:
Given that this paper is situated within the broader context of this research area, which already has a small community around it, I think the novelty in the approach is strong enough to warrant publication given that the additional metrics are reported in the experiments.
Papers/links that should be reviewed and cited:
1. E. Dunbar et al., "The Zero Resource Speech Challenge 2017," in Proc. ASRU, 2017.
2. H. Kamper, K. Livescu, and S. Goldwater. An embedded segmental k-means model for unsupervised segmentation and clustering of speech. in Proc. ASRU, 2017.
3. Lee, C.-y. and Glass, J. R. A nonparametric Bayesian approach to acoustic model discovery. ACL, 2012.
4. Ondel, Lucas, Lukaš Burget, Jan Černocký, and Santosh Kesiraju. "Bayesian phonotactic language model for acoustic unit discovery." In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pp. 5750-5754. IEEE, 2017.
5. Walter, O., Korthals, T., Haeb-Umbach, R., and Raj, B. (2013). A hierarchical system for word discovery exploiting DTW-based initialization. ASRU, 2013.
6. M. Versteegh, X. Anguera, A. Jansen, and E. Dupoux, "The Zero Resource Speech Challenge 2015: Proposed approaches and results,” in Proc. SLTU, 2016.
7. https://www.clsp.jhu.edu/wp-content/uploads/sites/75/2018/05/jsalt2016-burget-building-speech-recognition.pdf
8. https://www.clsp.jhu.edu/workshops/16-workshop/building-speech-recognition-system-from-untranscribed-data/