Paper:
We present a novel generative model that combines state-of-the-art neural textto-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 30 minutes supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. Audio samples are available on the web 1 .

Review:
Overview:
This paper proposes to use semi-supervised learning to enforce interpretability on latent variables corresponding to properties like affect and speaking rate for text-to-speech synthesis. During training, only a few training items are annotated for these types of properties; for items where these labels are not given, the variables are marginalised out. TTS experiments are performed and the approach is evaluated objectively by training classifiers on top of the synthesised speech and subjectively in terms of mean opinion score.
I should note that, although I am a speech researchers, I am not a TTS expert, and my review can be weighed accordingly.
Strengths:
The proposed approach is interesting. I think it differs from standard semi-supervised training in that at test time we aren't explicitly interested in predicting labels from the semi-supervised labelled classes; rather, we feed in these labels as input to affect the generated model output. I agree that this is a principled way to impart interpretability on latent spaces which are obtained through unsupervised modelling aiming to disentangle properties like affect and speaking rate.
Weaknesses:
This work misses some essential baselines, specifically a baseline that only makes use of the (small number of) labelled instances. In the experiments, the best performance is achieved when gamma is set very high, which (I think) correspond to the purely supervised case (I might be wrong). Nevertheless, I think a model that uses only the small amount of labelled data (i.e. without semi-supervised learning incorporating unlabelled data) should also be considered.
As a minor weakness, the evaluation seems lacking in that human evaluations are only performed on the audio quality, not any of the target properties that are being changed. For affect specifically, it would be helpful to know whether the changes can be perceived by humans. As a second minor weakness, some aspects of the paper's presentation can be improved (see below).
Overall assessment:
The paper currently does not contain some very relevant baselines, and I therefore assign a "weak reject".
Questions, suggestions, typos, grammar and style:
- p. 1: "control high level attributes *of of* speech"
- p. 2: It would be more helpful to state the absolute amount of labelled data (since 1% is somewhat meaningless).
- p. 2: I am not a TTS expert, but I believe the last of your contributions have already been achieved in other work.
- Figure 2: It would be helpful if these figures are vectorised.
- p. 4: "*where* summation would again ..."
- Figure 4: Is there a reason for the gamma=1000 experiment, which performs best in (a), not to be included in (b) to (d)?
- Section 5: Table 1 is not references in the text.
- Section 5.1: "P(x|y,z_s,z_u)" -> "p(x|y,z_s,z_u)"
- In a number of places, I think the paper meant to cite [1] but instead cited the older Kingma & Welling (2013) paper; for instance before equation (6) (this additional loss did not appear in the original VAE paper).
References:
[1] https://arxiv.org/abs/1406.5298
Edit: Based on the author's response, I am changing my rating from a 'weak reject' to a 'weak accept'.