Paper: LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR-2024)

Reviewer comment (Experiments):
It is not clear if the performance improvement comes from the key improvements, or the replacement of the vocoder, which can be seen as a post-processing step rather than a key part of the algorithm. It is well known that DiffWave produces much more natural-sounding output compared to the Griffin-Lim algorithm used by the previous works.

Author response:
As Figures 2 & 3 in the Appendix show, LipVoicer recovers mel-spectrograms that look like the mel-spectrogram of natural speech. The competitors, however, fail to do so and are missing crucial pitch information and hence sound unintelligible and metallic. Even an excellent vocoder cannot compensate for degraded input mel-spectrograms. Moreover, we trained the vocoder on mel-spectrograms which were extracted from natural speech signals, and not on the mel-spectrograms that we generated with LipVoicer. This gives a strong qualitative indication that the source of the improvement comes mainly from the quality of the mel-spectrogram reconstructed by LipVoicer.
We note that we compare favourably to SVTS that uses a modern neural vocoder, specifically Parallel WaveGan, and not Griffin-Lim. In general, Griffin-Lim is rarely used nowadays in recent papers in the text- and lip- to speech techniques, since more advanced vocoders currently exist.

Reviewer comment (Experiments):
The baseline models appear to be using pre-trained model weights. However, the models are not trained on the same data, so the results cannot be compared directly.

Author response:
We trained the baselines on LRS3 and LRS2, and we thank the reviewer for drawing our attention that we forgot to mention it in the text. Our comparisons are not based on the pre-trained models available on the project pages of the baselines. We will clarify this in the text.

Reviewer comment (Novelty):
The method appears to apply Guided-TTS techniques to the problem of lip-to-speech. Although this is well engineered, in my opinion this work is better suited to speech or CV conference compared to ICLR.

Author response:
We would like to draw the reviewer’s attention that papers of similar or related flavour have been published at ML conferences. For example, [1-3] were published at ICLR, [4] was published at NeurIPS and [5] was published at ICML.

[1] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: textually guided audio generation. The International Conference on Learning Representations (ICLR). 2023.

[2] Haoyue Cheng, Zhaoyang Liu, Wayne Wu and Limin Wang. Filter-recovery network for multi-speaker audio-visual speech separation. The International Conference on Learning Representations (ICLR). 2023.

[3] Kai Li, Runxuan Yang and Xiaolin Hu. An efficient encoder-decoder architecture with top-down attention for speech separation. The International Conference on Learning Representations (ICLR). 2023.

[4] Minsu Kim, Joanna Hong, and Yong Man Ro. Lip to speech synthesis with visual context attentional gan. Advances in Neural Information Processing Systems, 34:2758–2770, 2021.

[5] Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers. Proceedings of the 37th International Conference on Machine Learning, 2020.

Reviewer comment (Novelty):
In previous studies, a very important motivation for lip2speech tasks was to dispense with text modality (otherwise, this task can be transformed into lipreading+TTS), because 80% of the world's languages have no written text. However, this paper still depends on the text modality, so it is difficult to give a high score to this article.

Author response:
Our problem is fundamentally different from a lip-reading+TTS setup. The generated speech must be synchronised with the video, match the speaker appearance and convey prosody. Whereas lip-reading+TTS cannot achieve these objectives, we show how to successfully incorporate the text in a way which does manage to satisfy all of these tasks. Our claim is that the text is important, and should not be dispensed so fast. Importantly, we utilise it without resorting to transcribed videos by humans. The way that we acquire (and exploit) the textual modality is exactly one of the virtues of  LipVoicer. We suggest using a lip-reader, and therefore the text comes for free. We show in our experiments and the provided speech samples that text plays a major part in the success of our method.

Regarding the fact that 80% of the world's languages have no written text, while this might be true, the vast majority of speakers do speak languages with written text. While our solution does not solve all languages, it is relevant to the vast majority of the population and as such we believe this is significant enough to be of value to the community.

Reviewer comment (Novelty):
Lip2speech (Kim et al.(2023)) takes the ground-truth text as input to constrain the learning process and has shown the success of the role of text modality in this task. In this paper, the work uses the predicted text instead of the ground truth as Lip2Speech. But the manner is similar to Kim et al.(2022). So, besides using the predicted text with an existing method, is there some new contributions in the view of methodology?

Author response:
We would like to draw the reviewer’s attention that LipVoicer utilises the **unsupervised text** at **inference time**, whereas Lip2Speech uses the **ground-truth text** for **training** in the form of an additional loss term. Integrating the textual modality with the audio at inference time is not straightforward, and it considerably changes the algorithmic approach that needs to be taken. The words in the textual modality bear no information about their timing, duration and intonation. As Tables 1-4 demonstrate,  by using the classifier guidance, LipVoicer manages to deploy the text fairly better than Lip2Speech despite using noisy text from the lip-reader. Moreover, the ablation study in Table 6 shows that when the text is not used (w2=0), the WER score drops drastically. However when w2 is adequately chosen, we manage to achieve both excellent WER **and** audio quality. Another important difference between LipVoicer and Lip2Speech is that in LipVoicer we manage to extract vocal features of the speaker and make the generated audio reflect the identity of the speaker, whereas Lip2Speech lacks this ability.

Reviewer comment (Presentation):
I am a little confused about the fig.1(a) as described above.

Author response:
The inference procedure of LipVoicer involves the estimation of two noise terms. The first one, $\epsilon_{mg}$ is computed via classifier-free guidance using the MelGen module, whose inputs are the noisy mel-spectrogram $\mathbf{x}\_{t+1}$, the lip motion video and a face image. The second noise term is computed via classifier guidance. The ASR takes $\mathbf{x}\_{t+1}$ as an input, and computes the loss at the output with respect to the text predicted by the lip-reading network.
We have updated the figure so it better reflects the methodology of our method. We thank the reviewer for this comment which has helped us improve the clarity of the Fig. 1.a. We have also added a detailed Algorithm in the Appendix for further clarification.

Reviewer comment (Experiments):
The performance using the predicted text is already very appealing, but the involved lip reading model are almost the best two ones at present, with WER=19% and 26%. if the lip reading performance has been a much low value, e.g. WER=50%, what would be the performance here like?

Author response:
Our main claim is that the text modality, even if it is not perfect, can be of great help to audio generation from video. Therefore we do not argue that our method will perform well with a bad lip-reader, we rather say that the existing lip-readers can greatly boost performance. This is further corroborated in the ablation study we show in Table 7. A lip-reader with WER=32.3% is already not so good, and we show that LipVoicer is adversely affected by it. The better the lip-reader gets, the better our model performs.