Paper: SMILE: Audio-Visual Speech Recognition with Siamese Masked Interaction Learning (ICLR-2024)

Reviewer comment (Experiments):
It is not clear to me how much the visual information contributes to the WER. The WERs reported here are on par with state of the art acoustic-only speech recognition. Why not include an ablation of the modalities — show results for audio-only, video-only and audiovisual.

Author response:
ANS: We provide Table 6 in the appendix in our newly revised paper.

Reviewer comment (Experiments):
The motivation for the work is that the visual modality helps in the presence of acoustic noise. There are no experiments that actually demonstrate this, or quantify the effective gain in SNR provided by the visual information.

Author response:
ANS: We newly provide Table 2 which is trained and tested on LRS2 with noise conditions.

Reviewer comment (Novelty):
The paper describes the problem of AVSR as an “emerging research area”, but this ignores the 30+ years of prior research into multimodal speech recognition.

Author response:
ANS: Thanks for pointing it out. I revise it into "AVSR is an active and evolving research area derived from the Automatic Speech Recognition (ASR) task."

Reviewer comment (Writing):
I find the introductory paragraph that motivates multimodal speech recognition to be unclear. The authors state that “... speech contains detailed phonetic information, while lip movement conveys emphasis or stress ...”. Phonetic information is conveyed visually too — for example, the place of articulation for bilabial, labiodental, and dental sounds can be seen. Furthermore, emphasis and stress are both also conveyed acoustically. The main benefit of multimodal speech is that when acoustic noise masks the differences between two sounds so that they sound similar, those sounds can be disambiguated visually. Similar sounding speech events look different, and similar looking speech events sound different from one another. This is the complementary nature of audiovisual speech.

Author response:
ANS: I appreciate your clear and precise insights, which teach me a lot. According to your advise, we re-formulate our paper and provide   the corresponding experiments in our revised paper. Please see the first part of the introduction and new Table 2. Our motivation is : Lip movements primarily contribute to the improvement of Automatic Speech Recognition (ASR) in two main scenarios. Firstly, they offer distinct visual cues that aid in differentiating between speech words that sound similar. Secondly, they provide valuable visual cues that enhance speech recognition accuracy in the presence of various types of noise commonly encountered in real-world environments.

Reviewer comment (Novelty):
The authors refer to “The original Siamese networks” and cite papers from circa 2020, yet Siamese networks pre-dated these works by a couple of years.

Author response:
ANS: Thanks for pointing it out. We have corrected it in our revised paper.

Reviewer comment (Writing):
Equation (7) and the accompanying explanation feels a little redundant as this already is clear from Equation (6).

Author response:
ANS: We believe that this formula better captures the interaction process between the audio and video modalities, specifically between masked tokens and unmasked tokens.

Reviewer comment (Experiments):
In Equation (9), S is shared across all layers, which weighs the contribution of the visual information. What about using a dynamic, layer-dependent weight (as in Equation (10)) since the contribution of the visual information for the speech recognition task might be different for different layers (as you already note for the acoustic information: “... the shallower layers in audio models tend to prioritize speaker-related information, while the deeper layers tend to emphasize content-related information.”). The same might be true for visual features, and allowing the visual contribution to \hat{F}^{I}_{av} to vary might provide more robust features.

Author response:
ANS: We agree with your viewpoint. However, when conducting experiments with specific S_i values for each layer, we observed that this specific S_i setting performed worse compared to sharing the same S value across all layers. We postulate that S may represent the global weight assigned to visual cues that need to be incorporated into speech.

Reviewer comment (Experiments):
For Table 3(b), why were these four combinations of masking rate for the audio and visual selected? What about the other combinations?

Author response:
ANS: We firstly apply 0.3 masking rate and 6 mask length for both audio and visual modalities as our base setting and find it works well. Then we suppose that Siamese architecture may work with more masking rate and masking length, which is benefit to improve the audio-visual interaction and learn a higher-level audio-visual temporal correspondence. Considering that audio feaures are more redundant than visual features in terms of temporal information, we raise audio mask length to 12. We compare audio-to-video mask ratio with 0.4:0.6 and 0.6:0.4 under same mask length setting, and find that 0.6:0.4 is better. We suppose that masking more audio can force the model to pay more attention to visual cues, which may reduce the “audio-predominant” modality-imbalance problem. Further increasing the masking rate may lead to significant loss of modality information, resulting in a decline in model performance.

Reviewer comment (Experiments):
The authors state that AVSR "is primarily desgined to address ... high levels of noise and speech occlusion" but the paper does not address such scenarios.

Author response:
ANS: We re-formulated our motivation in the revised paper to make it more clearer. Our motivation is revised in the first paragraph in the introduction part.

Reviewer comment (Experiments):
Related to above points, the LRS2/3 dataset was collected using an automated pipeline using ASR. This means that the test set is biased towards examples that are easy for speech recognition systems -- the speech-only model in the data curation pipeline can achieve 100% accuracy (Sec 4 of 1809.02108 and Sec 2.1 of 1809.00496). This undermines the usefulness of this dataset for AVSR under clean conditions.

Author response:
ANS: We provide Table 2 in our paper, which is experimented on noise conditions.

Reviewer comment (Novelty):
The authors cite 2020 and later papers for original Siamese network, but the term was proposed much before by (Koch et al., 2015) for contrastive learning.

Author response:
ANS: Thanks for pointing out it. We correct it in the revised paper.

Reviewer comment (Experiments):
Does the model work in missing modality conditions?

Author response:
ANS: In the newly-revised paper, Table 6 in the appendix is added to provide the audio-only test and audio-only train&test results of our model in both clean and noise test conditions. All models are trained under noise conditions, with a fixed number of mixtures set to 1, a probability of noise set to 1, and a signal-to-noise ratio (SNR) set to 0.

Reviewer comment (Experiments):
What is the performance of audio-only ASR model trained using the same pretraining/finetuning data setup?

Author response:
ANS: In the newly-revised paper, Table 6 in the appendix is added to provide the audio-only test and audio-only train&test results of our model in both clean and noise test conditions. All models are trained under noise conditions, with a fixed number of mixtures set to 1, a probability of noise set to 1, and a signal-to-noise ratio (SNR) set to 0.

Reviewer comment (Experiments):
How sensitive is the result to the values of alpha and beta.

Author response:
ANS: In the newly-revised paper, Table 5(d) in the appendix is added.