Original: J OINTLY L EARNING V ISUAL AND A UDITORY S PEECH R EPRESENTATIONS FROM R AW D ATA
Revised: J OINTLY L EARNING V ISUAL AND A UDITORY S PEECH R EPRESENTATIONS FROM R AW D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations.
Revised: We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations.

Original: Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders.
Revised: Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders.

Original: Driven by the inherent differences between video and audio, our design is asymmetric w.r t. the two modalities’ pretext tasks:
Revised: Driven by the inherent differences between video and audio, our design is asymmetric w.r.t. the two modalities’ pretext tasks:

Original: Whereas the auditory stream predicts both the visual and auditory targets, the visual one predicts only the auditory targets.
Revised: Whereas the auditory stream predicts both the visual and auditory targets, the visual one predicts only the auditory targets.

Original: We observe strong results in lowand high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders were jointly trained.
Revised: We observe strong results in lowand high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.

Original: Notably,  combining RAVEnpre-training with self-training using only 30 hours of labelled data achieves competitive performance for visual speech recognition on LRS3, surpassing all self-supervised methods and a recent fullysupervised method trained on 90,000 hours of non-publiclabelled data.
Revised: Notably, RAVEn surpasses all self-supervised methods on visual speech recognition (VSR) on LRS3, and combining RAVEn with self-training using only 30 hours of labelled data even outperforms a recent semi-supervised method trained on 90,000 hours of non-public data.

Original: At the same time, we are on par with the state-of-the-art  for auditory speech recognition on LRS3.
Revised: At the same time, we achieve state-of-the-art results in the LRS3 low-resource setting for auditory speech recognition (as well as for VSR).

Original: Our findings point to the viability of learning powerful speech representations entirely from raw video and audio, i e. , without relying on handcrafted features.
Revised: Our findings point to the viability of learning powerful speech representations entirely from raw video and audio, i.e. , without relying on handcrafted features.

Original: Code and models will be made public
Revised: Code and models will be made public.

Original: The sound of someone articulating words coincides with the sight of movements in and around their mouth.
Revised: The sound of someone articulating words coincides with the sight of movements in and around their mouth.

Original: Both a recording of a speech waveform and a corresponding silent video of mouth motion provide rich - but not identical - information on which words were uttered.
Revised: Both a recording of a speech waveform and a corresponding silent video of mouth motion provide rich - but not identical - information on which words were uttered.

Original: Despite the difficulty of interpreting lip movements compared with an audio waveform, the task of visual speech recognition (VSR; also known as lipreading) has important applications, ranging from recognising utterances in a noisy environment (Ma et al., 2021b;
Revised: Despite the difficulty of interpreting lip movements compared with an audio waveform, the task of visual speech recognition (VSR; also known as lipreading) has important applications, ranging from recognising utterances in a noisy environment (Ma et al., 2021b;

Original: Afouras et al., 2018a; Martinez et al., 2020; Makino et al., 2019) and aiding people suffering from aphonia (an inability to speak), to transcribing archival silent films and detecting DeepFake videos (Haliassos et al., 2021).
Revised: Afouras et al., 2018a; Martinez et al., 2020; Makino et al., 2019) and aiding people suffering from aphonia (an inability to speak), to transcribing archival silent films and detecting DeepFake videos (Haliassos et al., 2021).

Original: Auditory (also known as automatic) speech recognition (ASR) and VSR benefit greatly from the combination of high-capacity neural networks and large datasets.
Revised: Auditory (also known as automatic) speech recognition (ASR) and VSR benefit greatly from the combination of high-capacity neural networks and large datasets.

Original: Rapid advances of modern hardware are enabling the use of ever-growing, data-hungry networks, but the effort required for transcription hinders the scaling of labelled data along with the models.
Revised: Rapid advances of modern hardware are enabling the use of ever-growing, data-hungry networks, but the effort required for transcription hinders the scaling of labelled data along with the models.

Original: One way to leverage unlabelled videos for VSR is to use an external ASR model for pseudo-labelling (Afouras et al., 2020; Ma et al., 2022).
Revised: One way to leverage unlabelled videos for VSR is to use an external ASR model for pseudo-labelling (Afouras et al., 2020; Ma et al., 2022).

Original: However, this requires a large amount of labelled data to train a strong ASR model in the first place, and fully-supervised VSR training with long sequences often poses optimisation problems, requiring costly curriculum learning strategies (Chung et al., 2017; Ma et al., 2022) or pre-training the feature extractor with isolated words (Afouras et al., 2018a; Ma et al., 2021b).
Revised: However, this requires a large amount of labelled data to train a strong ASR model in the first place, and supervised VSR training with long sequences often poses optimisation problems, requiring costly curriculum learning strategies (Chung et al., 2017; Ma et al., 2022) or pre-training the feature extractor with isolated words (Afouras et al., 2018a; Ma et al., 2021b).

Original: A promising solution is to first learn, in a self-supervised way, general representations from large corpora of unlabelled data, and then fine-tune them on smaller labelled datasets (Mohamed et al., 2022).
Revised: A promising solution is to first learn, in a self-supervised way, general representations from large corpora of unlabelled data, and then fine-tune them on smaller labelled datasets (Mohamed et al., 2022).

Original: The temporally fine-grained correspondence between the (synchronised) visual and auditory modalities provides a natural source of self-supervision, and can produce highly semantic representations invariant to nuisance factors not shared between the two modalities.
Revised: The temporally fine-grained correspondence between the (synchronised) visual and auditory modalities provides a natural source of self-supervision, and can produce highly semantic representations invariant to nuisance factors not shared between the two modalities.

Original: However, approaches leveraging this correspondence either (1) only work for word-level samples rather than continuous speech (Chung & Zisserman, 2016; Chung et al., 2019; 2020); (2) use handcrafted features ( e.g., spectrograms or MFCCs) as their inputs or targets (Ma et al., 2021a; Shi et al., 2022), which contain inductive biases that may influence the learned representations; (3) use multi-stage pre-training procedures (Ma et al., 2021a; Shi et al., 2022; Pan et al., 2022); and/or (4) use separate pre-training strategies for VSR and ASR (Shi et al., 2022), complicating the process of obtaining representations suitable for both tasks.
Revised: However, approaches leveraging this correspondence either (1) only work for word-level samples rather than continuous speech (Chung & Zisserman, 2016; Chung et al., 2019; 2020); (2) use handcrafted features ( e.g., spectrograms or MFCCs) as their inputs or targets (Ma et al., 2021a; Shi et al., 2022), which con- tain inductive biases that may influence the learned representations; (3) use multi-stage pre-training procedures (Ma et al., 2021a; Shi et al., 2022; Pan et al., 2022); and/or (4) use separate pre-training strategies for VSR and ASR (Shi et al., 2022), complicating the process of obtaining representations suitable for both tasks.

Original: In this paper, we present a single-stage self-supervised approach that jointly learns visual and auditory speech representations from raw video and audio only.
Revised: In this paper, we present a single-stage self-supervised approach that jointly learns visual and auditory speech representations from raw video and audio only.

Original: We dub our approach RAVEn ( R aw
Revised: We dub our approach RAVEn ( R aw

Original: A udio- V isual Speech En coders).
Revised: A udio- V isual Speech En coders).

Original: It involves a pair of student-teacher networks for each modality, whereby the students encode temporally-masked inputs, and, through the use of lightweight
Revised: It involves a pair of student-teacher networks for each modality, whereby the students encode temporally-masked inputs, and, through the use of lightweight

Original: Transformer-based predictors, regress outputs of momentum-based teachers (Grill et al., 2020;
Revised: Transformer-based predictors, regress outputs of momentum-based teachers (Grill et al., 2020;

Original: Caron et al., 2021) that are presented with unmasked inputs.
Revised: Caron et al., 2021) that are presented with unmasked inputs.

Original: Further, given that audio contains more information relevant to speech than video, we propose a learning strategy that accounts for the expected difference in the quality of targets between the modalities.
Revised: Further, given that audio contains more information relevant to speech than video, we propose a learning strategy that accounts for the expected difference in the quality of targets between the modalities.

Original: Namely, while the audio student predicts outputs from both video and audio teachers (cross- and within-modal learning), the video student predicts only auditory targets (cross-modal learning).
Revised: Namely, while the audio student predicts outputs from both video and audio teachers (cross- and within-modal learning), the video student predicts only auditory targets (cross-modal learning).

Original: As we show, this setup leads to better downstream performance for both VSR and ASR as opposed to other strategies.
Revised: As we show, this setup leads to better downstream performance for both VSR and ASR as opposed to other strategies.

Original: We conduct experiments with models and datasets of different sizes.
Revised: We conduct experiments with models and datasets of different sizes.

Original: We find that, when fine-tuning our pre-trained models for VSR and ASR with only 30 hours of labelled data from LRS3 (Afouras et al., 2018b), RAVEn surpasses recent self-supervised methods by a large margin in most settings.
Revised: We find that, when fine-tuning our pre-trained models for VSR and ASR with only 30 hours of labelled data from LRS3 (Afouras et al., 2018b), RAVEn surpasses recent self-supervised methods by a large margin in most settings.

Original: Coupling pre-training with self-training reaches 24.4% WER for VSR on LRS3, even outperforming a method trained on 3000 × more labelled hours (Serdyuk et al., 2021).
Revised: Coupling pre-training with self-training reaches 24.4% WER for VSR on LRS3, even outperforming a method trained on 3000 × more transcribed hours (Serdyuk et al., 2021).

Original: At the same time, we are better than or on par with the recent AV-Hubert method (Shi et al., 2022) on ASR, without using a task-dependent pre-training strategy nor handcrafted features.
Revised: At the same time, we are better than or on par with the recent AV-HuBERT method (Shi et al., 2022) on ASR, without using a task-dependent pre-training strategy nor handcrafted features.

Original: Using the full 433-hour LRS dataset for fine-tuning pushes the results even further, achieving 23.4% / 1.4% WER for VSR / ASR, respectively.
Revised: Using the full 433-hour LRS dataset for fine-tuning pushes the results even further, achieving 23.4% / 1.4% WER for VSR / ASR, respectively.

Original: Similarly strong performance is observed on the LRS2 dataset (Appendix B).
Revised: Similarly strong performance is observed on the LRS2 dataset (Appendix B).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Masked prediction.
Revised: Masked prediction.

Original: The pre-training task of predicting missing content given masked inputs has proven successful in various domains, such as natural language processing (Devlin et al., 2019;
Revised: The pre-training task of predicting missing content given masked inputs has proven successful in various domains, such as natural language processing (Devlin et al., 2019;

Original: Radford et al., 2018; 2019; Brown et al., 2020), image recognition (He et al., 2021; Xie et al., 2022;
Revised: Radford et al., 2018; 2019; Brown et al., 2020), image recognition (He et al., 2021; Xie et al., 2022;

Original: Bao et al., 2021), and speech recognition (Baevski et al., 2020; Hsu et al., 2021; Shi et al., 2022).
Revised: Bao et al., 2021), and speech recognition (Baevski et al., 2020; Hsu et al., 2021; Shi et al., 2022).

Original: An important aspect of masked prediction is the nature of the targets.
Revised: An important aspect of masked prediction is the nature of the targets.

Original: Some works (He et al., 2021; Xie et al., 2022) use pixels as targets; others use pre-trained tokenisers (Bao et al., 2021) or handcrafted features ( e.g. , HoG for images/video or MFCC for audio), whose nature depends on the modality at hand (Wei et al., 2021; Hsu et al., 2021; Shi et al., 2022).
Revised: Some works (He et al., 2021; Xie et al., 2022) use pixels as targets; others use pre-trained tokenisers (Bao et al., 2021) or handcrafted features ( e.g. , HoG for images/video or MFCC for audio), whose nature depends on the modality at hand (Wei et al., 2021; Hsu et al., 2021; Shi et al., 2022).

Original: Our method, in contrast, generates targets from raw video and audio using randomly-initialised momentum encoders.
Revised: Our method, in contrast, generates targets from raw video and audio using randomly-initialised momentum encoders.

Original: Self-distillation for unsupervised representation learning.
Revised: Self-distillation for unsupervised representation learning.

Original: RAVEn is partly inspired by the success of self-distillation in self-supervised learning with visual data (Grill et al., 2020; Caron et al., 2021); such workstend to target invariance w.r.t.
Revised: RAVEn is partly inspired by the success of self-distillation in self-supervised learning with visual data (Grill et al., 2020; Caron et al., 2021); such works target invariance w.r.t.

Original:  augmentations.
Revised: image-specific augmentations.

Original: In contrast, our method learns temporally-dense representations and does not rely on  augmentationsto define the pretext task but rather on a combination of multi-modal signals and masked inputs.
Revised: In contrast, RAVEn does not rely on domain-specific augmentations but rather on a combination of cross-modal learning and masked prediction to drive representation learning for visual and auditory speech signals.

Original: The recent method, data2vec (Baevski et al., 2022), also combines masked prediction with a momentum encoder, but, aside from being uni-modal, it is different methodologically in multiple ways.
Revised: data2vec (Baevski et al., 2022) combines masked prediction with a momentum encoder, but, aside from being uni-modal, it is different methodologically in multiple ways.

Original: For example, it applies ad-hoc normalisation and averaging techniques to the targets to prevent representation collapse, while our targets are simply the outputs of the encoders, which are regressed via Transformer-based heads.
Revised: For example, it applies ad-hoc normalisation and averaging techniques to the targets to prevent representation collapse, while our targets are simply the outputs of the encoders, which are regressed via Transformer-based heads.

Original: Self-supervised audiovisual learning.
Revised: Self-supervised audiovisual learning.

Original: Audiovisual correspondence has been used to learn global representations for action recognition through the use of clustering (Alwassel et al., 2020; Asano et al., 2020), contrastive learning (Arandjelovic & Zisserman, 2017; 2018; Korbar et al., 2018;
Revised: Audiovisual correspondence has been used to learn global representations for action recognition through the use of clustering (Alwassel et al., 2020; Asano et al., 2020), contrastive learning (Arandjelovic & Zisserman, 2017; 2018; Korbar et al., 2018;

Original: Patrick et al., 2020; Morgado et al., 2021; Ma et al., 2021c), or representation matching (Recasens et al., 2021).
Revised: Patrick et al., 2020; Morgado et al., 2021; Ma et al., 2021c), or representation matching (Recasens et al., 2021).

Original: Cross-modal learning has also found uses in biometric matching (Nagrani et al., 2018a;b), emotion recognition (Shukla et al., 2021), and DeepFake detection (Haliassos et al., 2022).
Revised: Cross-modal learning has also found uses in biometric matching (Nagrani et al., 2018a;b), emotion recognition (Shukla et al., 2021), and DeepFake detection (Haliassos et al., 2022).

Original: We employ cross- and within-modal losses to learn temporally-varying speech representations.
Revised: We employ cross- and within-modal losses to learn temporally-varying speech representations.

Original: Self-supervised audiovisual learning for speech recognition.
Revised: Self-supervised audiovisual learning for speech recognition.

Original: Earlier audiovisual selfsupervised works (Chung & Zisserman, 2016; Chung et al., 2019; 2020) tended to focus on wordlevel lipreading.
Revised: Earlier audiovisual selfsupervised works (Chung & Zisserman, 2016; Chung et al., 2019; 2020) tended to focus on word-

Original: Recently, some attention has been paid to the more realistic task of continuous speech recognition (Ma et al., 2021a; Shi et al., 2022; Sheng et al., 2021; Pan et al., 2022; Ma et al.,

Original: [Figure]
Revised: [Figure]

Original: 2021c).
Revised: level lipreading.

Original:  Sheng et al. ; Ma et al. (2021c) use contrastive learning and apply their to VSR.
Revised: Recently, some attention has been paid to the more realistic task of continuous speech recognition (Ma et al., 2021a; Shi et al., 2022; Sheng et al., 2021; Pan et al., 2022; Ma et al., 2021c).

Original: Ma et al. (2021a) predict speech features using an external PASE+ encoder
Revised: Sheng use contrastive learning and apply their predict speech features using an external PASE

Original: Ravanelli et al., 2020) Pan al.,
Revised: (Ravanelli et al., transfer which were separately trained via losses.

Original: It
Revised: HuBERT ( et

Original:  We 
Revised: ) predicts iteratively-refined cluster as masked and impressive and performance employs multiple stages of alternating between offline clustering and cluster assignment prediction and relies on hand-crafted audio features (MFCCs) for cluster initialisation which is shown to be crucial to the performance (Shi et al., 2022). We demonstrate that it is possible to jointly learn effective representations for VSR and ASR in a single stage simply from raw video and audio.

Original:  M ETHOD 2. P RE - TRAINING
Revised:  M ETHOD 3. P RE - TRAINING

Original: Our architecture consists of a pair of student-teacher networks per modality.
Revised: Our architecture consists of a pair of student-teacher networks per modality (see Figure 1).

Original: The students intake masked inputs and predict targets formed byslowly-improving teachers receiving unmasked inputs.
Revised: The students intake masked inputs and predict targets formed by teachers receiving unmasked inputs.

Original: Figure 1 depicts our approach.

Original: Masking.
Revised: Masking.

Original: We employ masking to encourage the students to take context into account when solving the task.
Revised: We employ masking to encourage the students to take context into account when solving the task.

Original: Given a grayscale video x v ∈ R T × H × W with resolution ( H, W ) and T frames, and an audio sample x a ∈ R N of length N , we randomly sample with probability 0.2 each video frame to be the starting mask index, and if selected, then the consecutive three frames are zeroed out (ablation in Section 3.4).
Revised: Given a grayscale video x v ∈ R T × H × W with resolution ( H, W ) and T frames, and an audio sample x a ∈ R N of length N , we randomly sample with probability 0.2 each video frame to be the starting mask index, and if selected, then the consecutive three frames are zeroed out (ablation in Section 4.4).

Original: A similar mask is applied to the auditory input, except that it is enlarged by a factor of 16 000 / 25 = 640 (since the audio is sampled at 16,000 Hz and the video at 25 fps).
Revised: A similar mask is applied to the auditory input, except that it is enlarged by a factor of 16 000 / 25 = 640 (since the audio is sampled at 16,000 Hz and the video at 25 fps).

Original: Encoders.
Revised: Encoders.

Original: The masked video and audio, denoted as ¯ x v and ¯ x a respectively, are fed to their corresponding student encoders f ve and f ae , yielding features f ve (¯ x v ) ∈ R T × D and f ae (¯ x a ) ∈ R T × D , where D is the dimensionality of each feature.
Revised: The masked video and audio, denoted as ¯ x v and ¯ x a respectively, are fed to their corresponding student encoders f ve and f ae , yielding features f ve (¯ x v ) ∈ R T × D and f ae (¯ x a ) ∈ R T × D , where D is the dimensionality of each feature.

Original: Both f ve and f ae consist of a modality-specific, convolutional feature extractor followed by a temporal encoder, as in related VSR/ASR works (Ma et al., ; Baevski et al., 2020).
Revised: Both f ve and f ae consist of a modality-specific, convo- lutional feature extractor followed by a temporal encoder, as in related VSR/ASR works (Ma et al., 2022; Baevski et al., 2020).

Original: The video feature extractor is a 2D ResNet18 (He et al., 2016) with a 3D convolutional stem, outputting an embedding per frame.
Revised: The video feature extractor is a 2D ResNet18 (He et al., 2016) with a 3D convolutional stem, outputting an embedding per frame.

Original: On the audio side, we use a 1D ResNet which produces features at 25 fps, to match the video sampling rate (see Appendix A.4).
Revised: On the audio side, we use a 1D ResNet which produces features at 25 fps, to match the video sampling rate (see Appendix A.5).

Original: The temporal encoder for each modality is a Transformer (Vaswani et al., 2017)
Revised: The temporal encoder for each modality is a Transformer (Vaswani et al., 2017)

Original: (without a classification head) with hidden size D .
Revised: (without a classification head) with hidden size D .

Original: We use relative positional embeddings (Dai et al., 2019), and following Chen et al.
Revised: We use relative positional embeddings (Dai et al., 2019), and following Chen et al.

Original: (2021), we replace layernorm (Ba et al., 2016) with batchnorm (Ioffe & Szegedy, 2015) before each multilayer perceptron (MLP) block (see Appendix C.1 for a comparison).
Revised: (2021), we replace layernorm (Ba et al., 2016) with batchnorm (Ioffe & Szegedy, 2015) before each multilayer perceptron (MLP) block (see Appendix C.1 for a comparison).

Original: Predictors.
Revised: Predictors.

Original: The students contain Transformer predictor networks at the top, which predict targets given 1) the student encoder outputs corresponding to the unmasked portions of the inputs and 2) mask tokens associated with the masked portions.
Revised: The students contain Transformer predictor networks at the top, which predict targets given 1) the student encoder outputs corresponding to the unmasked portions of the inputs and 2) mask tokens associated with the masked portions.

Original: Note that mask tokens are applied to the predictors rather than the encoders, a choice inspired by He et al.
Revised: Note that mask tokens are applied to the predictors rather than the encoders, a choice inspired by He et al.

Original: This reduces the discrepancy between pre-training and fine-tuning:
Revised: This reduces the discrepancy between pre-training and fine-tuning:

Original: In both cases, the encoders do not see mask tokens.
Revised: In both cases, the encoders do not see mask tokens.

Original: A predictor that takes representations corresponding to modality m 1 ∈ { v, a } and predicts targets associated with modality m 2 ∈ { v, a } is denoted as f m 1 → m 2 p .
Revised: A predictor that takes representations corresponding to modality m 1 ∈ { v, a } and predicts targets associated with modality m 2 ∈ { v, a } is denoted as f m 1 → m 2 p .

Original: For ease, the application of mask tokens to the encoder outputs is absorbed in the notation.
Revised: For ease, the application of mask tokens to the encoder outputs is absorbed in the notation.

Original: Unlike other works which output global representations (one embedding per sample) and thus use
Revised: Unlike other works which output global representations (one embedding per sample) and thus use

Original: MLPs as predictors (Grill et al., 2020; Chen et al., 2021), we use Transformers to allow modelling temporal dynamics, which we found greatly improves results.
Revised: MLPs as predictors (Grill et al., 2020; Chen et al., 2021), we use Transformers to allow modelling temporal dynamics, which we found greatly improves results.

Original: The predictors can be lightweight:
Revised: The predictors can be lightweight:

Original: Indeed, two-block Transformers with hidden size 512 work optimally in our experiments.
Revised: Indeed, two-block Transformers with hidden size 512 work optimally in our experiments.

Original: Targets.
Revised: Targets.

Original: The targets are simply the outputs of momentum-based teachers g v
Revised: The targets are simply the outputs of momentum-based teachers g v

Original: and g a (Grill et al., 2020; Caron et al., 2021), which are given as input the unmasked video or audio, in order to force the students to predict the missing information.
Revised: and g a (Grill et al., 2020; Caron et al., 2021), which are given as input the unmasked video or audio, in order to force the students to predict the missing information.

Original: Each teacher is architecturally identical to its student encoder counterpart.
Revised: Each teacher is architecturally identical to its student encoder counterpart.

Original: Denoting the parameters of the student encoders and teachers as s m and t m , respectively, at each iteration the following update is performed:
Revised: Denoting the parameters of the student encoders and teachers as s m and t m , respectively, at each iteration the following update is performed:

Original: [Equation]
Revised: [Equation]

Original: where m ∈ { v, a } specifies the modality and µ is a momentum parameter following a cosine schedule from 0.999 to 1.
Revised: where m ∈ { v, a } specifies the modality and µ is a momentum parameter following a cosine schedule from 0.999 to 1.

Original: A high value of µ leads to slowly-varying teachers and stable targets.
Revised: A high value of µ leads to slowly-varying teachers and stable targets.

Original: The use of momentum-based teachers obviates the need for handcrafted targets or multi-stage training.
Revised: The use of momentum-based teachers obviates the need for handcrafted targets or multi-stage training.

Original: Prediction tasks.
Revised: Prediction tasks.

Original: The auditory modality contains more information relevant to speech than the visual one: Mouth motion is inherently more ambiguous than a speech waveform due to the presence of homophemes (Chung et al., 2017).
Revised: The auditory modality contains more information relevant to speech than the visual one: Mouth motion is inherently more ambiguous than a speech waveform due to the presence of homophemes (Chung et al., 2017).

Original: We propose a loss structure which reflects this asymmetry between the modalities.
Revised: We propose a loss structure which reflects this asymmetry between the modalities.

Original: The audio student predicts the targets from both the video and audio teacher, thus benefiting from the ability of cross-modal learning to induce semantic representations, while at the same time being encouraged to retain information from the auditory input that is absent from the visual one.
Revised: The audio student predicts the targets from both the video and audio teacher, thus benefiting from the ability of cross-modal learning to induce semantic representations, while at the same time being encouraged to retain information from the auditory input that is absent from the visual one.

Original: As a result, two predictors are associated with the audio student, one for each target type.
Revised: As a result, two predictors are associated with the audio student, one for each target type.

Original: On the other hand, the video student only predicts the auditory targets, which are inevitably of higher quality.
Revised: On the other hand, the video student only predicts the auditory targets, which are inevitably of higher quality.

Original: Losses.
Revised: Losses.

Original: The loss function is the negative cosine similarity (Grill et al., 2020), denoted as sim.
Revised: The loss function is the negative cosine similarity (Grill et al., 2020), denoted as sim.

Original: Due to the temporal alignment of the inputs, the cosine similarity is applied between pairs of corresponding features and then summed across the time dimension.
Revised: Due to the temporal alignment of the inputs, the cosine similarity is applied between pairs of corresponding features and then summed across the time dimension.

Original: For the within-modal task (audio-to-audio prediction), the loss is applied only on targets corresponding to masked portions of the input (Devlin et al., 2019).
Revised: For the within-modal task (audio-to-audio prediction), the loss is applied only on targets corresponding to masked portions of the input (Devlin et al., 2019).

Original: For the cross-modal tasks, the loss is applied on all targets, which we found to work better.
Revised: For the cross-modal tasks, the loss is applied on all targets, which we found to work better.

Original: Note that predicting the unmasked portions in cross-modal learning is a non-trivial task (unlike in within-modal learning) and can bolster representation learning.
Revised: Note that predicting the unmasked portions in cross-modal learning is a non-trivial task (unlike in within-modal learning) and can bolster representation learning.

Original: Denoting the set of mask token indices for audio as M a , the audio-to-audio prediction loss and cross-modal losses can be respectively expressed as
Revised: Denoting the set of mask token indices for audio as M a , the audio-to-audio prediction loss and cross-modal losses can be respectively expressed as

Original: [Equation]
Revised: [Equation]

Original: where m 1 , m 2 ∈ { v, a } , m 1 ̸
Revised: where m 1 , m 2 ∈ { v, a } , m 1 ̸

Original: = m 2 , and sg denotes the “stop-gradient” operation, which indicates that no gradient is passed back to the teacher networks.
Revised: = m 2 , and sg denotes the “stop-gradient” operation, which indicates that no gradient is passed back to the teacher networks.

Original: Objectives.
Revised: Objectives.

Original: At each iteration, the objectives for the video and audio students are, respectively,
Revised: At each iteration, the objectives for the video and audio students are, respectively,

Original: [Equation]
Revised: [Equation]

Original: The teachers are updated via Equation 1.
Revised: The teachers are updated via Equation 1.

Original: 2. F INE - TUNING
Revised: 3. F INE - TUNING

Original: For fine-tuning, we keep the student encoders from the pre-training stage and discard the rest.
Revised: For fine-tuning, we keep the student encoders from the pre-training stage and discard the rest.

Original: We append a single linear layer and a Transformer decoder to perform joint CTC / attention decoding (Watanabe et al., 2017), as in Ma et al.
Revised: We append a single linear layer and a Transformer decoder to perform joint CTC / attention decoding (Watanabe et al., 2017), as in Ma et al.

Original: (2021b).
Revised: (2021b).

Original: Following Ma et al.
Revised: Following Ma et al.

Original: (2021b), we fix the beam size to 40 and the CTC weight(during both training and decoding) to 0.1.
Revised: (2021b), we set the CTC weight to 0.1.

Original: We use SentencePiece (Kudo & Richardson, 2018) subword units with a vocabulary size of 1,000 as our targets.
Revised: We use SentencePiece (Kudo & Richardson, 2018) subword units with a vocabulary size of 1,000 as our targets.

Original: Self-training.
Revised: Self-training.

Original: Combining pre-training with self-training tends to improve results over using either strategy in isolation (Xu et al., 2021; Shi et al., 2022).
Revised: Combining pre-training with self-training tends to improve results over using either strategy in isolation (Xu et al., 2021; Shi et al., 2022).

Original: To that end, we first fine-tune our pre-trained audio encoder on the labelled data, and then use the model for pseudo-labelling the unlabelled data.
Revised: To that end, we first fine-tune our pre-trained audio encoder on the labelled data, and then use the model for pseudo-labelling the unlabelled data.

Original: The pre-trained video and audio models are then fine-tuned using both the labels and pseudo-labels.
Revised: The pre-trained video and audio models are then fine-tuned using both the labels and pseudo-labels.

Original:  E XPERIMENTS 3. S ETUP
Revised:  E XPERIMENTS 4. S ETUP

Original: Datasets.
Revised: Datasets.

Original: For pre-training, we conduct experiments with LRS3 (Afouras et al., 2018b) (without the labels) as well as a combination of LRS3 and an English-only version of VoxCeleb2 (Chung et al., 2018) curated by Shi et al.
Revised: For pre-training, we conduct experiments with LRS3 (Afouras et al., 2018b) (without the labels) as well as a combination of LRS3 and an English-only version of VoxCeleb2 (Chung et al., 2018) curated by Shi et al.

Original: (2022), which we refer to as LRS3+Vox2-en.
Revised: (2022), which we refer to as LRS3+Vox2-en.

Original: The former features 433 hours of footage and the latter 1,759 hours.
Revised: The former features 433 hours of footage and the latter 1,759 hours.

Original: For fine-tuning, we use the full LRS3 with the labels as our high-resource labelled data setting, as well as a 30-hour subset (the “trainval” partition) as our low-resource setting.
Revised: For fine-tuning, we use the full LRS3 with the labels as our high-resource labelled data setting, as well as a 30-hour subset (the “trainval” partition) as our low-resource setting.

Original: We present results for the LRS2 dataset Chung et al. (2017) (223 hours of footage) in Appendix B.
Revised: We present results for the LRS2 dataset (Chung et al., 2017) (223 hours of footage) in Appendix B.

Original: Transformer encoder variants.
Revised: Transformer encoder.

Original: We show results for two configurations of the Transformer encoders, Base and Large, with 12 and 24 blocks respectively.
Revised: We show results for two configurations of the Transformer encoders, Base and Large, with 12 and 24 blocks respectively.

Original: The hidden size/MLP size/attention heads are 512/2048/8 for Base and 1024/4096/16 for Large, amounting to 41 million (M) and  M parameters, respectively.
Revised: The hidden size/MLP size/attention heads are 512/2048/8 for Base and 1024/4096/16 for Large, amounting to 41 million (M) and 328 M parameters, respectively.

Original: We note that our Base model is around half the size of the one used by Shi et al.
Revised: We note that our Base model is around half the size of the one used by Shi et al.

Original: and the Large models are similar in size.
Revised: and the Large models are similar in size.

Original: Training details are provided in Appendix A.
Revised: Training details are provided in Appendix A.

Original: 3. L OW -R ESOURCE L ABELLED D ATA S ETTING
Revised: 4. L OW -R ESOURCE L ABELLED D ATA S ETTING

Original: We pre-train our models on LRS3 and/or LRS3+Vox2-en and then fine-tune them on the 30-hour
Revised: We pre-train our models on LRS3 and/or LRS3+Vox2-en and then fine-tune them on the 30-hour

Original: LRS3 subset to evaluate performance when labels are scarce.
Revised: LRS3 subset to evaluate performance when labels are scarce.

Original: We reports results in Table 1.
Revised: We reports results in Table 1.

Original: Compared with training from scratch, RAVEn pre-training leads to dramatic performance improvements in all configurations.
Revised: Compared with training from scratch, RAVEn pre-training leads to dramatic performance improvements in all configurations.

Original: Notably, increasing the model size hurts VSR performance when training from scratch, but improves it when using pre-training.
Revised: Notably, increasing the model size hurts VSR performance when training from scratch, but improves it when using pre-training.

Original: Our Base variant outperforms all related methods on VSR.
Revised: Our Base variant outperforms all related methods on VSR.

Original: It surpasses the Base AV-Hubert model by 4.8% and 5.9% WER when using LRS3 and LRS3+Vox2-en, respectively, despite having roughly half the number of parameters.
Revised: It surpasses the Base AV-HuBERT model by 4.8% and 5.9% WER when using LRS3 and LRS3+Vox2-en, respectively, despite having roughly half the number of parameters.

Original: The Large model provides significant boosts over the Base model (33.1% vs 40.2% WER) when using LRS3+Vox2-en for pre-training, keeping the number of labelled data points fixed.
Revised: The Large model provides significant boosts over the Base model (33.1% vs 40.2% WER) when using LRS3+Vox2-en for pre-training, keeping the number of labelled data points fixed.

Original: Self-training further improves WER by 6.9%, indicating its complementarity with
Revised: Self-training further improves WER by 6.9%, indicating its complementarity with

Original: RAVEn pre-training.
Revised: RAVEn pre-training.

Original: Finally, using a language model (see Appendix A.3 for details) leads to a WER of 24.4%, better than afully-supervised method (Serdyuk et al., 2021) which was trained on 90, hours of non-publiclabelled data.
Revised: Finally, using a language model (see Appendix A.4 for details) leads to a WER of 24.4%, better than a method (Serdyuk et al., 2021) which was trained on 90,000 hours of non-public data.

Original: On ASR, RAVEn significantly outperforms the audio-only Hubert (Hsu et al., 2021) model, and in allcases is better than or on par with AV-Hubert. Our best ASR model without self-training achieves .6% WER vs AV-Hubert’s 2.9% WER, despite AV-Hubert using a different pre-training strategy for
Revised: On ASR, RAVEn significantly outperforms the audio-only Hubert (Hsu et al., 2021) model, and in all

Original: [Table]
Revised: [Table]

Original: Transf.

Original: ✗ -

Original: [Table]

Original: [Table]

Original: Transf.

Original: ✗

Original: [Table]

Original: [Table]

Original: [Table]

Original: Transf.

Original: ✗

Original: [Table]

Original: [Table] Large models, more pre-training data [Table] VSR than ASR.
Revised: 2.6% WER vs AV-HuBERT’s 2.9% WER, despite AV-HuBERT using a different pre-training strategy for VSR than ASR.

Original: For example, using the same pre-training hyperparameters increases  WER for ASR from 3.8% to 4.6% with the Base model (Shi et al., 2022).
Revised: For example, using the same pre-training hyperparameters increases AVHuBERT’s WER for ASR from 3.8% to 4.6% with the Base model (Shi et al., 2022).

Original: In contrast, the video and audio encoders we use for fine-tuning are the result of a single pre-training phase, where they were jointly learned.
Revised: In contrast, the video and audio encoders we use for fine-tuning are the result of a single pre-training phase, where they were jointly learned.

Original: All in all, our results suggest that transcribed data, which are costly to obtain, can be largely substituted with raw unlabelled audiovisual data.[Equation]
Revised: All in all, our results suggest that transcribed data, which are costly to obtain, can be largely substituted with raw unlabelled audiovisual data.

Original: [Table]
Revised: [Table]

Original: Table 2 reports results when fine-tuning on the full 433 hours of LRS3.
Revised: Table 2 reports results when fine-tuning on the full 433 hours of LRS3.

Original: Despite increasing the labelled data, training the model from scratch still leads to poor performance, as in the low-resource setting.
Revised: Despite increasing the labelled data, training the model from scratch still leads to poor performance, as in the low-resource setting.

Original: This is likely related to the long utterances in the LRS3 dataset, which may cause optimisation difficulties (Ma et al., 2022).
Revised: This is likely related to the long utterances in the LRS3 dataset, which may cause optimisation difficulties (Ma et al., 2022).

Original: A potential remedy is to employ curriculum learning, as proposed by Ma et al.
Revised: A potential remedy is to employ curriculum learning, as proposed by Ma et al.

Original: (2022), by training the network in multiple stages with increasingly longer sequences.
Revised: (2022), by training the network in multiple stages with increasingly longer sequences.

Original: We observe that this strategy indeed reduces WER from 87.3% to 39.8% for the Base model.Even [Table] ✗
Revised: We observe that this strategy indeed reduces WER from 87.3% to 39.8% for the Base model.

Original: [Table]

Original: Base models, less pre-training data

Original: Shi et al.

Original: Transf.

Original: ✗

Original: [Table]

Original: [Table]

Original: RAVEn

Original: Transf.

Original: ✗

Original: [Table]

Original: [Table]

Original: [Table]

Original: Base models, more pre-training data

Original: Ma et al.

Original: (2021a)

Original: Transf.

Original: [Table]

Original: Hsu et al.

Original: [Table]

Original: ✗

Original: [Table]

Original: [Table]

Original: Shi et al.

Original: Transf.

Original: ✗

Original: [Table]

Original: [Table]

Original: [Table]

Original: RAVEn

Original: Transf.

Original: ✗

Original: [Table]

Original: [Table]

Original: Hsu et al. Transf. ✗ 1,
Revised: Even so, pre-training with the same data used for fine-tuning leads to a better WER of 39.1%, without

Original: [Table]

Original: Shi et al.

Original: Transf.

Original: ✗

Original: [Table]

Original: Shi

Original: [Table]
Revised: [Table]

Original: al. (2022) w/ self-training

Original: Transf.

Original: ✗

Original: [Table]

Original: RAVEn

Original: [Table]

Original: [Table]

Original: RAVEn w/ self-training

Original: Transf.

Original: ✗ 1,  24.

Original: RAVEn w/ self-training Transf. ✓ 1,  23. so, pre-training with the same data used for fine-tuning leads to a better WER of 39.1%, without requiring a curriculum learning procedure.
Revised: requiring a curriculum learning procedure.

Original: This suggests that self-supervised pre-training can aid optimisability.
Revised: This suggests that self-supervised pre-training can aid optimisability.

Original: The impact of pre-training is even more pronounced with the Large model.
Revised: The impact of pre-training is even more pronounced with the Large model.

Original: RAVEn outperforms AV-Hubert under all configurations on VSR.
Revised: RAVEn outperforms AV-HuBERT under all configurations on VSR.

Original: Our best result is 23.4%, achieved using self-training and a language model.
Revised: Our best result is 23.4%, achieved using self-training and a language model.

Original: We note that unlike methods that use external ASR models (Afouras et al., 2020; Ma et al., 2022) for pseudo-labelling, we do not require extra data for the  phase.
Revised: We note that unlike methods that use external ASR models (Afouras et al., 2020; Ma et al., 2022) for pseudo-labelling, we do not require extra data for the self-training phase.

Original: We are on par with the state-of-the-art for ASR in the high-resource setting, achieving a WER of 1.4% with the Large model.
Revised: We are on par with the state-of-the-art for ASR in the high-resource setting, achieving a WER of 1.4% with the Large model.

Original: This is despite using raw audio as input (rather than spectrograms which
Revised: This is despite using raw audio as input (rather than spectrograms which

Original: Shi et al.
Revised: Shi et al.

Original: (2022) use).
Revised: (2022) use).

Original: We notice that additionally including self-training and a language model does not reduce the WER, suggesting that the ASR performance may have saturated in our environment.
Revised: We notice that additionally including self-training and a language model does not reduce the WER, suggesting that the ASR performance may have saturated in our environment.

Original: 3. P RE - TRAINING A BLATIONS
Revised: 4. P RE - TRAINING A BLATIONS

Original: Ablations are performed with our Base model in the low-resource setting using the validation set from Shi et al.
Revised: Ablations are performed with our Base model in the low-resource setting using the validation set from Shi et al.

Original: (2022) (as there is no official development set).
Revised: (2022) (as there is no official development set).

Original: For more ablations, see Appendix C.
Revised: For more ablations, see Appendix C.

Original: Prediction tasks.
Revised: Prediction tasks.

Original: We study in Table 3 the impact of cross- and within-modal prediction on the learned representations.
Revised: We study in Table 3 the impact of cross- and within-modal prediction on the learned representations.

Original: Figure 2 illustrates the various prediction tasks we consider.
Revised: Figure 2 illustrates the various prediction tasks we consider.

Original: We observe the following: The within-modal variant performs decently for ASR but poorly for VSR, which can be explained by the assumption that audio contains more  information than video
Revised: We observe the following. The within-modal variant performs decently for ASR but poorly for VSR, which can be explained by the assumption that audio contains more speech information than video. The crossmodal variant performs better than within-modal for both modalities.

Revised: Since lip movements and the corresponding waveform are correlated in terms of lexical content, the representations resulting from cross-modal prediction are expected to capture rich semantic information.

Revised: At the same time, they are likely to be, to a large extent, invariant to factors unshared between the two modalities,

Original: V
Revised: V

Original: A … …
Revised: A … …

Original: Within-modal
Revised: Within-modal

Original: V
Revised: V

Original: A … …
Revised: A … …

Original: Cross-modal
Revised: Cross-modal

Original: V
Revised: V

Original: A … …
Revised: A … …

Original: Cross-
Revised: Cross-

Original: + within-modal
Revised: + within-modal

Original: V
Revised: V

Original: A … …
Revised: A … …

Original: Cross-
Revised: Cross-

Original: + audio within-modal
Revised: + audio within-modal

Original: V video student
Revised: V video student

Original: A audio student video target audio target 
Revised: A audio student video target audio target V A … …

Revised: Cross-

Revised: + video within-modal

Original: Setting
Revised: Setting

Original: Prediction tasks
Revised: Prediction tasks

Original: WER (%)
Revised: WER (%)

Original: V → V
Revised: V → V

Original: A → A
Revised: A → A

Original: V → A A → V
Revised: V → A A → V

Original: VSR
Revised: VSR

Original: ASR
Revised: ASR

Original: Within-modal ✓ ✓ ✗ ✗ 92.
Revised: Within-modal ✓ ✓ ✗ ✗ 92.

Original: Cross-modal ✗ ✗ ✓ ✓ 40.
Revised: Cross-modal ✗ ✗ ✓ ✓ 40.

Original: Within-
Revised: Cross-

Original: + cross-modal ✓ ✓ ✓ ✓ 49.
Revised: + within-modal ✓ ✓ ✓ ✓ 49.

Original: Cross-
Revised: Cross-

Original: + audio within-modal✗ ✓  ✓ ✓ 32.
Revised: + video within-modal ✓ ✗ ✓ ✓ 55.

Original: w.r.t.
Revised: Cross-

Original: lexical content. The cross-modal variant performs better than the within-modal one for both modalities; cross-modal prediction forces the encoders to capture the information shared between the modalities and results in more semantic representations.
Revised: + audio within-modal ✗ ✓ ✓ ✓ 32. such as visual or auditory noise, benefiting generalisation.

Original: Combining cross- with within-modal learning hurts VSR relative to only cross-modal prediction.
Revised: Combining cross- with within-modal learning hurts VSR relative to only cross-modal prediction.

Original: However, cross-modal with withinmodal learning only for audio achieves the best results.
Revised: However, cross-modal with withinmodal learning only for audio achieves the best results.

Original: We hypothesise that predicting the auditory targets forces the auditory stream to keep relevant information absent from the visual stream.
Revised: We hypothesise that predicting the auditory targets forces the auditory stream to keep relevant information absent from the visual stream.

Original: We note that removing the video-to-video prediction (from row 3 to row 4 in Table 3) also improves the audio student, as the quality of the visual targets improves.
Revised: We note that removing the video-to-video prediction (from row 3 to row 5 in Table 3) also improves the audio student, as the quality of the visual targets improves.

Revised: Finally, cross-modal with within-modal learning only for video does not perform well, validating that the asymmetry only works one way.

Original: Masking sampling.
Revised: Masking sampling.

Original: Table 4a studies the effect of different masking strengths by varying the mask length and probability that a given index is chosen as the mask start.
Revised: Table 4a studies the effect of different masking strengths by varying the mask length and probability that a given index is chosen as the mask start.

Original: We observe that a probability of 0.2 and a length of three video frames (corresponding to 1,920 audio samples) works well.
Revised: We observe that a probability of 0.2 and a length of three video frames (corresponding to 1,920 audio samples) works well.

Original: Less masking allows the student to focus less on context and degrades both VSR and ASR.
Revised: Less masking allows the student to focus less on context and degrades both VSR and ASR.

Original: Interestingly, although more masking hurts VSR, it helps ASR, suggesting that an asymmetric masking strategy w.r [Table]
Revised: Interestingly, although more masking hurts VSR, it helps ASR, suggesting that an asymmetric masking strategy w.r.t.

Original: t. the two modalities may further improve results.
Revised: the two modalities may further improve results.

Original: We leave this exploration to future work.
Revised: We leave this exploration to future work.

Original: Our masking is less aggressive than what was found to be optimal in related self-supervised image and action recognition literature (where 75% or even 90% of the input is masked)
Revised: Our masking is less aggressive than what was found to be optimal in related self-supervised image and action recognition literature (where 75% or even 90% of the input is masked)

Original: (He et al., 2021;
Revised: (He et al., 2021;

Original: Tong et al., 2022).
Revised: Tong et al., 2022).

Original: We hypothesise that mouth movements are fast-varying and do not contain much temporal redundancy [Table] and thus such strong masking makes our pretext task overly difficult.
Revised: We hypothesise that mouth movements are fast-varying and do not contain much temporal redundancy, and thus such strong masking makes our pretext task overly difficult.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Mask token position.
Revised: Mask token position.

Original: We investigate in Table 4b the optimal position of the mask tokens.
Revised: We investigate in Table 4b the optimal position of the mask tokens.

Original: It is clear that better performance is achieved when applying them in the predictors rather than the encoders.
Revised: It is clear that better performance is achieved when applying them in the predictors rather than the encoders.

Original: Forgoing the use of mask tokens in the encoders leads to no input discrepancy between pre-training and fine-tuning, since no mask tokens are used during fine-tuning.
Revised: Forgoing the use of mask tokens in the encoders leads to no input discrepancy between pre-training and fine-tuning, since no mask tokens are used during fine-tuning.

Original: Mask loss.
Revised: Mask loss.

Original: It is common to apply the loss only on masked inputs for within-modal losses (Hsu et al., 2021;
Revised: It is common to apply the loss only on masked inputs for within-modal losses (Hsu et al., 2021;

Original: He et al., 2021), since predicting targets corresponding to unmasked inputs may be trivial.
Revised: He et al., 2021), since predicting targets corresponding to unmasked inputs may be trivial.

Original: However, this is not the case for cross-modal prediction, where the targets are not related to the inputs in an obvious way.
Revised: However, this is not the case for cross-modal prediction, where the targets are not related to the inputs in an obvious way.

Original: Indeed, Table 4c shows that applying the loss for both masked and unmasked inputs outperforms applying it only for masked inputs.
Revised: Indeed, Table 4c shows that applying the loss for both masked and unmasked inputs outperforms applying it only for masked inputs.

Original: Predictor design.
Revised: Predictor design.

Original: Table 5a shows the effect of different predictor types.
Revised: Table 5a shows the effect of different predictor types.

Original: We note that using no predictors at all leads to representation collapse, where all outputs are constant (Grill et al., 2020).
Revised: We note that using no predictors at all leads to representation collapse, where all outputs are constant (Grill et al., 2020).

Original: We compare linear layers and 2-layer MLPs (applied independently to each encoder output) with
Revised: We compare linear layers and 2-layer MLPs (applied independently to each encoder output) with

Original: Transformers of varying capacity.
Revised: Transformers of varying capacity.

Original: Following Grill et al.
Revised: Following Grill et al.

Original: (2020), the MLPs have hidden dimension of 4096 with batchnorm followed by ReLU (Nair & Hinton, 2010) after the hidden layer.
Revised: (2020), the MLPs have hidden dimension of 4096 with batchnorm followed by ReLU (Nair & Hinton, 2010) after the hidden layer.

Original: We find that a Transformer predictor works significantly better, even when the number of parameters in the (1-block) Transformer and the 2-layer MLP is similar.
Revised: We find that a Transformer predictor works significantly better, even when the number of parameters in the (1-block) Transformer and the 2-layer MLP is similar.

Original: Interestingly, the ability for the predictors to model temporal dependencies seems to be crucial to our representation learning phase.
Revised: Interestingly, the ability for the predictors to model temporal dependencies seems to be crucial to our representation learning phase.

Original: A two-block Transformer works optimally (Table 5a).
Revised: A two-block Transformer works optimally (Table 5a).

Original: A shallower Transformer likely results in representations too specialised to the pretext task.
Revised: A shallower Transformer likely results in representations too specialised to the pretext task.

Original: A deeper one might place a lot of the burden on the predictors, leaving the encoder representations too general.
Revised: A deeper one might place a lot of the burden on the predictors, leaving the encoder representations too general.

Original: Similar conclusions can be drawn regarding the Transformer width (see Table 5b).
Revised: Similar conclusions can be drawn regarding the Transformer width (see Table 5b).

Original: Overall, the lightweight predictor design means that using three predictors (one on the video student side and two on the audio) has relatively little effect on the total computational cost.
Revised: Overall, the lightweight predictor design means that using three predictors (one on the video student side and two on the audio) has relatively little effect on the total computational cost.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented RAVEn, a single-stage method that jointly learns visual and auditory speech representations entirely from raw data.
Revised: We presented RAVEn, a single-stage method that jointly learns visual and auditory speech representations entirely from raw data.

Original: It employs momentum encoders to generate targets, which, given masked inputs, are predicted by Transformer encoders and predictors.
Revised: It employs momentum encoders to generate targets, which, given masked inputs, are predicted by Transformer encoders and predictors.

Original: Especially salient to the quality of the representations are appropriate masking, lightweight Transformer predictors, and an asymmetric loss structure w.r.t.
Revised: Especially salient to the quality of the representations are appropriate masking, lightweight Transformer predictors, and an asymmetric loss structure w.r.t.

Original: the visual and auditory modalities.
Revised: the visual and auditory modalities.

Original: RAVEn achieves strong performance under many settings without requiring multiple pre-training stages, handcrafted audio targets, or separate pre-training strategies for VSR and ASR.
Revised: RAVEn achieves strong performance under many settings without requiring multiple pre-training stages, handcrafted audio targets, or separate pre-training strategies for VSR and ASR.

Original: As future work, it would be interesting to examine the effect of sharing weights between the visual and auditory encoders as a way of reducing the memory requirements for pre-training.
Revised: As future work, it would be interesting to examine the effect of sharing weights between the visual and auditory encoders as a way of reducing the memory requirements for pre-training.

Original: We would also like to apply RAVEn to otherrelated tasks such as speaker identification or emotion recognition.
Revised: We would also like to apply RAVEn to other tasks related to speech.

Original: We hope our study inspires future research extending beyond speech recognition.
Revised: We hope our study inspires future research extending beyond speech recognition.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Despite numerous positive applications our method can also be misused.
Revised: Despite numerous positive applications our method can also be misused.

Original: For example, lipreading technologies can be employed for surveillance, compromising the public’s privacy and trust.
Revised: For example, lipreading technologies can be employed for surveillance, compromising the public’s privacy and trust.

Original: This problem is likely to be exacerbated as the quality of CCTV cameras improves over time.
Revised: This problem is likely to be exacerbated as the quality of CCTV cameras improves over time.

Original: Appropriate government regulations will need to be put in place to limit such concerns.
Revised: Appropriate government regulations will need to be put in place to limit such concerns.

Original: Another issue relates to the potential biases embedded in the datasets used.
Revised: Another issue relates to the potential biases embedded in the datasets used.

Original: Such biases may have to do with gender, age, or ethnic backgrounds.
Revised: Such biases may have to do with gender, age, or ethnic backgrounds.

Original: A specific group being under-represented in the data would likely result in reduced model performance on samples belonging to said group.
Revised: A specific group being under-represented in the data would likely result in reduced model performance on samples belonging to said group.

Original: Making sure that the models are trained on balanced data, or using other bias-reduction techniques, can address such issues.
Revised: Making sure that the models are trained on balanced data, or using other bias-reduction techniques, can address such issues.

Original: R EPRODUCIBILITY
Revised: R EPRODUCIBILITY

Original: To ensure reproducibility, we provide as many implementation details as possible in the main paper as well as tables showing the hyperparameter values in the appendix.
Revised: To ensure reproducibility, we provide as many implementation details as possible in the main paper as well as tables showing the hyperparameter values in the appendix.

Original: Moreover, we plan on making the code and pre-trained models publicly available
Revised: Moreover, we plan on making the code and pre-trained models publicly available

Original: A  I MPLEMENTATION D ETAILS A. D ATASET PRE - PROCESSING
Revised: A D ATASET / I MPLEMENTATION D ETAILS A. D ATASETS

Revised: LRS3.

Revised: LRS3 (Afouras et al., 2018b) is the largest publicly available transcribed audio-visual dataset for continuous speech recognition.

Revised: It consists of around 430 hours of spoken sentences from TED talks, with a vocabulary of more than 50,000 words uttered by thousands of speakers.

Revised: The test set contains around 1 hour of utterances with speakers separate from those in the training set.

Revised: LRS2.

Revised: The 223-hour LRS2 dataset (Chung et al., 2017), collected from BBC programmes, is the second-largest publicly available transcribed audio-visual dataset for continuous speech recognition.

Revised: As LRS3, it contains an unconstrained vocabulary and thousands of diverse speakers.

Revised: VoxCeleb2.

Revised: VoxCeleb2 (Chung et al., 2018) is a non-transcribed dataset containing YouTubedownloaded videos.

Revised: It consists of around 2,500 hours of utterances with over 6,000 speakers.

Revised: Since

Revised: VoxCeleb2 is multi-lingual, as mentioned in the main text, we use an English-only version curated by Shi et al.

Revised: (2022), amounting to 1,759 hours.

Revised: A. D ATASET PRE - PROCESSING

Original: We follow common practices in the literature for dataset pre-processing (Ma et al., 2022;
Revised: We follow common practices in the literature for dataset pre-processing (Ma et al., 2022;

Original: Shi et al., 2022; Martinez et al., 2020).
Revised: Shi et al., 2022; Martinez et al., 2020).

Original: We use RetinaFace (Deng et al., 2020) to detect the faces and FAN (Bulat & Tzimiropoulos, 2017) to compute 68 facial landmarks for each video frame.
Revised: We use RetinaFace (Deng et al., 2020) to detect the faces and FAN (Bulat & Tzimiropoulos, 2017) to compute 68 facial landmarks for each video frame.

Original: We average the landmarks using a moving window of 12 frames to reduce motion jitter, and for each frame we perform affine warping to the mean face using the resulting smoothed landmarks.
Revised: We average the landmarks using a moving window of 12 frames to reduce motion jitter, and for each frame we perform affine warping to the mean face using the resulting smoothed landmarks.

Original: We crop a 96 × 96 region centred around the mouth and finally transform to grayscale.
Revised: We crop a 96 × 96 region centred around the mouth and finally transform to grayscale.

Original: Raw audio is used without any pre-processing nor normalisation.
Revised: Raw audio is used without any pre-processing nor normalisation.

Original: Utterances longer than 24 seconds are split into smaller constituents.
Revised: Utterances longer than 24 seconds are split into smaller constituents.

Original: A. P RE - TRAINING
Revised: A. P RE - TRAINING

Original: Table 6 provides the default setting for pre-training.
Revised: Table 6 provides the default setting for pre-training.

Original: We use the AdamW (Loshchilov & Hutter, 2019) optimiser with linear learning rate warmup (Goyal et al., 2017) and a cosine decay schedule (Loshchilov & Hutter, 2017).
Revised: We use the AdamW (Loshchilov & Hutter, 2019) optimiser with linear learning rate warmup (Goyal et al., 2017) and a cosine decay schedule (Loshchilov & Hutter, 2017).

Original: During training, we apply random spatial cropping of size (88 × 88) followed by horizontal flipping with probability 0.5.
Revised: During training, we apply random spatial cropping of size (88 × 88) followed by horizontal flipping with probability 0.5.

Original: These augmentations are applied in a time-consistent manner across the video clips to maintain temporal coherence.
Revised: These augmentations are applied in a time-consistent manner across the video clips to maintain temporal coherence.

Original: We do not use any augmentations for the raw audio.
Revised: We do not use any augmentations for the raw audio.

Original: We also use stochastic depth (Huang et al., 2016) for regularisation, as well as LayerScale (Touvron et al., 2021) with coefficient 0.1.
Revised: We also use stochastic depth (Huang et al., 2016) for regularisation, as well as LayerScale (Touvron et al., 2021) with coefficient 0.1.

Original: We train the Base model with
Revised: We train the Base model with

Original: A100 GPUs, and the Large model with 128.
Revised: A100 GPUs, and the Large model with 128.

Revised: It takes around 15 minutes / 1 hour per epoch to pre-

Revised: [Table]

Original: Our batching process is as follows:
Revised: Our batching process is as follows:

Original: The samples are sorted based on their length to minimise zeropadding, and then samples with the same length are randomly shuffled at the beginning of each training epoch.
Revised: The samples are sorted based on their length to minimise zeropadding [Table] and then samples with the same length are randomly shuffled at the beginning of each training epoch.

Original: Each batch contains a maximum of 96 / 36 seconds of footage for Base / Large.
Revised: Each batch contains a maximum of 96 / 36 seconds of footage for Base / Large.

Original: A. F INE - TUNING 
Revised: A. F INE - TUNING / DECODING

Original: The protocol for fine-tuning is similar to that for pre-training with some exceptions (see Table 7).
Revised: The protocol for fine-tuning is similar to that for pre-training with some exceptions (see Table 7).

Original: We use a higher learning rate for the decoder than for the pre-trained encoder, since the decoder is randomly-initialised.
Revised: We use a higher learning rate for the decoder than for the pre-trained encoder, since the decoder is randomly-initialised.

Original: We also employ layer-wise learning rate decay (Clark et al., 2020), which we found to reduce overfitting.
Revised: We also employ layer-wise learning rate decay (Clark et al., 2020), which we found to reduce overfitting.

Original: In addition to the augmentations from the pre-training stage, we apply time masking (Ma et al., 2022) for both video and audio clips.
Revised: In addition to the augmentations from the pre-training stage, we apply time masking (Ma et al., 2022) for both video and audio clips.

Original: Specifically, for each second of the sample, we use zero-masking with a duration that is uniformly sampled between 0 and 0.4 seconds.
Revised: Specifically, for each second of the sample, we use zero-masking with a duration that is uniformly sampled between 0 and 0.4 seconds.

Revised: Fine-tuning Base (for both VSR and ASR) takes around 30 seconds per epoch in the low-resource setting and 7 minutes per epoch in the high-resource setting.

Revised: Fine-tuning Large takes approximately 1 minute and 20 minutes per epoch in the low- and high-resource setting, respectively.

Original: CTC/attention decoding.
Revised: CTC/attention decoding.

Original: We use joint CTC/attention decoding to map the input sequence x of length T to a target sequence y =
Revised: We use joint CTC/attention decoding to map the input sequence x of length T to a target sequence y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y L ) of size L .
Revised: , y L ) of size L .

Original: The CTC loss  is given by
Revised: The CTC loss during fine-tuning is given by

Original: [Equation]
Revised: [Equation]

Original: where A x,y is the set of valid alignments, A is one such alignment, and a t is the token at time-step t .
Revised: where A x,y is the set of valid alignments, A is one such alignment, and a t is the token at time-step t .

Original: The attention loss, computed using the Transformer decoder outputs, can be expressed as
Revised: The attention loss, computed using the Transformer decoder outputs, can be expressed as

Original: [Equation]
Revised: [Equation]

Original: The final loss  is given by 
Revised: The final loss during fine-tuning is given by L = α L ctc + (1 − α ) L att , where α = 0 . We use the ESPnet framework (Watanabe et al., 2018) for decoding. We set the beam size to 40.

Revised: The final score used to choose the most likely sequence is given by S = λ S ctc

Revised: +(1 − λ ) S att + β S LM , where

Original: [Equation]
Revised: [Table]

Original: where L LM is an optional loss associated with a language model and α = 0 . When using a language model, β is chosen from { 0 . 1 , 0 .2 , 0 . 3 , 0 . 4 } using the validation set.
Revised: and S denote the from the and attention branches, respectively and = language model incorporated shallow fusion (Watanabe et al When using a language model β is chosen from { 0 . 1 , 0 .

Revised: Inference on one A100 GPU (without batching) takes around 3 seconds to decode 10 seconds of footage for Base and 5 seconds for Large.

Original: Language model.
Revised: Language model.

Revised: We use a 16-block Transformer-based language model, as proposed by Irie et al.

Revised: The hidden size/MLP size/attention heads are 512/2048/8.

Original: The language model is trained on the combination of the following datasets: Librispeech (Panayotov et al., 2015), LRS2/3, TED-LIUM 3 (Hernandez et al., 2018), VoxForge and Common Voice (Ardila et al., 2020).
Revised: The language model is trained on the combination of the following datasets: Librispeech (Panayotov et al., 2015), LRS2/3, TED-LIUM 3 (Hernandez et al., 2018), VoxForge and Common Voice (Ardila et al., 2020).

Original: The total number of characters is 166 million.
Revised: The total number of characters is 166 million.

Original: A. F EATURE E XTRACTORS
Revised: A. F EATURE E XTRACTORS

Original: The details of the visual and auditory convolutional feature extractors are provided in Tables 8a and 8b, respectively.
Revised: The details of the visual and auditory convolutional feature extractors are provided in Tables 8a and 8b, respectively.

Original: B LRS2 E XPERIMENTS
Revised: B LRS2 E XPERIMENTS

Original: We report results on the test set of the LRS2 dataset (Chung et al., 2017) in Table 9.
Revised: We report results on the test set of the LRS2 dataset (Chung et al., 2017) in Table 9.

Original: After pre-training on the LRS3 or the LRS3+Vox2-en datasets, we fine-tune on the “pre-training” and “training” sets of LRS2.
Revised: After pre-training on the LRS3 or the LRS3+Vox2-en datasets, we fine-tune on the “pre-training” and “training” sets of LRS2.

Original: We observe similar trends as in the LRS3 experiments (Table 2), namely that performance is benefited from large models, large unlabelled datasets, and self-training.
Revised: We observe similar trends as in the LRS3 experiments (Table 2), namely that performance is benefited from large models, large unlabelled datasets, and self-training.

Original: We significantly outperform all other methods, including one (Pan et al., 2022) pre-trained on 60,000 hours of audio data.
Revised: We significantly outperform all other methods, including one (Pan et al., 2022) pre-trained on 60,000 hours of audio data.

Original: C M ORE A BLATIONS C. M ORE PRE - TRAINING ABLATIONS
Revised: C M ORE A BLATIONS C. M ORE PRE - TRAINING ABLATIONS

Original: Momentum parameter.
Revised: Momentum parameter.

Original: Table 10a shows the effect of varying the momentum parameter for updating the teacher networks.
Revised: Table 10a shows the effect of varying the momentum parameter for updating the teacher networks.

Original: We show results at three coarse levels: 0 (teacher is a copy of the 
Revised: We show results at three coarse levels: 0 (teacher is a copy of the student at each iteration), 0.999 (with a cosine schedule to 1), and 1 (teacher does not get updated during training). We see that using a momentum value of 0 leads to representation collapse. At the other extreme, a value of 1 does not allow the teacher targets to improve during training, leading to poor representations. A slowly-evolving momentum encoder is most effective.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Method

Original: [Table]
Revised: [Table]

Original: Chung et al.
Revised: hours).

Original: [Table] [Table] Petridis et al.
Revised: Lab hours denotes the number of labelled hours, and LM denotes whether or not a language model was used during decoding. * Includes non-publicly available data.

Original: [Table]
Revised: [Table]

Original: ✓

Original: [Table][Table] Ren et al. Transf.
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: , a value of 1 does not allow the teacher targets to improve during training, leading to poor representations.

Original: A slowly-evolving momentum encoder is most effective.

Original: Pre-MLP normalisation.
Revised: Pre-MLP normalisation.

Original: As mentioned in the main text, we use batchnorm before each MLP module rather than layernorm, a choice inspired by Chen et al.
Revised: As mentioned in the main text, we use batchnorm before each MLP module rather than layernorm, a choice inspired by Chen et al.

Original: Table 10b shows that our method still works with layernorm, but lags behind batchnorm.
Revised: Table 10b shows that our method still works with layernorm, but lags behind batchnorm.

Original: This is an interesting observa- [Table] [Table] [Table]
Revised: This is an interesting observation worthy of future exploration.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] tion worthy of future exploration. A preliminary hypothesis is that batchnorm may improve the conditioning of the networks at initialisation, leading to better targets at the beginning of training (Richemond et al., 2020).
Revised: A preliminary hypothesis is that batchnorm may improve the conditioning of the networks at initialisation, leading to better targets at the beginning of training (Richemond et al., 2020).

Original: C. FINE - TUNING ABLATIONS
Revised: C. F

Revised: [Table] ABLATIONS

Original: Learning rates.
Revised: Learning rates.

Original: Table 11a studies the effect of using different learning rates for the encoder and decoder.
Revised: Table 11a studies the effect of using different learning rates for the encoder and decoder.

Original: We find that it is beneficial for VSR to use a higher learning rate for the decoder than the encoder, likely because the encoder is pre-trained, whereas the decoder is randomly-initialised and thus requires a larger learning rate to search for a more effective local optimum.
Revised: We find that it is beneficial for VSR to use a higher learning rate for the decoder than the encoder, likely because the encoder is pre-trained, whereas the decoder is randomly-initialised and thus requires a larger learning rate to search for a more effective local optimum.

Original: Learning rate decay.
Revised: Learning rate decay.

Original: Table 11b shows the influence of decaying the encoder’s learning rate as the depth of the model decreases.
Revised: Table 11b shows the influence of decaying the encoder’s learning rate as the depth of the model decreases.

Original: The learning rate at block b , r b , is given by r b = r B d B − b , where B is the index of the last block and d is the learning rate decay (Clark et al., 2020).
Revised: The learning rate at block b , r b , is given by r b = r B d B − b , where B is the index of the last block and d is the learning rate decay (Clark et al., 2020).

Original: A decay less than 1 works well for both VSR and ASR, suggesting that during fine-tuning it is useful to employ larger learning rates for the deeper layers, which are more task-specific.[Table]
Revised: A decay less than 1 works well for both VSR and ASR, suggesting that during fine-tuning it is useful to employ larger learning rates for the deeper layers, which are more task-specific.

Original: Tokenisation.
Revised: Tokenisation.

Original: In Table 11c, we compare the use of characters with SentencePiece (Kudo &
Revised: In Table 11c, we compare the use of characters with SentencePiece (Kudo &

Original: Richardson, 2018) subword units (vocabulary size of 1,000) as our target tokens.
Revised: Richardson, 2018) subword units (vocabulary size of 1,000) as our target tokens.

Original: We find that using subword units leads to superior results than character units.
Revised: We find that using subword units leads to superior results than character units.

Original: This may be explained by the language priors embedded in subword units, which facilitate speech recognition.
Revised: This may be explained by the language priors embedded in subword units, which facilitate speech recognition.

Original: C. A NALYSIS OF TRANSCRIPTION ERRORS
Revised: C. A UDIO - VISUAL FINE - TUNING

Revised: It is possible to use the learned auditory and visual representations for audio-visual speech recognition.

Revised: To that end, following Ma et al.

Revised: (2021b), we concatenate the outputs of the two encoders, feed the resulting embeddings to a 2-layer MLP module with hidden size 4096 and batchnorm, and

Revised: [Table]

Revised: [Table]

Revised: then fine-tune for speech recognition, as described in Section 3.2.

Revised: We initialise the video and audio encoders with weights obtained from uni-modal fine-tuning and randomly initialise the rest.

Revised: We fine-tune for 30 epochs with the hyperparameters used for audio-only fine-tuning, shown in Table 7.

Revised: Table 12 shows results in the low- and high-resource settings for the Large model with LRS3+Vox2en pre-training.

Revised: Audio-visual training gives marginal improvements (if any) compared to audio-only when the audio is clean (i.e., not noisy), consistent with findings by Ma et al.

Revised: (2021b).

Revised: Investigating the impact of audio-visual training and testing on audio corrupted with various noise types, is outside the scope of this work, but is interesting to take up in future work.

Revised: D A NALYSIS OF TRANSCRIPTION ERRORS

Original: We provide examples of transcription errors in Table 12.
Revised: We provide examples of transcription errors in Table 13.

Original: We consider the Base VSR and ASR models fine-tuned in the low-resource setting (47.0% and 4.7% WER, respectively) and the Large
Revised: We consider the Base VSR and ASR models fine-tuned in the low-resource setting (47.0% and 4.7% WER, respectively) and the Large

Original: VSR and ASR models fine-tuned in the high-resource setting with self-training and a language model (23.4% and 1.4% WER, respectively).
Revised: VSR and ASR models fine-tuned in the high-resource setting with self-training and a language model (23.4% and 1.4% WER, respectively).

Original: Although the worst model sometimes makes surprising errors ( e.g. , “They won the game”
Revised: Although the worst model sometimes makes surprising errors ( e.g. , “They won the game”

Original: → “Because I wasn’t happy”), most often the errors are related to words that are phonetically similar ( e.g. , “were taken” → “we’re taking”, “embedded” → “in better”, “won” → “want”).
Revised: → “Because I wasn’t happy”), most often the errors are related to words that are phonetically similar ( e.g. , “were taken” → “we’re taking”, “embedded” → “in better”, “won” → “want”).

Original: As expected, the quality of the transcriptions is higher for ASR than
Revised: As expected, the quality of the transcriptions is higher for ASR than

Original: VSR, and it improves as we increase the model size and number of unlabelled data points. 
Revised: VSR, and it improves as we increase the model size and number of unlabelled data points. [Table]

Revised: E RAVE N FOR V IDEO - TO -S PEECH S YNTHESIS

Revised: We evaluate here whether RAVEn pre-training can benefit tasks other than speech recognition by considering video-to-speech synthesis, which aims to output the speech waveform given the corresponding silent video of lip movements.

Revised: We follow the protocol of SVTS (Mira et al., 2022), the current state-of-the-art method, to perform video-to-speech on the LRS3 test set after training on

Revised: LRS3+Vox2-en.

Revised: We study the effect of initialising the video encoder with the weights from our

Revised: Large pre-trained model.

Revised: We use the same hyperparameters as Mira et al.

Revised: (2022), except that we use a learning rate of 2e-4 and train only for 30 epochs (rather than 150).

Revised: We train the “from scratch” baseline with a learning rate of 7e-4 for 50 epochs, as training longer resulted in overfitting.

Revised: Table 14 reports performance based on PESQ (Rix et al., 2001), which measures the perceptual quality of the generated samples, as well as STOI and ESTOI (Taal et al., 2011), which measure their intelligiblity.

Revised: The results show that using RAVEn pre-training results in significant improvements in performance, compared with training from scratch, across all metrics.

Revised: In fact, the performance boosts due to RAVEn pre-training (0.04 / 0.03 / 0.06 for PESQ / STOI / ESTOI) are larger than those observed by Mira et al.

Revised: (2022) (0.01, 0.02, 0.04) when increasing the training set size by around a factor of 4 (from LRS3 to LRS3+Vox2-en).