Original: u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality
Revised: u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality

Original: Abstract
Revised: Abstract

Original: While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality.
Revised: While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality.

Original: In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective.
Revised: In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective.

Original: By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models.
Revised: By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models.

Original: Moreover, our model fine-tuned only on audio can perform well with audio-visual and visual speech input, achieving zero-shot modality generalization for multiple speech processing tasks.
Revised: Moreover, our model fine-tuned only on audio can perform well with audio-visual and visual speech input, achieving zero-shot modality generalization for multiple speech processing tasks.

Original: In particular, our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input.
Revised: In particular, our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input.

Original:  Introduction
Revised:  Introduction

Original: Speech processing has been moving from developing domain-specific models toward building a general model for all domains.
Revised: Speech processing has been moving from developing domain-specific models toward building a general model for all domains.

Original: For example, early studies presented models dedicated for robust speech recognition [Kingsbury et al., 1998] and distant speech recognition
Revised: For example, early studies presented models dedicated for robust speech recognition [Kingsbury et al., 1998] and distant speech recognition

Original: [Feng et al., 2014, Kim et al., 2017] by taking into account acoustic properties for each domain.
Revised: [Feng et al., 2014, Kim et al., 2017] by taking into account acoustic properties for each domain.

Original: In contrast, recent work has built general domain models by pooling data from different domains and even different languages, including labeled ones for supervised learning [Chan et al., 2021, Likhomanenko et al., 2020, Pratap et al., 2020] and unlabeled ones for self-supervised learning [Kawakami et al., 2020, Hsu et al., 2021b, Conneau et al., 2020].
Revised: In contrast, recent work has built general domain models by pooling data from different domains and even different languages, including labeled ones for supervised learning [Chan et al., 2021, Likhomanenko et al., 2020, Pratap et al., 2020] and unlabeled ones for self-supervised learning [Kawakami et al., 2020, Hsu et al., 2021b, Conneau et al., 2020].

Original: These studies show that in addition to removing the need of deploying and maintaining multiple models, a single general domain model can in fact achieve performance on par or superior to domain-specific models, especially for domains with less data and improve generalization performance to unseen domains.
Revised: These studies show that in addition to removing the need of deploying and maintaining multiple models, a single general domain model can in fact achieve performance on par or superior to domain-specific models, especially for domains with less data and improve generalization performance to unseen domains.

Original: In contrast, there are few attempts and even fewer successful ones on developing a versatile model that can process many speech modalities [Makino et al., 2019, Sari et al., 2021].
Revised: In contrast, there are few attempts and even fewer successful ones on developing a versatile model that can process many speech modalities [Makino et al., 2019, Sari et al., 2021].

Original: In general, researchers develop models with different input speech modalities, such as visual speech [Chung and Zisserman, 2016], audio-visual speech [Xu et al., 2020], multi-channel speech [Haeb-Umbach et al., 2021], 3D talking faces [Richard et al., 2021], Electromyographic (EMG) muscle signals [Diener and Schultz, 2018], ultrasonic oral cavity scan [Hueber et al., 2010], or ultrasonic microphone [Livescu et al., 2009] in isolation, despite the fact that many of them are used to solve the same task.
Revised: In general, researchers develop models with different input speech modalities, such as visual speech [Chung and Zisserman, 2016], audio-visual speech [Xu et al., 2020], multi-channel speech [Haeb-Umbach et al., 2021], 3D talking faces [Richard et al., 2021], Electromyographic (EMG) muscle signals [Diener and Schultz, 2018], ultrasonic oral cavity scan [Hueber et al., 2010], or ultrasonic microphone [Livescu et al., 2009] in isolation, despite the fact that many of them are used to solve the same task.

Original: Moreover, while additional modalities, such as visual speech, can often boost the performance and robustness on top of audio speech, models taking such input are less studied and deployed in real world application.
Revised: Moreover, while additional modalities, such as visual speech, can often boost the performance and robustness on top of audio speech, models taking such input are less studied and deployed in real world application.

Original: This is mainly due to the lack of both labeled and unlabeled data for these alternative modalities, hindering the development of high-performing self-supervised or supervised modal-specific models that aretrained exclusively on one type of input.
Revised: This is mainly due to the lack of both labeled and unlabeled data for these alternative modalities, hindering the development of high-performing self-supervised or supervised modal-specific models that are

Original: We argue that to tackle the data scarcity issue, it essential to utilize not only the speech from the target modality, but also speech in other modalities to build a

Original: U-HuBERT [13,13,5,5,5] [12,5,5,3] [4,4,4,9,5,5]
Revised: U-HuBERT [13,13,5,5,5] [12,5,5,3] [4,4,4,9,5,5]

Original: Joint clustering
Revised: Joint clustering

Original: Modality-agnosticfeatures
Revised: Modality-agnosticfeatures

Original: Pseudo labels Multimodal / unimodal input
Revised: Pseudo labels Multimodal / unimodal input

Original: Pseudo labeling
Revised: Pseudo labeling

Original: U-HuBERT U-HuBERT
Revised: U-HuBERT U-HuBERT

Original: AND/OR
Revised: AND/OR

Original: AND/OR general speech model for many modalities.
Revised: AND/OR trained exclusively on one type of input.

Revised: We argue that to tackle the data scarcity issue, it essential to utilize not only the speech from the target modality, but also speech in other modalities to build a general speech model for many modalities.

Original: By doing so, more data can be used to learn components shared across different modalities, which is expected to bring bigger gains to low-resource modalities.
Revised: By doing so, more data can be used to learn components shared across different modalities, which is expected to bring bigger gains to low-resource modalities.

Original: Audio-visual HuBERT (AV-HuBERT)
Revised: Audio-visual HuBERT (AV-HuBERT)

Original: [Shi et al., 2022a] is a recent work progressing toward creating a versatile speech model: by pre-training on audio-visual speech, it demonstrates strong performance for not only audio-visual downstream tasks, but also audio-only and video-only tasks.
Revised: [Shi et al., 2022a] is a recent work progressing toward creating a versatile speech model: by pre-training on audio-visual speech, it demonstrates strong performance for not only audio-visual downstream tasks, but also audio-only and video-only tasks.

Original: However, it only explores pre-training exclusively on audio-visual speech, and fine-tuning and testing on matching modalities.
Revised: However, it only explores pre-training exclusively on audio-visual speech, and fine-tuning and testing on matching modalities.

Original: In this paper, we generalize AV-HuBERT and present unified hidden unit BERT (u-HuBERT), which leverages unlabeled speech data of many different modalities for pre-training, including both unimodal and multimodal speech.
Revised: In this paper, we generalize AV-HuBERT and present unified hidden unit BERT (u-HuBERT), which leverages unlabeled speech data of many different modalities for pre-training, including both unimodal and multimodal speech.

Original: The model is trained with a unified masked cluster prediction objective for both unimodal and multimodal speech data.
Revised: The model is trained with a unified masked cluster prediction objective for both unimodal and multimodal speech data.

Original: The pre-trained u-HuBERT can be fine-tuned with either multimodal or unimodal data, and we demonstrate that in both cases the resulting model can be used for many speech modalities.
Revised: The pre-trained u-HuBERT can be fine-tuned with either multimodal or unimodal data, and we demonstrate that in both cases the resulting model can be used for many speech modalities.

Original: Following AV-HuBERT, we apply modality dropout [Neverova et al., 2014] to multimodal data to simulate different combinations of input modalities.
Revised: Following AV-HuBERT, we apply modality dropout [Neverova et al., 2014] to multimodal data to simulate different combinations of input modalities.

Original: We present extensive analysis in this paper to show that modality dropout is crucial for learning modality-agnostic frame-level representation, which is the key to successful mixed modality pre-training and achieving good performance when fine-tuning and testing modalities mismatch.
Revised: We present extensive analysis in this paper to show that modality dropout is crucial for learning modality-agnostic frame-level representation, which is the key to successful mixed modality pre-training and achieving good performance when fine-tuning and testing modalities mismatch.

Original: We verify the effectiveness on two popular speech processing tasks: speech recognition and speech translation.
Revised: We verify the effectiveness on two popular speech processing tasks: speech recognition and speech translation.

Original: To the best of our knowledge, this is the first single model for many speech modalities that yields performance on par with the state-of-the-art modality-specific models and is capable of zero-shot modality generalization.
Revised: To the best of our knowledge, this is the first single model for many speech modalities that yields performance on par with the state-of-the-art modality-specific models and is capable of zero-shot modality generalization.

Original:  Related Work
Revised:  Related Work

Original: Comparison to [Shi et al., 2022a] u-HuBERT generalizes AV-HuBERT to utilize both multimodal and unimodal speech for pre-training by mapping various input to a modality-agnostic feature space and creating a shared target space for masked prediction.
Revised: Comparison to [Shi et al., 2022a] u-HuBERT generalizes AV-HuBERT to utilize both multimodal and unimodal speech for pre-training by mapping various input to a modality-agnostic feature space and creating a shared target space for masked prediction.

Original: In contrast, because AV-HuBERT is pre-trained exclusively on audio-visual speech, it always generates targets using features of the same modality (audio in the first iteration and audio-visual for the subsequent iterations).
Revised: In contrast, because AV-HuBERT is pre-trained exclusively on audio-visual speech, it always generates targets using features of the same modality (audio in the first iteration and audio-visual for the subsequent iterations).

Original: Therefore, learning modality-agnostic features to produce shared target space is not essential for AV-HuBERT.
Revised: Therefore, learning modality-agnostic features to produce shared target space is not essential for AV-HuBERT.

Original: Moreover, this paper presents a novel evaluation scenario: zero-shot transfer to unlabeled modalities, which is of great practical importance because while labeled data may not always be available in the target modality, it will most likely be available in some other modalities.
Revised: Moreover, this paper presents a novel evaluation scenario: zero-shot transfer to unlabeled modalities, which is of great practical importance because while labeled data may not always be available in the target modality, it will most likely be available in some other modalities.

Original: We demonstrate in this paper that modality dropout is key to achieving good zero-shot transfer performance.
Revised: We demonstrate in this paper that modality dropout is key to achieving good zero-shot transfer performance.

Original: Joint unimodal and multimodal pre-training Recent cross-modal contrastive approaches such as CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have shown impressive results yet they can only be trained with multimodal data and require a large-scale dataset, which can be hard to obtain for many speech modalities (e.g., paired EMG-audio speech).
Revised: Joint unimodal and multimodal pre-training Recent cross-modal contrastive approaches such as CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have shown impressive results yet they can only be trained with multimodal data and require a large-scale dataset, which can be hard to obtain for many speech modalities (e.g., paired EMG-audio speech).

Original: FLAVA [Singh et al., 2021] and SLAM [Bapna et al., 2021] are two frameworks that utilize both unimodal and multimodal data for image-text and speech-text pre-training, respectively.
Revised: FLAVA [Singh et al., 2021] and SLAM [Bapna et al., 2021] are two frameworks that utilize both unimodal and multimodal data for image-text and speech-text pre-training, respectively.

Original: There are three key differences between u-HuBERT and these two models.
Revised: There are three key differences between u-HuBERT and these two models.

Original: First, for multimodal speech, different modalities such as audio and visual speech are temporally aligned.
Revised: First, for multimodal speech, different modalities such as audio and visual speech are temporally aligned.

Original: Instead of concatenating the feature sequences from unimodal encoders, we fuse the features frame-by-frame before passing it to the shared encoder.
Revised: Instead of concatenating the feature sequences from unimodal encoders, we fuse the features frame-by-frame before passing it to the shared encoder.

Original: Second, while all three models utilize a masked token prediction objective, u-HuBERT considers a shared target token space for all unimodal and multimodal data, obtained by clustering modality-agnostic features extracted from the u-HuBERT model itself.
Revised: Second, while all three models utilize a masked token prediction objective, u-HuBERT considers a shared target token space for all unimodal and multimodal data, obtained by clustering modality-agnostic features extracted from the u-HuBERT model itself.

Original: In contrast, SLAM uses w2v-BERT [Chung et al., 2021] tokens for speech and SentencePiece [Schuster and Nakajima, 2012] for text.
Revised: In contrast, SLAM uses w2v-BERT [Chung et al., 2021] tokens for speech and SentencePiece [Schuster and Nakajima, 2012] for text.

Original: FLAVA on other hand uses a separately trained dVAE tokenizer for image and the same as SLAM for text, and it predicts tokens for unimodal and multimodal data at the output of different modules, differing from uHuBERT.
Revised: FLAVA on other hand uses a separately trained dVAE tokenizer for image and the same as SLAM for text, and it predicts tokens for unimodal and multimodal data at the output of different modules, differing from uHuBERT.

Original: In addition, both FLAVA and SLAM pre-train with a number of auxiliary objectives, while u-HuBERT only uses one, which simplifies hyperparameter selection.
Revised: In addition, both FLAVA and SLAM pre-train with a number of auxiliary objectives, while u-HuBERT only uses one, which simplifies hyperparameter selection.

Original: Third, different modalities of speech are often used to solve the same task.
Revised: Third, different modalities of speech are often used to solve the same task.

Original: Our evaluation focuses on building a single model that can solve the same task with different modalities despite not having labeled data for each modality, which is not considered by SLAM or FLAVA.
Revised: Our evaluation focuses on building a single model that can solve the same task with different modalities despite not having labeled data for each modality, which is not considered by SLAM or FLAVA.

Original: Single model for many input modalities Makino et al.
Revised: Single model for many input modalities Makino et al.

Original: [2019] explored building a single model for supervised audio-visual/audio/visual speech recognition.
Revised: [2019] explored building a single model for supervised audio-visual/audio/visual speech recognition.

Original: The studies show significant degradation comparing unified models with modality-specific ones.
Revised: The studies show significant degradation comparing unified models with modality-specific ones.

Original: Furthermore, these models are single-task supervised and hence not capable of zero-shot modality transfer when trained on unimodal data.
Revised: Furthermore, these models are single-task supervised and hence not capable of zero-shot modality transfer when trained on unimodal data.

Original: Omnivore [Girdhar et al., 2022] is a single supervised model for many vision modalities, including image (RGB), 3D image (RGBD), and video (RGB sequence), which also considers a model architecture with modality-specific encoders (RGB and depth) followed by a shared Swin Transformer backend [Liu et al., 2021].
Revised: Omnivore [Girdhar et al., 2022] is a single supervised model for many vision modalities, including image (RGB), 3D image (RGBD), and video (RGB sequence), which also considers a model architecture with modality-specific encoders (RGB and depth) followed by a shared Swin Transformer backend [Liu et al., 2021].

Original: Both Omnivore and u-HuBERT demonstrates cross-modal generalization, but Omnivore achieves it through supervised learning with different tasks, and u-HuBERT achieves it with unified self-supervised learning.
Revised: Both Omnivore and u-HuBERT demonstrates cross-modal generalization, but Omnivore achieves it through supervised learning with different tasks, and u-HuBERT achieves it with unified self-supervised learning.

Original: Finally, if we considered different languages as different modalities, our work is also related to mBERT [Devlin et al., 2019], mBART [Liu et al., 2020],
Revised: Finally, if we considered different languages as different modalities, our work is also related to mBERT [Devlin et al., 2019], mBART [Liu et al., 2020],

Original: TLM [Yao et al., 2021], and XLM [Lample and Conneau, 2019], fine-tuning which enables generalizing natural language understanding [Pires et al., 2019] and machine translation [Johnson et al., 2017] to languages without labeled data.
Revised: TLM [Yao et al., 2021], and XLM [Lample and Conneau, 2019], fine-tuning which enables generalizing natural language understanding [Pires et al., 2019] and machine translation [Johnson et al., 2017] to languages without labeled data.

Original: Pre-training on multimodal speech is analogous to pre-training on bitext, while the former requires less manual effort to obtain by collecting more measurements when speech is produced.
Revised: Pre-training on multimodal speech is analogous to pre-training on bitext, while the former requires less manual effort to obtain by collecting more measurements when speech is produced.

Original: In addition, time-synchronicity between modalities provides additional self-supervision, enabling u-HuBERT to more easily learn frame-level modality-agnostic features.
Revised: In addition, time-synchronicity between modalities provides additional self-supervision, enabling u-HuBERT to more easily learn frame-level modality-agnostic features.

Original:  Method 3.1 Background: audio-visual HuBERT
Revised:  Method 3.1 Background: audio-visual HuBERT

Original: We build our framework upon AV-HuBERT [Shi et al., 2022a], a self-supervised representation learning model designed to pre-train a single type of multimodal speech data: audio-visual speech.
Revised: We build our framework upon AV-HuBERT [Shi et al., 2022a], a self-supervised representation learning model designed to pre-train a single type of multimodal speech data: audio-visual speech.

Original: AV-HuBERT first encodes temporally aligned audio speech X a ∈ R T a × C a and visual speech
Revised: AV-HuBERT first encodes temporally aligned audio speech X a ∈ R T a × C a and visual speech

Original: X v ∈ R T v × C v × H × W into feature sequences h a ∈ R T × D h and h v ∈ R T ×
Revised: X v ∈ R T v × C v × H × W into feature sequences h a ∈ R T × D h and h v ∈ R T ×

Original: D h of the same lengths, which are then fused frame-by-frame FUSE ( h a , h v ) ∈ R T × D c and processed by a Transformer backend [Vaswani et al., 2017] to produce contextualized audio-visual representations c av ∈ R T × D c .
Revised: D h of the same lengths, which are then fused frame-by-frame FUSE ( h a , h v ) ∈ R T × D c and processed by a Transformer backend [Vaswani et al., 2017] to produce contextualized audio-visual representations c av ∈ R T × D c .

Original: This model is trained by iterating two steps.
Revised: This model is trained by iterating two steps.

Original: First, a K-means clustering model is trained on some audio-visual features to produce frame-level targets Y av ∈ [1 , · · · , K ] T , where K is the number of clusters.
Revised: First, a K-means clustering model is trained on some audio-visual features to produce frame-level targets Y av ∈ [1 , · · · , K ] T , where K is the number of clusters.

Original: Second, the AV-HuBERT model is trained with a masked cluster prediction task, where spans from the audio-visual input are randomly masked and the model learns to predict the clusters of the masked frames given the context.
Revised: Second, the AV-HuBERT model is trained with a masked cluster prediction task, where spans from the audio-visual input are randomly masked and the model learns to predict the clusters of the masked frames given the context.

Original: The model is forced to learn the underlying linguistic structure of audio-visual speech to infer the content in the unseen regions based on the context.
Revised: The model is forced to learn the underlying linguistic structure of audio-visual speech to infer the content in the unseen regions based on the context.

Original: The intermediate feature of a trained model, which is better than previous iteration, is used to produce targets for the subsequent iteration.
Revised: The intermediate feature of a trained model, which is better than previous iteration, is used to produce targets for the subsequent iteration.

Original: 3.2 Self-supervised pre-training with many modalities
Revised: 3.2 Self-supervised pre-training with many modalities

Original: We extend the framework to utilize not only multimodal data, but also unimodal data that can be more easily obtained in large quantities.
Revised: We extend the framework to utilize not only multimodal data, but also unimodal data that can be more easily obtained in large quantities.

Original: The main idea is to apply modality dropout to multimodalspeech along with a shared Transformer [Vaswani et al., 2017] to learn modality-agnostic features that can be quantized by a shared codebook to produce pseudo labels for masked prediction.
Revised: The main idea is to apply modality dropout to multimodal

Original: [Figure]
Revised: [Figure]

Original: Shared Transformer [13,13,5,5,5] [13,13,5,5,5] [13,13,5,5,5]
Revised: Transformer [13,13,5,5,5] [13,13,5,5,5] [13,13,5,5,5]

Original: Audioencoder
Revised: Audioencoder

Revised: [Figure]

Original: Dummy fuse
Revised: fuse

Original: Shared Transformer [12,5,5,3]
Revised: Shared Transformer [12,5,5,3]

Original: [Figure]
Revised: [Figure]

Revised: speech along with a shared Transformer [Vaswani et al., 2017] to learn modality-agnostic features that can be quantized by a shared codebook to produce pseudo labels for masked prediction.

Original: Given unimodal and multimodal speech data from many modalities, we assume that there is an anchor modality that is paired with all other modalities.
Revised: Given unimodal and multimodal speech data from many modalities, we assume that there is an anchor modality that is paired with all other modalities.

Original: For simplicity, assume the anchor modality is audio, and we have unlabeled audio, visual, and audio-visual speech.
Revised: For simplicity, assume the anchor modality is audio, and we have unlabeled audio, visual, and audio-visual speech.

Original: In the first iteration, because a modalityagnostic feature extractor is not available yet, we cluster the anchor modality features to produce targets for unimodal and multimodal speech that contain the anchor modality.
Revised: In the first iteration, because a modalityagnostic feature extractor is not available yet, we cluster the anchor modality features to produce targets for unimodal and multimodal speech that contain the anchor modality.

Original: Concretely speaking, this produces pseudo paired data for audio speech X a and audio-visual speech ( X a , X v ) with audiobased cluster assignment Y a as the target.
Revised: Concretely speaking, this produces pseudo paired data for audio speech X a and audio-visual speech ( X a , X v ) with audiobased cluster assignment Y a as the target.

Original: Like AV-HuBERT, each modality has its own feature extractor:
Revised: Like AV-HuBERT, each modality has its own feature extractor:

Original: f a for audio and f v for video, and feature sequences are fused frame-by-frame before being passed onto the Transformer g to produce contextualized features c av = g ( FUSE ( f a ( X a ) , f v ( X v ))) .
Revised: f a for audio and f v for video, and feature sequences are fused frame-by-frame before being passed onto the Transformer g to produce contextualized features c av = g ( FUSE ( f a ( X a ) , f v ( X v ))) .

Original: When a modality is absent, its feature sequence is simply replaced by zero vectors, for example, c a = g ( FUSE ( f a ( X a ) , 0 )) .
Revised: When a modality is absent, its feature sequence is simply replaced by zero vectors, for example, c a = g ( FUSE ( f a ( X a ) , 0 )) .

Original: Following HuBERT [Hsu et al., 2021a], this model is trained with a masked prediction objective.
Revised: Following HuBERT [Hsu et al., 2021a], this model is trained with a masked prediction objective.

Original: To encourage the model to learn modality-agnostic features, modality dropout is applied to multimodal data to randomly drop a subset of the modalities, effectively creating multiple copies of the data with the same target but different input modalities, as depicted in the top half of Figure 2.
Revised: To encourage the model to learn modality-agnostic features, modality dropout is applied to multimodal data to randomly drop a subset of the modalities, effectively creating multiple copies of the data with the same target but different input modalities, as depicted in the top half of Figure 2.

Original: While modality dropout was also used in AV-HuBERT, it was implemented for a different reason – to reduce the discrepancy between pre-training and fine-tuning for unimodal downstream tasks, which resulted in only minor improvement (reducing the WER from 57.0% to 55.3% as shown in Table D.1 in Shi et al.
Revised: While modality dropout was also used in AV-HuBERT, it was implemented for a different reason – to reduce the discrepancy between pre-training and fine-tuning for unimodal downstream tasks, which resulted in only minor improvement (reducing the WER from 57.0% to 55.3% as shown in Table D.1 in Shi et al.

Original: [2022a]).
Revised: [2022a]).

Original: In contrast, we will demonstrate that modality dropout is essential for mixed modality pre-training and zero-shot generalization.
Revised: In contrast, we will demonstrate that modality dropout is essential for mixed modality pre-training and zero-shot generalization.

Original: After the first iteration, the model can be used to extract modality-agnostic features for all unimodal and multimodal data, which are clustered to produce the pseudo label for next iteration of pre-training as illustrated in Figure 1.
Revised: After the first iteration, the model can be used to extract modality-agnostic features for all unimodal and multimodal data, which are clustered to produce the pseudo label for next iteration of pre-training as illustrated in Figure 1.

Original: For multimodal data like audio-visual speech, one can produce pseudo labels using features extracted from the complete multimodal input (e.g., Y av from audio-visual input), or those extracted from input with a subset of modalities (e.g., Y v using only the video stream).
Revised: For multimodal data like audio-visual speech, one can produce pseudo labels using features extracted from the complete multimodal input (e.g., Y av from audio-visual input), or those extracted from input with a subset of modalities (e.g., Y v using only the video stream).

Original: We opt for the former and study the impact of these choices in Secton 4.2.
Revised: We opt for the former and study the impact of these choices in Secton 4.2.

Original: 3.3 Multimodal and unimodal fine-tuning with partial modalities
Revised: 3.3 Multimodal and unimodal fine-tuning with partial modalities

Original: Once the model is pre-trained, we remove the cluster prediction head (the linear layer on top of transformer used to predict cluster assignment distribution) and add a downstream task-specific prediction head.
Revised: Once the model is pre-trained, we remove the cluster prediction head (the linear layer on top of transformer used to predict cluster assignment distribution) and add a downstream task-specific prediction head.

Original: The pre-trained model can be fine-tuned on labeled multimodal speech, unimodal speech, or speech with mixed modalities.
Revised: The pre-trained model can be fine-tuned on labeled multimodal speech, unimodal speech, or speech with mixed modalities.

Original: More importantly, the modalities included in the finetuning data do not necessarily cover the modalities seen during pre-training, yet the fine-tuned model can still handle the downstream task on all pre-trained modalities, which we refer to as zero-shot modality generalization .
Revised: More importantly, the modalities included in the finetuning data do not necessarily cover the modalities seen during pre-training, yet the fine-tuned model can still handle the downstream task on all pre-trained modalities, which we refer to as zero-shot modality generalization .

Original: This is because the pre-trained model has successfully learned modalityagnostic representations, where same speech manifested in different modalities are mapped to similar representations.
Revised: This is because the pre-trained model has successfully learned modalityagnostic representations, where same speech manifested in different modalities are mapped to similar representations.

Original: Thus mapping from the representation 
Revised: Thus mapping from the representation of one modality (e.g., audio) to a target output (e.g., text) can be directly applied to other pre-trained modalities (e.g., video).

Original: Similar to mBART [Liu et al., 2020] for zero-shot machine translation, catastrophic forgetting [Lee et al., 2019, Üstün et al., 2021] may occur during fine-tuning and hence hinders its generalization across modalities.
Revised: Similar to mBART [Liu et al., 2020] for zero-shot machine translation, catastrophic forgetting [Lee et al., 2019, Üstün et al., 2021] may occur during fine-tuning and hence hinders its generalization across modalities.

Original: Particularly, this is because the distributions of Transformer input and output at earlier layers are less modality-invariant.
Revised: Particularly, this is because the distributions of Transformer input and output at earlier layers are less modality-invariant.

Original: To alleviate the issue, we apply modality dropout when multimodal data is used, and explore a few techniques to prevent the representations from drifting too much, including freezing the entire pre-trained model or freezing a number of initial layers.
Revised: To alleviate the issue, we apply modality dropout when multimodal data is used, and explore a few techniques to prevent the representations from drifting too much, including freezing the entire pre-trained model or freezing a number of initial layers.

Original:  Experiments 4.1 Pre-training setup
Revised:  Experiments 4.1 Pre-training setup

Original: We evaluate u-HuBERT on audio, visual, and audio-visual speech, where larger quantities of unlabeled and labeled data are available.
Revised: We evaluate u-HuBERT on audio, visual, and audio-visual speech, where larger quantities of unlabeled and labeled data are available.

Original: Two audio-visual and one audio-only datasets are used for pre-training: (1) LRS3 [Afouras et al., 2018] with 433 hours of English audio-visual speech, (2) VoxCeleb2-En (VC2-En) with 1,326 hours of English YouTube audio-visual speech filtered from VoxCeleb2 [Chung et al., 2018] by Shi et al.
Revised: Two audio-visual and one audio-only datasets are used for pre-training: (1) LRS3 [Afouras et al., 2018] with 433 hours of English audio-visual speech, (2) VoxCeleb2-En (VC2-En) with 1,326 hours of English YouTube audio-visual speech filtered from VoxCeleb2 [Chung et al., 2018] by Shi et al.

Original: [2022a], and (3) TED-LIUM release 3 (TD)
Revised: [2022a], and (3) TED-LIUM release 3 (TD)

Original: [Hernandez et al., 2018] with 452 hours of English audio collected from the same domain as LRS3.
Revised: [Hernandez et al., 2018] with 452 hours of English audio collected from the same domain as LRS3.

Original: To improve noise robustness, we apply online noise augmentation where each utterance is corrupted with additive noise at 0dB sampled from MUSAN [Snyder et al., 2015] with a probability of 0.25.
Revised: To improve noise robustness, we apply online noise augmentation where each utterance is corrupted with additive noise at 0dB sampled from MUSAN [Snyder et al., 2015] with a probability of 0.25.

Original: More details can be found in
Revised: More details can be found in

Original: Appendix A. These datasets are solely used to perform experiments for the purpose of comparing with previous work in a fair setup.
Revised: Appendix A. These datasets are solely used to perform experiments for the purpose of comparing with previous work in a fair setup.

Original: We adopt the AV-HuBERT-LARGE model Shi et al.
Revised: We adopt the AV-HuBERT-LARGE model Shi et al.

Original: [2022a] 2 : the video encoder is a modified
Revised: [2022a] 3 : the video encoder is a modified

Original: ResNet-18 model [He et al., 2016] that takes image frames of 88x88 pixels sampled at 25Hz at input and produce a sequence of 512 dimensional embeddings at the same frame rate; audio is represented as filter bank features sampled at 100 Hz, stacked every four frames to match the video frame rate, and projected with a linear layer to 1024-D features.
Revised: ResNet-18 model [He et al., 2016] that takes image frames of 88x88 pixels sampled at 25Hz at input and produce a sequence of 512 dimensional embeddings at the same frame rate; audio is represented as filter bank features sampled at 100 Hz, stacked every four frames to match the video frame rate, and projected with a linear layer to 1024-D features.

Original: FUSE concatenates features frame-by-frame and linearly projects them to the embedding dimension of the shared Transformer.
Revised: FUSE concatenates features frame-by-frame and linearly projects them to the embedding dimension of the shared Transformer.

Original: The shared
Revised: The shared

Original: Transformer has 24 layers, with 16 heads, embedding dimension 1024, and FFN dimension 4096, using the pre-norm residual connection setup [Nguyen and Salazar, 2019].
Revised: Transformer has 24 layers, with 16 heads, embedding dimension 1024, and FFN dimension 4096, using the pre-norm residual connection setup [Nguyen and Salazar, 2019].

Original: To directly compare with the publicly available fifth iteration noise-augmented AV-HuBERT-LARGE model trained on LRS3 and VC2-En, the same feature extractor (fourth iteration BASE model) and the same codebook (learned from multimodal features C av ) are used to generate pseudo labels for multimodal LRS3 and VC2-En and unimodal TD.
Revised: To directly compare with the publicly available fifth iteration noise-augmented AV-HuBERT-LARGE model trained on LRS3 and VC2-En, the same feature extractor (fourth iteration BASE model) and the same codebook (learned from multimodal features C av ) are used to generate pseudo labels for multimodal LRS3 and VC2-En and unimodal TD.

Original: We then pre-train u-HuBERT on the combined data for 1M updates on 64 32GB V100 GPUs using the Adam optimizer [Kingma and Ba, 2015] and a learning rate of 0.002.
Revised: We then pre-train u-HuBERT on the combined data for 1M updates on 64 32GB V100 GPUs using the Adam optimizer [Kingma and Ba, 2015] and a learning rate of 0.002.

Original: Gradient norm is clipped at 1.0. to stablize training.
Revised: Gradient norm is clipped at 1.0. to stablize training.

Original: A batch size of maximal 40 seconds per GPU is used.
Revised: A batch size of maximal 40 seconds per GPU is used.

Original: When pre-trained on multimodal data, audio and video are dropped with a probability of 0.25 and 0.25, respectively.
Revised: When pre-trained on multimodal data, audio and video are dropped with a probability of 0.25 and 0.25, respectively.

Original: 4.2 Learning modality-agnostic features and codebooks with modality dropout
Revised: 4.2 Learning modality-agnostic features and codebooks with modality dropout

Original: We argue that modality dropout leads to learning modality-agnostic representations through forcing the model to predict the same target with different subsets of input modalities given multimodal input, which in turn enables construction of a shared codebook for all modalities.
Revised: We argue that modality dropout leads to learning modality-agnostic representations through forcing the model to predict the same target with different subsets of input modalities given multimodal input, which in turn enables construction of a shared codebook for all modalities.

Original: To verify this, we compare two models that are identical except for whether modality dropout is activated: the public noise-augmented AV-HuBERT LARGE model trained on LRS3 and VC2-En with modality dropout and its counterpart.
Revised: To verify this, we compare two models that are identical except for whether modality dropout is activated: the public noise-augmented AV-HuBERT LARGE model trained on LRS3 and VC2-En with modality dropout and its counterpart.

Original: For each model, we infer audio, visual, and audio-visual features for each LRS validation utterances, randomly sample 500 frames, pool the features, and run t-SNE [Van der Maaten and Hinton, 2008] dimension reduction for visualization (Figure 3).
Revised: For each model, we infer audio, visual, and audio-visual features for each LRS validation utterances, randomly sample 500 frames, pool the features, and run t-SNE [Van der Maaten and Hinton, 2008] dimension reduction for visualization (Figure 3).

Original: It is visually evident that without modality dropout, features extracted from different modalities have different distributions (bottom row), contrary to those from the model with modality dropout (top row).
Revised: It is visually evident that without modality dropout, features extracted from different modalities have different distributions (bottom row), contrary to those from the model with modality dropout (top row).

Original: We next adopt the phone normalized mutual information (PNMI) metric presented in Hsu et al.
Revised: We next adopt the phone normalized mutual information (PNMI) metric presented in Hsu et al.

Original: [2021a] to quantify cluster quality.
Revised: [2021a] to quantify cluster quality.

Original: For each model, we create four codebooks by clustering audiovisual features C av , audio features C a , visual features C v , or all three combined ∪ m C m , quantize each type of features with each codebook, and measure the normalized mutual information ∈ [0 , 1] between frame-level phone labels and cluster assignment (Table 1).
Revised: For each model, we create four codebooks by clustering audio- visual features C av , audio features C a , visual features C v , or all three combined ∪ m C m , quantize each type of features with each codebook, and measure the normalized mutual information ∈ [0 , 1] between frame-level phone labels and cluster assignment (Table 1).

Original: Because features inferred from different modalities exhibit similar distributions with modality dropout, all four codebooks lead to similar PNMI for a given feature.
Revised: Because features inferred from different modalities exhibit similar distributions with modality dropout, all four codebooks lead to similar PNMI for a given feature.

Original: In contrast, without modality dropout, cross-quantization (e.g., Y av with C v codebook) often leads to significantly worse PNMI, and PNMI is worse for all (codebook, feature) combinations compared with that from the modality dropout model.
Revised: In contrast, without modality dropout, cross-quantization (e.g., Y av with C v codebook) often leads to significantly worse PNMI, and PNMI is worse for all (codebook, feature) combinations compared with that from the modality dropout model.

Original: Note for a given codebook, the PNMI still differs among modalities which is due to the discrepancy of information carried per modality.
Revised: Note for a given codebook, the PNMI still differs among modalities which is due to the discrepancy of information carried per modality.

Original: [Table]
Revised: [Table]

Original: 4.3 Speech recognition
Revised: 4.3 Speech recognition

Original: Fine-tuning setup LRS3 trainval and pretrain , combined for 433 hours, are used for training, with the same 1,200 utterances as Shi et al.
Revised: Fine-tuning setup LRS3 trainval and pretrain , combined for 433 hours, are used for training, with the same 1,200 utterances as Shi et al.

Original: [2022a] split for validation.
Revised: [2022a] split for validation.

Original: Results on the test split are reported.
Revised: Results on the test split are reported.

Original: Noise augmentation with MUSAN used in pre-training is also adopted during fine-tuning.
Revised: Noise augmentation with MUSAN used in pre-training is also adopted during fine-tuning.

Original: A 9-layer randomly initialized Transformer decoder with 8 attention heads and 1024/4096-D embedding/FFN is appended to the pre-trained u-HuBERT, and the entire model is fine-tuned with a cross-entropy loss predicting the next text token given the input and the previous text tokens.
Revised: A 9-layer randomly initialized Transformer decoder with 8 attention heads and 1024/4096-D embedding/FFN is appended to the pre-trained u-HuBERT, and the entire model is fine-tuned with a cross-entropy loss predicting the next text token given the input and the previous text tokens.

Original: Text is represented as unigram-based subword units [Kudo, 2018] with a vocabulary size of 1,000.
Revised: Text is represented as unigram-based subword units [Kudo, 2018] with a vocabulary size of 1,000.

Original: Modality dropout is applied when fine-tuning on multimodal input.
Revised: Modality dropout is applied when fine-tuning on multimodal input.

Original: Additionally, we sweep learning rate, number of updates, steps to freeze, and layers to freeze throughout fine-tuning.
Revised: Additionally, we sweep learning rate, number of updates, steps to freeze, and layers to freeze throughout fine-tuning.

Original: We evaluate each model, regardless of the modality used for fine-tuning, on all kinds of input modalities and report the word error rates (WERs) without fusing external language models.
Revised: We evaluate each model, regardless of the modality used for fine-tuning, on all kinds of input modalities and report the word error rates (WERs) without fusing external language models.

Original: To gauge the robustness to noise, we follow the same protocol in Shi et al.
Revised: To gauge the robustness to noise, we follow the same protocol in Shi et al.

Original: [2022b] to noisy audio and audio-visual test sets by adding babble noise at 0 decibel (dB) to the audio stream for evaluation.
Revised: [2022b] to noisy audio and audio-visual test sets by adding babble noise at 0 decibel (dB) to the audio stream for evaluation.

Original: Detailed hyperpamater setups can be found in Appendix A, and more ablation studies can be found in Appendix B.1.
Revised: Detailed hyperpamater setups can be found in Appendix A, and more ablation studies can be found in Appendix B.1.

Original: Modality dropout is essential for building a single model Table 2 compares fine-tuning from scratch, as well as from pre-trained models with and without modality dropout, using labeled audio, visual, or audio-visual speech, and highlights the results of zero-shot modality transfer where labeled data in test modality are not used.
Revised: Modality dropout is essential for building a single model Table 2 compares fine-tuning from scratch, as well as from pre-trained models with and without modality dropout, using labeled audio, visual, or audio-visual speech, and highlights the results of zero-shot modality transfer where labeled data in test modality are not used.

Original: First, models without pre-training are significantly worse.
Revised: First, models without pre-training are significantly worse.

Original: Second, when fine-tuning on audio-visual data without pre-training modality dropout, performance drops significantly on unimodal data if pre-training modality dropout is not applied (e.g., 1.7% to 21.4% on clean audio).
Revised: Second, when fine-tuning on audio-visual data without pre-training modality dropout, performance drops significantly on unimodal data if pre-training modality dropout is not applied (e.g., 1.7% to 21.4% on clean audio).

Original: This can be alleviated with fine-tuning modality dropout, improving the average
Revised: This can be alleviated with fine-tuning modality dropout, improving the average

Original: WER from 24.5% to 12.3%, but is still behind applying it in both pre-training and fine-tuning (11.4% average).
Revised: WER from 24.5% to 12.3%, but is still behind applying it in both pre-training and fine-tuning (11.4% average).

Original: We also note that fine-tuning dropout leads to consistent improvement on noisy AV test set, likely because occasionally dropping the audio stream encourages the model to rely more on the visual input which improves noise robustness.
Revised: We also note that fine-tuning dropout leads to consistent improvement on noisy AV test set, likely because occasionally dropping the audio stream encourages the model to rely more on the visual input which improves noise robustness.

Original: The benefit of applying pre-training modality dropout is magnified in unimodal fine-tuning: the model fine-tuned only on labeled audio data yields the same performance on clean and noisy audio-visual input (1.3% and 4.6%) as the model fine-tuned with labeled audio-visual data, while being slight behind on video input compared to the model fine-tuned with labeled video data (31.6% vs. 28.7%).
Revised: The benefit of applying pre-training modality dropout is magnified in unimodal fine-tuning: the model fine-tuned only on labeled audio data yields the same performance on clean and noisy audio-visual input (1.3% and 4.6%) as the model fine-tuned with labeled audio-visual data, while being slight behind on video input compared to the model fine-tuned with labeled video data (31.6% vs. 28.7%).

Original: When it is compared to the model pre-trained without modality dropout and fine-tuned also with labeled audio, huge gains are observed on zero-shot scenarios, for example 96.8% versus 31.6% 
Revised: When it is compared to the model pre-trained without modality dropout and fine-tuned also with labeled audio, huge gains are observed on zero-shot scenarios, for example 96.8% versus 31.6% [Table] [Table]

Original: V-WER and 18.0% versus 4.6% noisy AV-WER.
Revised: V-WER and 18.0% versus 4.6% noisy AV-WER.

Original: Similar trend can be observed when fine-tuning with labeled visual speech.[Table]
Revised: Similar trend can be observed when fine-tuning with labeled visual speech.

Original: [Table] n [Table] Zero-shot modality transfer is not achieved through memorization Because the dataset used for fine-tuning is a subset of the pre-training data, there could be suspicions that the model memorizes the audio-visual pairs from pre-training and associates them with the labeled unimodal data to enable zero-shot transfer.
Revised: Zero-shot modality transfer is not achieved through memorization Because the dataset used for fine-tuning is a subset of the pre-training data, there could be suspicions that the model memorizes the audio-visual pairs from pre-training and associates them with the labeled unimodal data to enable zero-shot transfer.

Original: To avoid cheating, we fine-tune our model on LibriSpeech
Revised: To avoid cheating, we fine-tune our model on LibriSpeech

Original: [Panayotov et al., 2015], an audio-only read audiobook dataset that is out-of-domain relative to the pre-training data composed of oratory talks (Table 3).
Revised: [Panayotov et al., 2015], an audio-only read audiobook dataset that is out-of-domain relative to the pre-training data composed of oratory talks (Table 3).

Original: Despite the degradation compared to fine-tuning on in-domain LRS3 data, the model still performs decently on zero-shot transfer scenarios compared to models pre-trained without modality dropout in Table 2 and 3.
Revised: Despite the degradation compared to fine-tuning on in-domain LRS3 data, the model still performs decently on zero-shot transfer scenarios compared to models pre-trained without modality dropout in Table 2 and 3.

Original: [Table]
Revised: [Table]

Original: Comparison with state-of-the-art models Table 4 compares u-HuBERT with models from the literature, all of which are modality-specific.
Revised: Comparison with state-of-the-art models Table 4 compares u-HuBERT with models from the literature, all of which are modality-specific.

Original: In contrast, u-HuBERT is the only single model that can be fine-tuned on any modality and test on all modalities while achieving performance on par or better than the best modality-specific models using similar amounts of labeled data.
Revised: In contrast, u-HuBERT is the only single model that can be fine-tuned on any modality and test on all modalities while achieving performance on par or better than the best modality-specific models using similar amounts of labeled data.

Original: Furthermore, compared to AV-HuBERT, u-HuBERT can be pre-trained on additional unimodal data to yield even better performance.
Revised: Furthermore, compared to AV-HuBERT, u-HuBERT can be pre-trained on additional unimodal data to yield even better performance.

Original: 4.4 Speech translation
Revised: 4.4 Speech translation

Original: We repeat the experiment on speech translation to study if u-HuBERT enables construction of a single model and achieves zero-shot generalization on a different task.
Revised: We repeat the experiment on speech translation to study if u-HuBERT enables construction of a single model and achieves zero-shot generalization on a different task.

Original: Fine-tuning setup To simulate speech translation setting, we translate the English transcriptions into
Revised: Fine-tuning setup To simulate speech translation setting, we translate the English transcriptions into

Original: Spanish to create paired (English audio, English video, Spanish text) data.
Revised: Spanish to create paired (English audio, English video, Spanish text) data.

Original: The 30 hour LRS3 trainval split is used for fine-tuning.
Revised: The 30 hour LRS3 trainval split is used for fine-tuning.

Original: We use the same decoder for speech translation as for speech recognition, where a 9-layer randomly initialized Transformer is appended to the pretrained u-HuBERT encoder to decode English speech into Spanish text.
Revised: We use the same decoder for speech translation as for speech recognition, where a 9-layer randomly initialized Transformer is appended to the pretrained u-HuBERT encoder to decode English speech into Spanish text.

Original: Unigram-based sentencepiece units with a vocabulary  size of 1,000 are used as decoding output units.
Revised: Unigram-based sentencepiece units with a vocabulary [Table] size of 1,000 are used as decoding output units.

Original: The overall fine-tuning and decoding paradigms remain same as speech recognition.
Revised: The overall fine-tuning and decoding paradigms remain same as speech recognition.

Original: Detailed hyperparameters can be found in Appendix A. BLEU score Papineni et al.
Revised: Detailed hyperparameters can be found in Appendix A. BLEU score Papineni et al.

Original: [2002] is used to evaluate the translation model.[Table]
Revised: [2002] is used to evaluate the translation model.

Original: Translation performance The performance of u-HuBERT on speech translation is shown in Table [Table]
Revised: The performance of u-HuBERT on speech translation is shown in Table 5 4 .

Original: As is similar to the observation in speech recognition, the model hardly produces meaningful results if its encoder is randomly initialized (Avg-BLEU: 3.3).
Revised: As is similar to the observation in speech recognition, the model hardly produces meaningful results if its encoder is randomly initialized (Avg-BLEU: 3.3).

Original: u-HuBERT pre-training not only greatly improves translation quality by roughly 10 fold (Avg-BLEU: 3.3 → 35.0), but also shows strong performance in zero-shot modality transfer.
Revised: u-HuBERT pre-training not only greatly improves translation quality by roughly 10 fold (Avg-BLEU: 3.3 → 35.0), but also shows strong performance in zero-shot modality transfer.

Original: For instance, the model fine-tuned with audio-only speech achieves 34.3 BLEU score when it is tested using audio-visual speech under noisy environment, outperforming its audio-only counterpart (26.5).
Revised: For instance, the model fine-tuned with audio-only speech achieves 34.3 BLEU score when it is tested using audio-visual speech under noisy environment, outperforming its audio-only counterpart (26.5).

Original: In tasks such as speech translation, where labeled multimodal data can be difficult to harvest, our pre-training approach shows potential in enhancing the noise robustness supplied by the extra visual modality at test time even without exposure to any labeled visual data during training.
Revised: In tasks such as speech translation, where labeled multimodal data can be difficult to harvest, our pre-training approach shows potential in enhancing the noise robustness supplied by the extra visual modality at test time even without exposure to any labeled visual data during training.

Original: Within u-HuBERT, modality dropout is an essential component for acquiring such capacity, as is suggested by improvement in average BLEU score it brings (A: 27.3 → 34.7, V: 17.0 → 31.7).
Revised: Within u-HuBERT, modality dropout is an essential component for acquiring such capacity, as is suggested by improvement in average BLEU score it brings (A: 27.3 → 34.7, V: 17.0 → 31.7).

Original: In cases with available labeled multimodal data, a single audio-visual model is able to achieve performance superior or on par to a modality-specific model (A: 42.2 vs.
Revised: In cases with available labeled multimodal data, a single audio-visual model is able to achieve performance superior or on par to a modality-specific model (A: 42.2 vs.

Original: 42.9, V: 27.6 vs. 26.4) [Table]
Revised: 42.9, V: 27.6 vs. 26.4).

Original: [Table]

Original: 4.5 Ablation studies
Revised: 4.5 Ablation studies

Original: Pre-training modality dropout parameters To better understand the impact of pre-training modality dropout configurations on zero-shot modality transfer, we conducted pre-training ablation studies with a reduced setup: for each configuration, a BASE model is pre-trained on 433 hours of LRS3 data using 32 GPUs for 200K updates and then fine-tuned on 30 hours of LRS3 labeled data (the “trainval” split).
Revised: Pre-training modality dropout parameters To better understand the impact of pre-training modality dropout configurations on zero-shot modality transfer, we conducted pre-training ablation studies with a reduced setup: for each configuration, a BASE model is pre-trained on 433 hours of LRS3 data using 32 GPUs for 200K updates and then fine-tuned on 30 hours of LRS3 labeled data (the “trainval” split).

Original: Four modality dropout configurations are considered, represented by the probability of using both streams ( p av ), audio stream ( p a ) and video stream ( p v ) in Table 6.
Revised: Four modality dropout configurations are considered, represented by the probability of using both streams ( p av ), audio stream ( p a ) and video stream ( p v ) in Table 6.

Original: The first configuration ( p av , p a , p v ) = (1.00, 0.00, 0.00) is equivalent to not using modality dropout.
Revised: The first configuration ( p av , p a , p v ) = (1.00, 0.00, 0.00) is equivalent to not using modality dropout.

Original: When fine-tuning on audio-visual or audio-only data, among the models pre-trained with modality dropout, setting the p v higher leads to slightly worse performance on audio-visual and audio-only test sets.
Revised: When fine-tuning on audio-visual or audio-only data, among the models pre-trained with modality dropout, setting the p v higher leads to slightly worse performance on audio-visual and audio-only test sets.

Original: In contrast, when fine-tuning on visual-only data, setting the probability of V-only higher leads  to slightly better average performance.
Revised: In contrast, when fine-tuning on visual-only data, setting the probability of V-only higher leads [Table] to slightly better average performance.

Original: Nevertheless, all three configurations achieve significantly better performance compared to the model pre-trained without modality dropout, confirming that activating modality dropout is the key to successful zero-shot modality transfer. 
Revised: Nevertheless, all three configurations achieve significantly better performance compared to the model pre-trained without modality dropout, confirming that activating modality dropout is the key to successful zero-shot modality transfer. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: n

Revised: [Table]

Revised: a

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: n

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: n/a 11.

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  Pre-training dynamics with additional unimodal data While mixed modality pre-training allows the model to be trained on larger quantities of data, it also dilutes the in-domain data with respect to audio-visual or other unimodal tasks, which may lead to worse performance on those tasks.
Revised: [Table] [Table] [Table] Pre-training dynamics with additional unimodal data While mixed modality pre-training allows the model to be trained on larger quantities of data, it also dilutes the in-domain data with respect to audio-visual or other unimodal tasks, which may lead to worse performance on those tasks.

Original: We pre-train on audio-visual data and mixed-modal data (audio-visual and audio-only) respectively for different numbers of updates and study if degradation occurs to audio-visual and visual speech recognition in the mixed-modal case.
Revised: We pre-train on audio-visual data and mixed-modal data (audio-visual and audio-only) respectively for different numbers of updates and study if degradation occurs to audio-visual and visual speech recognition in the mixed-modal case.

Original: The baseline (PT on AV) is pre-trained on audio-visual LRS3, and the proposed mixed-modal pretraining (PT on AV+A) uses the same audio-visual data as the baseline plus the VC2-En audio data.
Revised: The baseline (PT on AV) is pre-trained on audio-visual LRS3, and the proposed mixed-modal pretraining (PT on AV+A) uses the same audio-visual data as the baseline plus the VC2-En audio data.

Original: For each setup we pre-train the model for {100, 200, 400, 600}K updates.
Revised: For each setup we pre-train the model for {100, 200, 400, 600}K updates.

Original: Results in Figure 4 showed that adding unlabeled audio data consistently improves the audio speech recognition performance at all number of updates.
Revised: Results in Figure 4 showed that adding unlabeled audio data consistently improves the audio speech recognition performance at all number of updates.

Original: In contrast, the performance of “PT on AV+A” lags behind “PT on
Revised: In contrast, the performance of “PT on AV+A” lags behind “PT on

Original: A” on audio-visual and visual speech recognition when a model is trained for fewer than 400K updates.
Revised: A” on audio-visual and visual speech recognition when a model is trained for fewer than 400K updates.

Original: Nevertheless, when a model is trained for longer (600K updates), mixed-modal pre-training outperforms the baseline on all three tasks.
Revised: Nevertheless, when a model is trained for longer (600K updates), mixed-modal pre-training outperforms the baseline on all three tasks.

Original: The observation suggests that u-HuBERT can benefit from training on more data and continue improving with more updates, while the performance of the baseline saturates earlier due to the lack of data. 
Revised: The observation suggests that u-HuBERT can benefit from training on more data and continue improving with more updates, while the performance of the baseline saturates earlier due to the lack of data. [Figure] [Figure] FT and

Revised: [Figure]

Revised: V

Original:   Discussion
Revised: PTonAV PTonAV [Figure] A  Discussion

Original: We assume that there is an anchor modality where all other modalities appear in some multimodal data paired with it.
Revised: We assume that there is an anchor modality where all other modalities appear in some multimodal data paired with it.

Original: This enables the model to predict shared targets and learn modality-agnostic representations for all modalities, such that non-anchor unimodal data can be used in the subsequent iterations.
Revised: This enables the model to predict shared targets and learn modality-agnostic representations for all modalities, such that non-anchor unimodal data can be used in the subsequent iterations.

Original: While this may seem limiting, most multimodal speech are paired with audio [Richard[Figure] [Figure] et al., 2021, Hueber et al., 2010, Richmond et al., 2011, Livescu et al., 2009] since audio is the primary measurement for speech and other modalities are often supplementary.
Revised: While this may seem limiting, most multimodal speech are paired with audio [Richard et al., 2021, Hueber et al., 2010, Richmond et al., 2011, Livescu et al., 2009] since audio is the primary measurement for speech and other modalities are often supplementary.

Original: This makes the assumption of having audio as the anchor modality practical.
Revised: This makes the assumption of having audio as the anchor modality practical.

Original: Furthermore, the idea can be easily extended to settings where any modality can be connected to the anchor via some intermediate modality.
Revised: Furthermore, the idea can be easily extended to settings where any modality can be connected to the anchor via some intermediate modality.

Original: In short, a modality that is N -hop away from the primary anchor can be used in the ( N + 1) -th iteration.
Revised: In short, a modality that is N -hop away from the primary anchor can be used in the ( N + 1) -th iteration.

Original: The only case not supported u-HuBERT is pre-training on modalities that cannot be connected through any multimodal data, for example, on a collection of audio, visual, audio-visual and EMG speech.
Revised: The only case not supported u-HuBERT is pre-training on modalities that cannot be connected through any multimodal data, for example, on a collection of audio, visual, audio-visual and EMG speech.

Original: A shared codebook cannot be derived for EMG and the rest of the modalities, and it is unclear if the model can still benefit from joint training.
Revised: A shared codebook cannot be derived for EMG and the rest of the modalities, and it is unclear if the model can still benefit from joint training.

Original: The setup has also been found much more challenging on other pairs of modalities: for example, SLAM Bapna et al.
Revised: The setup has also been found much more challenging on other pairs of modalities: for example, SLAM Bapna et al.

Original: [2021] shows that without paired text and speech data, joint pre-training on unlabeled speech and text degrades the performance compared to a model pre-trained only on speech.
Revised: [2021] shows that without paired text and speech data, joint pre-training on unlabeled speech and text degrades the performance compared to a model pre-trained only on speech.

Original: We leave such exploration to future work.
Revised: We leave such exploration to future work.

Original: In addition, we also include ethical discussion in Appendix C.
Revised: In addition, we also include ethical discussion in Appendix C.

Original:  Conclusion
Revised:  Conclusion

Original: Recent studies in applied machine learning trend toward building things that are general , for example, general perception module [Jaegle et al., 2021], general self-supervised objective [Baevski et al., 2022], single model for many tasks [Brown et al., 2020, Kaplan et al., 2020, Aghajanyan et al., 2022] and/or many input [Girdhar et al., 2022, Hu and Singh, 2021], instead of optimizing for performance on very specific setups.
Revised: Recent studies in applied machine learning trend toward building things that are general , for example, general perception module [Jaegle et al., 2021], general self-supervised objective [Baevski et al., 2022], single model for many tasks [Brown et al., 2020, Kaplan et al., 2020, Aghajanyan et al., 2022] and/or many input [Girdhar et al., 2022, Hu and Singh, 2021], instead of optimizing for performance on very specific setups.

Original: Our work contributes to this line of research by presenting a unified self-supervised objective to pre-train a model on speech of many modalities.
Revised: Our work contributes to this line of research by presenting a unified self-supervised objective to pre-train a model on speech of many modalities.

Original: Specifically, it can be fine-tuned for many different speech tasks on labeled data in any combination of pre-trained modalities, resulting in a single model that can process all combinations of pre-trained modalities to perform the fine-tuned task.
Revised: Specifically, it can be fine-tuned for many different speech tasks on labeled data in any combination of pre-trained modalities, resulting in a single model that can process all combinations of pre-trained modalities to perform the fine-tuned task.

Original: We envision our work can bring substantial benefit to multimodal speech processing.
Revised: We envision our work can bring substantial benefit to multimodal speech processing.

Original: Given there are significantly more unlabeled audio compared to unlabeled multimodal speech, being able to utilize unlabeled audio can greatly improve the quality of multimodal speech representations.
Revised: Given there are significantly more unlabeled audio compared to unlabeled multimodal speech, being able to utilize unlabeled audio can greatly improve the quality of multimodal speech representations.

Original: Furthermore, there are few or even no labeled multimodal speech for many tasks, such as speech translation.
Revised: Furthermore, there are few or even no labeled multimodal speech for many tasks, such as speech translation.

Original: Hence, it is essential to build a model capable of zero-shot input modality transfer to perform these tasks with modalities without labeled data.
Revised: Hence, it is essential to build a model capable of zero-shot input modality transfer to perform these tasks with modalities without labeled data.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See Section(b)
Revised: [Yes] See Section(b)

Original: Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] See Section(c)
Revised: [Yes] See Section(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] See Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 4 and Appendix A (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Section 4 and Appendix A (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Appendix B (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] See Appendix B (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4 and Appendix A 4.
Revised: [Yes] See Section 4 and Appendix A 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section(b)
Revised: [Yes] See Section(b)

Original: Did you mention the license of the assets?
Revised: Did you mention the license of the assets?

Original: [Yes] See Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] See Appendix A (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] See Appendix A (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] See Appendix A 5.
Revised: [Yes] See Appendix A 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Experiment Details A.1 Data
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Table 7 summarizes the datasets used in this paper, which are all licensed under CC BY-NC-ND or

Original: CC BY and have been used extensively by the research communities.

Original: Speech datasets are sourced from interviews, TED talks

Original: [Table]

Original: and audiobooks, which are not expected to contain offensive content.

Original: [Table]

Original: A.2 Fine-tuning

Original: Table 8 summarizes the hyperparameters used for ASR fine-tuning

Original: [Table]

Original: [Table]

Original: 9 summarizes the hyperparameters used for speech translation.

Original: [Table]

Original: A.3 Speech recognition decoding

Original: Beam search decoding is used with a length weight α , which searches for the hypothesis z 1: T that maximizes

Original: [Equation]

Original: For the results in the main paper, we do grid search from beam size ∈ { 1 , 5 , 10 , 15 , 20 , 25 } and α ∈ { 0 , 0 .

Original: For the ablation studies in the appendix a beam size of 10 and α = 1 . 0 is used.

Original: B Extend Experimental Results B.1 Impact of fine-tuning hyperparameters

Original: We conduct the ablation studies in this section with the models pre-trained on multimodal LRS3 and

Original: VC2-En.

Original: By default, the one pre-trained with modality dropout is used.

Original: B.1.1 Fine-tuning on multimodal data

Original: Table 10 shows how fine-tuning modality dropout configurations affect ASR performance.

Original: Models fine-tuned with dropout can yield better performance on all input modality than the one without (reported in the caption).

Original: When setting m-drop-p and a-drop-p within the range of [0 . 25 , 0 . 75] , the results have limited variation ( [1 . 43% , 1 . 65%] WER for AVSR, [1 . 82% , 2 .

Original: 22%] WER for ASR

Original: [Table]

Original: and [28 . 85% , 30 . 88%] WER for VSR).

Original: Similarly setting the values to 0.5 yields reasonable performance for all modalities.

Original: [Table]

Original: B.1.2 Fine-tuning on unimodal data

Original: Next

Original: [Table]

Original: we study the impact of hyperparameters when fine-tuning on unimodal data.

Original: Different from fine-tuning on multimodal data, fine-tuning on unimodal data is more prone to catastrophic forgetting, leading to huge performance degradation on modalities unseen during fine-tuning.

Original: Hence, we focus on studying the impact from three hyperparameters:

Original: • L frz : number of u-HuBERT Transformer layers that are frozen throughout fine-tuning.

Original: • N frz : number of the fine-tuning updates where the entire u-HuBERT is frozen.

Original: After this many updates, the layers above ( N frz )-th layer are optimized jointly with the prediction head.

Original: • LR : learning rate.

Original: Number of frozen layers ( L frz ) Setting this effectively treats the first L frz layers as a fixed feature extractor, while the layers above are considered pre-trained and jointly optimized with the added prediction head during fine-tuning.

Original: Results are shown in Table 11.

Original: When fine-tuned on audio speech from PT mod-drop , not freezing any layer leads to worse performance for all modalities, but freezing all the layers also reduces the model capacity and harms the performance for the fine-tuned modality and for audio-visual speech.

Original: For visual speech, the audio fine-tuned model achieves the best result when all the layers are frozen (30.60% compared to 35.68% when not freezing any layer), but the gap can be reduced from 5.08% to 1.23% when freezinglayers, in which audio-visual speech yields the best performance while audio is close to optimal.

Original: When fine-tuned on visual speech from PT mod-drop , the model yields similar performance on visual speech regardless how many layers are frozen (with a spread of 0.5% WER).

Original: In contrast, results are optimal for both audio-visual and audio input when 12 layers are frozen, with a good trade-off between mod el capacity and representation distribution shift.

Original: The trends are different when fine-tuning PT no-mod-drop .

Original: Because the model has only seen audiovisual input during pre-training, visual-only or audio-only input is considered out-of-distribution.

Original: Hence, when fine-tuning on visual speech with all the layers frozen ( L frz = 24 ), the performance on the fine-tuned input is significantly worse compared to fine-tuning PT mod-drop (45.18% versus 29.42%).

Original: Although this can be alleviated by unfreezing more layers, the model barely works for audio-only input with WERs ranging from 21.55% to 54.27%.

Original: This again verifies that pre-training modality dropout is essential for achieving zero-shot modality transfer.

Original: [Table]

Original: Number of frozen steps Results comparing the impact on freezing u-HuBERT for different numbers of fine-tuning steps are shown in Table 12.

Original: We can observe similar trends as those when varying the number of layers to freeze: not freezing at all leads worse performance on all input modalities, while treating the entire pre-trained model as a fixed feature extractor also leads to sub-optimal performance for the fine-tuned modality.

Original: Freezing for 30K updates out of 60K total updates leads to good balance for all modalities with PT mod-drop , which are 0.25%/0.11%/0.99% behind the optimal WERs when fine-tuned on audio, and 0%/0%/0.47% behind the optimal WERs when fine-tuned on visual speech.

Original: Similarly, the model pre-trained without modality dropout yields worse zero-shot modality transfer performance when fine-tuned on a single modality.

Original: [Table]

Original: Learning rate Table 13 presents the results.

Original: When fine-tuning on audio speech, increasing learning rate leads to significantly worse WER on visual input (30.62% → 37.43%)

Original: [Table]

Original: but benefits audio-visual and audio speech up to 6 × 10 − 4 and 8 × 10 − 4 , respectively, and the performance stays rather constant afterward.

Original: Similarly, when fine-tuning on visual speech, increasing learning rate to 10 − 3 hurts the result with audio input (2.73% → 3.41%); it also hurts the visual speech recognition performance, but the degradation is relatively minor (28.62% → 30.39%) compared to when fine-tuning on audio.

Original: In general, we observe a smaller learning is preferred for visual speech recognition

Original: [Table]
Revised: [N/A]

Original: and increasing learning rates harms the zero-shot modalities more.

Original: [Table]

Original: B.2 Per-layer representation analysis

Original: To better understand how u-HuBERT learns modality-agnostic features

Original: [Table]

Original: we show the clustering quality of different layers per modality of a pre-trained u-HuBERT model.

Original: Similar to Table 1, we report PNMI of layerwise clusters per modality quantized by audio-only ( C a ), video-only ( C v ), audiovisual ( C av ) and all-combined ( ∪ m C m ) codebook respectively (see Figure 5).

Original: When pre-trained without modality dropout, the model is unable to learn modality-agnostic features regardless of which layer to cluster, as can be shown from its overall lower PNMI of cross-modal clustering (e.g., PNMI of visual features quantized by C av codebook).

Original: To the model pre-trained with modality-dropout, the features of its intermediate layers become more agnostic to modality as the layer goes deeper, shown from the diminishing gap between its cross-modality and modality-specific performance in later layers.

Original: Furthermore, the approximately same quality of clusters from different codebooks towards the end suggests that the final layer is best suited to achieve a unified model that generalizes across modalities.

Original: B.3 Stability

Original: Table 14 shows the WER mean and standard deviation over five runs when fine-tuning the two u-HuBERT models.

Original: We see that the one pre-trained additionally with unimodal audio data (TD) is significantly better on all input modalities.

Original: [Table]

Original: C Ethical Discussion

Original: The ability to build a model that can process unimodal or multimodal speech without needing labeled data in the target modality opens many possibilities for real world applications, since except for audio-only speech, labeled data are extremely lacking for other speech modalities.

Original: In particular, we demonstrate its applications to audio-visual and visual speech recognition in this paper.

Original: The former can help hearing-impaired people to better “hear” speech in noisy environments with more accurate transcriptions, while the latter can help people with speech impairment (e.g., aphonia, dysphonia, dysarthria) to “speak” by transcribing silent speech.

Original: For visual speech recognition, there could be concerns about the technology being improperly used for CCTV surveillance.

Original: However, current visual speech recognition systems require mostly-frontal and high-resolution videos with a sufficiently high frame rate, such that motions are clearly captured.

Original: Hence, the type of data studied for audio-visual speech processing are face-to-face meeting scenarios (AMI, EasyCom, and Ego4D) and recorded speech (LRS3).

Original: In contrast, CCTV videos are low resolution, low frame rate

Original: [Table]

Original: and recorded from angles where faces are mostly not frontal, where visual speech processing models will very likely fail.

Original: D More Details of the Baseline ASR, VSR, and AVSR Methods

Original: Table 15 and 16 summarize the model architectures and training/testing setups of the baseline ASR,

Original: VSR

Original: [Table]

Original: and AVSR methods compared in Table

Original: [Table]

Original: [Table]

Original: [Table]