Original: A Single Self-Supervised Model for Many Speech Modalities Enables Zero-ShotModality Transfer Anonymous Author(s)
Revised: u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: While audio-visual speech models can yield superior performance and robustnesscompared to audio-only models, their development and adoption are hindered bythe lack of labeled and unlabeled audio-visual data and the cost to deploy one modelper modality.
Revised: While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality.

Original: In this paper, we present u-HuBERT, a self-supervised pre-trainingframework that can leverage both multimodal and unimodal speech with a unifiedmasked cluster prediction objective.
Revised: In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective.

Original: By utilizing modality dropout during pretraining, we demonstrate that a single fine-tuned model can achieve performanceon par or better than the state-of-the-art modality-specific models.
Revised: By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models.

Original: Moreover, ourmodel fine-tuned only on audio can perform well with audio-visual and visualspeech input, achieving zero-shot modality generalization for  speech recognitionand speaker verification.
Revised: Moreover, our model fine-tuned only on audio can perform well with audio-visual and visual speech input, achieving zero-shot modality generalization for multiple speech processing tasks.

Original: In particular, our single model yields 1.2%/1.4%/27.2%speech recognition word error rate on LRS3 with audio-visual/audio/visual input.
Revised: In particular, our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input.

Original:  Introduction
Revised:  Introduction

Original: Speech processing has been moving from developing domain-specific models toward building ageneral model for all domains.
Revised: Speech processing has been moving from developing domain-specific models toward building a general model for all domains.

Original: For example, early studies presented models dedicated for robustspeech recognition [Kingsbury et al., 1998] and distant speech recognition
Revised: For example, early studies presented models dedicated for robust speech recognition [Kingsbury et al., 1998] and distant speech recognition

Original: [Feng et al., 2014, Kimet al., 2017] by taking into account acoustic properties for each domain.
Revised: [Feng et al., 2014, Kim et al., 2017] by taking into account acoustic properties for each domain.

Original: In contrast, recent work hasbuilt general domain models by pooling data from different domains and even different languages,including labeled ones for supervised learning [Chan et al., 2021, Likhomanenko et al., 2020, Pratapet al., 2020] and unlabeled ones for self-supervised learning [Kawakami et al., 2020, Hsu et al. ,2021b, Conneau et al., 2020].
Revised: In contrast, recent work has built general domain models by pooling data from different domains and even different languages, including labeled ones for supervised learning [Chan et al., 2021, Likhomanenko et al., 2020, Pratap et al., 2020] and unlabeled ones for self-supervised learning [Kawakami et al., 2020, Hsu et al., 2021b, Conneau et al., 2020].

Original: These studies show that in addition to removing the need of deployingand maintaining multiple models, a single general domain model can in fact achieve performanceon par or superior to domain-specific models, especially for domains with less data and improvegeneralization performance to unseen domains.
Revised: These studies show that in addition to removing the need of deploying and maintaining multiple models, a single general domain model can in fact achieve performance on par or superior to domain-specific models, especially for domains with less data and improve generalization performance to unseen domains.

Original: In contrast, there are few attempts and even fewer successful ones on developing a versatile model thatcan process many speech modalities [Makino et al., 2019, Sari et al., 2021].
Revised: In contrast, there are few attempts and even fewer successful ones on developing a versatile model that can process many speech modalities [Makino et al., 2019, Sari et al., 2021].

Original: In general, researchersdevelop models with different input speech modalities, such as visual speech [Chung and Zisserman,2016], audio-visual speech [Xu et al., 2020], multi-channel speech [Haeb-Umbach et al., 2021], 3Dtalking faces [Richard et al., 2021], Electromyographic (EMG) muscle signals [Diener and Schultz,2018], ultrasonic oral cavity scan [Hueber et al., 2010], or ultrasonic microphone [Livescu et al. ,2009] in isolation, despite the fact that many of them are used to solve the same task, like speech andspeaker recognition.
Revised: In general, researchers develop models with different input speech modalities, such as visual speech [Chung and Zisserman, 2016], audio-visual speech [Xu et al., 2020], multi-channel speech [Haeb-Umbach et al., 2021], 3D talking faces [Richard et al., 2021], Electromyographic (EMG) muscle signals [Diener and Schultz, 2018], ultrasonic oral cavity scan [Hueber et al., 2010], or ultrasonic microphone [Livescu et al., 2009] in isolation, despite the fact that many of them are used to solve the same task.

Original: Moreover, while additional modalities, such as visual speech, can often boostthe performance and robustness on top of audio speech, models taking such input are less studied anddeployed in real world application.
Revised: Moreover, while additional modalities, such as visual speech, can often boost the performance and robustness on top of audio speech, models taking such input are less studied and deployed in real world application.

Original: This is mainly due to the lack of both labeled and unlabeled datafor these alternative modalities, hindering the development of high-performing self-supervised orsupervised modal-specific models that are trained exclusively on one type of input. We argue that to 
Revised: This is mainly due to the lack of both labeled and unlabeled data for these alternative modalities, hindering the development of high-performing self-supervised or supervised modal-specific models that are trained exclusively on one type of input. We argue that to tackle the data scarcity issue, it essential to utilize not only the speech from the target modality, but also speech in other modalities to build a

Original: U-HuBERT [13,13,5,5,5] [12,5,5,3] [4,4,4,9,5,5]
Revised: U-HuBERT [13,13,5,5,5] [12,5,5,3] [4,4,4,9,5,5]

Original: Joint clustering
Revised: Joint clustering

Original: Modality-agnosticfeatures
Revised: Modality-agnosticfeatures

Original: Pseudo labels Multimodal / unimodal input
Revised: Pseudo labels Multimodal / unimodal input

Original: Pseudo labeling
Revised: Pseudo labeling

Original: U-HuBERT U-HuBERT
Revised: U-HuBERT U-HuBERT

Original: AND/OR
Revised: AND/OR

Original: AND/ORtackle the data scarcity issue, it essential to utilize not only the speech from the target modality, butalso speech in other modalities to build a general speech model for many modalities.
Revised: AND/OR general speech model for many modalities.

Original: By doing so,more data can be used to learn components shared across different modalities, which is expected tobring bigger gains to low-resource modalities.
Revised: By doing so, more data can be used to learn components shared across different modalities, which is expected to bring bigger gains to low-resource modalities.

Original: In this paper, we present unified hidden unit BERT (u-HuBERT), which leverages unlabeled speechdata of many different modalities for pre-training, including both unimodal and multimodal speech.
Revised: Audio-visual HuBERT (AV-HuBERT) [Shi et al., 2022a] is a recent work progressing toward creating a versatile speech model: by pre-training on audio-visual speech, it demonstrates strong performance for not only audio-visual downstream tasks, but also audio-only and video-only tasks.

Original: Architecture-wise, u-HuBERT consists of a set of modality-specific feature extractors and a sharedtransformer backend.
Revised: However, it only explores pre-training exclusively on audio-visual speech, and fine-tuning and testing on matching modalities. In this paper, we generalize AV-HuBERT and present unified hidden unit BERT (u-HuBERT), which leverages unlabeled speech data of many different modalities for pre-training, including both unimodal and multimodal speech.

Original: The model is trained with a unified masked cluster prediction objective forboth unimodal and multimodal speech data.
Revised: The model is trained with a unified masked cluster prediction objective for both unimodal and multimodal speech data.

Original: Specifically, we adopt modality dropout [Neverovaet al., 2014] for multimodal data to simulate different combination of input modalities and encouragethe model to learn modality-agnostic frame-level representations. The pre-trained u-HuBERT canbe fine-tuned with either multimodal or unimodal data, and we demonstrate that in both cases theresulting model can be used for many speech modalities. 
Revised: The pre-trained u-HuBERT can be fine-tuned with either multimodal or unimodal data, and we demonstrate that in both cases the resulting model can be used for many speech modalities. Following AV-HuBERT, we apply modality dropout [Neverova et al., 2014] to multimodal data to simulate different combinations of input modalities. We present extensive analysis in this paper to show that modality dropout is crucial for learning modality-agnostic frame-level representation, which is the key to successful mixed modality pre-training and achieving good performance when fine-tuning and testing modalities mismatch.

Original: We verify the effectiveness on two popularspeech processing tasks: speech recognition and speaker verification.
Revised: We verify the effectiveness on two popular speech processing tasks: speech recognition and speech translation.

Original: To the best of our knowledge,this is the first single model for many speech modalities that yields performance on par with thestate-of-the-art modality-specific models and is capable of zero-shot modality generalization.
Revised: To the best of our knowledge, this is the first single model for many speech modalities that yields performance on par with the state-of-the-art modality-specific models and is capable of zero-shot modality generalization.

Original:  Related Work
Revised:  Related Work

Original: Comparison to AV-HuBERT Our work is an extension of AV-HuBERT, which studies pre-trainingexclusively with audio-visual data. While the model architecture and the form of pre-training objectiveare the same, u-HuBERT generalizes AV-HuBERT to utilize both multimodal and unimodal speechfor pre-training by mapping various input to a modality-agnostic feature space and creating a sharedtarget space for masked prediction. 
Revised: Comparison to [Shi et al., 2022a] u-HuBERT generalizes AV-HuBERT to utilize both multimodal and unimodal speech for pre-training by mapping various input to a modality-agnostic feature space and creating a shared target space for masked prediction. In contrast, because AV-HuBERT is pre-trained exclusively on audio-visual speech, it always generates targets using features of the same modality (audio in the first iteration and audio-visual for the subsequent iterations). Therefore, learning modality-agnostic features to produce shared target space is not essential for AV-HuBERT.

Original: We empirically demonstrate the benefits of including additionalunimodal data for pre-training. The fine-tuning and evaluation protocols also differ significantly. AV-HuBERT effectively learns modality-specific models (e.g., visual speech recognizer) given labeleddata in the target modality for fine-tuning. In contrast, this work focuses on building a single modelfor all modalities seen during pre-training with labeled data in any subset of modalities.
Revised: Moreover, this paper presents a novel evaluation scenario: zero-shot transfer to unlabeled modalities, which is of great practical importance because while labeled data may not always be available in the target modality, it will most likely be available in some other modalities.

Original:  Joint unimodal and multimodal pre-training Recent cross-modal contrastive approaches suchas CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have shown impressive results yet theycan only be trained with multimodal data and require a large-scale dataset, which can be hard toobtain for many speech modalities (e.g., paired EMG-audio speech).
Revised: We demonstrate in this paper that modality dropout is key to achieving good zero-shot transfer performance. Joint unimodal and multimodal pre-training Recent cross-modal contrastive approaches such as CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have shown impressive results yet they can only be trained with multimodal data and require a large-scale dataset, which can be hard to obtain for many speech modalities (e.g., paired EMG-audio speech).

Original: FLAVA [Singh et al., 2021]and SLAM [Bapna et al., 2021] are two frameworks that utilize both unimodal and multimodal datafor image-text and speech-text pre-training, respectively.
Revised: FLAVA [Singh et al., 2021] and SLAM [Bapna et al., 2021] are two frameworks that utilize both unimodal and multimodal data for image-text and speech-text pre-training, respectively.

Original: There are three key differences betweenu-HuBERT and these two models.
Revised: There are three key differences between u-HuBERT and these two models.

Original: First, for multimodal speech, different modalities such as audio andvisual speech are temporally aligned.
Revised: First, for multimodal speech, different modalities such as audio and visual speech are temporally aligned.

Original: Instead of concatenating the feature sequences from unimodalencoders, we fuse the features frame-by-frame before passing it to the shared encoder.
Revised: Instead of concatenating the feature sequences from unimodal encoders, we fuse the features frame-by-frame before passing it to the shared encoder.

Original: Second,while all three models utilize a masked token prediction objective, u-HuBERT considers a sharedtarget token space for all unimodal and multimodal data, obtained by clustering modality-agnosticfeatures extracted from the u-HuBERT model itself.
Revised: Second, while all three models utilize a masked token prediction objective, u-HuBERT considers a shared target token space for all unimodal and multimodal data, obtained by clustering modality-agnostic features extracted from the u-HuBERT model itself.

Original: In contrast, SLAM uses w2v-BERT [Chunget al., 2021] tokens for speech and SentencePiece [Schuster and Nakajima, 2012] for text.
Revised: In contrast, SLAM uses w2v-BERT [Chung et al., 2021] tokens for speech and SentencePiece [Schuster and Nakajima, 2012] for text.

Original: FLAVA onother hand uses a separately trained dVAE tokenizer for image and the same as SLAM for text, and itpredicts tokens for unimodal and multimodal data at the output of different modules, differing from u-
Revised: FLAVA on other hand uses a separately trained dVAE tokenizer for image and the same as SLAM for text, and it predicts tokens for unimodal and multimodal data at the output of different modules, differing from uHuBERT.

Original: HuBERT. In addition, both FLAVA and SLAM pre-train with a number of auxiliary objectives, whileu-HuBERT only uses one, which simplifies hyperparameter selection.
Revised: In addition, both FLAVA and SLAM pre-train with a number of auxiliary objectives, while u-HuBERT only uses one, which simplifies hyperparameter selection.

Original: Third, different modalities ofspeech are often used to solve the same task, such as speech recognition or speaker verification.
Revised: Third, different modalities of speech are often used to solve the same task.

Original: Ourevaluation focuses on building a single model that can solve the same task with different modalitiesdespite not having labeled data for each modality, which is not considered by SLAM or FLAVA.
Revised: Our evaluation focuses on building a single model that can solve the same task with different modalities despite not having labeled data for each modality, which is not considered by SLAM or FLAVA.

Original: Single model for many input modalities Makino et al.
Revised: Single model for many input modalities Makino et al.

Original: [2019] and Sari et al. [2021] exploredbuilding a single model for supervised audio-visual/audio/visual speech recognition and speakerverification, respectively.
Revised: [2019] explored building a single model for supervised audio-visual/audio/visual speech recognition.

Original: Both studies show significant degradation comparing unified models withmodality-specific ones.
Revised: The studies show significant degradation comparing unified models with modality-specific ones.

Original: Furthermore, these models are single-task supervised and hence not capable ofzero-shot modality transfer when trained on unimodal data.
Revised: Furthermore, these models are single-task supervised and hence not capable of zero-shot modality transfer when trained on unimodal data.

Original: Omnivore [Girdhar et al., 2022] is a singlesupervised model for many vision modalities, including image (RGB), 3D image (RGBD), and video(RGB sequence), which also considers a model architecture with modality-specific encoders (RGBand depth) followed by a shared Swin Transformer backend [Liu et al., 2021].
Revised: Omnivore [Girdhar et al., 2022] is a single supervised model for many vision modalities, including image (RGB), 3D image (RGBD), and video (RGB sequence), which also considers a model architecture with modality-specific encoders (RGB and depth) followed by a shared Swin Transformer backend [Liu et al., 2021].

Original: Transformer [Liuet al., 2021] to learn shared representations across modalities.

Original: Both Omnivore and u-HuBERTdemonstrates cross-modal generalization, but Omnivore achieves it through supervised learningwith different tasks, and u-HuBERT achieves it with unified self-supervised learning.
Revised: Both Omnivore and u-HuBERT demonstrates cross-modal generalization, but Omnivore achieves it through supervised learning with different tasks, and u-HuBERT achieves it with unified self-supervised learning.

Original: Finally, if weconsidered different languages as different modalities, our work is also related to mBERT [Devlinet al., 2019], mBART [Liu et al., 2020], TLM [Yao et al., 2021], and XLM [Lample and Conneau,2019], fine-tuning which enables generalizing natural language understanding [Pires et al., 2019]and machine translation [Johnson et al., 2017] to languages without labeled data.
Revised: Finally, if we considered different languages as different modalities, our work is also related to mBERT [Devlin et al., 2019], mBART [Liu et al., 2020], TLM [Yao et al., 2021], and XLM [Lample and Conneau, 2019], fine-tuning which enables generalizing natural language understanding [Pires et al., 2019] and machine translation [Johnson et al., 2017] to languages without labeled data.

Original: Pre-training onmultimodal speech is analogous to pre-training on bitext, while the former requires less manual effortto obtain by collecting more measurements when speech is produced.
Revised: Pre-training on multimodal speech is analogous to pre-training on bitext, while the former requires less manual effort to obtain by collecting more measurements when speech is produced.

Original: In addition, time-synchronicitybetween speech modalities provides additional self-supervision, enabling u-HuBERT to more easilylearn frame-level modality-agnostic representations.
Revised: In addition, time-synchronicity between modalities provides additional self-supervision, enabling u-HuBERT to more easily learn frame-level modality-agnostic features.

Original:  Method 
Revised:  Method 3.1 Background: audio-visual HuBERT

Original: 3.1 Background: audio-visual HuBERT

Original: We build our framework upon AV-HuBERT [Shi et al., 2022a], a self-supervised representationlearning model designed to pre-train a single type of multimodal speech data: audio-visual speech.
Revised: We build our framework upon AV-HuBERT [Shi et al., 2022a], a self-supervised representation learning model designed to pre-train a single type of multimodal speech data: audio-visual speech.

Original: AV-HuBERT first encodes temporally aligned audio speech X a ∈ R T a × C a and visual speech
Revised: AV-HuBERT first encodes temporally aligned audio speech X a ∈ R T a × C a and visual speech

Original: X v ∈ R T v × C v × H × W into feature sequences h a ∈ R T × D h and h v ∈ R T ×
Revised: X v ∈ R T v × C v × H × W into feature sequences h a ∈ R T × D h and h v ∈ R T ×

Original: D h of the same lengths,which are then fused frame-by-frame FUSE ( h a , h v ) ∈ R T × D c and processed by a Transformerbackend [Vaswani et al., 2017] to produce contextualized audio-visual representations c av ∈ R T × D c .
Revised: D h of the same lengths, which are then fused frame-by-frame FUSE ( h a , h v ) ∈ R T × D c and processed by a Transformer backend [Vaswani et al., 2017] to produce contextualized audio-visual representations c av ∈ R T × D c .

Original: This model is trained by iterating two steps.
Revised: This model is trained by iterating two steps.

Original: First, a K-means clustering model is trained on someaudio-visual features to produce frame-level targets Y av ∈ [1 , · · · , K ] T , where K is the number ofclusters.
Revised: First, a K-means clustering model is trained on some audio-visual features to produce frame-level targets Y av ∈ [1 , · · · , K ] T , where K is the number of clusters.

Original: Second, the AV-HuBERT model is trained with a masked cluster prediction task, wherespans from the audio-visual input are randomly masked and the model learns to predict the clusterassignment of the masked frames given the context.
Revised: Second, the AV-HuBERT model is trained with a masked cluster prediction task, where spans from the audio-visual input are randomly masked and the model learns to predict the clusters of the masked frames given the context.

Original: The model is forced to learn the underlyinglinguistic structure of audio-visual speech to infer the content in the unseen regions based on thecontext.
Revised: The model is forced to learn the underlying linguistic structure of audio-visual speech to infer the content in the unseen regions based on the context.

Original: The intermediate feature of a trained model, which is better than previous iteration, is usedto produce targets for the subsequent iteration.
Revised: The intermediate feature of a trained model, which is better than previous iteration, is used to produce targets for the subsequent iteration.

Original: The pre-trained AV-HuBERT model performs wellon both multimodal downstream tasks like audio-visual speech recognition [Shi et al., 2022b] andunimodal tasks like lip-reading [Shi et al., 2022a] when fine-tuned for a specific modality.

Original: 3.2 Self-supervised pre-training with many modalities
Revised: 3.2 Self-supervised pre-training with many modalities

Original: We extend the framework to utilize not only multimodal data, but also unimodal data that can bemore easily obtained in large quantities.
Revised: We extend the framework to utilize not only multimodal data, but also unimodal data that can be more easily obtained in large quantities.

Original: The main idea is to apply modality dropout to multimodalspeech along with a shared Transformer [Vaswani et al., 2017] to learn modality-agnostic featuresthat can be quantized by a shared codebook to produce pseudo labels for masked prediction.
Revised: The main idea is to apply modality dropout to multimodal speech along with a shared Transformer [Vaswani et al., 2017] to learn modality-agnostic features that can be quantized by a shared codebook to produce pseudo labels for masked prediction.

Original: Given unimodal and multimodal speech data from many modalities, we assume that there is an anchormodality that is paired with all other modalities.

Original: For simplicity, assume the anchor modality is audio,and we have unlabeled audio, visual, and audio-visual speech.

Original: In the first iteration, because a modalityagnostic feature extractor is not available yet, we cluster the anchor modality features to producetargets for unimodal and multimodal speech that contain the anchor modality.

Original: Concretely speaking,

Original: [Figure]
Revised: [Figure]

Original:  Transformer [13,13,5,5,5] [13,13,5,5,5] [13,13,5,5,5]
Revised: Shared Transformer [13,13,5,5,5] [13,13,5,5,5] [13,13,5,5,5]

Original: Audioencoder
Revised: Audioencoder

Original: [Figure]

Original:  fuse
Revised: Dummy fuse

Original: Shared Transformer [12,5,5,3]
Revised: Shared Transformer [12,5,5,3]

Original: [Figure]
Revised: [Figure]

Revised: Given unimodal and multimodal speech data from many modalities, we assume that there is an anchor modality that is paired with all other modalities.

Revised: For simplicity, assume the anchor modality is audio, and we have unlabeled audio, visual, and audio-visual speech.

Revised: In the first iteration, because a modalityagnostic feature extractor is not available yet, we cluster the anchor modality features to produce targets for unimodal and multimodal speech that contain the anchor modality.

Original:  this produces pseudo paired data for audio speech X a and audio-visual speech ( X a , X v ) with audiobased cluster assignment Y a as the target.
Revised: Concretely speaking, this produces pseudo paired data for audio speech X a and audio-visual speech ( X a , X v ) with audiobased cluster assignment Y a as the target.

Original: Like AV-HuBERT, each modality has its own featureextractor: f a for audio and f v for video, and feature sequences are fused frame-by-frame before beingpassed onto the Transformer g to produce contextualized features c av = g ( FUSE ( f a ( X a ) , f v ( X v ))) .
Revised: Like AV-HuBERT, each modality has its own feature extractor: f a for audio and f v for video, and feature sequences are fused frame-by-frame before being passed onto the Transformer g to produce contextualized features c av = g ( FUSE ( f a ( X a ) , f v ( X v ))) .

Original: When a modality is absent, its feature sequence is simply replaced by zero vectors, for example,c a = g ( FUSE ( f a ( X a ) , 0 )) .
Revised: When a modality is absent, its feature sequence is simply replaced by zero vectors, for example, c a = g ( FUSE ( f a ( X a ) , 0 )) .

Original: Following HuBERT [Hsu et al., 2021a], this model is trained with amasked prediction objective.
Revised: Following HuBERT [Hsu et al., 2021a], this model is trained with a masked prediction objective.

Original: To encourage the model to learn modality-agnostic features, modality dropout is applied to multimodaldata to randomly drop a subset of the modalities, effectively creating multiple copies of the data withthe same target but different input modalities, as depicted in the top half of Figure 2.
Revised: To encourage the model to learn modality-agnostic features, modality dropout is applied to multimodal data to randomly drop a subset of the modalities, effectively creating multiple copies of the data with the same target but different input modalities, as depicted in the top half of Figure 2.

Original: While modalitydropout was also used in AV-HuBERT, it was implemented for a different reason – to reduce thediscrepancy between pre-training and fine-tuning for unimodal downstream tasks, which resultedin only minor improvement (reducing the WER from 57.0% to 55.3% as shown in Table D.1 in Shiet al.
Revised: While modality dropout was also used in AV-HuBERT, it was implemented for a different reason – to reduce the discrepancy between pre-training and fine-tuning for unimodal downstream tasks, which resulted in only minor improvement (reducing the WER from 57.0% to 55.3% as shown in Table D.1 in Shi et al.

Original: [2022a]).
Revised: [2022a]).

Original: In contrast, we will demonstrate that modality dropout is essential for mixed modalitypre-training and zero-shot generalization.
Revised: In contrast, we will demonstrate that modality dropout is essential for mixed modality pre-training and zero-shot generalization.

Original: After the first iteration, the model can be used to extract modality-agnostic features for all unimodaland multimodal data, which are clustered to produce the pseudo label for next iteration of pre-trainingas illustrated in Figure 1.
Revised: After the first iteration, the model can be used to extract modality-agnostic features for all unimodal and multimodal data, which are clustered to produce the pseudo label for next iteration of pre-training as illustrated in Figure 1.

Original: For multimodal data like audio-visual speech, one can produce pseudolabels using features extracted from the complete multimodal input (e.g., Y av from audio-visualinput), or those extracted from input with a subset of modalities (e.g., Y v using only the video stream).
Revised: For multimodal data like audio-visual speech, one can produce pseudo labels using features extracted from the complete multimodal input (e.g., Y av from audio-visual input), or those extracted from input with a subset of modalities (e.g., Y v using only the video stream).

Original: We opt for the former and study the impact of these choices in Secton 4.2.
Revised: We opt for the former and study the impact of these choices in Secton 4.2.

Original: 3.3 Multimodal and unimodal fine-tuning with partial modalities
Revised: 3.3 Multimodal and unimodal fine-tuning with partial modalities

Original: Once the model is pre-trained, we remove the cluster prediction head (the linear layer on top oftransformer used to predict cluster assignment distribution) and add a downstream task-specificprediction head.
Revised: Once the model is pre-trained, we remove the cluster prediction head (the linear layer on top of transformer used to predict cluster assignment distribution) and add a downstream task-specific prediction head.

Original: The pre-trained model can be fine-tuned on labeled multimodal speech, unimodalspeech, or speech with mixed modalities.
Revised: The pre-trained model can be fine-tuned on labeled multimodal speech, unimodal speech, or speech with mixed modalities.

Original: More importantly, the modalities included in the finetuning data do not necessarily cover the modalities seen during pre-training, yet the fine-tuned modelcan still handle the downstream task on all pre-trained modalities, which we refer to as zero-shotmodality generalization .
Revised: More importantly, the modalities included in the finetuning data do not necessarily cover the modalities seen during pre-training, yet the fine-tuned model can still handle the downstream task on all pre-trained modalities, which we refer to as zero-shot modality generalization .

Original: This is because the pre-trained model has successfully learned modalityagnostic representations, where same speech manifested in different modalities are mapped to similarrepresentations.
Revised: This is because the pre-trained model has successfully learned modalityagnostic representations, where same speech manifested in different modalities are mapped to similar representations.

Original: Thus mapping from the representationof one modality (e.g., audio) to a target output(e.g. , text) can be directly applied to other pre-trained modalities (e.g., video).
Revised: Thus mapping from the representation

Original: Similar to mBART [Liu et al., 2020] for zero-shot machine translation, catastrophic forgetting [Leeet al., 2019, Üstün et al., 2021] may occur during fine-tuning and hence hinders its generalizationacross modalities.
Revised: Similar to mBART [Liu et al., 2020] for zero-shot machine translation, catastrophic forgetting [Lee et al., 2019, Üstün et al., 2021] may occur during fine-tuning and hence hinders its generalization across modalities.

Original: Particularly, this is because the distributions of Transformer input and output atearlier layers are less modality-invariant.
Revised: Particularly, this is because the distributions of Transformer input and output at earlier layers are less modality-invariant.

Original: To alleviate the issue, we apply modality dropout whenmultimodal data is used, and explore a few techniques to prevent the representations from drifting toomuch, including freezing the entire pre-trained model for a number of steps and freezing a number of initial layers (up to where more modality-agnostic representations are produced) throughoutfine-tuning.
Revised: To alleviate the issue, we apply modality dropout when multimodal data is used, and explore a few techniques to prevent the representations from drifting too much, including freezing the entire pre-trained model or freezing a number of initial layers.

Original:  Experiments4.1 Pre-training setup
Revised:  Experiments 4.1 Pre-training setup

Original: We evaluate u-HuBERT on audio, visual, and audio-visual speech, where larger quantities of unlabeledand labeled data are available.
Revised: We evaluate u-HuBERT on audio, visual, and audio-visual speech, where larger quantities of unlabeled and labeled data are available.

Original: Two audio-visual and one audio-only datasets are used for pre-training:(1) LRS3 [Afouras et al., 2018] with 433 hours of English audio-visual speech, (2) VoxCeleb2-En(VC2-En) with 1,326 hours of English YouTube audio-visual speech filtered from VoxCeleb2 [Chunget al., 2018] by Shi et al.
Revised: Two audio-visual and one audio-only datasets are used for pre-training: (1) LRS3 [Afouras et al., 2018] with 433 hours of English audio-visual speech, (2) VoxCeleb2-En (VC2-En) with 1,326 hours of English YouTube audio-visual speech filtered from VoxCeleb2 [Chung et al., 2018] by Shi et al.

Original: [2022a], and (3) TED-LIUM release 3 (TD)
Revised: [2022a], and (3) TED-LIUM release 3 (TD)

Original: [Hernandez et al., 2018] with452 hours of English audio collected from the same domain as LRS3.
Revised: [Hernandez et al., 2018] with 452 hours of English audio collected from the same domain as LRS3.

Original: To improve noise robustness,we apply online noise augmentation where each utterance is corrupted with additive noise at 0dBsampled from MUSAN [Snyder et al., 2015] with a probability of 0.25.
Revised: To improve noise robustness, we apply online noise augmentation where each utterance is corrupted with additive noise at 0dB sampled from MUSAN [Snyder et al., 2015] with a probability of 0.25.

Original: More details can be found in
Revised: More details can be found in

Revised: Appendix A. These datasets are solely used to perform experiments for the purpose of comparing with previous work in a fair setup.

Original: Appendix A. We adopt the AV-HuBERT-LARGE model Shi et al.
Revised: We adopt the AV-HuBERT-LARGE model Shi et al.

Original: [2022a] 1 : the video encoder is amodified ResNet-18 model [He et al., 2016] that takes lip region-of-interest (ROI) frames  sampled at25Hz at input and produce a sequence of 512 dimensional embeddings at the same frame rate; audiois represented as filter bank features sampled at 100 Hz, stacked every four frames to match the videoframe rate, and projected with a linear layer to 1024-D features.
Revised: [2022a] 2 : the video encoder is a modified ResNet-18 model [He et al., 2016] that takes image frames of 88x88 pixels sampled at 25Hz at input and produce a sequence of 512 dimensional embeddings at the same frame rate; audio is represented as filter bank features sampled at 100 Hz, stacked every four frames to match the video frame rate, and projected with a linear layer to 1024-D features.

Original: FUSE concatenates features frameby-frame and linearly projects them to the embedding dimension of the shared Transformer.
Revised: FUSE concatenates features frame-by-frame and linearly projects them to the embedding dimension of the shared Transformer.

Original: Theshared Transformer has 24 layers, with 16 heads, embedding dimension 1024, and FFN dimension4096, using the pre-norm residual connection setup [Nguyen and Salazar, 2019].
Revised: The shared Transformer has 24 layers, with 16 heads, embedding dimension 1024, and FFN dimension 4096, using the pre-norm residual connection setup [Nguyen and Salazar, 2019].

Original: To directly compare with the publicly available fifth iteration noise-augmented AV-HuBERT-LARGEmodel trained on LRS3 and VC2-En, the same feature extractor (fourth iteration BASE model) andthe same codebook (learned from multimodal features C av ) are used to generate pseudo labels formultimodal LRS3 and VC2-En and unimodal TD.
Revised: To directly compare with the publicly available fifth iteration noise-augmented AV-HuBERT-LARGE model trained on LRS3 and VC2-En, the same feature extractor (fourth iteration BASE model) and the same codebook (learned from multimodal features C av ) are used to generate pseudo labels for multimodal LRS3 and VC2-En and unimodal TD.

Original: We then pre-train u-HuBERT on the combineddata for 1M updates on 64 32GB V100 GPUs using the Adam optimizer [Kingma and Ba, 2015] anda learning rate of 0.002.
Revised: We then pre-train u-HuBERT on the combined data for 1M updates on 64 32GB V100 GPUs using the Adam optimizer [Kingma and Ba, 2015] and a learning rate of 0.002.

Original: Gradient norm is clipped at 1.0. to stablize training.
Revised: Gradient norm is clipped at 1.0. to stablize training.

Original: A batch size of maximal40 seconds per GPU is used.
Revised: A batch size of maximal 40 seconds per GPU is used.

Original: When pre-trained on multimodal data, audio and video are droppedwith a probability of 0.25 and 0.25, respectively.
Revised: When pre-trained on multimodal data, audio and video are dropped with a probability of 0.25 and 0.25, respectively.

Original: 4.2 Learning modality-agnostic features and codebooks with modality dropout
Revised: 4.2 Learning modality-agnostic features and codebooks with modality dropout

Original: We argue that modality dropout leads to learning modality-agnostic representations through forcingthe model to predict the same target with different subsets of input modalities given multimodalinput, which in turn enables construction of a shared codebook for all modalities.
Revised: We argue that modality dropout leads to learning modality-agnostic representations through forcing the model to predict the same target with different subsets of input modalities given multimodal input, which in turn enables construction of a shared codebook for all modalities.

Original: To verify this, wecompare two models that are identical except for whether modality dropout is activated: the publicnoise-augmented AV-HuBERT LARGE model trained on LRS3 and VC2-En with modality dropoutand its counterpart.
Revised: To verify this, we compare two models that are identical except for whether modality dropout is activated: the public noise-augmented AV-HuBERT LARGE model trained on LRS3 and VC2-En with modality dropout and its counterpart.

Original: For each model, we infer audio, visual, and audio-visual features for each LRS3validation utterances, randomly sample 500 frames, pool the features, and run t-SNE [Van der Maatenand Hinton, 2008] dimension reduction for visualization (Figure 3).
Revised: For each model, we infer audio, visual, and audio-visual features for each LRS validation utterances, randomly sample 500 frames, pool the features, and run t-SNE [Van der Maaten and Hinton, 2008] dimension reduction for visualization (Figure 3).

Original: It is visually evident that withoutmodality dropout, features extracted from different modalities have different distributions (bottomrow), contrary to those from the model with modality dropout (top row).
Revised: It is visually evident that without modality dropout, features extracted from different modalities have different distributions (bottom row), contrary to those from the model with modality dropout (top row).

Original: We next adopt the phone normalized mutual information (PNMI) metric presented in Hsu et al.[2021a] to quantify cluster quality.
Revised: We next adopt the phone normalized mutual information (PNMI) metric presented in Hsu et al. [2021a] to quantify cluster quality.

Original: For each model, we create four codebooks by clustering audiovisual features C av , audio features C a , visual features C v , or all three combined ∪ m C m , quantizeeach type of features with each codebook, and measure the normalized mutual information ∈ [0 , 1]between frame-level phone labels and cluster assignment (Table 1).
Revised: For each model, we create four codebooks by clustering audiovisual features C av , audio features C a , visual features C v , or all three combined ∪ m C m , quantize each type of features with each codebook, and measure the normalized mutual information ∈ [0 , 1] between frame-level phone labels and cluster assignment (Table 1).

Original: Because features inferred fromdifferent modalities exhibit similar distributions with modality dropout, all four codebooks lead tosimilar PNMI for a given feature.
Revised: Because features inferred from different modalities exhibit similar distributions with modality dropout, all four codebooks lead to similar PNMI for a given feature.

Original: In contrast, without modality dropout, cross-quantization (e.g., Y avwith C v codebook) often leads to significantly worse PNMI, and PNMI is worse for all (codebook,feature) combinations compared with that from the modality dropout model.
Revised: In contrast, without modality dropout, cross-quantization (e.g., Y av with C v codebook) often leads to significantly worse PNMI, and PNMI is worse for all (codebook, feature) combinations compared with that from the modality dropout model.

Original: Note for a givencodebook, the PNMI still differs among modalities which is due to the discrepancy of informationcarried per modality.[Figure] [Table] [Figure]
Revised: Note for a given codebook, the PNMI still differs among modalities which is due to the discrepancy of information carried per modality.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 4.3 Speech recognition
Revised: 4.3 Speech recognition

Original: Fine-tuning setup LRS3 trainval and pretrain , combined for 433 hours, are used for training, withthe same 1,200 utterances as Shi et al.
Revised: Fine-tuning setup LRS3 trainval and pretrain , combined for 433 hours, are used for training, with the same 1,200 utterances as Shi et al.

Original: [2022a] split for validation.
Revised: [2022a] split for validation.

Original: Results on the test split are reported.
Revised: Results on the test split are reported.

Original: Noise augmentation with MUSAN used in pre-training is also adopted during fine-tuning.
Revised: Noise augmentation with MUSAN used in pre-training is also adopted during fine-tuning.

Original: A 9-layerrandomly initialized Transformer decoder with 8 attention heads and 1024/4096-D embedding/FFNis appended to the pre-trained u-HuBERT, and the entire model is fine-tuned with a cross-entropyloss predicting the next text token given the input and the previous text tokens.
Revised: A 9-layer randomly initialized Transformer decoder with 8 attention heads and 1024/4096-D embedding/FFN is appended to the pre-trained u-HuBERT, and the entire model is fine-tuned with a cross-entropy loss predicting the next text token given the input and the previous text tokens.

Original: Text is representedas unigram-based subword units [Kudo, 2018] with a vocabulary size of 1,000.
Revised: Text is represented as unigram-based subword units [Kudo, 2018] with a vocabulary size of 1,000.

Original: Modality dropoutis applied when fine-tuning on multimodal input.
Revised: Modality dropout is applied when fine-tuning on multimodal input.

Original: Additionally, we sweep learning rate, numberof updates, steps to freeze, and layers to freeze throughout fine-tuning.
Revised: Additionally, we sweep learning rate, number of updates, steps to freeze, and layers to freeze throughout fine-tuning.

Original: We evaluate each model,regardless of the modality used for fine-tuning, on all kinds of input modalities and report the worderror rates (WERs) without fusing external language models.
Revised: We evaluate each model, regardless of the modality used for fine-tuning, on all kinds of input modalities and report the word error rates (WERs) without fusing external language models.

Original: To gauge the robustness to noise, wefollow the same protocol in Shi et al.
Revised: To gauge the robustness to noise, we follow the same protocol in Shi et al.

Original: [2022b] to noisy audio and audio-visual test sets by addingbabble noise at 0 decibel (dB) to the audio stream for evaluation.
Revised: [2022b] to noisy audio and audio-visual test sets by adding babble noise at 0 decibel (dB) to the audio stream for evaluation.

Original: Detailed hyperpamater setups canbe found in Appendix A, and more ablation studies can be found in Appendix B.1.
Revised: Detailed hyperpamater setups can be found in Appendix A, and more ablation studies can be found in Appendix B.1.

Original: Modality dropout is essential for building a single model Table 2 compares fine-tuning fromscratch, as well as from pre-trained models with and without modality dropout, using labeled audio,visual, or audio-visual speech, and highlights the results of zero-shot modality transfer where labeleddata in test modality are not used.
Revised: Modality dropout is essential for building a single model Table 2 compares fine-tuning from scratch, as well as from pre-trained models with and without modality dropout, using labeled audio, visual, or audio-visual speech, and highlights the results of zero-shot modality transfer where labeled data in test modality are not used.

Original: First, models without pre-training are significantly worse.
Revised: First, models without pre-training are significantly worse.

Original: Second,when fine-tuning on audio-visual data without  modality dropout, performance drops significantly onunimodal data if pre-training modality dropout is not applied (e.g., 1.7% to 21.4% on clean audio).
Revised: Second, when fine-tuning on audio-visual data without pre-training modality dropout, performance drops significantly on unimodal data if pre-training modality dropout is not applied (e.g., 1.7% to 21.4% on clean audio).

Original: This can be alleviated with fine-tuning modality dropout, improving the average WER from 24.5% to12.3%, but is still behind applying it in both pre-training and fine-tuning (11.4% average).
Revised: This can be alleviated with fine-tuning modality dropout, improving the average WER from 24.5% to 12.3%, but is still behind applying it in both pre-training and fine-tuning (11.4% average).

Original:  The benefit of applying pre-training modality dropout is magnified in unimodal fine-tuning: the modelfine-tuned only on labeled audio data yields the same performance on clean and noisy audio-visualinput (1.3% and 4.6%) as the model fine-tuned with labeled audio-visual data, while being slightbehind on video input compared to the model fine-tuned with labeled video data (31.6% vs. 28.7%).
Revised: We also note that fine-tuning dropout leads to consistent improvement on noisy AV test set, likely because occasionally dropping the audio stream encourages the model to rely more on the visual input which improves noise robustness. The benefit of applying pre-training modality dropout is magnified in unimodal fine-tuning: the model fine-tuned only on labeled audio data yields the same performance on clean and noisy audio-visual input (1.3% and 4.6%) as the model fine-tuned with labeled audio-visual data, while being slight behind on video input compared to the model fine-tuned with labeled video data (31.6% vs. 28.7%).

Original: When it is compared to the model pre-trained without modality dropout and fine-tuned also withlabeled audio, huge gains are observed on zero-shot scenarios, for example 96.8% versus 31.6%
Revised: When it is compared to the model pre-trained without modality dropout and fine-tuned also with labeled audio, huge gains are observed on zero-shot scenarios, for example 96.8% versus 31.6%

Original: V-WER and 18.0% versus 4.6% noisy AV-WER.
Revised: V-WER and 18.0% versus 4.6% noisy AV-WER.

Original: Similar trend can be observed when fine-tuningwith labeled visual speech. 
Revised: Similar trend can be observed when fine-tuning with labeled visual speech. [Table]

Original:  Zero-shot modality transfer is not achieved through memorization Because the dataset used forfine-tuning is a subset of the pre-training data, there could be suspicions that the model memorizesthe audio-visual pairs from pre-training and associates them with the labeled unimodal data to enablezero-shot transfer.
Revised: [Table] n [Table] Zero-shot modality transfer is not achieved through memorization Because the dataset used for fine-tuning is a subset of the pre-training data, there could be suspicions that the model memorizes the audio-visual pairs from pre-training and associates them with the labeled unimodal data to enable zero-shot transfer.

Original: To avoid cheating, we fine-tune our model on LibriSpeech [Panayotov et al., 2015],an audio-only read audiobook dataset that is out-of-domain relative to the pre-training data composedof oratory talks (Table 3).
Revised: To avoid cheating, we fine-tune our model on LibriSpeech [Panayotov et al., 2015], an audio-only read audiobook dataset that is out-of-domain relative to the pre-training data composed of oratory talks (Table 3).

Original: Despite the degradation compared to fine-tuning on in-domain LRS3 data,the model still performs decently on zero-shot transfer scenarios compared to models pre-trainedwithout modality dropout in Table 2. [Table] [Table]
Revised: Despite the degradation compared to fine-tuning on in-domain LRS3 data, the model still performs decently on zero-shot transfer scenarios compared to models pre-trained without modality dropout in Table 2 and 3.

Original: [Table]
Revised: [Table]

Original: Comparison with state-of-the-art models Table 4 compares u-HuBERT with models from theliterature, all of which are modality-specific.
Revised: Comparison with state-of-the-art models Table 4 compares u-HuBERT with models from the literature, all of which are modality-specific.

Original: In contrast, u-HuBERT is the only single model that canbe fine-tuned on any modality and test on all modalities while achieving performance on par or betterthan the best modality-specific models.
Revised: In contrast, u-HuBERT is the only single model that can be fine-tuned on any modality and test on all modalities while achieving performance on par or better than the best modality-specific models using similar amounts of labeled data.

Original: Furthermore, compared to AV-HuBERT, u-HuBERT can bepre-trained on additional unimodal data to yield even better performance.[Table]
Revised: Furthermore, compared to AV-HuBERT, u-HuBERT can be pre-trained on additional unimodal data to yield even better performance.

Original: 4.4 Speaker verification
Revised: 4.4 Speech translation

Original: We repeat the experiment on speaker verification to study if u-HuBERT enables construction of asingle model and achieves zero-shot generalization on a different task.
Revised: We repeat the experiment on speech translation to study if u-HuBERT enables construction of a single model and achieves zero-shot generalization on a different task.

Original: Fine-tuning setup We follow the protocol of SUPERB
Revised: Fine-tuning setup To simulate speech translation setting, we translate the English transcriptions into

Original: [Yang et al., 2021], a standard benchmark forself-supervised speech representation evaluation.
Revised: Spanish to create paired (English audio, English video, Spanish text) data.

Original: Specifically, we use VoxCeleb1 (VC1) [Nagraniet al., 2017] which contains 352 hours of speech from 1,251 speakers for fine-tuning. Here theutterance-level speaker identity is  the target label.
Revised: The 30 hour LRS3 trainval split is used for fine-tuning. We use the same decoder for speech translation as for speech recognition, where a 9-layer randomly initialized Transformer is appended to the pretrained u-HuBERT encoder to decode English speech into Spanish text.

Original: For visual modality, we use the face image framesprovided by Nagrani et al.
Revised: Unigram-based sentencepiece units with a vocabulary size of 1,000 are used as decoding output units.

Original: [2018] and follow the samepreprocessing steps as Shi et al. [2022c]to get the lip regions.
Revised: The overall fine-tuning and decoding paradigms remain same as speech recognition.

Original: Noise augmentation following Shi et al.
Revised: Detailed hyperparameters can be found in Appendix A. BLEU score Papineni et al.

Original: [2022b,c] is used to simulate noisyenvironments.
Revised: [2002] is used to evaluate the translation model. [Table]

Original:  The noise is sampled from {Babble, Speech, Music, Noise} categories in MUSANcorpus
Revised: Translation performance The performance of u-HuBERT on speech translation is shown in Table [Table]

Original: [Snyder et al., 2015] at SNR in {-10, -5, 0, 5, 10} dB and we report the average performanceover 20 configurations. Speaker verification aims toevaluate how well the utterance-level representation captures speakersimilarity based on some distance metric in  the space. An X-vector
Revised: As is similar to the observation in speech recognition, the model hardly produces meaningful results if its encoder is randomly initialized (Avg-BLEU: 3.2).

Original: [Snyder et al., 2018] network isadded following the SUPERB protocol, which first weighted-sums frame-level u-HuBERT representations over all Transformer layers, and then pools them temporally into a sequence-level representation. The X-vector is randomly initialized and only added during fine-tuning.
Revised: u-HuBERT pre-training not only greatly improves translation quality by roughly 10 fold (Avg-BLEU: 3.2 → 33.0), but also shows strong performance in zero-shot modality transfer. For instance, the model fine-tuned with audio-only speech achieves 28.7 BLEU score when it is tested using audio-visual speech under noisy environment, outperforming its audio-only counterpart (23.9).

Original: The model is trained withthe widely used AMS-Softmax loss [Wang et al., 2018] with scale and margin being set to 0.4and 30 respectively. For evaluation, we compute the cosine similarity between two utterances andmeasure the equal error rate (EER). During fine-tuning, the u-HuBERT model is frozen while onlythe downstream X-Vector network is fine-tuned.
Revised: In tasks such as speech translation, where labeled multimodal data can be difficult to harvest, our pre-training approach shows potential in enhancing the noise robustness supplied by the extra visual modality at test time even without exposure to any labeled visual data during training. Within u-HuBERT, modality dropout is an essential component for acquiring such capacity, as is suggested by improvement in average BLEU score it brings (A: 23.1 → 31.9, V: 17.1 → 29.0).

Original: The model is trained with Adam [Kingma and Ba,2015].
Revised: In cases with available labeled multimodal data, a single audio-visual model is able to achieve performance superior or on par to a modality-specific model (A: 40.2 vs.

Original: The learning rate is searched within the range of [1e-7, 1e-1] and is kept fixed throughouttraining.
Revised: 41.0, V: 27.2 vs. 26.1)

Original: Note the above fine-tuning and evaluation paradigm is based on SUPERB, which is for adirect comparison with prior works on the same benchmark. See Appendix A and B.2.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Zero-shot performance Good audio-visual speaker representations aim to capture a number of relatedfeatures, such as voice, appearance, and speaking style (wording, intonation, rhythm).
Revised: 4.5 Ablation studies

Original: However, incontrast to speech recognition where content is highly correlated with each modality, forspeakerverification each modality by itself can only infer a subset of these speaker features (e.g., appearancecan barely be inferred from audio). This imposes an inherent limitation to zero-shot transfer.
Revised: Pre-training modality dropout parameters To better understand the impact of pre-training modality dropout configurations on zero-shot modality transfer, we conducted pre-training ablation studies with a reduced setup: for each configuration, a BASE model is pre-trained on 433 hours of LRS3 data using 32 GPUs for 200K updates and then fine-tuned on 30 hours of LRS3 labeled data (the “trainval” split).

Original: We present the results in Table 5. We found u-HuBERT fine-tuned with audio-only data can be used foraudio-visual speaker verification and yield better performance.
Revised: Four modality dropout configurations are considered, represented by the probability of using both streams ( p av ), audio stream ( p a ) and video stream ( p v ) in Table 6.

Original: In particular, it achieves 3.85%/8.92% EER with audio-visual input compared to 4.25%/14.46% with audio input on clean/noisy test set. With labeled audio data, the model learns to map voice and speaking style features to speakerrepresentations.
Revised: The first configuration ( p av , p a , p v ) = (1.00, 0.00, 0.00) is equivalent to not using modality dropout.

Original: Hence, when tested with additional visual input, the model gains robustness oninferring voice and speaking style features, such that the  performance on noisy test set is significantlyimproved (14.46% → 8.92% EER).
Revised: When fine-tuning on audio-visual or audio-only data, among the models pre-trained with modality dropout, setting the p v higher leads to slightly worse performance on audio-visual and audio-only test sets.

Original: However, this model does not learn to map appearance features(which could still be preserved in early Transformer layers that the X-vector head also takes asinput [Chang et al., 2022]) to speaker representations. As a result, the model performs much worsewith video input (17.90%) compared to the model fine-tuned on visual speech (5.01%), which learnsto map appearance and potentially speaking style features to speaker representations. Similarly, the audio fine-tuned model is also slightly behind audio-visual fine-tuned model when tested on audiovisual input (3.85% vs. 2.54%), as the latter could learn to map all features to speaker representations. On the other hand, when fine-tuning u-HuBERT with only visual speech, the model struggles togeneralize for audio-visual and audio input because mapping from voice to speaker is not learned.
Revised: In contrast, when fine-tuning on visual-only data, setting the probability of V-only higher leads to slightly better average performance.

Original: Effect of modality dropout Modality dropout plays an important role in building a single modelwith multimodal labeled data and enabling zero-shot transfer with only unimodal labeled data.
Revised: Nevertheless, all three configurations achieve significantly better performance compared to the model pre-trained without modality dropout, confirming that activating modality dropout is the key to successful zero-shot modality transfer. [Table] [Table]

Original: Inunimodal fine-tuning (audio-only or video-only), the performance consistently drops across allsettings but one (noisy AV-EER of model fine-tuned on V) when the model is pre-trained withoutmodality dropout.
Revised: [Table]

Original: In audio-visual fine-tuning, modality dropout greatly improves theperformanceof a unified audio-visual model under single-modal test setting (A-EER: 7.40% → 4.23%, V-EER:10.73% → 6.53%). Generally, the single model  trained  with modality dropout in both pre-trainingand fine-tuning achieves a comparable performance compared to modality-specific systems (2.54%vs.
Revised: [Table] Pre-training dynamics with additional unimodal data While mixed modality pre-training allows the model to be trained on larger quantities of data, it also dilutes the in-domain data with respect to audio-visual or other unimodal tasks, which may lead to worse performance on those tasks. We pre-train on audio-visual data and mixed-modal data (audio-visual and audio-only) respectively for different numbers of updates and study if degradation occurs to audio-visual and visual speech recognition in the mixed-modal case.

Original: 2.31% in clean AV-EER, 4.23% vs. 4.25% in clean A-EER, 6.52% vs. 5.00% in V-EER).
Revised: The baseline (PT on AV) is pre-trained on audio-visual LRS3, and the proposed mixed-modal pretraining (PT on AV+A) uses the same audio-visual data as the baseline plus the VC2-En audio data. For each setup we pre-train the model for {100, 200, 400, 600}K updates. Results in Figure 4 showed that adding unlabeled audio data consistently improves the audio speech recognition performance at all number of updates. In contrast, the performance of “PT on AV+A” lags behind “PT on

Original: Thevalue of dropout can be tuned in order to favor test performance ofone specific modality, which weshow in the Appendix B.1.1.
Revised: A” on audio-visual and visual speech recognition when a model is trained for fewer than 400K updates. Nevertheless, when a model is trained for longer (600K updates), mixed-modal pre-training outperforms the baseline on all three tasks. The observation suggests that u-HuBERT can benefit from training on more data and continue improving with more updates, while the performance of the baseline saturates earlier due to the lack of data.

Original:  Discussion
Revised:  Discussion

Original: We assume that there is an anchor modality where all other modalities appear in some multimodaldata paired with it.
Revised: We assume that there is an anchor modality where all other modalities appear in some multimodal data paired with it.

Original: This enables the model to predict shared targets and learn modality-agnosticrepresentations for all modalities, such that non-anchor unimodal data can be used in the subsequentiterations.
Revised: This enables the model to predict shared targets and learn modality-agnostic representations for all modalities, such that non-anchor unimodal data can be used in the subsequent iterations.

Original: While this may seem limiting, most multimodal speech are paired with audio [Richardet al., 2021, Hueber et al., 2010, Richmond et al., 2011, Livescu et al., 2009] since audio is the primarymeasurement for speech and other modalities are often supplementary.
Revised: While this may seem limiting, most multimodal speech are paired with audio [Richard [Figure] [Figure] et al., 2021, Hueber et al., 2010, Richmond et al., 2011, Livescu et al., 2009] since audio is the primary measurement for speech and other modalities are often supplementary.

Original: This makes the assumptionof having audio as the anchor modality practical.
Revised: This makes the assumption of having audio as the anchor modality practical.

Original: Furthermore, the idea can be easily extended tosettings where any modality can be connected to the anchor via some intermediate modality.
Revised: Furthermore, the idea can be easily extended to settings where any modality can be connected to the anchor via some intermediate modality.

Original: In short,a modality that is N -hop away from the primary anchor can be used in the ( N + 1) -th iteration.
Revised: In short, a modality that is N -hop away from the primary anchor can be used in the ( N + 1) -th iteration.

Original: The only case not supported u-HuBERT is pre-training on modalities that cannot be connectedthrough any multimodal data, for example, on a collection of audio, visual, audio-visual and EMGspeech.
Revised: The only case not supported u-HuBERT is pre-training on modalities that cannot be connected through any multimodal data, for example, on a collection of audio, visual, audio-visual and EMG speech.

Original: A shared codebook cannot be derived for EMG and the rest of the modalities, and it isunclear if the model can still benefit from joint training.
Revised: A shared codebook cannot be derived for EMG and the rest of the modalities, and it is unclear if the model can still benefit from joint training.

Original: The setup has also been found much morechallenging on other pairs of modalities: for example, SLAM Bapna et al.
Revised: The setup has also been found much more challenging on other pairs of modalities: for example, SLAM Bapna et al.

Original: [2021] shows that withoutpaired text and speech data, joint pre-training on unlabeled speech and text degrades the performancecompared to a model pre-trained only on speech.
Revised: [2021] shows that without paired text and speech data, joint pre-training on unlabeled speech and text degrades the performance compared to a model pre-trained only on speech.

Original: We leave such exploration to future work.
Revised: We leave such exploration to future work.

Original: Inaddition, we also include ethical discussion in Appendix C.
Revised: In addition, we also include ethical discussion in Appendix C.

Original:  Conclusion
Revised:  Conclusion

Original: Recent studies in applied machine learning trend toward building things that are general , for example,general perception module [Jaegle et al., 2021], general self-supervised objective [Baevski et al. ,2022], single model for many tasks [Brown et al., 2020, Kaplan et al., 2020, Aghajanyan et al. ,2022] and/or many input [Girdhar et al., 2022, Hu and Singh, 2021], instead of optimizing forperformance on very specific setups.
Revised: Recent studies in applied machine learning trend toward building things that are general , for example, general perception module [Jaegle et al., 2021], general self-supervised objective [Baevski et al., 2022], single model for many tasks [Brown et al., 2020, Kaplan et al., 2020, Aghajanyan et al., 2022] and/or many input [Girdhar et al., 2022, Hu and Singh, 2021], instead of optimizing for performance on very specific setups.

Original: Our work contributes to this line of research by presenting aunified self-supervised objective to pre-train a model on speech of many modalities.
Revised: Our work contributes to this line of research by presenting a unified self-supervised objective to pre-train a model on speech of many modalities.

Original: Specifically, itcan be fine-tuned for many different speech tasks on labeled data in any combination of pre-trainedmodalities, resulting in a single model that can process all combinations of pre-trained modalities toperform the fine-tuned task.
Revised: Specifically, it can be fine-tuned for many different speech tasks on labeled data in any combination of pre-trained modalities, resulting in a single model that can process all combinations of pre-trained modalities to perform the fine-tuned task.

Original: We envision our work can bring substantial benefit to multimodal speech processing.
Revised: We envision our work can bring substantial benefit to multimodal speech processing.

Original: Given there aresignificantly more unlabeled audio compared to unlabeled multimodal speech, being able to utilizeunlabeled audio can greatly improve the quality of multimodal speech representations.
Revised: Given there are significantly more unlabeled audio compared to unlabeled multimodal speech, being able to utilize unlabeled audio can greatly improve the quality of multimodal speech representations.

Original: Furthermore,there are few or even no labeled multimodal speech for many tasks, such as speech translation.
Revised: Furthermore, there are few or even no labeled multimodal speech for many tasks, such as speech translation.

Original: Hence,it is essential to build a model capable of zero-shot input modality transfer to perform these taskswith modalities without labeled data. 
Revised: Hence, it is essential to build a model capable of zero-shot input modality transfer to perform these tasks with modalities without labeled data. Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] See Section(b)

Revised: Did you describe the limitations of your work?

Revised: [Yes] See Section(c)

Revised: Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] See Appendix (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Revised: [N/A] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] See Section 4 and Appendix A (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] See Appendix B (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Section 4 and Appendix A 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] See Section(b)

Revised: Did you mention the license of the assets?

Revised: [Yes] See Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [No] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [Yes] See Appendix A (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [Yes] See Appendix A 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]

Revised: A Experiment Details A.1 Data

Revised: Table 7 summarizes the datasets used in this paper, which are all licensed under CC BY-NC-ND or

Revised: CC BY and have been used extensively by the research communities.

Revised: Speech datasets are sourced from interviews, TED talks

Revised: [Table]

Revised: and audiobooks, which are not expected to contain offensive content.

Revised: [Table]

Revised: A.2 Fine-tuning

Revised: Table 8 summarizes the hyperparameters used for ASR fine-tuning

Revised: [Table]

Revised: [Table]

Revised: 9 summarizes the hyperparameters used for speech translation.

Revised: [Table]

Revised: A.3 Speech recognition decoding

Revised: Beam search decoding is used with a length weight α , which searches for the hypothesis z 1: T that maximizes

Revised: [Equation]

Revised: For the results in the main paper, we do grid search from beam size ∈ { 1 , 5 , 10 , 15 , 20 , 25 } and α ∈ { 0 , 0 .

Revised: For the ablation studies in the appendix a beam size of 10 and α = 1 . 0 is used.

Revised: B Extend Experimental Results B.1 Impact of fine-tuning hyperparameters

Revised: We conduct the ablation studies in this section with the models pre-trained on multimodal LRS3 and

Revised: VC2-En.

Revised: By default, the one pre-trained with modality dropout is used.

Revised: B.1.1 Fine-tuning on multimodal data

Revised: Table 10 shows how fine-tuning modality dropout configurations affect ASR performance.

Revised: Models fine-tuned with dropout can yield better performance on all input modality than the one without (reported in the caption).

Revised: When setting m-drop-p and a-drop-p within the range of [0 . 25 , 0 . 75] , the results have limited variation ( [1 . 43% , 1 . 65%] WER for AVSR, [1 . 82% , 2 .

Revised: 22%] WER for ASR

Revised: [Table]

Revised: and [28 . 85% , 30 . 88%] WER for VSR).

Revised: Similarly setting the values to 0.5 yields reasonable performance for all modalities.

Revised: [Table]

Revised: B.1.2 Fine-tuning on unimodal data

Revised: Next

Revised: [Table]

Revised: we study the impact of hyperparameters when fine-tuning on unimodal data.

Revised: Different from fine-tuning on multimodal data, fine-tuning on unimodal data is more prone to catastrophic forgetting, leading to huge performance degradation on modalities unseen during fine-tuning.

Revised: Hence, we focus on studying the impact from three hyperparameters:

Revised: • L frz : number of u-HuBERT Transformer layers that are frozen throughout fine-tuning.

Revised: • N frz : number of the fine-tuning updates where the entire u-HuBERT is frozen.

Revised: After this many updates, the layers above ( N frz )-th layer are optimized jointly with the prediction head.

Revised: • LR : learning rate.

Revised: Number of frozen layers ( L frz ) Setting this effectively treats the first L frz layers as a fixed feature extractor, while the layers above are considered pre-trained and jointly optimized with the added prediction head during fine-tuning.

Revised: Results are shown in Table 11.

Revised: When fine-tuned on audio speech from PT mod-drop , not freezing any layer leads to worse performance for all modalities, but freezing all the layers also reduces the model capacity and harms the performance for the fine-tuned modality and for audio-visual speech.

Revised: For visual speech, the audio fine-tuned model achieves the best result when all the layers are frozen (30.60% compared to 35.68% when not freezing any layer), but the gap can be reduced from 5.08% to 1.23% when freezinglayers, in which audio-visual speech yields the best performance while audio is close to optimal.

Revised: When fine-tuned on visual speech from PT mod-drop , the model yields similar performance on visual speech regardless how many layers are frozen (with a spread of 0.5% WER).

Revised: In contrast, results are optimal for both audio-visual and audio input when 12 layers are frozen, with a good trade-off between mod el capacity and representation distribution shift.

Revised: The trends are different when fine-tuning PT no-mod-drop .

Revised: Because the model has only seen audiovisual input during pre-training, visual-only or audio-only input is considered out-of-distribution.

Revised: Hence, when fine-tuning on visual speech with all the layers frozen ( L frz = 24 ), the performance on the fine-tuned input is significantly worse compared to fine-tuning PT mod-drop (45.18% versus 29.42%).

Revised: Although this can be alleviated by unfreezing more layers, the model barely works for audio-only input with WERs ranging from 21.55% to 54.27%.

Revised: This again verifies that pre-training modality dropout is essential for achieving zero-shot modality transfer.

Revised: [Table]

Revised: Number of frozen steps Results comparing the impact on freezing u-HuBERT for different numbers of fine-tuning steps are shown in Table 12.

Revised: We can observe similar trends as those when varying the number of layers to freeze: not freezing at all leads worse performance on all input modalities, while treating the entire pre-trained model as a fixed feature extractor also leads to sub-optimal performance for the fine-tuned modality.

Revised: Freezing for 30K updates out of 60K total updates leads to good balance for all modalities with PT mod-drop , which are 0.25%/0.11%/0.99% behind the optimal WERs when fine-tuned on audio, and 0%/0%/0.47% behind the optimal WERs when fine-tuned on visual speech.

Revised: Similarly, the model pre-trained without modality dropout yields worse zero-shot modality transfer performance when fine-tuned on a single modality.

Revised: [Table]

Revised: Learning rate Table 13 presents the results.

Revised: When fine-tuning on audio speech, increasing learning rate leads to significantly worse WER on visual input (30.62% → 37.43%)

Revised: [Table]

Revised: but benefits audio-visual and audio speech up to 6 × 10 − 4 and 8 × 10 − 4 , respectively, and the performance stays rather constant afterward.

Revised: Similarly, when fine-tuning on visual speech, increasing learning rate to 10 − 3 hurts the result with audio input (2.73% → 3.41%); it also hurts the visual speech recognition performance, but the degradation is relatively minor (28.62% → 30.39%) compared to when fine-tuning on audio.

Revised: In general, we observe a smaller learning is preferred for visual speech recognition

Revised: [Table]

Revised: and increasing learning rates harms the zero-shot modalities more.

Revised: [Table]

Revised: B.2 Per-layer representation analysis

Revised: To better understand how u-HuBERT learns modality-agnostic features

Revised: [Table]

Revised: we show the clustering quality of different layers per modality of a pre-trained u-HuBERT model.

Revised: Similar to Table 1, we report PNMI of layerwise clusters per modality quantized by audio-only ( C a ), video-only ( C v ), audiovisual ( C av ) and all-combined ( ∪ m C m ) codebook respectively (see Figure 5).

Revised: When pre-trained without modality dropout, the model is unable to learn modality-agnostic features regardless of which layer to cluster, as can be shown from its overall lower PNMI of cross-modal clustering (e.g., PNMI of visual features quantized by C av codebook).

Revised: To the model pre-trained with modality-dropout, the features of its intermediate layers become more agnostic to modality as the layer goes deeper, shown from the diminishing gap between its cross-modality and modality-specific performance in later layers.

Revised: Furthermore, the approximately same quality of clusters from different codebooks towards the end suggests that the final layer is best suited to achieve a unified model that generalizes across modalities.

Revised: B.3 Stability

Revised: Table 14 shows the WER mean and standard deviation over five runs when fine-tuning the two u-HuBERT models.

Revised: We see that the one pre-trained additionally with unimodal audio data (TD) is significantly better on all input modalities.

Revised: [Table]

Revised: C Ethical Discussion

Revised: The ability to build a model that can process unimodal or multimodal speech without needing labeled data in the target modality opens many possibilities for real world applications, since except for audio-only speech, labeled data are extremely lacking for other speech modalities.

Revised: In particular, we demonstrate its applications to audio-visual and visual speech recognition in this paper.

Revised: The former can help hearing-impaired people to better “hear” speech in noisy environments with more accurate transcriptions, while the latter can help people with speech impairment (e.g., aphonia, dysphonia, dysarthria) to “speak” by transcribing silent speech.

Revised: For visual speech recognition, there could be concerns about the technology being improperly used for CCTV surveillance.

Revised: However, current visual speech recognition systems require mostly-frontal and high-resolution videos with a sufficiently high frame rate, such that motions are clearly captured.

Revised: Hence, the type of data studied for audio-visual speech processing are face-to-face meeting scenarios (AMI, EasyCom, and Ego4D) and recorded speech (LRS3).

Revised: In contrast, CCTV videos are low resolution, low frame rate

Revised: [Table]

Revised: and recorded from angles where faces are mostly not frontal, where visual speech processing models will very likely fail.

Revised: D More Details of the Baseline ASR, VSR, and AVSR Methods

Revised: Table 15 and 16 summarize the model architectures and training/testing setups of the baseline ASR,

Revised: VSR

Revised: [Table]

Revised: and AVSR methods compared in Table

Revised: [Table]

Revised: [Table]

Revised: [Table]