Original: A Single Self-Supervised Model for Many Speech Modalities Enables Zero-Shot Modality Transfer
Revised: A Single Self-Supervised Model for Many Speech Modalities Enables Zero-Shot Modality Transfer

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: While audio-visual speech models can yield superior performance and robustnesscompared to audio-only models, their development and adoption are hindered bythe lack of labeled and unlabeled audio-visual data and the cost to deploy one modelper modality.
Revised: While audio-visual speech models can yield superior performance and robustnesscompared to audio-only models, their development and adoption are hindered bythe lack of labeled and unlabeled audio-visual data and the cost to deploy one modelper modality.

Original: In this paper, we present u-HuBERT, a self-supervised pre-trainingframework that can leverage both multimodal and unimodal speech with a unifiedmasked cluster prediction objective.
Revised: In this paper, we present u-HuBERT, a self-supervised pre-trainingframework that can leverage both multimodal and unimodal speech with a uniﬁedmasked cluster prediction objective.

Original: By utilizing modality dropout during pretraining, we demonstrate that a single fine-tuned model can achieve performanceon par or better than the state-of-the-art modality-specific models.
Revised: By utilizing modality dropout during pretraining, we demonstrate that a single ﬁne-tuned model can achieve performanceon par or better than the state-of-the-art modality-speciﬁc models.

Original: Moreover, ourmodel fine-tuned only on audio can perform well with audio-visual and visualspeech input, achieving zero-shot modality generalization for speech recognitionand speaker verification.
Revised: Moreover, ourmodel ﬁne-tuned only on audio can perform well with audio-visual and visualspeech input, achieving zero-shot modality generalization for speech recognitionand speaker veriﬁcation.

Original: In particular, our single model yields 1.2%/1.4%/27.2%speech recognition word error rate on LRS3 with audio-visual/audio/visual input.
Revised: In particular, our single model yields 1.2%/1.4%/27.2%speech recognition word error rate on LRS3 with audio-visual/audio/visual input.

Original:  Introduction
Revised:  Introduction

Original: Speech processing has been moving from developing domain-specific models toward building ageneral model for all domains.
Revised: Speech processing has been moving from developing domain-speciﬁc models toward building ageneral model for all domains.

Original: For example, early studies presented models dedicated for robustspeech recognition [Kingsbury et al., 1998] and distant speech recognition
Revised: For example, early studies presented models dedicated for robustspeech recognition [Kingsbury et al., 1998] and distant speech recognition

Original: [Feng et al., 2014, Kimet al., 2017] by taking into account acoustic properties for each domain.
Revised: [Feng et al., 2014, Kimet al., 2017] by taking into account acoustic properties for each domain.

Original: In contrast, recent work hasbuilt general domain models by pooling data from different domains and even different languages,including labeled ones for supervised learning [Chan et al., 2021, Likhomanenko et al., 2020, Pratapet al., 2020] and unlabeled ones for self-supervised learning [Kawakami et al., 2020, Hsu et al.
Revised: In contrast, recent work hasbuilt general domain models by pooling data from different domains and even different languages,including labeled ones for supervised learning [Chan et al., 2021, Likhomanenko et al., 2020, Pratapet al., 2020] and unlabeled ones for self-supervised learning [Kawakami et al., 2020, Hsu et al.

Original: ,2021b, Conneau et al., 2020].
Revised: ,2021b, Conneau et al., 2020].

Original: These studies show that in addition to removing the need of deployingand maintaining multiple models, a single general domain model can in fact achieve performanceon par or superior to domain-specific models, especially for domains with less data and improvegeneralization performance to unseen domains.
Revised: These studies show that in addition to removing the need of deployingand maintaining multiple models, a single general domain model can in fact achieve performanceon par or superior to domain-speciﬁc models, especially for domains with less data, and improvegeneralization performance to unseen domains.

Original: In contrast, there are few attempts and even fewer successful ones on developing a versatile model thatcan process many speech modalities [Makino et al., 2019, Sari et al., 2021].
Revised: In contrast, there are few attempts and even fewer successful ones on developing a versatile model thatcan process many speech modalities [Makino et al., 2019, Sari et al., 2021].

Original: In general, researchersdevelop models with different input speech modalities, such as visual speech [Chung and Zisserman,2016], audio-visual speech [Xu et al., 2020], multi-channel speech [Haeb-Umbach et al., 2021], 3Dtalking faces [Richard et al., 2021], Electromyographic (EMG) muscle signals [Diener and Schultz,2018], ultrasonic oral cavity scan [Hueber et al., 2010], or ultrasonic microphone [Livescu et al.
Revised: In general, researchersdevelop models with different input speech modalities, such as visual speech [Chung and Zisserman,2016], audio-visual speech [Xu et al., 2020], multi-channel speech [Haeb-Umbach et al., 2021], 3Dtalking faces [Richard et al., 2021], Electromyographic (EMG) muscle signals [Diener and Schultz,2018], ultrasonic oral cavity scan [Hueber et al., 2010], or ultrasonic microphone [Livescu et al.

Original: ,2009] in isolation, despite the fact that many of them are used to solve the same task, like speech andspeaker recognition.
Revised: ,2009] in isolation, despite the fact that many of them are used to solve the same task, like speech andspeaker recognition.

Original: Moreover, while additional modalities, such as visual speech, can often boostthe performance and robustness on top of audio speech, models taking such input are less studied anddeployed in real world application.
Revised: Moreover, while additional modalities, such as visual speech, can often boostthe performance and robustness on top of audio speech, models taking such input are less studied anddeployed in real world application.

Original: This is mainly due to the lack of both labeled and unlabeled datafor these alternative modalities, hindering the development of high-performing self-supervised orsupervised modal-specific models that are trained exclusively on one type of input.
Revised: This is mainly due to the lack of both labeled and unlabeled datafor these alternative modalities, hindering the development of high-performing self-supervised or supervised modal-speciﬁc models that are trained exclusively on one type of input.

Original: We argue that to
Revised: We argue that to

Original: U-HuBERT [13,13,5,5,5] [12,5,5,3] [4,4,4,9,5,5]
Revised: U-HuBERT [13,13,5,5,5] [12,5,5,3] [4,4,4,9,5,5]

Original: Joint clustering
Revised: Joint clustering

Original: Modality-agnosticfeatures
Revised: Modality-agnosticfeatures

Original: Pseudo labels Multimodal / unimodal input
Revised: Pseudo labels Multimodal / unimodal input

Original: Pseudo labeling
Revised: Pseudo labeling

Original: U-HuBERT U-HuBERT
Revised: U-HuBERT U-HuBERT

Original: AND/OR
Revised: AND/OR

Original: AND/OR tackle the data scarcity issue, it essential to utilize not only the speech from the target modality, butalso speech in other modalities to build a general speech model for many modalities.
Revised: AND/OR tackle the data scarcity issue, it essential to utilize not only the speech from the target modality, butalso speech in other modalities to build a general speech model for many modalities.

Original: By doing so,more data can be used to learn components shared across different modalities, which is expected tobring bigger gains to low-resource modalities.
Revised: By doing so,more data can be used to learn components shared across different modalities, which is expected tobring bigger gains to low-resource modalities.

Original: In this paper, we present unified hidden unit BERT (u-HuBERT), which leverages unlabeled speechdata of many different modalities for pre-training, including both unimodal and multimodal speech.
Revised: In this paper, we present uniﬁed hidden unit BERT (u-HuBERT), which leverages unlabeled speechdata of many different modalities for pre-training, including both unimodal and multimodal speech.

Original: Architecture-wise, u-HuBERT consists of a set of modality-specific feature extractors and a sharedtransformer backend.
Revised: Architecture-wise, u-HuBERT consists of a set of modality-speciﬁc feature extractors and a sharedtransformer backend.

Original: The model is trained with a unified masked cluster prediction objective forboth unimodal and multimodal speech data.
Revised: The model is trained with a uniﬁed masked cluster prediction objective forboth unimodal and multimodal speech data.

Original: Specifically, we adopt modality dropout [Neverovaet al., 2014] for multimodal data to simulate different combination of input modalities and encouragethe model to learn modality-agnostic frame-level representations.
Revised: Speciﬁcally, we adopt modality dropout [Neverovaet al., 2014] for multimodal data to simulate different combination of input modalities and encouragethe model to learn modality-agnostic frame-level representations.

Original: The pre-trained u-HuBERT canbe fine-tuned with either multimodal or unimodal data, and we demonstrate that in both cases theresulting model can be used for many speech modalities.
Revised: The pre-trained u-HuBERT canbe ﬁne-tuned with either multimodal or unimodal data, and we demonstrate that in both cases theresulting model can be used for many speech modalities.

Original: We verify the effectiveness on two popularspeech processing tasks: speech recognition and speaker verification.
Revised: We verify the effectiveness on two popularspeech processing tasks: speech recognition and speaker veriﬁcation.

Original: To the best of our knowledge,this is the first single model for many speech modalities that yields performance on par with thestate-of-the-art modality-specific models and is capable of zero-shot modality generalization.
Revised: To the best of our knowledge,this is the ﬁrst single model for many speech modalities that yields performance on par with thestate-of-the-art modality-speciﬁc models and is capable of zero-shot modality generalization.

Original:  Related Work
Revised:  Related Work

Original: Comparison to AV-HuBERT Our work is an extension of AV-HuBERT, which studies pre-trainingexclusively with audio-visual data.
Revised: Comparison to AV-HuBERT Our work is an extension of AV-HuBERT, which studies pre-trainingexclusively with audio-visual data.

Original: While the model architecture and the form of pre-training objectiveare the same, u-HuBERT generalizes AV-HuBERT to utilize both multimodal and unimodal speechfor pre-training by mapping various input to a modality-agnostic feature space and creating a sharedtarget space for masked prediction.
Revised: While the model architecture and the form of pre-training objectiveare the same, u-HuBERT generalizes AV-HuBERT to utilize both multimodal and unimodal speechfor pre-training by mapping various input to a modality-agnostic feature space and creating a sharedtarget space for masked prediction.

Original: We empirically demonstrate the benefits of including additionalunimodal data for pre-training.
Revised: We empirically demonstrate the beneﬁts of including additionalunimodal data for pre-training.

Original: The fine-tuning and evaluation protocols also differ significantly.
Revised: The ﬁne-tuning and evaluation protocols also differ signiﬁcantly.

Original: AV-HuBERT effectively learns modality-specific models (e.g., visual speech recognizer) given labeleddata in the target modality for fine-tuning.
Revised: AV-HuBERT effectively learns modality-speciﬁc models (e.g., visual speech recognizer) given labeleddata in the target modality for ﬁne-tuning.

Original: In contrast, this work focuses on building a single modelfor all modalities seen during pre-training with labeled data in any subset of modalities.
Revised: In contrast, this work focuses on building a single modelfor all modalities seen during pre-training with labeled data in any subset of modalities.

Original: Joint unimodal and multimodal pre-training Recent cross-modal contrastive approaches suchas CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have shown impressive results yet theycan only be trained with multimodal data and require a large-scale dataset, which can be hard toobtain for many speech modalities (e.g., paired EMG-audio speech).
Revised: Joint unimodal and multimodal pre-training Recent cross-modal contrastive approaches suchas CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have shown impressive results yet theycan only be trained with multimodal data and require a large-scale dataset, which can be hard toobtain for many speech modalities (e.g., paired EMG-audio speech).

Original: FLAVA [Singh et al., 2021]and SLAM [Bapna et al., 2021] are two frameworks that utilize both unimodal and multimodal datafor image-text and speech-text pre-training, respectively.
Revised: FLAVA [Singh et al., 2021]and SLAM [Bapna et al., 2021] are two frameworks that utilize both unimodal and multimodal datafor image-text and speech-text pre-training, respectively.

Original: There are three key differences betweenu-HuBERT and these two models.
Revised: There are three key differences betweenu-HuBERT and these two models.

Original: First, for multimodal speech, different modalities such as audio andvisual speech are temporally aligned.
Revised: First, for multimodal speech, different modalities such as audio andvisual speech are temporally aligned.

Original: Instead of concatenating the feature sequences from unimodalencoders, we fuse the features frame-by-frame before passing it to the shared encoder.
Revised: Instead of concatenating the feature sequences from unimodalencoders, we fuse the features frame-by-frame before passing it to the shared encoder.

Original: Second,while all three models utilize a masked token prediction objective, u-HuBERT considers a sharedtarget token space for all unimodal and multimodal data, obtained by clustering modality-agnosticfeatures extracted from the u-HuBERT model itself.
Revised: Second,while all three models utilize a masked token prediction objective, u-HuBERT considers a sharedtarget token space for all unimodal and multimodal data, obtained by clustering modality-agnosticfeatures extracted from the u-HuBERT model itself.

Original: In contrast, SLAMuses w2v-BERT [Chunget al., 2021] tokens for speech and SentencePiece [Schuster and Nakajima, 2012] for text. FLAVA onother hand uses a separately trained dVAE tokenizer for image and the same as SLAM for text, and itpredicts tokens for unimodal and multimodal data at the output of different modules, differing from u-
Revised: In contrast, SLAM

Original: HuBERT.
Revised: HuBERT.

Original: In addition, both FLAVA and SLAM pre-train with a number of auxiliary objectives, whileu-HuBERT only uses one, which simplifies hyperparameter selection.
Revised: In addition, both FLAVA and SLAM pre-train with a number of auxiliary objectives, whileu-HuBERT only uses one, which simpliﬁes hyperparameter selection.

Original: Third, different modalities ofspeech are often used to solve the same task, such as speech recognition or speaker verification.
Revised: Third, different modalities ofspeech are often used to solve the same task, such as speech recognition or speaker veriﬁcation.

Original: Ourevaluation focuses on building a single model that can solve the same task with different modalitiesdespite not having labeled data for each modality, which is not considered by SLAM or FLAVA.
Revised: Ourevaluation focuses on building a single model that can solve the same task with different modalitiesdespite not having labeled data for each modality, which is not considered by SLAM or FLAVA.

Original: Single model for many input modalities Makino et al.
Revised: Single model for many input modalities Makino et al.

Original: [2019] and Sari et al.
Revised: [2019] and Sari et al.

Original: [2021] exploredbuilding a single model for supervised audio-visual/audio/visual speech recognition and speakerverification, respectively.
Revised: [2021] exploredbuilding a single model for supervised audio-visual/audio/visual speech recognition and speakerveriﬁcation, respectively.

Original: Both studies show significant degradation comparing unified models withmodality-specific ones.
Revised: Both studies show signiﬁcant degradation comparing uniﬁed models withmodality-speciﬁc ones.

Original: Furthermore, these models are single-task supervised and hence not capable ofzero-shot modality transfer when trained on unimodal data.
Revised: Furthermore, these models are single-task supervised and hence not capable ofzero-shot modality transfer when trained on unimodal data.

Original: Omnivore [Girdhar et al., 2022] is a singlesupervised model for many vision modalities, including image (RGB), 3D image (RGBD), and video(RGB sequence), which also considers a model architecture with modality-specific encoders (RGBand depth) followed by a shared Swin Transformer backend [Liu et al., 2021].
Revised: Omnivore [Girdhar et al., 2022] is a singlesupervised model for many vision modalities, including image (RGB), 3D image (RGBD), and video(RGB sequence), which also considers a model architecture with modality-speciﬁc encoders (RGBand depth) followed by a shared Swin Transformer backend [Liu et al., 2021].

Original: Transformer [Liuet al., 2021] to learn shared representations across modalities.

Original: Both Omnivore and u-HuBERTdemonstrates cross-modal generalization, but Omnivore achieves it through supervised learningwith different tasks, and u-HuBERT achieves it with unified self-supervised learning.
Revised: Both Omnivore andu-HuBERT demonstrates cross-modal generalization, but Omnivore achieves it through supervisedlearning with different tasks, and u-HuBERT achieves it with uniﬁed self-supervised learning.

Original: Finally, if weconsidered different languages as different modalities, our work is also related to mBERT [Devlinet al., 2019], mBART [Liu et al., 2020], TLM [Yao et al., 2021], and XLM [Lample and Conneau,2019], fine-tuning which enables generalizing natural language understanding [Pires et al., 2019]and machine translation [Johnson et al., 2017] to languages without labeled data.
Revised: Finally, if we considered different languages as different modalities, our work is also related tomBERT [Devlin et al., 2019], mBART [Liu et al., 2020], TLM [Yao et al., 2021], and XLM [Lampleand Conneau, 2019], ﬁne-tuning which enables generalizing natural language understanding [Pireset al., 2019] and machine translation [Johnson et al., 2017] to languages without labeled data.

Original: Pre-training onmultimodal speech is analogous to pre-training on bitext, while the former requires less manual effortto obtain by collecting more measurements when speech is produced.
Revised: Pretraining on multimodal speech is analogous to pre-training on bitext, while the former requires lessmanual effort to obtain by collecting more measurements when speech is produced.

Original: In addition, time-synchronicitybetween speech modalities provides additional self-supervision, enabling u-HuBERT to more easilylearn frame-level modality-agnostic representations.
Revised: In addition, timesynchronicity between speech modalities provides additional self-supervision, enabling u-HuBERTto more easily learn frame-level modality-agnostic representations.

Original:  Method
Revised:  Method

Revised: [Figure]

Original: 3.1 Background: audio-visual HuBERT
Revised: 3.1 Background: audio-visual HuBERT

Original:  We build our framework upon AV-HuBERT [Shi et al., 2022a], a self-supervised representationlearning model designed to pre-train a single type of multimodal speech data: audio-visual speech.
Revised: [Figure] We build our framework upon AV-HuBERT [Shi et al., 2022a], a self-supervised representationlearning model designed to pre-train a single type of multimodal speech data: audio-visual speech.

Original: AV-HuBERT first encodes temporally aligned audio speech X a ∈ R T a × C a and visual speech
Revised: AV-HuBERT ﬁrst encodes temporally aligned audio speech X a 2 R T a ⇥ C a and visual speech

Original: X v ∈ R T v × C v × H × W into feature sequences h a ∈ R T × D h andh v ∈ R T ×
Revised: X v 2 R T v ⇥ C v ⇥ H ⇥ W into feature sequences h a 2 R T ⇥ D h and

Original:  D h of the same lengths,which are then fused frame-by-frame FUSE ( h a , h v ) ∈ R T × D c and processed by a Transformerbackend [Vaswani et al., 2017] to produce contextualized audio-visual representations c av ∈ R T × D c .
Revised: h v 2 R T ⇥ D h of the same lengths,which are then fused frame-by-frame FUSE ( h a , h v ) 2 R T ⇥ D c and processed by a Transformerbackend [Vaswani et al., 2017] to produce contextualized audio-visual representations c av 2 R T ⇥ D c .

Original: This model is trained by iterating two steps.
Revised: This model is trained by iterating two steps.

Original: First, a K-means clustering model is trained on someaudio-visual features to produce frame-level targets Y av ∈ [1 , · · · , K ] T , where K is the number ofclusters.
Revised: First, a K-means clustering model is trained on someaudio-visual features to produce frame-level targets Y av 2 [1 , · · · , K ] T , where K is the number ofclusters.

Original: Second, the AV-HuBERT model is trained with a masked cluster prediction task, wherespans from the audio-visual input are randomly masked and the model learns to predict the clusterassignment of the masked frames given the context.
Revised: Second, the AV-HuBERT model is trained with a masked cluster prediction task, wherespans from the audio-visual input are randomly masked and the model learns to predict the clusterassignment of the masked frames given the context.

Original: The model is forced to learn the underlyinglinguistic structure of audio-visual speech to infer the content in the unseen regions based on thecontext.
Revised: The model is forced to learn the underlyinglinguistic structure of audio-visual speech to infer the content in the unseen regions based on thecontext.

Original: The intermediate feature of a trained model, which is better than previous iteration, is usedto produce targets for the subsequent iteration.
Revised: The intermediate feature of a trained model, which is better than previous iteration, is usedto produce targets for the subsequent iteration.

Original: The pre-trained AV-HuBERT model performs wellon both multimodal downstream tasks like audio-visual speech recognition [Shi et al., 2022b] andunimodal tasks like lip-reading [Shi et al., 2022a] when fine-tuned for a specific modality.
Revised: The pre-trained AV-HuBERT model performs wellon both multimodal downstream tasks like audio-visual speech recognition [Shi et al., 2022b] andunimodal tasks like lip-reading [Shi et al., 2022a] when ﬁne-tuned for a speciﬁc modality.

Original: 3.2 Self-supervised pre-training with manymodalities
Revised: 3.2 Self-supervised pre-training with many

Revised: modalities

Original: We extend the framework to utilize not only multimodal data, but also unimodal data that can bemore easily obtained in large quantities.
Revised: We extend the framework to utilize not only multimodal data, but also unimodal data that can bemore easily obtained in large quantities.

Original: The main idea is to apply modality dropout to multimodalspeech along with a shared Transformer [Vaswani et al., 2017] to learn modality-agnostic featuresthat can be quantized by a shared codebook to produce pseudo labels for masked prediction.
Revised: The main idea is to apply modality dropout to multimodalspeech along with a shared Transformer [Vaswani et al., 2017] to learn modality-agnostic featuresthat can be quantized by a shared codebook to produce pseudo labels for masked prediction.

Original: Given unimodal and multimodal speech data from many modalities, we assume that there is an anchormodality that is paired with all other modalities.
Revised: Given unimodal and multimodal speech data from many modalities, we assume that there is an anchormodality that is paired with all other modalities.

Original: For simplicity, assume the anchor modality is audio,and we have unlabeled audio, visual, and audio-visual speech.
Revised: For simplicity, assume the anchor modality is audio,and we have unlabeled audio, visual, and audio-visual speech.

Original: In the first iteration, because a modalityagnostic feature extractor is not available yet, we cluster the anchor modality features to producetargets for unimodal and multimodal speech that contain the anchor modality.
Revised: In the ﬁrst iteration, because a modalityagnostic feature extractor is not available yet, we cluster the anchor modality features to producetargets for unimodal and multimodal speech that contain the anchor modality.

Original: Concretely speaking,
Revised: Concretely speaking,

Original: [Figure]
Revised: [Figure]

Original:  Transformer [13,13,5,5,5] [13,13,5,5,5] [13,13,5,5,5]
Revised: Shared Transformer [13,13,5,5,5] [13,13,5,5,5] [13,13,5,5,5]

Original: Audioencoder
Revised: Audioencoder

Original: [Figure] fuse
Revised: Dummy fuse

Original: Shared Transformer [12,5,5,3]
Revised: Shared Transformer [12,5,5,3]

Original: [Figure]
Revised: [Figure]

Original: this produces pseudo paired data for audio speech X a and audio-visual speech ( X a , X v ) with audiobased cluster assignment Y a as the target.
Revised: this produces pseudo paired data for audio speech X a and audio-visual speech ( X a , X v ) with audiobased cluster assignment Y a as the target.

Original: Like AV-HuBERT, each modality has its own featureextractor: f a for audio and f v for video, and feature sequences are fused frame-by-frame before beingpassed onto the Transformer g to produce contextualized features c av = g ( FUSE ( f a ( X a ) , f v ( X v ))).
Revised: Like AV-HuBERT, each modality has its own featureextractor: f a for audio and f v for video, and feature sequences are fused frame-by-frame before beingpassed onto the Transformer g to produce contextualized features c av = g ( FUSE ( f a ( X a ) , f v ( X v )))

Original: When a modality is absent, its feature sequence is simply replaced by zero vectors, for example,c a = g ( FUSE ( f a ( X a ) , 0 )) .
Revised: When a modality is absent, its feature sequence is simply replaced by zero vectors, for example,c a = g ( FUSE ( f a ( X a ) , 0 )) .

Original: Following HuBERT [Hsu et al., 2021a], this model is trained with amasked prediction objective.
Revised: Following HuBERT [Hsu et al., 2021a], this model is trained with amasked prediction objective.

Original: To encourage the model to learn modality-agnostic features, modality dropout is applied to multimodaldata to randomly drop a subset of the modalities, effectively creating multiple copies of the data withthe same target but different input modalities, as depicted in the top half of Figure 2.
Revised: To encourage the model to learn modality-agnostic features, modality dropout is applied to multimodaldata to randomly drop a subset of the modalities, effectively creating multiple copies of the data withthe same target but different input modalities, as depicted in the top half of Figure 2.

Original: While modalitydropout was also used in AV-HuBERT, it was implemented for a different reason – to reduce thediscrepancy between pre-training and fine-tuning for unimodal downstream tasks, which resultedin only minor improvement (reducing the WER from 57.0% to 55.3% as shown in Table D.1 in Shiet al.
Revised: While modalitydropout was also used in AV-HuBERT, it was implemented for a different reason – to reduce thediscrepancy between pre-training and ﬁne-tuning for unimodal downstream tasks, which resultedin only minor improvement (reducing the WER from 57.0% to 55.3% as shown in Table D.1 in Shiet al.

Original: [2022a]).
Revised: [2022a]).

Original: In contrast, we will demonstrate that modality dropout is essential for mixed modalitypre-training and zero-shot generalization.
Revised: In contrast, we will demonstrate that modality dropout is essential for mixed modalitypre-training and zero-shot generalization.

Original: After the first iteration, the model can be used to extract modality-agnostic features for all unimodaland multimodal data, which are clustered to produce the pseudo label for next iteration of pre-trainingas illustrated in Figure 1.
Revised: After the ﬁrst iteration, the model can be used to extract modality-agnostic features for all unimodaland multimodal data, which are clustered to produce the pseudo label for next iteration of pre-trainingas illustrated in Figure 1.

Original: For multimodal data like audio-visual speech, one can produce pseudolabels using features extracted from the complete multimodal input (e.g., Y av from audio-visualinput), or those extracted from input with a subset of modalities (e.g., Y v using only the video stream).
Revised: For multimodal data like audio-visual speech, one can produce pseudolabels using features extracted from the complete multimodal input (e.g., Y av from audio-visualinput), or those extracted from input with a subset of modalities (e.g., Y v using only the video stream).

Original: We opt for the former and study the impact of these choices in Secton 4.2.
Revised: We opt for the former and study the impact of these choices in Secton 4.2.

Original: 3.3 Multimodal and unimodal fine-tuning with partial modalities
Revised: 3.3 Multimodal and unimodal ﬁne-tuning with partial modalities

Original: Once the model is pre-trained, we remove the cluster prediction head (the linear layer on top oftransformer used to predict cluster assignment distribution) and add a downstream task-specificprediction head.
Revised: Once the model is pre-trained, we remove the cluster prediction head (the linear layer on top oftransformer used to predict cluster assignment distribution) and add a downstream task-speciﬁcprediction head.

Original: The pre-trained model can be fine-tuned on labeled multimodal speech, unimodalspeech, or speech with mixed modalities.
Revised: The pre-trained model can be ﬁne-tuned on labeled multimodal speech, unimodalspeech, or speech with mixed modalities.

Original: More importantly, the modalities included in the finetuning data do not necessarily cover the modalities seen during pre-training, yet the fine-tuned modelcan still handle the downstream task on all pre-trained modalities, which we refer to as zero-shotmodality generalization .
Revised: More importantly, the modalities included in the ﬁnetuning data do not necessarily cover the modalities seen during pre-training, yet the ﬁne-tuned modelcan still handle the downstream task on all pre-trained modalities, which we refer to as zero-shotmodality generalization .

Original: This is because the pre-trained model has successfully learned modalityagnostic representations, where same speech manifested in different modalities are mapped to similarrepresentations.
Revised: This is because the pre-trained model has successfully learned modalityagnostic representations, where same speech manifested in different modalities are mapped to similarrepresentations.

Original: Thus mapping from the representation of one modality (e.g., audio) to a target output(e.g.
Revised: Thus mapping from the representation of one modality (e.g., audio) to a target output(e.g.

Original: , text) can be directly applied to other pre-trained modalities (e.g., video).
Revised: , text) can be directly applied to other pre-trained modalities (e.g., video).

Original: Similar to mBART [Liu et al., 2020] for zero-shot machine translation, catastrophic forgetting [Leeet al., 2019, Üstün et al., 2021] may occur during fine-tuning and hence hinders its generalizationacross modalities.
Revised: Similar to mBART [Liu et al., 2020] for zero-shot machine translation, catastrophic forgetting [Leeet al., 2019, Üstün et al., 2021] may occur during ﬁne-tuning and hence hinders its generalizationacross modalities.

Original: Particularly, this is because the distributions of Transformer input and output atearlier layers are less modality-invariant.
Revised: Particularly, this is because the distributions of Transformer input and output atearlier layers are less modality-invariant.

Original: To alleviate the issue, we apply modality dropout whenmultimodal data is used, and explore a few techniques to prevent the representations from drifting toomuch, including freezing the entire pre-trained model for a number of steps and freezing a number of initial layers (up to where more modality-agnostic representations are produced) throughoutfine-tuning.
Revised: To alleviate the issue, we apply modality dropout whenmultimodal data is used, and explore a few techniques to prevent the representations from drifting toomuch, including freezing the entire pre-trained model for a number of steps and freezing a number of initial layers (up to where more modality-agnostic representations are produced) throughoutﬁne-tuning.

Original:  Experiments4.1 Pre-training setup
Revised:  Experiments

Revised: 4.1 Pre-training setup

Original: We evaluate u-HuBERT on audio, visual, and audio-visual speech, where larger quantities of unlabeledand labeled data are available.
Revised: We evaluate u-HuBERT on audio, visual, and audio-visual speech, where larger quantities of unlabeledand labeled data are available.

Original: Two audio-visual and one audio-only datasets are used for pre-training:(1) LRS3 [Afouras et al., 2018] with 433 hours of English audio-visual speech, (2) VoxCeleb2-En(VC2-En) with 1,326 hours of English YouTube audio-visual speech filtered from VoxCeleb2 [Chunget al., 2018] by Shi et al.
Revised: Two audio-visual and one audio-only datasets are used for pre-training:(1) LRS3 [Afouras et al., 2018] with 433 hours of English audio-visual speech, (2) VoxCeleb2-En(VC2-En) with 1,326 hours of English YouTube audio-visual speech ﬁltered from VoxCeleb2 [Chunget al., 2018] by Shi et al.

Original: [2022a], and (3) TED-LIUM release 3 (TD)
Revised: [2022a], and (3) TED-LIUM release 3 (TD)

Original: [Hernandez et al., 2018]
Revised: [Hernandez et al., 2018]

Original: with452 hours of English audio collected from the same domain as LRS3.
Revised: with452 hours of English audio collected from the same domain as LRS3.

Original: To improve noise robustness,we apply online noise augmentation where each utterance is corrupted with additive noise at 0dBsampled from MUSAN [Snyder et al., 2015] with a probability of 0.25.
Revised: To improve noise robustness,we apply online noise augmentation where each utterance is corrupted with additive noise at 0dBsampled from MUSAN [Snyder et al., 2015] with a probability of 0.25.

Original: More details can be found in
Revised: More details can be found in

Original: Appendix A. We adopt the AV-HuBERT-LARGE model Shi et al.
Revised: Appendix A. We adopt the AV-HuBERT-LARGE model Shi et al.

Original: [2022a] 1 : the video encoder is amodified ResNet-18 model [He et al., 2016] that takes lip region-of-interest (ROI) frames sampled at25Hz at input and produce a sequence of 512 dimensional embeddings at the same frame rate; audiois represented as filter bank features sampled at 100 Hz, stacked every four frames to match the videoframe rate, and projected with a linear layer to 1024-D features.
Revised: [2022a] 1 : the video encoder is amodiﬁed ResNet-18 model [He et al., 2016] that takes lip region-of-interest (ROI) frames sampled at25Hz at input and produce a sequence of 512 dimensional embeddings at the same frame rate; audiois represented as ﬁlter bank features sampled at 100 Hz, stacked every four frames to match the videoframe rate, and projected with a linear layer to 1024-D features.

Original: FUSE concatenates features frameby-frame and linearly projects them to the embedding dimension of the shared Transformer.
Revised: FUSE concatenates features frameby-frame and linearly projects them to the embedding dimension of the shared Transformer.

Original: Theshared Transformer has 24 layers, with 16 heads, embedding dimension 1024, and FFN dimension4096, using the pre-norm residual connection setup [Nguyen and Salazar, 2019].
Revised: Theshared Transformer has 24 layers, with 16 heads, embedding dimension 1024, and FFN dimension4096, using the pre-norm residual connection setup [Nguyen and Salazar, 2019].

Original: To directly compare with the publicly available fifth iteration noise-augmented AV-HuBERT-LARGEmodel trained on LRS3 and VC2-En, the same feature extractor (fourth iteration BASE model) andthe same codebook (learned from multimodal features C av ) are used to generate pseudo labels formultimodal LRS3 and VC2-En and unimodal TD.
Revised: To directly compare with the publicly available ﬁfth iteration noise-augmented AV-HuBERT-LARGEmodel trained on LRS3 and VC2-En, the same feature extractor (fourth iteration BASE model) andthe same codebook (learned from multimodal features C av ) are used to generate pseudo labels formultimodal LRS3 and VC2-En and unimodal TD.

Original: We then pre-train u-HuBERT on the combineddata for 1M updates on 64 32GB V100 GPUs using the Adam optimizer [Kingma and Ba, 2015] anda learning rate of 0.002.
Revised: We then pre-train u-HuBERT on the combineddata for 1M updates on 64 32GB V100 GPUs using the Adam optimizer [Kingma and Ba, 2015] anda learning rate of 0.002.

Original: Gradient norm is clipped at 1.0. to stablize training.
Revised: Gradient norm is clipped at 1.0. to stablize training.

Original: A batch size of maximal40 seconds per GPU is used.
Revised: A batch size of maximal40 seconds per GPU is used.

Original: When pre-trained on multimodal data, audio and video are droppedwith a probability of 0.25 and 0.25, respectively.
Revised: When pre-trained on multimodal data, audio and video are droppedwith a probability of 0.25 and 0.25, respectively.

Original: 4.2 Learning modality-agnostic features and codebooks with modality dropout
Revised: 4.2 Learning modality-agnostic features and codebooks with modality dropout

Original: We argue that modality dropout leads to learning modality-agnostic representations through forcingthe model to predict the same target with different subsets of input modalities given multimodalinput, which in turn enables construction of a shared codebook for all modalities.
Revised: We argue that modality dropout leads to learning modality-agnostic representations through forcingthe model to predict the same target with different subsets of input modalities given multimodalinput, which in turn enables construction of a shared codebook for all modalities.

Original: To verify this, wecompare two models that are identical except for whether modality dropout is activated: the publicnoise-augmented AV-HuBERT LARGE model trained on LRS3 and VC2-En with modality dropoutand its counterpart.
Revised: To verify this, wecompare two models that are identical except for whether modality dropout is activated: the publicnoise-augmented AV-HuBERT LARGE model trained on LRS3 and VC2-En with modality dropoutand its counterpart.

Original: For each model, we infer audio, visual, and audio-visual features for each LRS3validation utterances, randomly sample 500 frames, pool the features, and run t-SNE [Van der Maatenand Hinton, 2008] dimension reduction for visualization (Figure 3).
Revised: For each model, we infer audio, visual, and audio-visual features for each LRS3validation utterances, randomly sample 500 frames, pool the features, and run t-SNE [Van der Maatenand Hinton, 2008] dimension reduction for visualization (Figure 3).

Original: It is visually evident that withoutmodality dropout, features extracted from different modalities have different distributions (bottomrow), contrary to those from the model with modality dropout (top row).
Revised: It is visually evident that withoutmodality dropout, features extracted from different modalities have different distributions (bottomrow), contrary to those from the model with modality dropout (top row).

Original: We next adopt the phone normalized mutual information (PNMI) metric presented in Hsu et al.[2021a] to quantify cluster quality.
Revised: We next adopt the phone normalized mutual information (PNMI) metric presented in Hsu et al.[2021a] to quantify cluster quality.

Original: For each model, we create four codebooks by clustering audiovisual features C av , audio features C a , visual features C v , or all three combined ∪ m C m , quantizeeach type of features with each codebook, and measure the normalized mutual information ∈ [0 , 1]between frame-level phone labels and cluster assignment (Table 1).
Revised: For each model, we create four codebooks by clustering audiovisual features C av , audio features C a , visual features C v , or all three combined [ m C m , quantizeeach type of features with each codebook, and measure the normalized mutual information 2 [0 , 1]between frame-level phone labels and cluster assignment (Table 1).

Original: Because features inferred fromdifferent modalities exhibit similar distributions with modality dropout, all four codebooks lead tosimilar PNMI for a given feature.
Revised: Because features inferred fromdifferent modalities exhibit similar distributions with modality dropout, all four codebooks lead tosimilar PNMI for a given feature.

Original: In contrast, without modality dropout, cross-quantization (e.g., Y avwith C v codebook) often leads to significantly worse PNMI, and PNMI is worse for all (codebook,feature) combinations compared with that from the modality dropout model.
Revised: In contrast, without modality dropout, cross-quantization (e.g., Y avwith C v codebook) often leads to signiﬁcantly worse PNMI, and PNMI is worse for all (codebook,feature) combinations compared with that from the modality dropout model.

Original: Note for a givencodebook, the PNMI still differs among modalities which is due to the discrepancy of informationcarried per modality.
Revised: Note for a givencodebook, the PNMI still differs among modalities which is due to the discrepancy of informationcarried per modality.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure]
Revised: means

Original: [Table] [Figure] 
Revised: [Figure] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.3 Speech recognition
Revised: 4.3 Speech recognition

Original: Fine-tuning setup LRS3 trainval and pretrain , combined for 433 hours, are used for training, withthe same 1,200 utterances as Shi et al.
Revised: Fine-tuning setup LRS3 trainval and pretrain , combined for 433 hours, are used for training, withthe same 1,200 utterances as Shi et al.

Original: [2022a] split for validation.
Revised: [2022a] split for validation.

Original: Results on the test split are reported.
Revised: Results on the test split are reported.

Original: Noise augmentation with MUSAN used in pre-training is also adopted during fine-tuning.
Revised: Noise augmentation with MUSAN used in pre-training is also adopted during ﬁne-tuning.

Original: A 9-layerrandomly initialized Transformer decoder with 8 attention heads and 1024/4096-D embedding/FFNis appended to the pre-trained u-HuBERT, and the entire model is fine-tuned with a cross-entropyloss predicting the next text token given the input and the previous text tokens.
Revised: A 9-layerrandomly initialized Transformer decoder with 8 attention heads and 1024/4096-D embedding/FFNis appended to the pre-trained u-HuBERT, and the entire model is ﬁne-tuned with a cross-entropyloss predicting the next text token given the input and the previous text tokens.

Original: Text is representedas unigram-based subword units [Kudo, 2018] with a vocabulary size of 1,000.
Revised: Text is representedas unigram-based subword units [Kudo, 2018] with a vocabulary size of 1,000.

Original: Modality dropoutis applied when fine-tuning on multimodal input.
Revised: Modality dropoutis applied when ﬁne-tuning on multimodal input.

Original: Additionally, we sweep learning rate, numberof updates, steps to freeze, and layers to freeze throughout fine-tuning.
Revised: Additionally, we sweep learning rate, numberof updates, steps to freeze, and layers to freeze throughout ﬁne-tuning.

Original: We evaluate each model,regardless of the modality used for fine-tuning, on all kinds of input modalities and report the worderror rates (WERs) without fusing external language models.
Revised: We evaluate each model,regardless of the modality used for ﬁne-tuning, on all kinds of input modalities and report the worderror rates (WERs) without fusing external language models.

Original: To gauge the robustness to noise, wefollow the same protocol in Shi et al.
Revised: To gauge the robustness to noise, wefollow the same protocol in Shi et al.

Original: [2022b] to noisy audio and audio-visual test sets by addingbabble noise at 0 decibel (dB) to the audio stream for evaluation.
Revised: [2022b] to noisy audio and audio-visual test sets by addingbabble noise at 0 decibel (dB) to the audio stream for evaluation.

Original: Detailed hyperpamater setups canbe found in Appendix A, and more ablation studies can be found in Appendix B.1.
Revised: Detailed hyperpamater setups canbe found in Appendix A, and more ablation studies can be found in Appendix B.1.

Original: Modality dropout is essential for building a single model Table 2 compares fine-tuning fromscratch, as well as from pre-trained models with and without modality dropout, using labeled audio,visual, or audio-visual speech, and highlights the results of zero-shot modality transfer where labeleddata in test modality are not used.
Revised: Modality dropout is essential for building a single model Table 2 compares ﬁne-tuning fromscratch, as well as from pre-trained models with and without modality dropout, using labeled audio,visual, or audio-visual speech, and highlights the results of zero-shot modality transfer where labeleddata in test modality are not used.

Original: First, models without pre-training are significantly worse.
Revised: First, models without pre-training are signiﬁcantly worse.

Original: Second,when fine-tuning on audio-visual data without modality dropout, performance drops significantly onunimodal data if pre-training modality dropout is not applied (e.g., 1.7% to 21.4% on clean audio).
Revised: Second,when ﬁne-tuning on audio-visual data without modality dropout, performance drops signiﬁcantly onunimodal data if pre-training modality dropout is not applied (e.g., 1.7% to 21.4% on clean audio).

Original: This can be alleviated with fine-tuning modality dropout, improving the average WER from 24.5% to12.3%, but is still behind applying it in both pre-training and fine-tuning (11.4% average).
Revised: This can be alleviated with ﬁne-tuning modality dropout, improving the average WER from 24.5% to12.3%, but is still behind applying it in both pre-training and ﬁne-tuning (11.4% average).

Original: The benefit of applying pre-training modality dropout is magnified in unimodal fine-tuning: the modelfine-tuned only on labeled audio data yields the same performance on clean and noisy audio-visualinput (1.3% and 4.6%) as the model fine-tuned with labeled audio-visual data, while being slightbehind on video input compared to the model fine-tuned with labeled video data (31.6% vs. 28.7%).
Revised: The beneﬁt of applying pre-training modality dropout is magniﬁed in unimodal ﬁne-tuning: the modelﬁne-tuned only on labeled audio data yields the same performance on clean and noisy audio-visualinput (1.3% and 4.6%) as the model ﬁne-tuned with labeled audio-visual data, while being slightbehind on video input compared to the model ﬁne-tuned with labeled video data (31.6% vs. 28.7%).

Original: When it is compared to the model pre-trained without modality dropout and fine-tuned also withlabeled audio, huge gains are observed on zero-shot scenarios, for example 96.8% versus 31.6%
Revised: When it is compared to the model pre-trained without modality dropout and ﬁne-tuned also withlabeled audio, huge gains are observed on zero-shot scenarios, for example 96.8% versus 31.6%

Original: V-WER and 18.0% versus 4.6% noisy AV-WER.
Revised: V-WER and 18.0% versus 4.6% noisy AV-WER.

Original: Similar trend can be observed when fine-tuningwith labeled visual speech.
Revised: Similar trend can be observed when ﬁne-tuningwith labeled visual speech.

Original: Zero-shot modality transfer is not achieved through memorization Because the dataset used forfine-tuning is a subset of the pre-training data, there could be suspicions that the model memorizesthe audio-visual pairs from pre-training and associates them with the labeled unimodal data to enablezero-shot transfer.
Revised: Zero-shot modality transfer is not achieved through memorization Because the dataset used forﬁne-tuning is a subset of the pre-training data, there could be suspicions that the model memorizesthe audio-visual pairs from pre-training and associates them with the labeled unimodal data to enablezero-shot transfer.

Original: To avoid cheating, we fine-tune our model on LibriSpeech [Panayotov et al., 2015],an audio-only read audiobook dataset that is out-of-domain relative to the pre-training data composedof oratory talks (Table 3).
Revised: To avoid cheating, we ﬁne-tune our model on LibriSpeech [Panayotov et al., 2015],an audio-only read audiobook dataset that is out-of-domain relative to the pre-training data composedof oratory talks (Table 3).

Original: Despite the degradation compared to fine-tuning on in-domain LRS3 data,the model still performs decently on zero-shot transfer scenarios compared to models pre-trainedwithout modality dropout in Table 2.
Revised: Despite the degradation compared to ﬁne-tuning on in-domain LRS3 data,the model still performs decently on zero-shot transfer scenarios compared to models pre-trainedwithout modality dropout in Table 2.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: Comparison with state-of-the-art models Table 4 compares u-HuBERT with models from theliterature, all of which are modality-specific.
Revised: Comparison with state-of-the-art models Table 4 compares u-HuBERT with models from theliterature, all of which are modality-speciﬁc.

Original: In contrast, u-HuBERT is the only single model that canbe fine-tuned on any modality and test on all modalities while achieving performance on par or betterthan the best modality-specific models.
Revised: In contrast, u-HuBERT is the only single model that canbe ﬁne-tuned on any modality and test on all modalities while achieving performance on par or betterthan the best modality-speciﬁc models.

Original: Furthermore, compared to AV-HuBERT, u-HuBERT can bepre-trained on additional unimodal data to yield even better performance.[Table]
Revised: Furthermore, compared to AV-HuBERT, u-HuBERT can bepre-trained on additional unimodal data to yield even better performance.

Original: 4.4 Speaker verification
Revised: 4.4 Speaker veriﬁcation

Original: We repeat the experiment on speaker verification to study if u-HuBERT enables construction of asingle model and achieves zero-shot generalization on a different task.
Revised: We repeat the experiment on speaker veriﬁcation to study if u-HuBERT enables construction of asingle model and achieves zero-shot generalization on a different task.

Original: Fine-tuning setup We follow the protocol of SUPERB
Revised: Fine-tuning setup We follow the protocol of SUPERB

Original: [Yang et al., 2021], a standard benchmark forself-supervised speech representation evaluation.
Revised: [Yang et al., 2021], a standard benchmark forself-supervised speech representation evaluation.

Original: Specifically, we use VoxCeleb1 (VC1)
Revised: Speciﬁcally, we use VoxCeleb1 (VC1)

Original: [Nagraniet al., 2017] which contains 352 hours of speech from 1,251 speakers for fine-tuning.
Revised: [Nagraniet al., 2017] which contains 352 hours of speech from 1,251 speakers for ﬁne-tuning.

Original: Here theutterance-level speaker identity is the target label.
Revised: Here theutterance-level speaker identity is the target label.

Original: For visual modality, we use the face image framesprovided by Nagrani et al.
Revised: For visual modality, we use the face image framesprovided by Nagrani et al.

Original: [2018] and follow the same preprocessing steps as Shi et al.
Revised: [2018] and follow the same preprocessing steps as Shi et al.

Original: [2022c]to get the lip regions.
Revised: [2022c]to get the lip regions.

Original: Noise augmentation following Shi et al.
Revised: Noise augmentation following Shi et al.

Original: [2022b,c] is used to simulate noisyenvironments.
Revised: [2022b,c] is used to simulate noisyenvironments.

Original: The noise is sampled from {Babble, Speech, Music, Noise} categories in MUSANcorpus
Revised: The noise is sampled from {Babble, Speech, Music, Noise} categories in MUSANcorpus

Original: [Snyder et al., 2015] at SNR in {-10, -5, 0, 5, 10} dB
Revised: [Snyder et al., 2015] at SNR in {-10, -5, 0, 5, 10} dB

Original: and we report the average performanceover 20 configurations.
Revised: and we report the average performanceover 20 conﬁgurations.

Original: Speaker verification aims to evaluate how well the utterance-level representation captures speakersimilarity based on some distance metric in the space.
Revised: Speaker veriﬁcation aims to evaluate how well the utterance-level representation captures speakersimilarity based on some distance metric in the space.

Original: An X-vector
Revised: An X-vector

Original: [Snyder et al., 2018] network isadded following the SUPERB protocol, which first weighted-sums frame-level u-HuBERT representations over all Transformer layers, and then pools them temporally into a sequence-level representation.
Revised: [Snyder et al., 2018] network isadded following the SUPERB protocol, which ﬁrst weighted-sums frame-level u-HuBERT representations over all Transformer layers, and then pools them temporally into a sequence-level representation.

Original: The X-vector is randomly initialized and only added during fine-tuning.
Revised: The X-vector is randomly initialized and only added during ﬁne-tuning.

Original: The model is trained withthe widely used AMS-Softmax loss [Wang et al., 2018] with scale and margin being set to 0.4and 30 respectively.
Revised: The model is trained withthe widely used AMS-Softmax loss [Wang et al., 2018] with scale and margin being set to 0.4and 30 respectively.

Original: For evaluation, we compute the cosine similarity between two utterances andmeasure the equal error rate (EER).
Revised: For evaluation, we compute the cosine similarity between two utterances andmeasure the equal error rate (EER).

Original: During fine-tuning, the u-HuBERT model is frozen while onlythe downstream X-Vector network is fine-tuned.
Revised: During ﬁne-tuning, the u-HuBERT model is frozen while onlythe downstream X-Vector network is ﬁne-tuned.

Original: The model is trained with Adam [Kingma and Ba,2015].
Revised: The model is trained with Adam [Kingma and Ba,2015].

Original: The learning rate is searched within the range of [1e-7, 1e-1] and is kept fixed throughouttraining.
Revised: The learning rate is searched within the range of [1e-7, 1e-1] and is kept ﬁxed throughout [Table] training.

Original: Note the above fine-tuning and evaluation paradigm is based on SUPERB, which is for adirect comparison with prior works on the same benchmark.
Revised: Note the above ﬁne-tuning and evaluation paradigm is based on SUPERB, which is for adirect comparison with prior works on the same benchmark.

Original: See Appendix A and B.2.
Revised: See Appendix A and B.2.

Original: [Table]
Revised: [Table]

Original: Zero-shot performance Good audio-visual speaker representations aim to capture a number of relatedfeatures, such as voice, appearance, and speaking style (wording, intonation, rhythm).
Revised: Zero-shot performance Good audio-visual speaker representations aim to capture a number of relatedfeatures, such as voice, appearance, and speaking style (wording, intonation, rhythm).

Original: However, incontrast to speech recognition where content is highly correlated with each modality, for speakerverification each modality by itself can only infer a subset of these speaker features (e.g., appearancecan barely be inferred from audio).
Revised: However, incontrast to speech recognition where content is highly correlated with each modality, for speakerveriﬁcation each modality by itself can only infer a subset of these speaker features (e.g., appearancecan barely be inferred from audio).

Original: This imposes an inherent limitation to zero-shot transfer.
Revised: This imposes an inherent limitation to zero-shot transfer.

Original: We present the results in Table 5.
Revised: We present the results in Table 5.

Original: We found u-HuBERT fine-tuned with audio-only data can be used foraudio-visual speaker verification and yield better performance.
Revised: We found u-HuBERT ﬁne-tuned with audio-only data can be used foraudio-visual speaker veriﬁcation and yield better performance.

Original: In particular, it achieves 3.85%/8.92%
Revised: In particular, it achieves 3.85%/8.92%

Original: EER with audio-visual input compared to 4.25%/14.46% with audio input on clean/noisy test set.
Revised: EER with audio-visual input compared to 4.25%/14.46% with audio input on clean/noisy test set.

Original: With labeled audio data, the model learns to map voice and speaking style features to speakerrepresentations.
Revised: With labeled audio data, the model learns to map voice and speaking style features to speakerrepresentations.

Original: Hence, when tested with additional visual input, the model gains robustness oninferring voice and speaking style features, such that the performance on noisy test set is significantlyimproved (14.46% → 8.92% EER).
Revised: Hence, when tested with additional visual input, the model gains robustness oninferring voice and speaking style features, such that the performance on noisy test set is signiﬁcantlyimproved (14.46% ! 8.92% EER).

Original: However, this model does not learn to map appearance features(which could still be preserved in early Transformer layers that the X-vector head also takes asinput [Chang et al., 2022]) to speaker representations.
Revised: However, this model does not learn to map appearance features(which could still be preserved in early Transformer layers that the X-vector head also takes as input [Chang et al., 2022]) to speaker representations.

Original: As a result, the model performs much worsewith video input (17.90%) compared to the model fine-tuned on visual speech (5.01%), which learnsto map appearance and potentially speaking style features to speaker representations.
Revised: As a result, the model performs much worsewith video input (17.90%) compared to the model ﬁne-tuned on visual speech (5.01%), which learnsto map appearance and potentially speaking style features to speaker representations.

Original: Similarly, the audio fine-tuned model is also slightly behind audio-visual fine-tuned model when tested on audiovisual input (3.85% vs. 2.54%), as the latter could learn to map all features to speaker representations.
Revised: Similarly, theaudio ﬁne-tuned model is also slightly behind audio-visual ﬁne-tuned model when tested on audiovisual input (3.85% vs. 2.54%), as the latter could learn to map all features to speaker representations.

Original: On the other hand, when fine-tuning u-HuBERT with only visual speech, the model struggles togeneralize for audio-visual and audio input because mapping from voice to speaker is not learned.
Revised: On the other hand, when ﬁne-tuning u-HuBERT with only visual speech, the model struggles togeneralize for audio-visual and audio input because mapping from voice to speaker is not learned.

Original: Effect of modality dropout Modality dropout plays an important role in building a single modelwith multimodal labeled data and enabling zero-shot transfer with only unimodal labeled data.
Revised: Effect of modality dropout Modality dropout plays an important role in building a single modelwith multimodal labeled data and enabling zero-shot transfer with only unimodal labeled data.

Original: Inunimodal fine-tuning (audio-only or video-only), the performance consistently drops across allsettings but one (noisy AV-EER of model fine-tuned on V) when the model is pre-trained withoutmodality dropout.
Revised: Inunimodal ﬁne-tuning (audio-only or video-only), the performance consistently drops across allsettings but one (noisy AV-EER of model ﬁne-tuned on V) when the model is pre-trained withoutmodality dropout.

Original: In audio-visual fine-tuning, modality dropout greatly improves the performanceof a unified audio-visual model under single-modal test setting (A-EER: 7.40% → 4.23%, V-EER:10.73% → 6.53%).
Revised: In audio-visual ﬁne-tuning, modality dropout greatly improves the performanceof a uniﬁed audio-visual model under single-modal test setting (A-EER: 7.40% ! 4.23%, V-EER:10.73% !

Original: Generally, the single model trained with modality dropout in both pre-trainingand fine-tuning achieves a comparable performance compared to modality-specific systems (2.54%vs.
Revised: Generally, the single model trained with modality dropout in both pre-trainingand ﬁne-tuning achieves a comparable performance compared to modality-speciﬁc systems (2.54%vs.

Original: 2.31% in clean AV-EER, 4.23% vs. 4.25% in clean A-EER, 6.52% vs. 5.00% in V-EER).
Revised: 2.31% in clean AV-EER, 4.23% vs. 4.25% in clean A-EER, 6.52% vs. 5.00% in V-EER).

Original: Thevalue of dropout can be tuned in order to favor test performance of one specific modality, which weshow in the Appendix B.1.1.
Revised: Thevalue of dropout can be tuned in order to favor test performance of one speciﬁc modality, which weshow in the Appendix B.1.1.

Original:  Discussion
Revised:  Discussion

Original: We assume that there is an anchor modality where all other modalities appear in some multimodaldata paired with it.
Revised: We assume that there is an anchor modality where all other modalities appear in some multimodaldata paired with it.

Original: This enables the model to predict shared targets and learn modality-agnosticrepresentations for all modalities, such that non-anchor unimodal data can be used in the subsequentiterations.
Revised: This enables the model to predict shared targets and learn modality-agnosticrepresentations for all modalities, such that non-anchor unimodal data can be used in the subsequentiterations.

Original: While this may seem limiting, most multimodal speech are paired with audio [Richardet al., 2021, Hueber et al., 2010, Richmond et al., 2011, Livescu et al., 2009] since audio is the primarymeasurement for speech and other modalities are often supplementary.
Revised: While this may seem limiting, most multimodal speech are paired with audio [Richardet al., 2021, Hueber et al., 2010, Richmond et al., 2011, Livescu et al., 2009] since audio is the primarymeasurement for speech and other modalities are often supplementary.

Original: This makes the assumptionof having audio as the anchor modality practical.
Revised: This makes the assumptionof having audio as the anchor modality practical.

Original: Furthermore, the idea can be easily extended tosettings where any modality can be connected to the anchor via some intermediate modality.
Revised: Furthermore, the idea can be easily extended tosettings where any modality can be connected to the anchor via some intermediate modality.

Original: In short,a modality that is N -hop away from the primary anchor can be used in the ( N + 1) -th iteration.
Revised: In short,a modality that is N -hop away from the primary anchor can be used in the ( N + 1) -th iteration.

Original: The only case not supported u-HuBERT is pre-training on modalities that cannot be connectedthrough any multimodal data, for example, on a collection of audio, visual, audio-visual and EMGspeech.
Revised: The only case not supported u-HuBERT is pre-training on modalities that cannot be connectedthrough any multimodal data, for example, on a collection of audio, visual, audio-visual and EMGspeech.

Original: A shared codebook cannot be derived for EMG and the rest of the modalities, and it isunclear if the model can still benefit from joint training.
Revised: A shared codebook cannot be derived for EMG and the rest of the modalities, and it isunclear if the model can still beneﬁt from joint training.

Original: The setup has also been found much morechallenging on other pairs of modalities: for example, SLAM Bapna et al.
Revised: The setup has also been found much morechallenging on other pairs of modalities: for example, SLAM Bapna et al.

Original: [2021] shows that withoutpaired text and speech data, joint pre-training on unlabeled speech and text degrades the performancecompared to a model pre-trained only on speech.
Revised: [2021] shows that withoutpaired text and speech data, joint pre-training on unlabeled speech and text degrades the performancecompared to a model pre-trained only on speech.

Original: We leave such exploration to future work.
Revised: We leave such exploration to future work.

Original: Inaddition, we also include ethical discussion in Appendix C.
Revised: Inaddition, we also include ethical discussion in Appendix C.

Original:  Conclusion
Revised:  Conclusion

Original: Recent studies in applied machine learning trend toward building things that are general , for example,general perception module [Jaegle et al., 2021], general self-supervised objective [Baevski et al.
Revised: Recent studies in applied machine learning trend toward building things that are general , for example,general perception module [Jaegle et al., 2021], general self-supervised objective [Baevski et al.

Original: ,2022], single model for many tasks [Brown et al., 2020, Kaplan et al., 2020, Aghajanyan et al.
Revised: ,2022], single model for many tasks [Brown et al., 2020, Kaplan et al., 2020, Aghajanyan et al.

Original: ,2022] and/or many input [Girdhar et al., 2022, Hu and Singh, 2021], instead of optimizing forperformance on very specific setups.
Revised: ,2022] and/or many input [Girdhar et al., 2022, Hu and Singh, 2021], instead of optimizing forperformance on very speciﬁc setups.

Original: Our work contributes to this line of research by presenting aunified self-supervised objective to pre-train a model on speech of many modalities.
Revised: Our work contributes to this line of research by presenting auniﬁed self-supervised objective to pre-train a model on speech of many modalities.

Original: Specifically, itcan be fine-tuned for many different speech tasks on labeled data in any combination of pre-trainedmodalities, resulting in a single model that can process all combinations of pre-trained modalities toperform the fine-tuned task.
Revised: Speciﬁcally, itcan be ﬁne-tuned for many different speech tasks on labeled data in any combination of pre-trainedmodalities, resulting in a single model that can process all combinations of pre-trained modalities toperform the ﬁne-tuned task.

Original: We envision our work can bring substantial benefit to multimodal speech processing.
Revised: We envision our work can bring substantial beneﬁt to multimodal speech processing.

Original: Given there aresignificantly more unlabeled audio compared to unlabeled multimodal speech, being able to utilizeunlabeled audio can greatly improve the quality of multimodal speech representations.
Revised: Given there aresigniﬁcantly more unlabeled audio compared to unlabeled multimodal speech, being able to utilizeunlabeled audio can greatly improve the quality of multimodal speech representations.

Original: Furthermore,there are few or even no labeled multimodal speech for many tasks, such as speech translation.
Revised: Furthermore,there are few or even no labeled multimodal speech for many tasks, such as speech translation.

Original: Hence,it is essential to build a model capable of zero-shot input modality transfer to perform these taskswith modalities without labeled data. 
Revised: Hence,it is essential to build a model capable of zero-shot input modality transfer to perform these taskswith modalities without labeled data. Checklist For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: {+[Yes] See Section 
+}
Revised: {+[Yes] See Section 
+}

Revised: (b) Did

Revised: you describe the limitations of your work

Revised: [Yes]

Original: {+See Section 
+}
Revised: {+See Section 
+}

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] See

Revised: Appendix

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Revised: them?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A](b)

Revised: Did you include complete proofs of all theoretical results?

Revised: [N/A]3.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Revised: mental results (either in the supplemental material or as a URL)?

Revised: [No] Data are publicly

Revised: available.

Revised: Code and instructions will be released in the camera-ready version

Revised: (b) Did you

Revised: specify all the training details

Revised: (e.g., data splits, hyperparameters, how they

Revised: were

Revised: chosen

Revised: [Yes] See Section

Revised: 4 and

Revised: Appendix

Revised: A

Revised: (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [Yes] See Appendix B

Revised: (d) Did you include the total amount of compute and the type of resources used (e

Revised: g., type

Revised: of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Section 4 and

Revised: Appendix A

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] See Section

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] See Appendix A

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [No]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [Yes] See Appendix A

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [Yes] See Appendix A

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]