Original: O N U NI - MODAL FEATURE LEARNING IN S UPERVISED MULTI - MODAL LEARNING
Revised: O N U NI - MODAL FEATURE LEARNING IN S UPERVISED MULTI - MODAL LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We abstract the features of multi-modal data into 1) uni-modal features , which can be learned from uni-modal training, and 2) paired features , which can only be learned from cross-modal interaction.
Revised: We abstract the features of multi-modal data into 1) uni-modal features , which can be learned from uni-modal training, and 2) paired features , which can only be learned from cross-modal interaction.

Original: Multi-modal joint training is expected to benefit from cross-modal interaction on the basis of ensuring uni-modal feature learning.
Revised: Multi-modal joint training is expected to benefit from cross-modal interaction on the basis of ensuring uni-modal feature learning.

Original: However, recent late-fusion training approaches still suffer from insufficient learning of uni-modal features on each modality, and we prove that this phenomenon does hurt the model’s generalization ability.
Revised: However, recent late-fusion training approaches still suffer from insufficient learning of uni-modal features on each modality, and we prove that this phenomenon does hurt the model’s generalization ability.

Original: Given a multi-modal task, we propose to choose targeted late-fusion learning method from U ni-
Revised: Given a multi-modal task, we propose to choose targeted late-fusion learning method from U ni-

Original: M odal
Revised: M odal

Original: E nsemble (UME) and the proposed U ni-
Revised: E nsemble (UME) and the proposed U ni-

Original: M odal T eacher (UMT), according to the distribution of uni-modal and paired features.
Revised: M odal T eacher (UMT), according to the distribution of uni-modal and paired features.

Original: We demonstrate that, under a simple guiding strategy, we can achieve comparable results to other complex late-fusion or intermediate-fusion methods on multi-modal datasets, including VGG-Sound,
Revised: We demonstrate that, under a simple guiding strategy, we can achieve comparable results to other complex late-fusion or intermediate-fusion methods on multi-modal datasets, including VGG-Sound,

Original: Kinetics-400, UCF101, and ModelNet40.
Revised: Kinetics-400, UCF101, and ModelNet40.

Original: [Figure]
Revised: [Figure]

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-modal signals, e.g. , vision, sound, text, are ubiquitous in our daily life, allowing us to perceive the world through multiple sensory systems.
Revised: Multi-modal signals, e.g. , vision, sound, text, are ubiquitous in our daily life, allowing us to perceive the world through multiple sensory systems.

Original: Inspired by the crucial role that multi-modal interactions play in human perception and decision (Smith & Gasser, 2005), substantial efforts have been made to build effective and reliable computational multi-modal systems in fields like multimedia computing (Wang et al., 2020; Xiao et al., 2020), representation learning (Arandjelovic &
Revised: Inspired by the crucial role that multi-modal interactions play in human perception and decision (Smith & Gasser, 2005), substantial efforts have been made to build effective and reliable computational multi-modal systems in fields like multimedia computing (Wang et al., 2020; Xiao et al., 2020), representation learning (Arandjelovic &

Original: Zisserman, 2017; Radford et al., 2021) and robotics (Chen et al., 2020a).
Revised: Zisserman, 2017; Radford et al., 2021) and robotics (Chen et al., 2020a).

Original: According to how the features of multi-modal data can be learned, we abstract them into two categories: (1) uni-modal features , which can be learned from uni-modal training, and (2) paired features , which can only be learned from cross-modal interaction.
Revised: According to how the features of multi-modal data can be learned, we abstract them into two categories: (1) uni-modal features , which can be learned from uni-modal training, and (2) paired features , which can only be learned from cross-modal interaction.

Original: In this paper, we focus on multimodal tasks where uni-modal priors are meaningful 

Revised: In this paper, we focus on multimodal tasks where uni-modal priors are meaningful

Original: (Kay et al., 2017; Chen et al., 2020b).
Revised: (Kay et al., 2017; Chen et al., 2020b).

Original: Ideally, we hope that multi-modal joint training can learn paired features through cross-modal interactions on the basis of ensuring that enough uni-modal features are learned.
Revised: Ideally, we hope that multi-modal joint training can learn paired features through cross-modal interactions on the basis of ensuring that enough uni-modal features are learned.

Original: However, recent late-fusion methods still suffer from learning insufficient uni-modal representations of each modality (Peng et al., 2022).
Revised: However, recent late-fusion methods still suffer from learning insufficient uni-modal representations of each modality (Peng et al., 2022).

Original: We term this phenomenon as Modality Laziness and illustrate that in Figure 1.
Revised: We term this phenomenon as Modality Laziness and illustrate that in Figure 1.

Original: We theoretically characterize Modality Laziness and prove that it does hurt the generalization ability of the model , especially when uni-modal features are dominant in the given task.
Revised: We theoretically characterize Modality Laziness and prove that it does hurt the generalization ability of the model , especially when uni-modal features are dominant in the given task.

Original: Besides the laziness problem, another shortcoming of recent late-fusion approaches is that they are complex to implement.
Revised: Besides the laziness problem, another shortcoming of recent late-fusion approaches is that they are complex to implement.

Original: For example, G-Blending (Wang et al., 2020) needs an extra split of data to estimate the overfitting-to-generalization ratio to re-weight the losses and then re-train the model again and again.
Revised: For example, G-Blending (Wang et al., 2020) needs an extra split of data to estimate the overfitting-to-generalization ratio to re-weight the losses and then re-train the model again and again.

Original: Peng et al.
Revised: Peng et al.

Original: (2022) proposes OGM-GE, which dynamically adjusts the gradients of different modalities during training.
Revised: (2022) proposes OGM-GE, which dynamically adjusts the gradients of different modalities during training.

Original: However, it needs to tune too many hyper-parameters 2 , including the start and end epoch of the gradient modulation, an “alpha” used to calculate the coefficients for the modulation and whether adaptive Gaussian noise Enhancement (GE) is needed.
Revised: However, it needs to tune too many hyper-parameters 2 , including the start and end epoch of the gradient modulation, an “alpha” used to calculate the coefficients for the modulation and whether adaptive Gaussian noise Enhancement (GE) is needed.

Original: The more complicated thing is that these hyper-parameters need to be re-tuned on new datasets.
Revised: The more complicated thing is that these hyper-parameters need to be re-tuned on new datasets.

Original: To this end, more simple and effective methods are urgently needed.
Revised: To this end, more simple and effective methods are urgently needed.

Original: We pay attention to the learning of uni-modal features and propose to choose targeted late-fusion training method from U
Revised: We pay attention to the learning of uni-modal features and propose to choose targeted late-fusion training method from U

Original: ni-
Revised: ni-

Original: M odal
Revised: M odal

Original: E nsemble (UME) and proposed U ni-
Revised: E nsemble (UME) and proposed U ni-

Original: M odal T eacher (UMT) according to the distribution of unimodal and paired features.
Revised: M odal T eacher (UMT) according to the distribution of unimodal and paired features.

Original: If both uni-modal and paired features are essential, UMT is effective, which helps multi-modal models better learn uni-modal features via uni-modal distillation; if both modalities have strong uni-modal features and paired features are not important enough, UME is properer, which combines predictions of uni-modal models and completely avoids insufficient learning of uni-modal features.
Revised: If both uni-modal and paired features are essential, UMT is effective, which helps multi-modal models better learn uni-modal features via uni-modal distillation; if both modalities have strong uni-modal features and paired features are not important enough, UME is properer, which combines predictions of uni-modal models and completely avoids insufficient learning of uni-modal features.

Original: We also provide an empirical trick to decide which one to use.
Revised: We also provide an empirical trick to decide which one to use.

Original: Under this guidance, we achieve comparable results to other complex late-fusion or intermediate-fusion methods on multiple multi-modal datasets, including VGG-Sound (Chen et al., 2020b),
Revised: Under this guidance, we achieve comparable results to other complex late-fusion or intermediate-fusion methods on multiple multi-modal datasets, including VGG-Sound (Chen et al., 2020b),

Original: Kinetics400 (Kay et al., 2017), UCF101 (Soomro et al., 2012) and ModelNet40 (Wu et al., 2022).
Revised: Kinetics400 (Kay et al., 2017), UCF101 (Soomro et al., 2012) and ModelNet40 (Wu et al., 2022).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-modal training approaches aim to train a multi-modal model by using all modalities simultaneously (Liang et al., 2021), including audio-visual classification (Peng et al., 2022; Xiao et al., 2020; Panda et al., 2021), action recognition (Wang et al., 2020; Panda et al., 2021), visual question answering (Agrawal et al., 2018) and RGB-D segmentation (Park et al., 2017;
Revised: Multi-modal training approaches aim to train a multi-modal model by using all modalities simultaneously (Liang et al., 2021), including audio-visual classification (Peng et al., 2022; Xiao et al., 2020; Panda et al., 2021), action recognition (Wang et al., 2020; Panda et al., 2021), visual question answering (Agrawal et al., 2018) and RGB-D segmentation (Park et al., 2017;

Original: Hu et al., 2019;
Revised: Hu et al., 2019;

Original: Seichter et al., 2020).
Revised: Seichter et al., 2020).

Original: There are several different fusion methods, including early/middle fusion (Seichter et al., 2020; Nagrani et al., 2021;
Revised: There are several different fusion methods, including early/middle fusion (Seichter et al., 2020; Nagrani et al., 2021;

Original: Wu et al., 2022) and late fusion (Wang et al., 2020; Peng et al., 2022; Fayek & Kumar, 2020).
Revised: Wu et al., 2022) and late fusion (Wang et al., 2020; Peng et al., 2022; Fayek & Kumar, 2020).

Original: In this paper, we mainly consider the late-fusion methods following Wang et al.
Revised: In this paper, we mainly consider the late-fusion methods following Wang et al.

Original: (2020), which is convenient and straightforward to evaluate the learning of uni-modal features.
Revised: (2020), which is convenient and straightforward to evaluate the learning of uni-modal features.

Original: We demonstrate that simple late-fusion approaches can outperform approaches with more complex model architecture (Wu et al., 2022; Xiao et al., 2020).
Revised: We demonstrate that simple late-fusion approaches can outperform approaches with more complex model architecture (Wu et al., 2022; Xiao et al., 2020).

Original: Multi-modal learning theory.
Revised: Multi-modal learning theory.

Original: The research on multi-modal learning theory is still at an early age.
Revised: The research on multi-modal learning theory is still at an early age.

Original: A line of work focuses on understanding multi-view tasks (Amini et al., 2009; Xu et al., 2013; Arora et al., 2016; Allen-Zhu & Li, 2020), and our assumption on the data structure partially stems from
Revised: A line of work focuses on understanding multi-view tasks (Amini et al., 2009; Xu et al., 2013; Arora et al., 2016; Allen-Zhu & Li, 2020), and our assumption on the data structure partially stems from

Original: Allen-Zhu & Li (2020).
Revised: Allen-Zhu & Li (2020).

Original: Huang et al.
Revised: Huang et al.

Original: (2021) explains multi-modal learning is potentially better than uni-modal learning and Huang et al.
Revised: (2021) explains multi-modal learning is potentially better than uni-modal learning and Huang et al.

Original: (2022) explains why failure exists in multi-modal learning.
Revised: (2022) explains why failure exists in multi-modal learning.

Original: Our paper investigates the different types of features in multi-modal data and provides solutions for the weakness of multi-modal learning.
Revised: Our paper investigates the different types of features in multi-modal data and provides solutions for the weakness of multi-modal learning.

Original: Knowledge distillation was introduced to compress the knowledge from an ensemble into a smaller and faster model but still preserve competitive generalization power (Buciluˇa et al., 2006; Hinton et al., 2015; Tian et al., 2019; Gou et al., 2021; Allen-Zhu & Li, 2020).
Revised: Knowledge distillation was introduced to compress the knowledge from an ensemble into a smaller and faster model but still preserve competitive generalization power (Buciluˇa et al., 2006; Hinton et al., 2015; Tian et al., 2019; Gou et al., 2021; Allen-Zhu & Li, 2020).

Original: In this paper, we propose
Revised: In this paper, we propose

Original: Uni-Modal Teacher to leverage uni-modal distillation for joint training to help the learning of unimodal features, without involving cross-modal knowledge distillation (Pham et al., 2019; Gupta et al., 2016; Tan & Bansal, 2020; Garcia et al., 2018; Luo et al., 2018).
Revised: Uni-Modal Teacher to leverage uni-modal distillation for joint training to help the learning of unimodal features, without involving cross-modal knowledge distillation (Pham et al., 2019; Gupta et al., 2016; Tan & Bansal, 2020; Garcia et al., 2018; Luo et al., 2018).

Original:  A NALYSIS , L EARNING G UIDANCE AND T HEORY
Revised:  A NALYSIS , L EARNING G UIDANCE AND T HEORY

Original: In this section, we show the drawbacks and advantages of joint training.
Revised: In this section, we show the drawbacks and advantages of joint training.

Original: On one hand, joint training results in insufficient learning of uni-modal features (Modality Laziness).
Revised: On one hand, joint training results in insufficient learning of uni-modal features (Modality Laziness).

Original: On the other hand, it
Revised: On the other hand, it

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: allows interactions between modalities to learn representations beyond uni-modal features, namely paired features.
Revised: allows interactions between modalities to learn representations beyond uni-modal features, namely paired features.

Original: Based on this, we offer guidance on multi-modal late-fusion learning.
Revised: Based on this, we offer guidance on multi-modal late-fusion learning.

Original: Finally
Revised: Finally

Original: [Table]
Revised: [Table]

Original: we provide a theoretical analysis of Modality Laziness and justification for our solution.
Revised: we provide a theoretical analysis of Modality Laziness and justification for our solution.

Original: Discussion.
Revised: Discussion.

Original: The importance of uni-modal prior varies across different multi-modal tasks.
Revised: The importance of uni-modal prior varies across different multi-modal tasks.

Original: In tasks like video classification (Chen et al., 2020b) and action recognition (Feichtenhofer et al., 2016;
Revised: In tasks like video classification (Chen et al., 2020b) and action recognition (Feichtenhofer et al., 2016;

Original: Wang et al., 2020), uni-modal models can achieve good performance alone, suggesting that unimodal priors in these settings are essential.
Revised: Wang et al., 2020), uni-modal models can achieve good performance alone, suggesting that unimodal priors in these settings are essential.

Original: Visual question and answering (VQA) (Agrawal et al., 2018) is a counter example.
Revised: Visual question and answering (VQA) (Agrawal et al., 2018) is a counter example.

Original: Specifically, the same image with different text questions may have totally different labels, making it pointless to check its uni-modal accuracy.
Revised: Specifically, the same image with different text questions may have totally different labels, making it pointless to check its uni-modal accuracy.

Original: In this paper, we focus on the tasks where uni-modal priors are essential , following Wu et al.
Revised: In this paper, we focus on the tasks where uni-modal priors are essential , following Wu et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: 3. I NSUFFICIENT L EARNING OF U NI - MODAL F EATURES IN M ULTI - MODAL T RAINING
Revised: 3. I NSUFFICIENT L EARNING OF U NI - MODAL F EATURES IN M ULTI - MODAL T RAINING

Original: This subsection illustrates that existing multi-modal late-fusion training methods suffer from Modality Laziness.
Revised: This subsection illustrates that existing multi-modal late-fusion training methods suffer from Modality Laziness.

Original: Even recent methods, G-Blending (Wang et al., 2020) and OGM-GE (Peng et al., 2022), are no exception.
Revised: Even recent methods, G-Blending (Wang et al., 2020) and OGM-GE (Peng et al., 2022), are no exception.

Original: In multi-modal late-fusion learning , each modality is encoded by its corresponding encoder and then a fusion module is applied on top of them to produce outputs.
Revised: In multi-modal late-fusion learning , each modality is encoded by its corresponding encoder and then a fusion module is applied on top of them to produce outputs.

Original: By building a classifier on frozen encoder (Chen et al., 2020c), we assess the learned representations of the encoder:
Revised: By building a classifier on frozen encoder (Chen et al., 2020c), we assess the learned representations of the encoder:

Original: • As Table 1 shows, all encoders from multi-modal training are worse than those from unimodal training, especially the RGB encoder in VGG-Sound and optical flow encoder in UCF101.
Revised: • As Table 1 shows, all encoders from multi-modal training are worse than those from unimodal training, especially the RGB encoder in VGG-Sound and optical flow encoder in UCF101.

Original: No matter which optimizer is used (Appendix A.3).
Revised: No matter which optimizer is used (Appendix A.3).

Original: [Table]
Revised: [Table]

Original: Input: Uni-modal supervised pre-trained models F m 1 pretrain , F m 2 pretrain , random initialized latefusion multi-modal model F mm , iteration number N , loss weight λ task , λ distill .
Revised: Input: Uni-modal supervised pre-trained models F m 1 pretrain , F m 2 pretrain , random initialized latefusion multi-modal model F mm , iteration number N , loss weight λ task , λ distill .

Original: for 0 to N do
Revised: for 0 to N do

Original: Sample multi-modal data { X m 1 , X m 2 , Y } ∼ D .
Revised: Sample multi-modal data { X m 1 , X m 2 , Y } ∼ D .

Original: Compute uni-modal pre-trained features f m 1 pre , f m 2 pre of the data by F m 1 pretrain , F m 2 pretrain .
Revised: Compute uni-modal pre-trained features f m 1 pre , f m 2 pre of the data by F m 1 pretrain , F m 2 pretrain .

Original: Compute the prediction and features ˆ Y , f m 1 , f m 2 from multi-modal model.
Revised: Compute the prediction and features ˆ Y , f m 1 , f m 2 from multi-modal model.

Original: Compute the losses between ˆ Y , f m 1 , f m 2 and Y, f m 1 pre , f m 2 pre and multiply by the λ task , λ distill , λ distill , respectively.
Revised: Compute the losses between ˆ Y , f m 1 , f m 2 and Y, f m 1 pre , f m 2 pre and multiply by the λ task , λ distill , λ distill , respectively.

Original: Update the multi-modal model by SGD or its variant.
Revised: Update the multi-modal model by SGD or its variant.

Original: end for
Revised: end for

Original: Return: A multi-modal model trained by UMT.
Revised: Return: A multi-modal model trained by UMT.

Original: • As Figure 2 shows, throughout the training process, the two encoders mentioned above not only cannot achieve comparable performance to their uni-modal counterparts but are far worse than them.
Revised: • As Figure 2 shows, throughout the training process, the two encoders mentioned above not only cannot achieve comparable performance to their uni-modal counterparts but are far worse than them.

Original: 3. H OW DOES A M ODEL B ENEFIT FROM M ULTI - MODAL T RAINING ?
Revised: 3. H OW DOES A M ODEL B ENEFIT FROM M ULTI - MODAL T RAINING ?

Original: In Sec 3.1, we empirically show that recent late-fusion methods suffer from insufficient learning of uni-modal features.
Revised: In Sec 3.1, we empirically show that recent late-fusion methods suffer from insufficient learning of uni-modal features.

Original: Combining predictions from uni-modal trained models avoids laziness by nature, which raises another question: How does a multi-modal model benefit from multi-modal training?
Revised: Combining predictions from uni-modal trained models avoids laziness by nature, which raises another question: How does a multi-modal model benefit from multi-modal training?

Original: We answer this question by investigating different models on VGG-Sound and find that the model learns some representations beyond uni-modal features.
Revised: We answer this question by investigating different models on VGG-Sound and find that the model learns some representations beyond uni-modal features.

Original: As Table 2 shows, in certain classes of VGG-Sound, the accuracy of naive fusion (navie fusion or naive multi-modal late-fusion learning means no carefully designed tricks are used) exceeds the sum of the accuracy of the two uni-modal models.
Revised: As Table 2 shows, in certain classes of VGG-Sound, the accuracy of naive fusion (navie fusion or naive multi-modal late-fusion learning means no carefully designed tricks are used) exceeds the sum of the accuracy of the two uni-modal models.

Original: Besides, we evaluate two other methods.
Revised: Besides, we evaluate two other methods.

Original: One is to directly average the uni-modal models’ predictions, which has little cross-modal interaction.
Revised: One is to directly average the uni-modal models’ predictions, which has little cross-modal interaction.

Original: The other one is to train a multi-modal linear classifier on top of uni-modal pre-trained encoders, where modalities can interact with each other through the linear layer.
Revised: The other one is to train a multi-modal linear classifier on top of uni-modal pre-trained encoders, where modalities can interact with each other through the linear layer.

Original: We find that naive fusion training, which owns maximum freedom of cross-modal interaction among these models, gets the best mean accuracy across these classes, suggesting that joint training enables the model to learn representations beyond uni-modal features, which we term as paired features .
Revised: We find that naive fusion training, which owns maximum freedom of cross-modal interaction among these models, gets the best mean accuracy across these classes, suggesting that joint training enables the model to learn representations beyond uni-modal features, which we term as paired features .

Original: They are a type of feature that uni-modal training cannot learn.
Revised: They are a type of feature that uni-modal training cannot learn.

Original: We offer more explanations on paired features in Appendix A.10.
Revised: We offer more explanations on paired features in Appendix A.10.

Original: We also analyze more datasets and find that different datasets have different characteristics, we put more experimental analysis and interpretation in Appendix A.7.
Revised: We also analyze more datasets and find that different datasets have different characteristics, we put more experimental analysis and interpretation in Appendix A.7.

Original: 3. G UIDANCE ON M ULTI - MODAL L EARNING
Revised: 3. G UIDANCE ON M ULTI - MODAL L EARNING

Original: Given a multi-modal task, if both uni-modal features and paired features are essential,
Revised: Given a multi-modal task, if both uni-modal features and paired features are essential,

Original: U ni-
Revised: U ni-

Original: M odal
Revised: M odal

Original: T eacher (UMT) is effective; if both modalities have strong uni-modal features and paired features are not important enough, simply combining the predictions of uni-modal models works well, which is named as U
Revised: T eacher (UMT) is effective; if both modalities have strong uni-modal features and paired features are not important enough, simply combining the predictions of uni-modal models works well, which is named as U

Original: ni-
Revised: ni-

Original: M odal E nsemble (UME).
Revised: M odal E nsemble (UME).

Original: UMT.
Revised: UMT.

Original: U ni-
Revised: U ni-

Original: M odal T eacher (UMT) is proposed for late-fusion joint training.
Revised: M odal T eacher (UMT) is proposed for late-fusion joint training.

Original: It distills the pre-trained uni-modal features to the corresponding parts of multi-modal late-fusion models.
Revised: It distills the pre-trained uni-modal features to the corresponding parts of multi-modal late-fusion models.

Original: Distilling knowledge from uni-modal models can help multi-modal models learn uni-modal features better, which happens in feature-level.
Revised: Distilling knowledge from uni-modal models can help multi-modal models learn uni-modal features better, which happens in feature-level.

Original: The framework of UMT is shown in Algorithm 1 and Figure 4.
Revised: The framework of UMT is shown in Algorithm 1 and Figure 4.

Original: Noting that we use the same backbone in uni-modal and multi-modal model for a specified modality.
Revised: Noting that we use the same backbone in uni-modal and multi-modal model for a specified modality.

Original: The backbones and loss function used can be found in Sec 4.
Revised: The backbones and loss function used can be found in Sec 4.

Original: More details can be found in Appendix A.4.
Revised: More details can be found in Appendix A.4.

Original: UME.
Revised: UME.

Original: If both modalities have strong uni-modal features, joint training does more harm than good.
Revised: If both modalities have strong uni-modal features, joint training does more harm than good.

Original: Combining predictions of uni-modal models avoids insufficient learning of uni-modal features by nature.
Revised: Combining predictions of uni-modal models avoids insufficient learning of uni-modal features by nature.

Original: Firstly, we can train uni-modal models independently.
Revised: Firstly, we can train uni-modal models independently.

Original: Then, we can give final output by weighting the predictions of uni-modal models.
Revised: Then, we can give final output by weighting the predictions of uni-modal models.

Original: The simple ensemble method is named as U niM odal E nsemble (UME).
Revised: The simple ensemble method is named as U niM odal E nsemble (UME).

Original: We demonstrate that UME can show competitive performance on certain multi-modal datasets.
Revised: We demonstrate that UME can show competitive performance on certain multi-modal datasets.

Original: An empirical trick to decide which method to use.
Revised: An empirical trick to decide which method to use.

Original: We can train a multi-modal linear classifier on uni-modal pre-trained encoders and compare that with averaging predictions of uni-modal models.
Revised: We can train a multi-modal linear classifier on uni-modal pre-trained encoders and compare that with averaging predictions of uni-modal models.

Original: If the performance of the classifier is better, it means we can benefit from cross-modal interaction in this task and we can choose UMT, where cross-modal interactions are preserved while guaranteeing improved learning of uni-modal features; otherwise, the simple cross-modal interaction does more harm than good because of the strong uni-modal features of each modality, and we can choose UME, which avoids Modality Laziness completely.
Revised: If the performance of the classifier is better, it means we can benefit from cross-modal interaction in this task and we can choose UMT, where cross-modal interactions are preserved while guaranteeing improved learning of uni-modal features; otherwise, the simple cross-modal interaction does more harm than good because of the strong uni-modal features of each modality, and we can choose UME, which avoids Modality Laziness completely.

Original: 3. T HEORETICAL C HARACTERIZATION AND J USTIFICATION
Revised: 3. T HEORETICAL C HARACTERIZATION AND J USTIFICATION

Original: In this subsection, we characterize Modality Laziness of Sec 3.1 from a feature learning perspective and prove it does hurt the generalization of the model.
Revised: In this subsection, we characterize Modality Laziness of Sec 3.1 from a feature learning perspective and prove it does hurt the generalization of the model.

Original: And then, we give justification for the learning guidance proposed in Sec 3.3.
Revised: And then, we give justification for the learning guidance proposed in Sec 3.3.

Original: Before diving into the technical details, we first provide some intuition behind the proof.
Revised: Before diving into the technical details, we first provide some intuition behind the proof.

Original: Our goal is to show that how Modality Laziness happens in multi-modal joint training, and we refer to Figureas an illustration.
Revised: Our goal is to show that how Modality Laziness happens in multi-modal joint training, and we refer to Figureas an illustration.

Original: Here we omit the effect of paired features for easier to understand the intuition.
Revised: Here we omit the effect of paired features for easier to understand the intuition.

Original: During the naive multi-modal training process, learning those easy-to-learn features suffices to reach zero training error (point A in Figure 3).
Revised: During the naive multi-modal training process, learning those easy-to-learn features suffices to reach zero training error (point A in Figure 3).

Original: However, the model is under-trained at point A, and the zero-training-error region stops us from further training.
Revised: However, the model is under-trained at point A, and the zero-training-error region stops us from further training.

Original: As a comparison, uni-modal models can learn more features and achieve point B, outperforming point A.
Revised: As a comparison, uni-modal models can learn more features and achieve point B, outperforming point A.

Original: We next give Modality Laziness a theoretical explanation under a simple but effective regime.
Revised: We next give Modality Laziness a theoretical explanation under a simple but effective regime.

Original: We mainly consider cases with two modalities x m 1 and x m 2 , and similar techniques can be directly generalized to the cases with more modalities .
Revised: We mainly consider cases with two modalities x m 1 and x m 2 , and similar techniques can be directly generalized to the cases with more modalities .

Original: Data distribution.
Revised: Data distribution.

Original: We formalize the distribution of the multi-modal features.
Revised: We formalize the distribution of the multi-modal features.

Original: Specifically, we abstract the features into uni-modal features (Definition 3.1) and paired features (Definition 3.2) to describe the core differences between uni-modal training and multi-modal joint training.
Revised: Specifically, we abstract the features into uni-modal features (Definition 3.1) and paired features (Definition 3.2) to describe the core differences between uni-modal training and multi-modal joint training.

Original: We consider the binary classification regime where the label y has a uniform distribution over {− 1 , 1 } without loss of generality.
Revised: We consider the binary classification regime where the label y has a uniform distribution over {− 1 , 1 } without loss of generality.

Original: Such simplification is self-contained to describe the differences between uni-modal features and paired features.
Revised: Such simplification is self-contained to describe the differences between uni-modal features and paired features.

Original: Definition 3.1 (Uni-modal features, which can be learned from uni-modal training ) .
Revised: Definition 3.1 (Uni-modal features, which can be learned from uni-modal training ) .

Original: The i -th unimodal feature ( f i ( x m 1 ) ) in modality x m 1 is generated as 3 :
Revised: The i -th unimodal feature ( f i ( x m 1 ) ) in modality x m 1 is generated as 3 :

Original: [Equation]
Revised: [Equation]

Original: < 0 .
Revised: < 0 .

Original: The i -th uni-modal feature ( g i ( x m 2 ) ) in modality x m 2 is similarly generated with parameters p
Revised: The i -th uni-modal feature ( g i ( x m 2 ) ) in modality x m 2 is similarly generated with parameters p

Original: ( g i ) and ϵ ( g i ) .
Revised: ( g i ) and ϵ ( g i ) .

Original: ℱ ! ℱ !
Revised: ℱ ! ℱ !

Original: 𝑥 " !
Revised: 𝑥 " !

Original: ℱ
Revised: ℱ

Original: # ′ ℱ
Revised: # ′ ℱ

Original: # 𝑥 " " ℱ # 𝑥 " " ℱ !
Revised: # 𝑥 " " ℱ # 𝑥 " " ℱ !

Original: 𝑥 " !
Revised: 𝑥 " !

Original: 𝑨 𝑨 𝑩 𝑩
Revised: 𝑨 𝑨 𝑩 𝑩

Original: Training Procedure
Revised: Training Procedure

Original: Testing Procedure
Revised: Testing Procedure

Original: [Figure]
Revised: [Figure]

Original: error decreases
Revised: error decreases

Original: [Figure]
Revised: [Figure]

Original: Region
Revised: Region

Original: [Figure]
Revised: [Figure]

Original: training error ≠
Revised: training error ≠

Original: [Figure]
Revised: [Figure]

Original: : Region where training
Revised: : Region where training

Original: [Figure]
Revised: [Figure]

Original: /𝓕 𝒊 ′
Revised: /𝓕 𝒊 ′

Original: [Figure]
Revised: [Figure]

Original: learned
Revised: learned

Original: [Figure]
Revised: [Figure]

Original: modality 𝑥
Revised: modality 𝑥

Original: [Figure]
Revised: [Figure]

Original: uni
Revised: uni

Original: [Figure]
Revised: [Figure]

Original: multi
Revised: multi

Original: [Figure]
Revised: [Figure]

Original: modal
Revised: modal

Original: [Figure]
Revised: [Figure]

Original: Definition 3.2 (Paired features, which can only be learned from cross-modal interaction ) .
Revised: Definition 3.2 (Paired features, which can only be learned from cross-modal interaction ) .

Original: The j -th paired feature 4 h j is generated as:
Revised: The j -th paired feature 4 h j is generated as:

Original: [Equation]
Revised: [Equation]

Original: When the context is clear, we abuse the notation r i to represent either f i (uni-modal feature in modality x m 1 ), g i (uni-modal feature in modality x m 2 ), or h i (paired feature).
Revised: When the context is clear, we abuse the notation r i to represent either f i (uni-modal feature in modality x m 1 ), g i (uni-modal feature in modality x m 2 ), or h i (paired feature).

Original: We name p ( r i ) as the predicting probability of feature r i .
Revised: We name p ( r i ) as the predicting probability of feature r i .

Original: When r i is present (meaning that r i ̸ = 0 ), we use I ( r i > 0) − I ( r i < 0) to predict y .
Revised: When r i is present (meaning that r i ̸ = 0 ), we use I ( r i > 0) − I ( r i < 0) to predict y .

Original: Otherwise ( r i = 0 ), we random guess y uniformly over {− 1 , 1 } .
Revised: Otherwise ( r i = 0 ), we random guess y uniformly over {− 1 , 1 } .

Original: To simplify the discussion, we always assume ϵ ( f i ) =
Revised: To simplify the discussion, we always assume ϵ ( f i ) =

Original: p ( f i ) /c , where c > 1 is a fixed constant.
Revised: p ( f i ) /c , where c > 1 is a fixed constant.

Original: For the ease of notations, we define the empty feature in Definition 3.3.
Revised: For the ease of notations, we define the empty feature in Definition 3.3.

Original: Definition 3.3 (Empty Feature) .
Revised: Definition 3.3 (Empty Feature) .

Original: Empty feature e i is a kind of uni-modal feature (or paired feature) with p ( e i )
Revised: Empty feature e i is a kind of uni-modal feature (or paired feature) with p ( e i )

Original: =
Revised: =

Original: ϵ ( e i )
Revised: ϵ ( e i )

Original: = 0 .
Revised: = 0 .

Original: Evaluation procedure.
Revised: Evaluation procedure.

Original: When the context is clear, we abuse r i to denote the learned features.
Revised: When the context is clear, we abuse r i to denote the learned features.

Original: For each data point, we random guess ˆ y on {− 1 , 1 } uniformly when (cid:80) i I ( r i > 0)
Revised: For each data point, we random guess ˆ y on {− 1 , 1 } uniformly when (cid:80) i I ( r i > 0)

Original: = (cid:80) i
Revised: = (cid:80) i

Original: I ( r i < 0) .
Revised: I ( r i < 0) .

Original: Otherwise, we predict the label by ˆ y =
Revised: Otherwise, we predict the label by ˆ y =

Original: I ( (cid:80) i I ( r i > 0) > (cid:80) i I ( r i < 0)) − 1 .
Revised: I ( (cid:80) i I ( r i > 0) > (cid:80) i I ( r i < 0)) − 1 .

Original: We define the error as (cid:80) i I (
Revised: We define the error as (cid:80) i I (

Original: yr i < 0) − (cid:80)
Revised: yr i < 0) − (cid:80)

Original: i
Revised: i

Original: I ( yr i > 0) .
Revised: I ( yr i > 0) .

Original: Training procedure.
Revised: Training procedure.

Original: (a.) multi-modal joint training , which directly train the model using both modality x m 1 and modality x m 2 ; (b.) uni-modal ensemble , which firstly train the features via independent training ( x m 1 and x m 2 separately), and then combine the x m 1 -learned features and x m 2 -learned features.
Revised: (a.) multi-modal joint training , which directly train the model using both modality x m 1 and modality x m 2 ; (b.) uni-modal ensemble , which firstly train the features via independent training ( x m 1 and x m 2 separately), and then combine the x m 1 -learned features and x m 2 -learned features.

Original: During the training process, we first initialize all the features with empty features e i to imitate random initialization.
Revised: During the training process, we first initialize all the features with empty features e i to imitate random initialization.

Original: The models then learn the features in descending order of predicting probability, meaning that the powerful features (with large predicting probability) are learned first 5 .
Revised: The models then learn the features in descending order of predicting probability, meaning that the powerful features (with large predicting probability) are learned first 5 .

Original: Our goal is to minimize the training error to zero 6 .
Revised: Our goal is to minimize the training error to zero 6 .

Original: We now state our main theorem in Theorem 3.4, demonstrating naive joint training learn fewer uni-modal features compared to uni-modal training, which hurts the model’s generalization.
Revised: We now state our main theorem in Theorem 3.4, demonstrating naive joint training learn fewer uni-modal features compared to uni-modal training, which hurts the model’s generalization.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: In uni-modal ensemble, assume that the training procedure learns b m1 features in modality x m 1 and learns b m2 features in modality x m 2 .
Revised: In uni-modal ensemble, assume that the training procedure learns b m1 features in modality x m 1 and learns b m2 features in modality x m 2 .

Original: We order the probability of uni-modal fea- tures (both x m 1 and x m 2 ) in decreasing order of predicting probability p , namely, p [1] , p [2] , . . . .
Revised: We order the probability of uni-modal fea- tures (both x m 1 and x m 2 ) in decreasing order of predicting probability p , namely, p [1] , p [2] , . . . .

Original: In multi-modal training approaches, assume that the training procedure learns k m1 uni-modal features in modality x m 1 , learns k m2 uni-modal features in modality x m 2 , and learns k pa paired features with predicting probability p ( h 1 ) , . .
Revised: In multi-modal training approaches, assume that the training procedure learns k m1 uni-modal features in modality x m 1 , learns k m2 uni-modal features in modality x m 2 , and learns k pa paired features with predicting probability p ( h 1 ) , . .

Original: , p ( h k pa ) .
Revised: , p ( h k pa ) .

Original: We provide three types of laziness:
Revised: We provide three types of laziness:

Original: (a. ) Quantity Laziness : k m1 + k m2 + k pa ≤ min { b m1 , b m2 } .
Revised: (a. ) Quantity Laziness : k m1 + k m2 + k pa ≤ min { b m1 , b m2 } .

Original: (b. )
Revised: (b. )

Original: Uni-modal Laziness : Each modality in multi-modal training approaches performs worse than uni-modal training.
Revised: Uni-modal Laziness : Each modality in multi-modal training approaches performs worse than uni-modal training.

Original: (c. )
Revised: (c. )

Original: Performance Laziness : Consider a new testing point, then for every δ > 0 , if the following inequality holds:
Revised: Performance Laziness : Consider a new testing point, then for every δ > 0 , if the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where ∆( δ ) =
Revised: where ∆( δ ) =

Original: (cid:112) 8( k pa + b m1 − k m1 + b m2 − k m2 ) log(1 /δ ) , then with probability 7 at least 1 − δ , uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.
Revised: (cid:112) 8( k pa + b m1 − k m1 + b m2 − k m2 ) log(1 /δ ) , then with probability 7 at least 1 − δ , uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.

Original: In theorem 3.4, we describe three notations of laziness problem: Quantity Laziness indicates that the number of features learned in naive multi-modal training is less than uni-modal training.
Revised: In theorem 3.4, we describe three notations of laziness problem: Quantity Laziness indicates that the number of features learned in naive multi-modal training is less than uni-modal training.

Original: Unimodal Laziness shows encoders from multi-modal training perform worse than from uni-modal training because of Quantity Laziness.
Revised: Unimodal Laziness shows encoders from multi-modal training perform worse than from uni-modal training because of Quantity Laziness, which fits the experimental results in sec3.1.

Original: Performance Laziness compares the performance of multimodal joint training approaches with Uni-Modal Ensemble, demonstrating that when uni-modal features dominate, combining uni-modal predictions is more effective.
Revised: Performance Laziness compares the performance of multi-modal joint training approaches with Uni-Modal Ensemble, demonstrating that when uni-modal features dominate, combining uni-modal predictions is more effective.

Original: We defer the complete proof to Appendix B.1 andit’s easy to generalize the theorem to more modalities (Appendix B.2).
Revised: We defer the complete proof to Appendix B.1 and generalize that to more modalities (Appendix B.2).

Original: Besides, we give a concrete example in Appendix B.3 to better illustrate Theorem 3.4.
Revised: We give a concrete example in Appendix B.3 to better illustrate Theorem 3.4.

Original: We next prove that UMT proposed in Sec 3.3 indeed helps uni-modal feature learning and can also learn some easy-to-learn paired features in Theorem 3.5 and Appendix B.3.
Revised: We next prove that UMT proposed in Sec 3.3 indeed helps uni-modal feature learning and can also learn some easy-to-learn paired features in Theorem 3.5 and Appendix B.3.

Original: Theorem 3.5.
Revised: Theorem 3.5.

Original: Denote the paired features by h 1 , . . .
Revised: Denote the paired features by h 1 , . . .

Original: h L with corresponding predicting probability p ( h 1 ) , . .
Revised: h L with corresponding predicting probability p ( h 1 ) , . .

Original: , p ( h L ) .
Revised: , p ( h L ) .

Original: Assume that distillation can boost the training priority by p 0 > 0 .
Revised: Assume that distillation can boost the training priority by p 0 > 0 .

Original: If there exists paired features whose predicting probability exceeds the boosting probability p 0 , namely, the set S is not empty:
Revised: If there exists paired features whose predicting probability exceeds the boosting probability p 0 , namely, the set S is not empty:

Original: [Equation]
Revised: [Equation]

Original: Then UMT helps uni-modal feature learning and can also learn easy-to-learn paired features.
Revised: Then UMT helps uni-modal feature learning and can also learn easy-to-learn paired features.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In Sec 3.4, we justify our method theoretically.
Revised: In Sec 3.4, we justify our method theoretically.

Original: In this section, we firstly introduce the experimental setup and then demonstrate that choosing a suitable learning method from UMT and UME can outperforming other complex late-fusion or intermediate-fusion methods in various multi-modal tasks.
Revised: In this section, we firstly introduce the experimental setup and then demonstrate that choosing a suitable learning method from UMT and UME can outperforming other complex late-fusion or intermediate-fusion methods in various multi-modal tasks.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Dataset.
Revised: Dataset.

Original: We run experiments on four datasets.
Revised: We run experiments on four datasets.

Original: Kinetics-400 (Kay et al., 2017) is a video recognition dataset with 240k videos for training and 19k for validation.
Revised: Kinetics-400 (Kay et al., 2017) is a video recognition dataset with 240k videos for training and 19k for validation.

Original: We treat the two modalities, RGB and audio, as the inputs.
Revised: We treat the two modalities, RGB and audio, as the inputs.

Original: VGG-Sound (Chen et al., 2020b) is an audio-visual classification dataset which contains over 200k video clips for 309 different sound classes.
Revised: VGG-Sound (Chen et al., 2020b) is an audio-visual classification dataset which contains over 200k video clips for 309 different sound classes.

Original: UCF101 (Soomro et al., 2012) is an action recognition dataset with 101 action categories, including 7k videos for training and 3k for testing.
Revised: UCF101 (Soomro et al., 2012) is an action recognition dataset with 101 action categories, including 7k videos for training and 3k for testing.

Original: ModelNet40 is a 3D object classification task with 9,483 training samples and 2,468 test samples.
Revised: ModelNet40 is a 3D object classification task with 9,483 training samples and 2,468 test samples.

Original: Following Wu et al.
Revised: Following Wu et al.

Original: (2022), we treat the front and rear view as two modalities.
Revised: (2022), we treat the front and rear view as two modalities.

Original: Training Settings.
Revised: Training Settings.

Original: In VGG-Sound, UCF101 and ModelNet40, we use ResNet as our backbone, all with 18 layers.
Revised: In VGG-Sound, UCF101 and ModelNet40, we use ResNet as our backbone, all with 18 layers.

Original: As for Kinetics-400, we use 50 or 101 layers’ ResNet to encode the inputs.
Revised: As for Kinetics-400, we use 50 or 101 layers’ ResNet to encode the inputs.

Original: Noting that 3D CNN is used for visual data of VGG-Sound and Kinetics-400.
Revised: Noting that 3D CNN is used for visual data of VGG-Sound and Kinetics-400.

Original: The data preprocessing, hyper-parameters, optimizer can be found in the Appendix A.1 and A.2.
Revised: The data preprocessing, hyper-parameters, optimizer can be found in the Appendix A.1 and A.2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. A N EMPIRICAL TRICK TO DECIDE WHICH LEARNING METHOD TO USE .
Revised: 4. A N EMPIRICAL TRICK TO DECIDE WHICH LEARNING METHOD TO USE .

Original: We train a multi-modal linear classifier on frozen uni-modal pre-trained encoders and compare that with averaging uni-modal predictions.
Revised: We train a multi-modal linear classifier on frozen uni-modal pre-trained encoders and compare that with averaging uni-modal predictions.

Original: As Table 3 shows, in VGG-Sound and Kinetics-400, the classifier is better, meaning cross-modal interaction can benefit the classifier in the two datasets.
Revised: As Table 3 shows, in VGG-Sound and Kinetics-400, the classifier is better, meaning cross-modal interaction can benefit the classifier in the two datasets.

Original: However, in UCF101 and ModelNet40, averaging uni-modal predictions performs well.
Revised: However, in UCF101 and ModelNet40, averaging uni-modal predictions performs well.

Original: To explore why the classifier fails in UCF101, we check the uni-modal classifiers of the newly trained multimodal linear layer (details in Appendix A.7.1).
Revised: To explore why the classifier fails in UCF101, we check the uni-modal classifiers of the newly trained multimodal linear layer (details in Appendix A.7.1).

Original: As Table 4 shows, they are far worse than the unimodal models.
Revised: As Table 4 shows, they are far worse than the unimodal models.

Original: The result shows the simple linear classifier suffers from serious Modality Laziness in UCF101 , which negatively impact the performance.
Revised: The result shows the simple linear classifier suffers from serious Modality Laziness in UCF101 , which negatively impact the performance.

Original: Both modalities in ModelNet40 also have strong uni-modal features and can achieve 89% accuracy individually.
Revised: Both modalities in ModelNet40 also have strong uni-modal features and can achieve 89% accuracy individually.

Original: Averaging uni-modal predictions avoids laziness problem and achieves competitive performance.
Revised: Averaging uni-modal predictions avoids laziness problem and achieves competitive performance.

Original: Based on the above analysis, we perform UMT on VGG-Sound and Kinetics-400, and UME on UCF101 and ModelNet40.
Revised: Based on the above analysis, we perform UMT on VGG-Sound and Kinetics-400, and UME on UCF101 and ModelNet40.

Original: 4. UMT IS AN EFFECTIVE REGULARIZER FOR UNI - MODAL FEATURE LEARNING
Revised: 4. UMT IS AN EFFECTIVE REGULARIZER FOR UNI - MODAL FEATURE LEARNING

Original: In this subsection, we demonstrate that Uni-Modal Teacher outperforms other multi-modal training methods in VGG-Sound and Kinetics-400.
Revised: In this subsection, we demonstrate that Uni-Modal Teacher outperforms other multi-modal training methods in VGG-Sound and Kinetics-400.

Original: We then do ablation experiments to explain why UMT works.
Revised: We then do ablation experiments to explain why UMT works.

Original: In all experiments, we use MSELoss as the distillation loss and set the weight of that as 50.
Revised: In all experiments, we use MSELoss as the distillation loss and set the weight of that as 50.

Original: 4.3. UMT IS EFFECTIVE ON VGG-S OUND AND K INETICS -400.
Revised: 4.3. UMT IS EFFECTIVE ON VGG-S OUND AND K INETICS -400.

Original: UMT vs Other Late-Fusion Methods.
Revised: UMT vs Other Late-Fusion Methods.

Original: The late-fusion architecture is commonly used for multimodal classification tasks (Wang et al., 2020; Peng et al., 2022).
Revised: The late-fusion architecture is commonly used for multimodal classification tasks (Wang et al., 2020; Peng et al., 2022).

Original: In late-fusion architecture, the features are extracted from different modalities by the corresponding encoders, and then the head layer is applied to output predictions.
Revised: In late-fusion architecture, the features are extracted from different modalities by the corresponding encoders, and then the head layer is applied to output predictions.

Original: We compare different heads, including linear layer, MLP, and attention layer.
Revised: We compare different heads, including linear layer, MLP, and attention layer.

Original: In UMT, we use a simple linear layer as the multi-modal head.
Revised: In UMT, we use a simple linear layer as the multi-modal head.

Original: We also conduct another experiment, which adds extra uni-modal linear heads to receive the uni-modal features and generating additional losses to joint optimize the model, namely Auxiliary-CEloss.
Revised: We also conduct another experiment, which adds extra uni-modal linear heads to receive the uni-modal features and generating additional losses to joint optimize the model, namely Auxiliary-CEloss.

Original: Auxiliary-CEloss gives all losses equal weights, while G-Blending reweights the losses according to the overfitting-togeneralization-ratio (OGR) (Wang et al., 2020).
Revised: Auxiliary-CEloss gives all losses equal weights, while G-Blending reweights the losses according to the overfitting-togeneralization-ratio (OGR) (Wang et al., 2020).

Original: OGM-GE (Peng et al., 2022) controls the optimization of each modality by online gradient modulation.
Revised: OGM-GE (Peng et al., 2022) controls the optimization of each modality by online gradient modulation.

Original: However, both OGM-GE and G-Blending are complex to implement.
Revised: However, both OGM-GE and G-Blending are complex to implement.

Original: As shown in Table 5, UMT outperforms other methods.
Revised: As shown in Table 5, UMT outperforms other methods.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: UMT vs AVSlowFast.
Revised: UMT vs AVSlowFast.

Original: Audio-Visual SlowFast is an representative intermediate fusion method.
Revised: Audio-Visual SlowFast is an representative intermediate fusion method.

Original: We compare UMT with AVSlowFast in Kinetics-400.
Revised: We compare UMT with AVSlowFast in Kinetics-400.

Original: As Table 6 shows, under different RGB encoders, UMT consistently exceeds AVSlowFast, although we cannot reproduce their results due to the dynamics of Kinetics-400 (Appendix A.11).
Revised: As Table 6 shows, under different RGB encoders, UMT consistently exceeds AVSlowFast, although we cannot reproduce their results due to the dynamics of Kinetics-400 (Appendix A.11).

Original: Ablation Study of UMT.
Revised: Ablation Study of UMT.

Original: We first evaluate the encoders of UMT by training linear classifiers on them to verify that UMT does improve the uni-modal feature learning.
Revised: We first evaluate the encoders of UMT by training linear classifiers on them to verify that UMT does improve the uni-modal feature learning.

Original: As Table 7 shows, UMT makes its encoders stand out.
Revised: As Table 7 shows, UMT makes its encoders stand out.

Original: Benefiting from uni-modal distillation, some encoders even outperform their uni-modal counterparts.
Revised: Benefiting from uni-modal distillation, some encoders even outperform their uni-modal counterparts.

Original: We then compare UMT with classic self-distillation methods (distillation on soft label (Hinton et al., 2015) and feature (Romero et al., 2014)).
Revised: We then compare UMT with classic self-distillation methods (distillation on soft label (Hinton et al., 2015) and feature (Romero et al., 2014)).

Original: As Tableshows, naive self-distillation can only bring limited improvement, showing that UMT improves overall performance by improving the uni-modal feature learning instead of knowledge distillation.
Revised: As Tableshows, naive self-distillation can only bring limited improvement, showing that UMT improves overall performance by improving the uni-modal feature learning instead of knowledge distillation.

Original: 4.3. U NI -M ODAL E NSEMBLE IN M ULTI - MODAL LEARNING
Revised: 4.3. U NI -M ODAL E NSEMBLE IN M ULTI - MODAL LEARNING

Original: In this subsection, we demonstrate that Uni-Modal Ensemble is effective on multi-modal datasets where modalities have strong uni-modal features, outperforming other complex methods.
Revised: In this subsection, we demonstrate that Uni-Modal Ensemble is effective on multi-modal datasets where modalities have strong uni-modal features, outperforming other complex methods.

Original: Even though we don’t combine these uni-modal predictions in any special way, but simply average.
Revised: Even though we don’t combine these uni-modal predictions in any special way, but simply average.

Original: In UCF101 , we compare Uni-Modal Ensemble with various multi-modal late-fusion methods.
Revised: In UCF101 , we compare Uni-Modal Ensemble with various multi-modal late-fusion methods.

Original: As
Revised: As

Original: Table 9 shows, although Gradient Blending (Wang et al., 2020) and OGM-GE (Peng et al., 2022) outperforms baseline methods, they are far worse than Uni-Modal Ensemble.
Revised: Table 9 shows, although Gradient Blending (Wang et al., 2020) and OGM-GE (Peng et al., 2022) outperforms baseline methods, they are far worse than Uni-Modal Ensemble.

Original: In ModelNet40 , the main comparing methods come from Wu et al.
Revised: In ModelNet40 , the main comparing methods come from Wu et al.

Original: (2022), which uses a multimodal DNN with intermediate fusion.
Revised: (2022), which uses a multimodal DNN with intermediate fusion.

Original: It proposes a balanced multi-modal algorithm which balances conditional utilization of each modality by re-balancing the optimization step.
Revised: It proposes a balanced multi-modal algorithm which balances conditional utilization of each modality by re-balancing the optimization step.

Original: UME surpass their balanced multi-modal algorithm, as Table 10 shows.
Revised: UME surpass their balanced multi-modal algorithm, as Table 10 shows.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper analyzes the phenomenon of insufficient uni-modal feature learning in multi-modal training and proves that it does hurt the overall performance.
Revised: This paper analyzes the phenomenon of insufficient uni-modal feature learning in multi-modal training and proves that it does hurt the overall performance.

Original: We propose to choose proper learning method from Uni-Modal Ensemble and proposed Uni-Modal Teacher according to the distribution of uni-modal and paired features and demonstrate the effectiveness of the guiding principle.
Revised: We propose to choose proper learning method from Uni-Modal Ensemble and proposed Uni-Modal Teacher according to the distribution of uni-modal and paired features and demonstrate the effectiveness of the guiding principle.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: In our paper, we detail the specific implementation steps of the methods in Sec 3.3 and also give the values of the hyper-parameters we use in detail in Appendix A.2.
Revised: In our paper, we detail the specific implementation steps of the methods in Sec 3.3 and also give the values of the hyper-parameters we use in detail in Appendix A.2.

Original: We also provide the codes in supplementary material, which can reproduce the results of the experiments
Revised: We also provide the codes in supplementary material, which can reproduce the results of the experiments

Original: A E XPERIMENTAL D ETAILS AND A DDITIONAL E XPERIMENTS A. D ATASETS
Revised: A E XPERIMENTAL D ETAILS AND A DDITIONAL E XPERIMENTS A. D ATASETS

Original: Here, we describe the preprocessing of Kinetics-400, VGG-Sound, UCF101 and ModelNet40 in detail.
Revised: Here, we describe the preprocessing of Kinetics-400, VGG-Sound, UCF101 and ModelNet40 in detail.

Original: Kinetics-400 dataset (Kay et al., 2017) contains over 240k videos for training and 19k for validation, which we download from cvdfoundation 8 .
Revised: Kinetics-400 dataset (Kay et al., 2017) contains over 240k videos for training and 19k for validation, which we download from cvdfoundation 8 .

Original: Kinetics-400 is a commonly used dataset withclasses, and we mainly follow the open source preprocessing methods to process that.
Revised: Kinetics-400 is a commonly used dataset withclasses, and we mainly follow the open source preprocessing methods to process that.

Original: For RGB modality, we follow the procedure of PySlowFast 9 , which resizes the video to the short edge size of 256.
Revised: For RGB modality, we follow the procedure of PySlowFast 9 , which resizes the video to the short edge size of 256.

Original: and for audio modality, we follow mmaction2 10 to extract specgram features.
Revised: and for audio modality, we follow mmaction2 10 to extract specgram features.

Original: When performing joint training, we take consecutive 64 frames from a video with fps of 30 and random crop the video to 224*224, and for audio inputs, we take the specgram that can be aligned in time with the clip extracted from the video.
Revised: When performing joint training, we take consecutive 64 frames from a video with fps of 30 and random crop the video to 224*224, and for audio inputs, we take the specgram that can be aligned in time with the clip extracted from the video.

Original: When testing, we ensemble the predictions from uniformly sampled clips with RGB and audio from a video and give the final outputs, following PySlowfast.
Revised: When testing, we ensemble the predictions from uniformly sampled clips with RGB and audio from a video and give the final outputs, following PySlowfast.

Original: VGG-Sound dataset (Chen et al., 2020b), which contains over 200k video clips for 309 different sound classes, is also used for evaluating our method.
Revised: VGG-Sound dataset (Chen et al., 2020b), which contains over 200k video clips for 309 different sound classes, is also used for evaluating our method.

Original: It is an audio-visual dataset in the wild where each object that emits sound is also visible in the corresponding video clip, making it suitable for scene classification tasks.
Revised: It is an audio-visual dataset in the wild where each object that emits sound is also visible in the corresponding video clip, making it suitable for scene classification tasks.

Original: Please note that some clips in the dataset are no longer available on
Revised: Please note that some clips in the dataset are no longer available on

Original: YouTube, and we actually use about 175k videos for training and 15k for testing, but the number of classes remains the same.
Revised: YouTube, and we actually use about 175k videos for training and 15k for testing, but the number of classes remains the same.

Original: We design a preprocessing paradigm to improve training efficiency as follows: (1) each video is interpolated to 256 × 256 and saved as stacked images; (2) each audio is first converted to 16 kHz and 32-bit precision in the floating-point PCM format, then randomly cropped or tiled to a fixed duration of 10s.
Revised: We design a preprocessing paradigm to improve training efficiency as follows: (1) each video is interpolated to 256 × 256 and saved as stacked images; (2) each audio is first converted to 16 kHz and 32-bit precision in the floating-point PCM format, then randomly cropped or tiled to a fixed duration of 10s.

Original: For video input, 32 frames are uniformly sampled from each clip before feeding to the video encoder.
Revised: For video input, 32 frames are uniformly sampled from each clip before feeding to the video encoder.

Original: While for the audio input, a 1024-point discrete
Revised: While for the audio input, a 1024-point discrete

Original: Fourier transform is performed using nnAudio (Cheuk et al., 2020), with 64 ms frame length andms frame-shift.
Revised: Fourier transform is performed using nnAudio (Cheuk et al., 2020), with 64 ms frame length andms frame-shift.

Original: And we only feed the magnitude spectrogram to the audio encoder.
Revised: And we only feed the magnitude spectrogram to the audio encoder.

Original: UCF101 dataset (Soomro et al., 2012) is an action recognition dataset with 101 action categories, including 7k videos for training and 3k for testing.
Revised: UCF101 dataset (Soomro et al., 2012) is an action recognition dataset with 101 action categories, including 7k videos for training and 3k for testing.

Original: And we use the rgb and flow provided by (Feichtenhofer et al., 2016).
Revised: And we use the rgb and flow provided by (Feichtenhofer et al., 2016).

Original: For RGB, we use one image of (3 ∗ 224 ∗ 224) as the input; while for flow, we use a stack of optical flow images which contained 10 x-channel and 10 y-channel images, So its input shape is (20 ∗ 224 ∗ 224) .
Revised: For RGB, we use one image of (3 ∗ 224 ∗ 224) as the input; while for flow, we use a stack of optical flow images which contained 10 x-channel and 10 y-channel images, So its input shape is (20 ∗ 224 ∗ 224) .

Original: During training, we perform random crop and random horizontal flip as the data augmentation; while testing, we resize the image to 224 and do not perform data augmentation operations.
Revised: During training, we perform random crop and random horizontal flip as the data augmentation; while testing, we resize the image to 224 and do not perform data augmentation operations.

Original: ModalNet40 is a 3D object classification dataset with 9,483 training samples and 2,468 test samples.
Revised: ModalNet40 is a 3D object classification dataset with 9,483 training samples and 2,468 test samples.

Original: We base on the front view and the rear view of the 3D object to classify that, following Wu et al.
Revised: We base on the front view and the rear view of the 3D object to classify that, following Wu et al.

Original: A. T RAINING H YPERPARAMETERS
Revised: A. T RAINING H YPERPARAMETERS

Original: In this subsection, we show the hyperparameters of our experiments in UCF101 and VGG-Sound in
Revised: In this subsection, we show the hyperparameters of our experiments in UCF101 and VGG-Sound in

Original: Table 11.
Revised: Table 11.

Original: As for Kinetics-400’s RGB modality, we totally follow the hyperparameters and settings of PySlowFast 11 .
Revised: As for Kinetics-400’s RGB modality, we totally follow the hyperparameters and settings of PySlowFast 11 .

Original: As for audio modality, we modify the hyperparameters 12 to be as consistent as possible with the RGB training for further joint training.
Revised: As for audio modality, we modify the hyperparameters 12 to be as consistent as possible with the RGB training for further joint training.

Original: Specifically, we use the same learning rate and batch size as RGB training used.
Revised: Specifically, we use the same learning rate and batch size as RGB training used.

Original: As for ModelNet40, we totally follow the experimental settings of Wu et al.
Revised: As for ModelNet40, we totally follow the experimental settings of Wu et al.

Original: [Table]
Revised: [Table]

Original: A. C AN EXISTING OPTIMIZERS SOLVE M ODALITY L AZINESS ?
Revised: A. C AN EXISTING OPTIMIZERS SOLVE M ODALITY L AZINESS ?

Original: [Table]
Revised: [Table]

Original: While the results in Table 1 show that different multi-modal methods suffer from learning insufficient uni-modal features, how about changing the optimizer ?
Revised: While the results in Table 1 show that different multi-modal methods suffer from learning insufficient uni-modal features, how about changing the optimizer ?

Original: To answer this question
Revised: To answer this question

Original: [Table]
Revised: [Table]

Original: we try different optimizers for multi-modal late-fusion training (with a linear multi-modal head), including
Revised: we try different optimizers for multi-modal late-fusion training (with a linear multi-modal head), including

Original: SGD, RMSprop, Adagrad, Adadelta, Adamw and Adam.
Revised: SGD, RMSprop, Adagrad, Adadelta, Adamw and Adam.

Original: As Table 12 shows, Modality Laziness exists no matter which optimizer is used.
Revised: As Table 12 shows, Modality Laziness exists no matter which optimizer is used.

Original: A. D ETAILS ON U NI -M ODAL T EACHER (UMT)
Revised: A. D ETAILS ON U NI -M ODAL T EACHER (UMT)

Original: In this subsection, we describe how Uni-Molda Teacher (UMT) applies on multi-modal late-fusion tasks.
Revised: In this subsection, we describe how Uni-Molda Teacher (UMT) applies on multi-modal late-fusion tasks.

Original: The overall architecture can be found in Figure 4.
Revised: The overall architecture can be found in Figure 4.

Original: UMT in late-fusion classification.
Revised: UMT in late-fusion classification.

Original: In multi-modal late-fusion architecture, modalities are first encoded by the corresponding encoders and then mapped to the output space by a multi-modal fusion head (Figure 4 left).
Revised: In multi-modal late-fusion architecture, modalities are first encoded by the corresponding encoders and then mapped to the output space by a multi-modal fusion head (Figure 4 left).

Original: Uni-Modal Teacher distills the pre-trained uni-modal features to the corresponding parts in multi-modal networks in multi-modal training (Figure 4 right).
Revised: Uni-Modal Teacher distills the pre-trained uni-modal features to the corresponding parts in multi-modal networks in multi-modal training (Figure 4 right).

Original: Uni-modal distillation happens before fusion, so it’s suitable for late-fusion multi-modal architecture.
Revised: Uni-modal distillation happens before fusion, so it’s suitable for late-fusion multi-modal architecture.

Original: The pre-trained unimodal features are generated by inputting the data to the pre-trained uni-modal models.
Revised: The pre-trained unimodal features are generated by inputting the data to the pre-trained uni-modal models.

Original: UMT’s weights.
Revised: UMT’s weights.

Original: For VGG-Sound and Kinetics, we use 50 (both audio feature distillation and RGB feature distillation) as the distillation loss’s weight.
Revised: For VGG-Sound and Kinetics, we use 50 (both audio feature distillation and RGB feature distillation) as the distillation loss’s weight.

Original: We test different distillation weights on VGGSound and Kinetics-400.
Revised: We test different distillation weights on VGGSound and Kinetics-400.

Original: As shown in Table 13, on both datasets, UMT performs well with an distillation weight of 50.
Revised: As shown in Table 13, on both datasets, UMT performs well with an distillation weight of 50.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: A. D ROPOUT IN M ULTI - MODAL T RAINING .
Revised: A. D ROPOUT IN M ULTI - MODAL T RAINING .

Original: Here we consider the common regularizer, dropout (Srivastava et al., 2014), and a variant of it, namely modality-wise dropout, which randomly drops (with probability 1 / 3 ) the feature from one modality in each iteration.
Revised: Here we consider the common regularizer, dropout (Srivastava et al., 2014), and a variant of it, namely modality-wise dropout, which randomly drops (with probability 1 / 3 ) the feature from one modality in each iteration.

Original: Modality dropout is akin to the ModDrop in Neverova et al.
Revised: Modality dropout is akin to the ModDrop in Neverova et al.

Original: As Table 14 shows, modality-wise dropout is significantly better than dropout, which implies that modality-wise laziness is serious and modality-wise dropout is also effective.
Revised: As Table 14 shows, modality-wise dropout is significantly better than dropout, which implies that modality-wise laziness is serious and modality-wise dropout is also effective.

Original: A. F INETUNING THE UNI - MODAL PRE - TRAINED ENCODERS
Revised: A. F INETUNING THE UNI - MODAL PRE - TRAINED ENCODERS

Original: In this subsection, we use the uni-modal pre-trained encoders’ parameters as the initialized weights in multi-modal training and randomly initialize a multi-modal linear classifier on the encoders.
Revised: In this subsection, we use the uni-modal pre-trained encoders’ parameters as the initialized weights in multi-modal training and randomly initialize a multi-modal linear classifier on the encoders.

Original: We set the classifier’s learning rate as 1 e − 3 and try different learning rates on the encoders.
Revised: We set the classifier’s learning rate as 1 e − 3 and try different learning rates on the encoders.

Original: As Table 15 shows, using the uni-modal supervised pre-trained encoder’s weights in multi-modal training and then fine-tuning the whole multi-modal model can bring some improvement compared to naive fusion ( 49 . 46 ) but is worse than UMT, which gets 53 .
Revised: As Table 15 shows, using the uni-modal supervised pre-trained encoder’s weights in multi-modal training and then fine-tuning the whole multi-modal model can bring some improvement compared to naive fusion ( 49 . 46 ) but is worse than UMT, which gets 53 .

Original: 46 accuracy.
Revised: 46 accuracy.

Original: When the learning rate of encoders is large, the encoders forget some abilities to extract uni-modal features.
Revised: When the learning rate of encoders is large, the encoders forget some abilities to extract uni-modal features.

Original: Ngiam et al.
Revised: Ngiam et al.

Original: (2011) proposes to use Bimodal Deep Autoencoder to pre-train the encoders with multiple modalities.
Revised: (2011) proposes to use Bimodal Deep Autoencoder to pre-train the encoders with multiple modalities.

Original: It is a direction worth exploring to address Modality Laziness of deep multimodal models.
Revised: It is a direction worth exploring to address Modality Laziness of deep multimodal models.

Original: A. T HE ROLE OF CROSS - MODAL INTERACTION ON DIFFERENT DATASETS
Revised: A. T HE ROLE OF CROSS - MODAL INTERACTION ON DIFFERENT DATASETS

Original: In this subsection, we conduct various experiments to further investigate the effect of cross-modal interaction and explore the benefits and harms that cross-modal interaction brings in different multi-
Revised: In this subsection, we conduct various experiments to further investigate the effect of cross-modal interaction and explore the benefits and harms that cross-modal interaction brings in different multi-

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: modal tasks/datasets.
Revised: modal tasks/datasets.

Original: We find that in different datasets, cross-modal interaction has a different effect on the performance.
Revised: We find that in different datasets, cross-modal interaction has a different effect on the performance.

Original: A.7. A VERAGING THE UNI - MODAL PREDICTIONS vs T HE LINEAR CLASSIFIER TRAINED ON UNI - MODAL PRE - TRAINED ENCODERS .
Revised: A.7. A VERAGING THE UNI - MODAL PREDICTIONS vs T HE LINEAR CLASSIFIER TRAINED ON UNI - MODAL PRE - TRAINED ENCODERS .

Original: In Sec 4.2, we train a multi-modal linear classifier on frozen uni-modal pre-trained encoders and compare this classifier with directly averaging uni-modal models’ predictions.
Revised: In Sec 4.2, we train a multi-modal linear classifier on frozen uni-modal pre-trained encoders and compare this classifier with directly averaging uni-modal models’ predictions.

Original: As Table 3 shows, this classifier does not consistently outperform simply averaging the uni-modal predictions on all datasets.
Revised: As Table 3 shows, this classifier does not consistently outperform simply averaging the uni-modal predictions on all datasets.

Original: It shows better performance on VGG-Sound and Kinetics-400, but worse performance on
Revised: It shows better performance on VGG-Sound and Kinetics-400, but worse performance on

Original: UCF101 and ModelNet40.
Revised: UCF101 and ModelNet40.

Original: To further explain this phenomenon, we check and disassemble this new trained multi-modal classifier on UCF101.
Revised: To further explain this phenomenon, we check and disassemble this new trained multi-modal classifier on UCF101.

Original: In the late-fusion multi-modal training, the features of different modalities are concatenated first and then the multi-modal classifier receives them and output predictions.
Revised: In the late-fusion multi-modal training, the features of different modalities are concatenated first and then the multi-modal classifier receives them and output predictions.

Original: Different modalities in the classifier do not share the parameters.
Revised: Different modalities in the classifier do not share the parameters.

Original: So we split the new trained multi-modal linear classifier into uni-modal classifiers.
Revised: So we split the new trained multi-modal linear classifier into uni-modal classifiers.

Original: We use the uni-modal pre-trained encoders to extract features and then the uni-modal classifiers receive the corresponding features and output predictions.
Revised: We use the uni-modal pre-trained encoders to extract features and then the uni-modal classifiers receive the corresponding features and output predictions.

Original: Noting that OGM-GE (Peng et al., 2022) uses similar technique to check how well different modalities are trained.
Revised: Noting that OGM-GE (Peng et al., 2022) uses similar technique to check how well different modalities are trained.

Original: As Table 4 shows, the uni-modal classifiers from new trained multi-modal classifiers are significantly worse than uni-modal models, implying that the multi-modal classifier trained on uni-modal pre-trained encoders suffers from serious Modality Laziness on UCF101, although it is just a simple linear layer, resulting in worse performance than directly averaging the uni-modal predictions.
Revised: As Table 4 shows, the uni-modal classifiers from new trained multi-modal classifiers are significantly worse than uni-modal models, implying that the multi-modal classifier trained on uni-modal pre-trained encoders suffers from serious Modality Laziness on UCF101, although it is just a simple linear layer, resulting in worse performance than directly averaging the uni-modal predictions.

Original: A.7. C LASS - LEVEL E VALUATION ON D IFFERENT M ULTI - MODAL D ATASETS
Revised: A.7. C LASS - LEVEL E VALUATION ON D IFFERENT M ULTI - MODAL D ATASETS

Original: In this subsection, we compare naive late-fusion learning with averaging predictions of uni-modal models in class level.
Revised: In this subsection, we compare naive late-fusion learning with averaging predictions of uni-modal models in class level.

Original: It’s obvious that there are more cross-modal interactions in naive fusion.
Revised: It’s obvious that there are more cross-modal interactions in naive fusion.

Original: Although naive fusion suffers from learning insufficient uni-modal features, we find in some classes in Kinetics and VGG-Sound, the accuracy of naive fusion model outperforms averaging the unimodal models’ predictions, and even outperforms the sum of the accuracy of the two uni-modal models in VGG-Sound and Kinetics-400, as shown in Table 2 and 16.
Revised: Although naive fusion suffers from learning insufficient uni-modal features, we find in some classes in Kinetics and VGG-Sound, the accuracy of naive fusion model outperforms averaging the unimodal models’ predictions, and even outperforms the sum of the accuracy of the two uni-modal models in VGG-Sound and Kinetics-400, as shown in Table 2 and 16.

Original: However, We cannot find any class that naive fusion can exceed the sum of the accuracy of the uniRGB model and uni-flow model in UCF101.
Revised: However, We cannot find any class that naive fusion can exceed the sum of the accuracy of the uniRGB model and uni-flow model in UCF101.

Original: We select classes in UCF101 by sorting the differences of accuracy between naive fusion and the best uni-modal model in class level and the top ten with the largest difference are selected.
Revised: We select classes in UCF101 by sorting the differences of accuracy between naive fusion and the best uni-modal model in class level and the top ten with the largest difference are selected.

Original: In these classes where naive fusion has advantages, averaging the predictions can outperform naive fusion in some classes (ID:29, 67, 71), and this phenomenon is not found in VGG-Sound and Kinetics.
Revised: In these classes where naive fusion has advantages, averaging the predictions can outperform naive fusion in some classes (ID:29, 67, 71), and this phenomenon is not found in VGG-Sound and Kinetics.

Original: And as Tabla 4 and Table 17 show, both RGB and optical flow in UCF101 can get strong performance individually.
Revised: And as Tabla 4 and Table 17 show, both RGB and optical flow in UCF101 can get strong performance individually.

Original: All the evidence shows that in UCF101, the uni-modal features are totally dominate and any joint training can lead to serious Modality Laziness.
Revised: All the evidence shows that in UCF101, the uni-modal features are totally dominate and any joint training can lead to serious Modality Laziness.

Original: The mapping betwenn class ID and class name in different datasets The correspondence between id and name of the selected class in VGG-Sound is: 164: People Sniggering, 303: Wood
Revised: The mapping betwenn class ID and class name in different datasets The correspondence between id and name of the selected class in VGG-Sound is: 164: People Sniggering, 303: Wood

Original: Thrush Calling, 33: Cat Meowing, 255: Sea Waves, 91:
Revised: Thrush Calling, 33: Cat Meowing, 255: Sea Waves, 91:

Original: Footsteps On Snow, 4: Alligators
Revised: Footsteps On Snow, 4: Alligators

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Crocodiles Hissing,
Revised: Crocodiles Hissing,

Original: [Table]
Revised: [Table]

Original: People Gargling
Revised: People Gargling

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Mynah Bird Singing
Revised: Mynah Bird Singing

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Door Slamming
Revised: Door Slamming

Original: [Table] [Table]
Revised: [Table] [Table]

Original: People Humming
Revised: People Humming

Original: [Table]
Revised: [Table]

Original: For Kinetics-400, we sort the classes alphabetically from smallest to largest according to the class name, and then we can get the mapping between class name and id.
Revised: For Kinetics-400, we sort the classes alphabetically from smallest to largest according to the class name, and then we can get the mapping between class name and id.

Original: In UCF101, the mapping can be found in classInd.txt, a given file of UCF101.
Revised: In UCF101, the mapping can be found in classInd.txt, a given file of UCF101.

Original: A. C ROSS - MODAL INTERACTION IN U NI -M ODAL T EACHER (UMT)
Revised: A. C ROSS - MODAL INTERACTION IN U NI -M ODAL T EACHER (UMT)

Original: [Table]
Revised: [Table]

Original: In order to verify whether the multimodal loss in UMT makes sense, we train uni-modal models by knowledge distillation to get better performance than encoders trained by UMT and then combine them by introducing a new multi-modal classifier on these encoders.
Revised: In order to verify whether the multimodal loss in UMT makes sense, we train uni-modal models by knowledge distillation to get better performance than encoders trained by UMT and then combine them by introducing a new multi-modal classifier on these encoders.

Original: As Table 18 shows, UMT works better in multi-modal performance, although the encoders of UMT in unimodal evaluation are worse, showing that UMT indeed benefits from cross-modal interaction.
Revised: As Table 18 shows, UMT works better in multi-modal performance, although the encoders of UMT in unimodal evaluation are worse, showing that UMT indeed benefits from cross-modal interaction.

Original: A. E XPLORING UMT FOR M ULTI - MODAL S EGMENTATION
Revised: A. E XPLORING UMT FOR M ULTI - MODAL S EGMENTATION

Original: NYU Depth V2 dataset (Silberman et al., 2012) contains 1449 indoor RGB-Depth data totally and we use 40-class label setting.
Revised: NYU Depth V2 dataset (Silberman et al., 2012) contains 1449 indoor RGB-Depth data totally and we use 40-class label setting.

Original: The number of training set and testing set is 795 and 654 respectively.
Revised: The number of training set and testing set is 795 and 654 respectively.

Original: All perprocessing operations are following (Seichter et al., 2020).
Revised: All perprocessing operations are following (Seichter et al., 2020).

Original: In contrast to the late fusion classification task, the RGB-Depth semantic segmentation belongs to middle fusion.
Revised: In contrast to the late fusion classification task, the RGB-Depth semantic segmentation belongs to middle fusion.

Original: The main encoder receives RGB inputs, and the depth inputs are fed into the depth encoder.
Revised: The main encoder receives RGB inputs, and the depth inputs are fed into the depth encoder.

Original: At each intermediate layer, the main encoder fuses its own intermediate outputs and the depth features obtained from the depth encoder, which makes it a mid-fusion task (Seichter et al., 2020).
Revised: At each intermediate layer, the main encoder fuses its own intermediate outputs and the depth features obtained from the depth encoder, which makes it a mid-fusion task (Seichter et al., 2020).

Original: Since features generated by each layer matter, we distill multi-scale depth feature maps using the MSE loss.
Revised: Since features generated by each layer matter, we distill multi-scale depth feature maps using the MSE loss.

Original: For feature maps from the RGB encoder, however, since they are generated by fusing
Revised: For feature maps from the RGB encoder, however, since they are generated by fusing

Original: RGB and depth modalities, we cannot distill RGB feature maps directly like depth feature maps.
Revised: RGB and depth modalities, we cannot distill RGB feature maps directly like depth feature maps.

Original: To mitigate this effect, we curate predictors, namely 2 layers CNNs, aiming to facilitate the fused feature maps to predict the RGB feature maps trained by the RGB modality before distillation.
Revised: To mitigate this effect, we curate predictors, namely 2 layers CNNs, aiming to facilitate the fused feature maps to predict the RGB feature maps trained by the RGB modality before distillation.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: full schematic diagram is presented in Figure 5.
Revised: full schematic diagram is presented in Figure 5.

Original: As shown in Table 19, UMT can also improve multi-modal segmentation whether the encoder is pre-trained on ImageNet or not.
Revised: As shown in Table 19, UMT can also improve multi-modal segmentation whether the encoder is pre-trained on ImageNet or not.

Original: A. E XPLANATIONS ON P AIRED F EATURES
Revised: A. E XPLANATIONS ON P AIRED F EATURES

Original: We revisit the definitions of uni-modal features and paired features: uni-modal features , which can be learned by uni-modal training; paired features , which can only be learned by cross-modal interaction in joint training.
Revised: We revisit the definitions of uni-modal features and paired features: uni-modal features , which can be learned by uni-modal training; paired features , which can only be learned by cross-modal interaction in joint training.

Original: Different datasets contain different proportions of these features.
Revised: Different datasets contain different proportions of these features.

Original: In this subsection, we use synthetic datasets to explain the uni-modal features and paired features in multi-modal tasks.
Revised: In this subsection, we use synthetic datasets to explain the uni-modal features and paired features in multi-modal tasks.

Original: Understanding different types of features in multi-modal tasks by synthetic datasets.
Revised: Understanding different types of features in multi-modal tasks by synthetic datasets.

Original: Three different multi-modal datasets are generated to help us understand the uni-modal features and paired features in multi-modal tasks.
Revised: Three different multi-modal datasets are generated to help us understand the uni-modal features and paired features in multi-modal tasks.

Original: The process of data generation mainly refers to Hessel & Lee (2020).
Revised: The process of data generation mainly refers to Hessel & Lee (2020).

Original: Firstly, we generate a dataset where each modality can extract the features to give correct predictions.
Revised: Firstly, we generate a dataset where each modality can extract the features to give correct predictions.

Original: We name this dataset as Dataset α .
Revised: We name this dataset as Dataset α .

Original: The data generation process is as follows:
Revised: The data generation process is as follows:

Original: Sample random projection P 1 ∈ R d 1 × d and P 2 ∈ R d 2 × d from U ( − 0 .
Revised: Sample random projection P 1 ∈ R d 1 × d and P 2 ∈ R d 2 × d from U ( − 0 .

Original: Sample
Revised: Sample

Original: z ∈ R d ∼ N (0 , 1) .
Revised: z ∈ R d ∼ N (0 , 1) .

Original: Normalize z to unit length 3.
Revised: Normalize z to unit length 3.

Original: Sample x ∈ R d ∼ N (0 , 1) .
Revised: Sample x ∈ R d ∼ N (0 , 1) .

Original: Normalize x to unit length 4.
Revised: Normalize x to unit length 4.

Original: if | x · z | ≤ 0 .
Revised: if | x · z | ≤ 0 .

Original: 1 , return to the Step 3.
Revised: 1 , return to the Step 3.

Original: If x · z > 0 . 1 , then y = 1 ; else y = 0 . 6.
Revised: If x · z > 0 . 1 , then y = 1 ; else y = 0 . 6.

Original: Get the data point ( P 1 x, P 2 x, y ) .
Revised: Get the data point ( P 1 x, P 2 x, y ) .

Original: If the amount of data generated is less than N , return to the Step 3; else break
Revised: If the amount of data generated is less than N , return to the Step 3; else break

Original: The P 1 x, P 2 x represents two modalities of the multi-modal dataset and we set d 1 , d 2 , d, N as 200 , 100 , 50 , 5000 , respectively.
Revised: The P 1 x, P 2 x represents two modalities of the multi-modal dataset and we set d 1 , d 2 , d, N as 200 , 100 , 50 , 5000 , respectively.

Original: And we randomly split 80% of the generated data as train set, and the rest serves as a test set.
Revised: And we randomly split 80% of the generated data as train set, and the rest serves as a test set.

Original: In this dataset, uni-modal models can extract useful features to give correct predictions and multi-modal joint training is not necessary, as Table 20 shows ( Dataset α ).
Revised: In this dataset, uni-modal models can extract useful features to give correct predictions and multi-modal joint training is not necessary, as Table 20 shows ( Dataset α ).

Original: We name the features, that uni-modal models can learns to give correct predictions in the given task, as uni-modal features.
Revised: We name the features, that uni-modal models can learns to give correct predictions in the given task, as uni-modal features.

Original: Secondly, we generate another dataset where the model must rely on both the two modalities to make correct predictions.
Revised: Secondly, we generate another dataset where the model must rely on both the two modalities to make correct predictions.

Original: We name this dataset as Dataset β .
Revised: We name this dataset as Dataset β .

Original: The data generation process is as follows:
Revised: The data generation process is as follows:

Original: Sample random projection P 1 ∈ R d 1 × d and P 2 ∈ R d 2 × d from U ( − 0 .
Revised: Sample random projection P 1 ∈ R d 1 × d and P 2 ∈ R d 2 × d from U ( − 0 .

Original: Synthetic Dataset α mainly contains uni-modal features which can be learned in uni-modal training;
Revised: Synthetic Dataset α mainly contains uni-modal features which can be learned in uni-modal training;

Original: Synthetic Dataset β mainly contains paired features which need joint training to learn;
Revised: Synthetic Dataset β mainly contains paired features which need joint training to learn;

Original: Synthetic
Revised: Synthetic

Original: Dataset γ contains uni-modal features and paired features.
Revised: Dataset γ contains uni-modal features and paired features.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Sample x 1 , x 2 ∈ R d ∼ N (0 , 1) .
Revised: Sample x 1 , x 2 ∈ R d ∼ N (0 , 1) .

Original: Normalize x 1 , x 2 to unit length 3.
Revised: Normalize x 1 , x 2 to unit length 3.

Original: if | x 1 · x 2 | ≤ 0 .
Revised: if | x 1 · x 2 | ≤ 0 .

Original: 25 , return to the Step 2.
Revised: 25 , return to the Step 2.

Original: If x 1 · x 2 > 0 .
Revised: If x 1 · x 2 > 0 .

Original: then y = 1 ; else y = 0 .
Revised: then y = 1 ; else y = 0 .

Original: Get the data point ( P 1 x 1 , P 2 x 2 , y ) .
Revised: Get the data point ( P 1 x 1 , P 2 x 2 , y ) .

Original: If the amount of data generated is less than N , return to the Step 2; else break
Revised: If the amount of data generated is less than N , return to the Step 2; else break

Original: This multi-modal dataset is different from the first dataset, because the labels in this dataset are highly dependent on the relationship between the two modalities.
Revised: This multi-modal dataset is different from the first dataset, because the labels in this dataset are highly dependent on the relationship between the two modalities.

Original: As we can see in Table 20 ( Dataset β ), the uni-modal models can only gives about 50 percent accuracy, while the multi-modal models can give about 90 percent accuracy.
Revised: As we can see in Table 20 ( Dataset β ), the uni-modal models can only gives about 50 percent accuracy, while the multi-modal models can give about 90 percent accuracy.

Original: In binary classification tasks, 50 percent accuracy is no different from guessing.
Revised: In binary classification tasks, 50 percent accuracy is no different from guessing.

Original: In this dataset, because labels are heavily relied on the relationships of the two modalities, we must train both modalities simultaneously to extract the joint representations to learn the relationship of the two modalities, which are beyond uni-modal features.
Revised: In this dataset, because labels are heavily relied on the relationships of the two modalities, we must train both modalities simultaneously to extract the joint representations to learn the relationship of the two modalities, which are beyond uni-modal features.

Original: In order to better carry out theoretical analysis, we abstract these representations into paired features, which can only be learned from multi-modal joint training in multi-modal tasks.
Revised: In order to better carry out theoretical analysis, we abstract these representations into paired features, which can only be learned from multi-modal joint training in multi-modal tasks.

Original: Finally, we generate a dataset that contains both uni-modal features and paired features.
Revised: Finally, we generate a dataset that contains both uni-modal features and paired features.

Original: We name this dataset as Dataset γ .
Revised: We name this dataset as Dataset γ .

Original: The data generation process is as follows:
Revised: The data generation process is as follows:

Original: Sample random projection P 1 ∈ R d 1 × d and P 2 ∈ R d 2 × d from U ( − 0 .
Revised: Sample random projection P 1 ∈ R d 1 × d and P 2 ∈ R d 2 × d from U ( − 0 .

Original: Sample
Revised: Sample

Original: z ∈ R d ∼ N (0 , 1) .
Revised: z ∈ R d ∼ N (0 , 1) .

Original: Normalize z to unit length 3.
Revised: Normalize z to unit length 3.

Original: Sample x ∈ R d ∼ N (0 , 1) .
Revised: Sample x ∈ R d ∼ N (0 , 1) .

Original: Normalize x to unit length 4.
Revised: Normalize x to unit length 4.

Original: if x · z ≥ 0 . 1 , get the data point ( P 1 x, P 2 x, y = 0) .
Revised: if x · z ≥ 0 . 1 , get the data point ( P 1 x, P 2 x, y = 0) .

Original: else, return to the step 3 until collecting 2500 data points.
Revised: else, return to the step 3 until collecting 2500 data points.

Original: Sample x 1 , x 2 ∈ R d ∼ N (0 , 1) .
Revised: Sample x 1 , x 2 ∈ R d ∼ N (0 , 1) .

Original: Normalize x 1 , x 2 to unit length.
Revised: Normalize x 1 , x 2 to unit length.

Original: If x 1 · z > − 0 .
Revised: If x 1 · z > − 0 .

Original: 1 or x 2 · z > − 0 .
Revised: 1 or x 2 · z > − 0 .

Original: 1 , resample.
Revised: 1 , resample.

Original: if | x 1 · x 2 | ≤ 0 .
Revised: if | x 1 · x 2 | ≤ 0 .

Original: 25 , return to the Step 5.
Revised: 25 , return to the Step 5.

Original: If x 1 · x 2 > 0 .
Revised: If x 1 · x 2 > 0 .

Original: then y = 2 ; else y = 1 . 8.
Revised: then y = 2 ; else y = 1 . 8.

Original: Get the data point ( P 1 x 1 , P 2 x 2 , y ) .
Revised: Get the data point ( P 1 x 1 , P 2 x 2 , y ) .

Original: If
Revised: If

Original: the total amount
Revised: the total amount

Original: of data generated
Revised: of data generated

Original: is less than
Revised: is less than

Original: , return to the Step 5; else break
Revised: , return to the Step 5; else break

Original: In the data labeled 0, each modality contains features that can give a correct prediction, while in the data labeled 1 or 2, we need both modalities together to make the right predictions.
Revised: In the data labeled 0, each modality contains features that can give a correct prediction, while in the data labeled 1 or 2, we need both modalities together to make the right predictions.

Original: To further understand how uni-modal models give predictions in dataset that containing both uni-modal and paired features, we give the confusion matrix of the uni-modal model.
Revised: To further understand how uni-modal models give predictions in dataset that containing both uni-modal and paired features, we give the confusion matrix of the uni-modal model.

Original: As Table 21 shows, the unimodal model can give correct predictions for data labeled 0, while for data labeled 1 or 2, it fails and gives a random predictions.
Revised: As Table 21 shows, the unimodal model can give correct predictions for data labeled 0, while for data labeled 1 or 2, it fails and gives a random predictions.

Original: Because for data labeled 1 or 2, we need to learn the relationship of the two modalities to give the correct predictions.
Revised: Because for data labeled 1 or 2, we need to learn the relationship of the two modalities to give the correct predictions.

Original: In this subsection, we mainly discuss the synthetic multi-modal datasets and in Appendix A.7, we conduct various experiments on real-world multi-modal datasets to help us understand the uni-modal features, paired features and cross-modal interaction in multi-modal training better.
Revised: In this subsection, we mainly discuss the synthetic multi-modal datasets and in Appendix A.7, we conduct various experiments on real-world multi-modal datasets to help us understand the uni-modal features, paired features and cross-modal interaction in multi-modal training better.

Original: Training settings on synthetics datasets.
Revised: Training settings on synthetics datasets.

Original: We use a two layer MLP with ReLU as activation function.
Revised: We use a two layer MLP with ReLU as activation function.

Original: As for hidden layer, we use 200 dimensions for multi-modal training and 100 dimensions for uni-modal training.
Revised: As for hidden layer, we use 200 dimensions for multi-modal training and 100 dimensions for uni-modal training.

Original: We use SGD as the optimizer and the learning rate is 0.2.
Revised: We use SGD as the optimizer and the learning rate is 0.2.

Original: In each iteration, we use the whole training set to compute the gradients.
Revised: In each iteration, we use the whole training set to compute the gradients.

Original: And we provide the code in supplement materials.
Revised: And we provide the code in supplement materials.

Original: A. U NI -M ODAL P ERFORMANCE IN K INETICS - 
Revised: A. U NI -M ODAL P ERFORMANCE IN K INETICS -

Original: Kinetics-400 is a dynamic dataset, because videos may be removed from YouTube.
Revised: Kinetics-400 is a dynamic dataset, because videos may be removed from YouTube.

Original: In this subsection, we report the uni-modal performance of ours and Xiao et al.
Revised: In this subsection, we report the uni-modal performance of ours and Xiao et al.

Original: ’s on Kinetics-400.
Revised: ’s on Kinetics-400.

Original: As
Revised: As

Original: Table 22 shows, we cannot reproduce their uni-modal performance and ours are lower than theirs.
Revised: Table 22 shows, we cannot reproduce their uni-modal performance and ours are lower than theirs.

Original: But we demonstrate that UMT outperforms AVSlowFast in Sec 4.3.1, which shows UMT’s effectiveness.
Revised: But we demonstrate that UMT outperforms AVSlowFast in Sec 4.3.1, which shows UMT’s effectiveness.

Original: [Table]
Revised: [Table]

Original: B P ROOF B. P ROOF OF T HEOREM 3. 
Revised: B P ROOF B. P ROOF OF T HEOREM 3.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: In uni-modal ensemble, assume that the training procedure learns b m1 features in modality x m 1 and learns b m2 features in modality x m 2 .
Revised: In uni-modal ensemble, assume that the training procedure learns b m1 features in modality x m 1 and learns b m2 features in modality x m 2 .

Original: We order the probability of uni-modal features (both x m 1 and x m 2 ) in decreasing order of predicting probability p , namely, p [1] , p [2] , . . . .
Revised: We order the probability of uni-modal features (both x m 1 and x m 2 ) in decreasing order of predicting probability p , namely, p [1] , p [2] , . . . .

Original: In multi-modal training approaches, assume that the training procedure learns k m1 uni-modal features in modality x m 1 , learns k m2 uni-modal features in modality x m 2 , and learns k pa paired features with predicting probability p ( h 1 ) , . .
Revised: In multi-modal training approaches, assume that the training procedure learns k m1 uni-modal features in modality x m 1 , learns k m2 uni-modal features in modality x m 2 , and learns k pa paired features with predicting probability p ( h 1 ) , . .

Original: , p ( h k pa ) .
Revised: , p ( h k pa ) .

Original: We provide three types of laziness:
Revised: We provide three types of laziness:

Original: (a. ) Quantity Laziness : k m1 + k m2 + k pa ≤
Revised: (a. ) Quantity Laziness : k m1 + k m2 + k pa ≤

Original: [Equation]
Revised: [Equation]

Original: b m1 , b m2 } .
Revised: b m1 , b m2 } .

Original: (b. )
Revised: (b. )

Original: Uni-modal Laziness : Each modality in multi-modal training approaches performs worse than uni-modal training.
Revised: Uni-modal Laziness : Each modality in multi-modal training approaches performs worse than uni-modal training.

Original: (c. )
Revised: (c. )

Original: Performance Laziness : Consider a new testing point, then for every δ > 0 , if the following inequality holds:
Revised: Performance Laziness : Consider a new testing point, then for every δ > 0 , if the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where ∆( δ ) =
Revised: where ∆( δ ) =

Original: (cid:112) 8( k pa + b m1 − k m1 + b m2 − k m2 ) log(1 /δ ) , then with probability 14 at least 1 − δ , uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.
Revised: (cid:112) 8( k pa + b m1 − k m1 + b m2 − k m2 ) log(1 /δ ) , then with probability 14 at least 1 − δ , uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.

Original: We prove the theorem, which shows that naive joint training indeed suffers from overfitting issues, meaning that it learns less features compared to uni-modal ensemble.
Revised: We prove the theorem, which shows that naive joint training indeed suffers from overfitting issues, meaning that it learns less features compared to uni-modal ensemble.

Original: Proof.
Revised: Proof.

Original: We first introduce some additional notations used in the proof.
Revised: We first introduce some additional notations used in the proof.

Original: We define the features trained in x m 1 -uni-modal training as f 1 ( x m 1 ) , . . .
Revised: We define the features trained in x m 1 -uni-modal training as f 1 ( x m 1 ) , . . .

Original: , f b m1 ( x m 1 ) , define the features trained in x m 1uni-modal training as g 1 ( x m 2 ) , . . .
Revised: , f b m1 ( x m 1 ) , define the features trained in x m 1uni-modal training as g 1 ( x m 2 ) , . . .

Original: , g b m2 ( x m 2 ) .
Revised: , g b m2 ( x m 2 ) .

Original: Therefore, there are in total b m1 + b m2 features learned in uni-modal ensemble, namely, f 1 ( x m 1 ) , . . .
Revised: Therefore, there are in total b m1 + b m2 features learned in uni-modal ensemble, namely, f 1 ( x m 1 ) , . . .

Original: , f b m1 ( x m 1 ) , g 1 ( x m 2 ) , . . .
Revised: , f b m1 ( x m 1 ) , g 1 ( x m 2 ) , . . .

Original: , g b m2 ( x m 2 ) .
Revised: , g b m2 ( x m 2 ) .

Original: Besides, We define the features trained in multi-modal training approaches as f 1 ( x m 1 ) , . . .
Revised: Besides, We define the features trained in multi-modal training approaches as f 1 ( x m 1 ) , . . .

Original: , f k m1 ( x m 1 ) , g 1 ( x m 2 ) , . . . , g k m2 ( x m 2 ) , h 1 ( x m 1 , x m 2 ) , . .
Revised: , f k m1 ( x m 1 ) , g 1 ( x m 2 ) , . . . , g k m2 ( x m 2 ) , h 1 ( x m 1 , x m 2 ) , . .

Original: , h k pa ( x m 1 , x m 2 ) .
Revised: , h k pa ( x m 1 , x m 2 ) .

Original: When the context is clear, we omit the dependency of x m 1 , x m 2 and denote them as f i , g i , h i for simplicity.
Revised: When the context is clear, we omit the dependency of x m 1 , x m 2 and denote them as f i , g i , h i for simplicity.

Original: When the context is clear, we abuse the notation r to represent arbitrary f , g or h .
Revised: When the context is clear, we abuse the notation r to represent arbitrary f , g or h .

Original: The corresponding predicting probability of feature r i is denoted as p ( r i ) .
Revised: The corresponding predicting probability of feature r i is denoted as p ( r i ) .

Original: To summary, there are b m1 + b m2 features in uni-modal ensemble, k m1 + k m2 + k pa features in multi-modal training approaches.
Revised: To summary, there are b m1 + b m2 features in uni-modal ensemble, k m1 + k m2 + k pa features in multi-modal training approaches.

Original: We first prove statement (a.), which claims that the number of features learned in multi-modal training approaches are provably less than any of the number of features learned in uni-modal training.
Revised: We first prove statement (a.), which claims that the number of features learned in multi-modal training approaches are provably less than any of the number of features learned in uni-modal training.

Original: The proof depends on the following Lemma B.1.
Revised: The proof depends on the following Lemma B.1.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Assume there exists T features r
Revised: Assume there exists T features r

Original: i , i = 1 , . . .
Revised: i , i = 1 , . . .

Original: , T .
Revised: , T .

Original: If we replace one of the T features (without loss of generality, r T ) with a more powerful feature r ′ , where p ( r ′ ) > p ( r T ) , then the predicting probability for each data point increases (where the probability is taken over the randomness of the training data).
Revised: If we replace one of the T features (without loss of generality, r T ) with a more powerful feature r ′ , where p ( r ′ ) > p ( r T ) , then the predicting probability for each data point increases (where the probability is taken over the randomness of the training data).

Original: We next provide the proof of statements (a.): based on Lemma B.1.
Revised: We next provide the proof of statements (a.): based on Lemma B.1.

Original: We shall prove k m1 + k m2 + k pa < b m1 without loss of generality.
Revised: We shall prove k m1 + k m2 + k pa < b m1 without loss of generality.

Original: Start from the features f 1 ( x m 1 ) , . . .
Revised: Start from the features f 1 ( x m 1 ) , . . .

Original: , f k m1 ( x m 1 ) which are common features in both multi-modal training approaches and Uni-modal training.
Revised: , f k m1 ( x m 1 ) which are common features in both multi-modal training approaches and Uni-modal training.

Original: Next step, we add feature f k m1 +1 in uni-modal approachesand g 1 in multi-modal training approaches.
Revised: Next step, we add feature f k m1 +1 in uni-modal approachesand g 1 in multi-modal training approaches.

Original: Obviously, p ( g 1 ) > p ( f k m1 +1 ) due to the training priority (or multi-modal training approaches should learn f k m1 +1 instead of g 1 ).
Revised: Obviously, p ( g 1 ) > p ( f k m1 +1 ) due to the training priority (or multi-modal training approaches should learn f k m1 +1 instead of g 1 ).

Original: Therefore, the predicting probability of multi-modal training approaches is larger than uni-modal approaches.
Revised: Therefore, the predicting probability of multi-modal training approaches is larger than uni-modal approaches.

Original: Repeating the procedure by comparing g i with f k m1 + i and comparing h j with f k m1 + k m2 + j , the pre- dicting probability of multi-modal training approaches is always larger than uni-modal approaches.
Revised: Repeating the procedure by comparing g i with f k m1 + i and comparing h j with f k m1 + k m2 + j , the pre- dicting probability of multi-modal training approaches is always larger than uni-modal approaches.

Original: Note that b m1 should be always larger than k m1 + k m2 , or the predicting probability of uni-modal approaches would be smaller than multi-modal training approaches.
Revised: Note that b m1 should be always larger than k m1 + k m2 , or the predicting probability of uni-modal approaches would be smaller than multi-modal training approaches.

Original: At the end of the comparison, the predicting probability of multi-modal training approaches is still larger than uni-modal approaches.
Revised: At the end of the comparison, the predicting probability of multi-modal training approaches is still larger than uni-modal approaches.

Original: This requires that uni-modal approaches should learn more features, which can be regarded as uni-modal approacheslearns a features while multi-modal training approaches learns an empty feature.
Revised: This requires that uni-modal approaches should learn more features, which can be regarded as uni-modal approacheslearns a features while multi-modal training approaches learns an empty feature.

Original: In conclusion, uni-modal approaches learns more features compared to multi-modal training approaches, leading to b m1 > k m1 + k m2 + k pa .
Revised: In conclusion, uni-modal approaches learns more features compared to multi-modal training approaches, leading to b m1 > k m1 + k m2 + k pa .

Original: We next prove the statement (b.).
Revised: We next prove the statement (b.).

Original: The proof of (b.) is based on (a.).
Revised: The proof of (b.) is based on (a.).

Original: We next only consider modality x m 1 , the proof for modality x m 2 is similar.
Revised: We next only consider modality x m 1 , the proof for modality x m 2 is similar.

Original: Note that the since the number of features learned in multi-modal training approaches is less than b m1 , the number of features learned in x m 1 must be less than b m1 (Note that those features can be either paired feature or uni-modal feature, namely, f 1 , . . .
Revised: Note that the since the number of features learned in multi-modal training approaches is less than b m1 , the number of features learned in x m 1 must be less than b m1 (Note that those features can be either paired feature or uni-modal feature, namely, f 1 , . . .

Original: , f k m1 and h 1 , . .
Revised: , f k m1 and h 1 , . .

Original: . , h k pa ).
Revised: . , h k pa ).

Original: Therefore, multi-modal training approaches learns less features compared to uni-modal approaches in modality x m 1 .
Revised: Therefore, multi-modal training approaches learns less features compared to uni-modal approaches in modality x m 1 .

Original: On the other hand, the predicting probability of features learned in multi-modal training approaches ( f 1 , . . .
Revised: On the other hand, the predicting probability of features learned in multi-modal training approaches ( f 1 , . . .

Original: , f k m1 and h 1 , . .
Revised: , f k m1 and h 1 , . .

Original: , h k pa , considering only modality x m 1 for the paired feature) is less than that learned in uni-modal approaches ( f 1 , . .
Revised: , h k pa , considering only modality x m 1 for the paired feature) is less than that learned in uni-modal approaches ( f 1 , . .

Original: . , f b m1 ), because otherwise, uni-modal approaches will learn the features in h instead of f .
Revised: . , f b m1 ), because otherwise, uni-modal approaches will learn the features in h instead of f .

Original: In conclusion, when considering only modality x m 1 , the number of features learned in multi-modal training approaches is less and its corresponding predicting probability is small.
Revised: In conclusion, when considering only modality x m 1 , the number of features learned in multi-modal training approaches is less and its corresponding predicting probability is small.

Original: Therefore, each modality in multi-modal training approaches performs worse than uni-modal approaches.
Revised: Therefore, each modality in multi-modal training approaches performs worse than uni-modal approaches.

Original: We finally prove the statement (c.).
Revised: We finally prove the statement (c.).

Original: Recall that the loss is − (cid:80) i u ( r i ) where u ( r i )
Revised: Recall that the loss is − (cid:80) i u ( r i ) where u ( r i )

Original: =
Revised: =

Original: I ( yr i > 0) − I ( yr i < 0) .
Revised: I ( yr i > 0) − I ( yr i < 0) .

Original: Note that E ( u ( r i ))
Revised: Note that E ( u ( r i ))

Original: = 12 p ( r i ) and | u ( r i ) | ≤ 1 .
Revised: = 12 p ( r i ) and | u ( r i ) | ≤ 1 .

Original: We derive that:
Revised: We derive that:

Original: P   − (cid:88) i ∈
Revised: P   − (cid:88) i ∈

Original: [ k m1 ] u ( f i ) − (cid:88) i ∈
Revised: [ k m1 ] u ( f i ) − (cid:88) i ∈

Original: [ k m2 ] u ( g i ) − (cid:88)
Revised: [ k m2 ] u ( g i ) − (cid:88)

Original: i ∈
Revised: i ∈

Original: [ k pa ] u ( h i ) ≤ − (cid:88) i ∈ [ b m1 ] u ( f i ) − (cid:88)
Revised: [ k pa ] u ( h i ) ≤ − (cid:88) i ∈ [ b m1 ] u ( f i ) − (cid:88)

Original: i ∈ [ b m2 ] u ( g i ) 
Revised: i ∈ [ b m2 ] u ( g i ) 

Original:  =
Revised:  =

Original: P   (cid:88) k m1 <i ≤ b m u ( f i ) + (cid:88) k m2 <i ≤ b m u ( g i ) − (cid:88)
Revised: P   (cid:88) k m1 <i ≤ b m u ( f i ) + (cid:88) k m2 <i ≤ b m u ( g i ) − (cid:88)

Original: i ∈
Revised: i ∈

Original: [ k pa ] u ( h i ) ≤
Revised: [ k pa ] u ( h i ) ≤

Original: 
Revised: 

Original: =
Revised: =

Original: P   (cid:88)
Revised: P   (cid:88)

Original: k m
Revised: k m

Original: [Equation]
Revised: [Equation]

Original: m u (
Revised: m u (

Original: [Equation]
Revised: [Equation]

Original: + (cid:88) k m2 <i ≤ b m u ( g i ) − (cid:88) i ∈
Revised: + (cid:88) k m2 <i ≤ b m u ( g i ) − (cid:88) i ∈

Original: [ k pa ] u ( h i ) + 1 2 E ≤ 1 2 E 
Revised: [ k pa ] u ( h i ) + 1 2 E ≤ 1 2 E 

Original:  , where E = − E ( (cid:80) k m1 <i ≤ b m1 u ( f i ) + (cid:80)
Revised:  , where E = − E ( (cid:80) k m1 <i ≤ b m1 u ( f i ) + (cid:80)

Original: k m2 <i ≤ b m2 u ( g i ) − (cid:80) i ∈ [ k pa ] u ( h i )) =
Revised: k m2 <i ≤ b m2 u ( g i ) − (cid:80) i ∈ [ k pa ] u ( h i )) =

Original: (cid:80) i ∈
Revised: (cid:80) i ∈

Original: [ k pa ] p ( h i ) − (cid:80) k m1 <i ≤ b m1 p ( f i ) − (cid:80) k m2 <i ≤ b m2 p ( g i ) .
Revised: [ k pa ] p ( h i ) − (cid:80) k m1 <i ≤ b m1 p ( f i ) − (cid:80) k m2 <i ≤ b m2 p ( g i ) .

Original: Due to the training priority and the conclusion in (a.), (cid:88) i ∈ [ b m1 +1 ,b m1 + b m2 ] p [ i ] ≤ (cid:88) k m1 <i
Revised: Due to the training priority and the conclusion in (a.), (cid:88) i ∈ [ b m1 +1 ,b m1 + b m2 ] p [ i ] ≤ (cid:88) k m1 <i

Original: ≤ b m p ( f i ) + (cid:88) k m2 <i ≤ b m p ( g i ) .
Revised: ≤ b m p ( f i ) + (cid:88) k m2 <i ≤ b m p ( g i ) .

Original: Therefore,
Revised: Therefore,

Original: E ≤ (cid:80)
Revised: E ≤ (cid:80)

Original: [Equation]
Revised: [Equation]

Original: We next apply Hoeffding inequality on Equation B.1 and derive that
Revised: We next apply Hoeffding inequality on Equation B.1 and derive that

Original: [Equation]
Revised: [Equation]

Original: To conclude
Revised: To conclude

Original: [Equation]
Revised: [Equation]

Original: multi-modal training approaches outperform uni-modal ensemble concerning the testing loss with probability at least 1 − δ .
Revised: multi-modal training approaches outperform uni-modal ensemble concerning the testing loss with probability at least 1 − δ .

Original: Compared to uni-modal ensemble, denote the additional paired feature are indexed by c , and the additional uni-modal feature in uni-modal ensemble are indexed by v .
Revised: Compared to uni-modal ensemble, denote the additional paired feature are indexed by c , and the additional uni-modal feature in uni-modal ensemble are indexed by v .

Original: We have that:
Revised: We have that:

Original: [Equation]
Revised: [Equation]

Original: Therefore, if (cid:80) j ∈ [ v ] p j − (cid:80) i ∈ [ c ] p i ≥ (cid:112) 8( c + v ) log(1 /δ ) , the probability is done.
Revised: Therefore, if (cid:80) j ∈ [ v ] p j − (cid:80) i ∈ [ c ] p i ≥ (cid:112) 8( c + v ) log(1 /δ ) , the probability is done.

Original: Therefore, for a new data point, uni-modal ensemble can outperforms multi-modal training approaches with high probability.
Revised: Therefore, for a new data point, uni-modal ensemble can outperforms multi-modal training approaches with high probability.

Original: Proof of Lemma B.1 .
Revised: Proof of Lemma B.1 .

Original: We define r [ − T ] as the features r 1 , . . .
Revised: We define r [ − T ] as the features r 1 , . . .

Original: , r T − 1 .
Revised: , r T − 1 .

Original: The proof is divided into two parts, depending on whether (cid:80) i ∈ [ T − 1] I ( r i ̸ = 0) is even or odd.
Revised: The proof is divided into two parts, depending on whether (cid:80) i ∈ [ T − 1] I ( r i ̸ = 0) is even or odd.

Original: We regard the term (cid:80) i ∈
Revised: We regard the term (cid:80) i ∈

Original: [ T − 1] I ( r i ̸ = 0) as the number of effective features in r [ − T ] .
Revised: [ T − 1] I ( r i ̸ = 0) as the number of effective features in r [ − T ] .

Original: To simplify the discussion, we rescale r such that | yr | = 1 (when r ̸ = 0 ) or | yr | = 0 (when r = 0 ).
Revised: To simplify the discussion, we rescale r such that | yr | = 1 (when r ̸ = 0 ) or | yr | = 0 (when r = 0 ).

Original: Case 1 : When the number of effective features in r [ − T ] is even.
Revised: Case 1 : When the number of effective features in r [ − T ] is even.

Original: (a. )
Revised: (a. )

Original: If | (cid:80) i ∈ [ T − 1]
Revised: If | (cid:80) i ∈ [ T − 1]

Original: yr i | ≥ 2 , adding r T or r ′ does not alter the predicting probability, namely
Revised: yr i | ≥ 2 , adding r T or r ′ does not alter the predicting probability, namely

Original: [Table]
Revised: [Table]

Original:   y  
Revised:   y  

Original: [Table]
Revised: [Table]

Original: ′ +
Revised: ′ +

Original: [Table]
Revised: [Table]

Original: ∈
Revised: ∈

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table]
Revised: [Table]

Original: ] yr
Revised: ] yr

Original: [Table]
Revised: [Table]

Original: (b. )
Revised: (b. )

Original: When the number of effective features in r [ − T ] is even, | (cid:80)
Revised: When the number of effective features in r [ − T ] is even, | (cid:80)

Original: i ∈
Revised: i ∈

Original: [ T − 1] yr i | ̸ = 1 .
Revised: [ T − 1] yr i | ̸ = 1 .

Original: (c. )
Revised: (c. )

Original: When | (cid:80) i ∈ [ T − 1]
Revised: When | (cid:80) i ∈ [ T − 1]

Original: yr i | = 0 , due to the assumption
Revised: yr i | = 0 , due to the assumption

Original: that p ( r ′ ) > p ( r T ) and ϵ ( r ) = p ( r ) /c , adding r ′ helps increase the predicting probability compared to r T , namely
Revised: that p ( r ′ ) > p ( r T ) and ϵ ( r ) = p ( r ) /c , adding r ′ helps increase the predicting probability compared to r T , namely

Original: P   y  
Revised: P   y  

Original: r T + (cid:88) i ∈
Revised: r T + (cid:88) i ∈

Original: [ T − 1] yr i 
Revised: [ T − 1] yr i 

Original: [Table]
Revised: [Table]

Original: >
Revised: >

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: P   y
Revised: P   y

Original: [Table]
Revised: [Table]

Original: ∈
Revised: ∈

Original: [Table] [Table]
Revised: [Table] [Table]

Original: i ∈
Revised: i ∈

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original:  > P   y 
Revised:  > P   y 

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: i   = 0 (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) (cid:88) i ∈
Revised: i   = 0 (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) (cid:88) i ∈

Original: [ T − 1] yr i (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) =
Revised: [ T − 1] yr i (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) =

Original:  .
Revised:  .

Original: The above inequality is derived based on the following equation:
Revised: The above inequality is derived based on the following equation:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ] yr
Revised: ] yr

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Since we assume c >
Revised: Since we assume c >

Original: [Table]
Revised: [Table]

Original: the probability increases with probability p ( r T ) .
Revised: the probability increases with probability p ( r T ) .

Original: Therefore, under the three conditions
Revised: Therefore, under the three conditions

Original: [Table]
Revised: [Table]

Original: adding r ′ increase the predicting probability more compared to r T .
Revised: adding r ′ increase the predicting probability more compared to r T .

Original: In summary, under case 1 (a-c), adding r ′ increase the predicting probability compared to r T .
Revised: In summary, under case 1 (a-c), adding r ′ increase the predicting probability compared to r T .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the
Revised: the

Original: [Table]
Revised: [Table]

Original: r [
Revised: r [

Original: [Table]
Revised: [Table]

Original: odd.
Revised: odd.

Original: The
Revised: The

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: be a little bit more complex
Revised: be a little bit more complex

Original: [Table]
Revised: [Table]

Original: case
Revised: case

Original: [Table] [Table]
Revised: [Table] [Table]

Original: If | (cid:80) i ∈ [ T − 1]
Revised: If | (cid:80) i ∈ [ T − 1]

Original: yr i | ≥ 2 , similar to case 1, adding r T or r ′ does not alter the predicting probability, namely
Revised: yr i | ≥ 2 , similar to case 1, adding r T or r ′ does not alter the predicting probability, namely

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  =
Revised:  =

Original: 12 p ( r T )
Revised: 12 p ( r T )

Original: [Equation]
Revised: [Equation]

Original: (b.2 )
Revised: (b.2 )

Original: If (cid:80) i ∈ [ T − 1] yr
Revised: If (cid:80) i ∈ [ T − 1] yr

Original: i = +1 :
Revised: i = +1 :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Note that the probability of event (b.1) and the probability of event (b.2) satisfy the following equation by Lemma B.2:
Revised: Note that the probability of event (b.1) and the probability of event (b.2) satisfy the following equation by Lemma B.2:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the total probability under case (b) is
Revised: Therefore, the total probability under case (b) is

Original: [Equation]
Revised: [Equation]

Original: which is independent of p ( r T ) .
Revised: which is independent of p ( r T ) .

Original: Therefore, adding r T or r ′ share the same predicting probability.
Revised: Therefore, adding r T or r ′ share the same predicting probability.

Original: (c
Revised: (c

Original: ) When the number of effective features in r [
Revised: ) When the number of effective features in r [

Original: −
Revised: −

Original: T ] is
Revised: T ] is

Original: odd
Revised: odd

Original: , |
Revised: , |

Original: (cid:80)
Revised: (cid:80)

Original: i ∈
Revised: i ∈

Original: [ T − 1] yr i | ̸ = 0 .
Revised: [ T − 1] yr i | ̸ = 0 .

Original: In summary, under case 2 (a-c), adding r ′ do not decrease the predicting probability compared to r T .
Revised: In summary, under case 2 (a-c), adding r ′ do not decrease the predicting probability compared to r T .

Original: The following lemmas are used during the proof.
Revised: The following lemmas are used during the proof.

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Consider T − 1 features r 1 , . .
Revised: Consider T − 1 features r 1 , . .

Original: , r T − 1 , the following equation holds:
Revised: , r T − 1 , the following equation holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It can be proved to compare the events
Revised: It can be proved to compare the events

Original: A = { (cid:80) i ∈
Revised: A = { (cid:80) i ∈

Original: [ T − 1] yr i = 1 } and B = { (cid:80) i ∈
Revised: [ T − 1] yr i = 1 } and B = { (cid:80) i ∈

Original: [ T − 1] yr i = − 1 } .
Revised: [ T − 1] yr i = − 1 } .

Original: Every event in A has a complementary event in B , namely,
Revised: Every event in A has a complementary event in B , namely,

Original: [Equation]
Revised: [Equation]

Original: Comparing each event in A with its complementary event in B leads to the conclusion.
Revised: Comparing each event in A with its complementary event in B leads to the conclusion.

Original: Combining case 1 and case 2 together leads to the final conclusion.
Revised: Combining case 1 and case 2 together leads to the final conclusion.

Original: B. G ENERALIZE T HEOREM 3.4 TO MORE MODALITIES
Revised: B. G ENERALIZE T HEOREM 3.4 TO MORE MODALITIES

Original: We next show that the results in Theorem 3.4 can be generalized to the regime of more modals.
Revised: We next show that the results in Theorem 3.4 can be generalized to the regime of more modals.

Original: Specifically, we assume a T -modal regimes, and denote the modals as x m i , i ∈
Revised: Specifically, we assume a T -modal regimes, and denote the modals as x m i , i ∈

Original: [ T ] .
Revised: [ T ] .

Original: In uni-modal pre-training approaches, let b m i denote the number of returned features in modal i .
Revised: In uni-modal pre-training approaches, let b m i denote the number of returned features in modal i .

Original: In multi-modal joint training, let k m i denote the number of uni-modal features for modal i , and k pa denote the number of returned paired features.
Revised: In multi-modal joint training, let k m i denote the number of uni-modal features for modal i , and k pa denote the number of returned paired features.

Original: We derive the following Theorem B.3 for the multi-modal regimes.
Revised: We derive the following Theorem B.3 for the multi-modal regimes.

Original: Theorem B.3.
Revised: Theorem B.3.

Original: Based on the above notations, we provide three types of laziness from three perspectives:
Revised: Based on the above notations, we provide three types of laziness from three perspectives:

Original: (a. ) Quantity Laziness : (cid:80) i
Revised: (a. ) Quantity Laziness : (cid:80) i

Original: k
Revised: k

Original: m i + k pa ≤ min i { b m i } .
Revised: m i + k pa ≤ min i { b m i } .

Original: (b. )
Revised: (b. )

Original: Uni-modal Laziness : Each modality in multi-modal training approaches performs worse than uni-modal training.
Revised: Uni-modal Laziness : Each modality in multi-modal training approaches performs worse than uni-modal training.

Original: (c. )
Revised: (c. )

Original: Performance Laziness : Consider a new testing point, then for every δ > 0 , if the following inequality holds:
Revised: Performance Laziness : Consider a new testing point, then for every δ > 0 , if the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: where ∆( δ ) =
Revised: where ∆( δ ) =

Original: (cid:113) 8( k pa + (cid:80) j
Revised: (cid:113) 8( k pa + (cid:80) j

Original: [ b m j − k
Revised: [ b m j − k

Original: m j ]) log(1 /δ )
Revised: m j ]) log(1 /δ )

Original: , then with probability 15 at least 1 − δ , uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.
Revised: , then with probability 15 at least 1 − δ , uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF T HEOREM 3.

Original: Theorem 3.5.
Revised: Theorem 3.5.

Original: Denote the paired features by h 1 , . . .
Revised: Denote the paired features by h 1 , . . .

Original: h L with corresponding predicting probability p ( h 1 ) , . .
Revised: h L with corresponding predicting probability p ( h 1 ) , . .

Original: , p ( h L ) .
Revised: , p ( h L ) .

Original: Assume that distillation can boost the training priority by p 0 > 0 .
Revised: Assume that distillation can boost the training priority by p 0 > 0 .

Original: If there exists paired features whose predicting probability exceeds the boosting probability p 0 , namely, the set S is not empty:
Revised: If there exists paired features whose predicting probability exceeds the boosting probability p 0 , namely, the set S is not empty:

Original: [Equation]
Revised: [Equation]

Original: Then UMT helps uni-modal feature learning and can also learn easy-to-learn paired features.
Revised: Then UMT helps uni-modal feature learning and can also learn easy-to-learn paired features.

Original: [Table]
Revised: [Table]

Original: Proof.
Revised: Proof.

Original: The core of Theorem 3.5 is to clarify the training priority.
Revised: The core of Theorem 3.5 is to clarify the training priority.

Original: We revisit the notations of Theorem 3.4 as follows without further clarification.
Revised: We revisit the notations of Theorem 3.4 as follows without further clarification.

Original: At the end of the training, uni-modal ensemble learn b m1 + b m2 useful features, namely, f 1 , . . .
Revised: At the end of the training, uni-modal ensemble learn b m1 + b m2 useful features, namely, f 1 , . . .

Original: , f b m1 , g 1 , . .
Revised: , f b m1 , g 1 , . .

Original: . , g b m2 .
Revised: . , g b m2 .

Original: And multi-modal training approaches learn k m1 + k m2 + k pa features: f 1 , . .
Revised: And multi-modal training approaches learn k m1 + k m2 + k pa features: f 1 , . .

Original: , f k m1 , g 1 , . .
Revised: , f k m1 , g 1 , . .

Original: . , g k m2 , h 1 , . .
Revised: . , g k m2 , h 1 , . .

Original: h k pa .
Revised: h k pa .

Original: We note that there are still many empty features e i in the model due to the initialization.
Revised: We note that there are still many empty features e i in the model due to the initialization.

Original: By distillation, the model learns the features according to the new priority.
Revised: By distillation, the model learns the features according to the new priority.

Original: Since the set S is not empty, there exists paired features that is learned before the empty features.
Revised: Since the set S is not empty, there exists paired features that is learned before the empty features.

Original: By distillation, the model would learn all the useful features that appear in uni-modal approaches, as well as those features in set S .
Revised: By distillation, the model would learn all the useful features that appear in uni-modal approaches, as well as those features in set S .

Original: Therefore, UMT outperforms uni-modal ensemblewhen there exists useful paired features.
Revised: Therefore, UMT outperforms uni-modal ensemblewhen there exists useful paired features.

Original: [Table]
Revised: [Table]

Original:  A CONCRETE EXAMPLE TO ILLUSTRATE T HEOREM 3. 
Revised:  A CONCRETE EXAMPLE TO ILLUSTRATE T HEOREM 3.

Original: We next provide a concrete example to better illustrate the Modality Laziness issues.
Revised: We next provide a concrete example to better illustrate the Modality Laziness issues.

Original: For Example B.4, we aim to show the Modality Laziness issues.
Revised: For Example B.4, we aim to show the Modality Laziness issues.

Original: For Example B.5, we aim to show the role of the pushing force.
Revised: For Example B.5, we aim to show the role of the pushing force.

Original: Example B.4 .
Revised: Example B.4 .

Original: Consider modality x m 1 with features f 1 , f 2 , f 3 (corresponding prediction probability p = 0 .
Revised: Consider modality x m 1 with features f 1 , f 2 , f 3 (corresponding prediction probability p = 0 .

Original: 2 , 0 . 1 , 0 . 05 ), and modality x m 2 with features g 1 , g 2 , g 3 (corresponding prediction probability p = 0 .
Revised: 2 , 0 . 1 , 0 . 05 ), and modality x m 2 with features g 1 , g 2 , g 3 (corresponding prediction probability p = 0 .

Original: We show the dataset in Table 23 and aim to minimize the training loss to zero.
Revised: We show the dataset in Table 23 and aim to minimize the training loss to zero.

Original: In uni-modal approaches, we learn features f 1 , f 2 and f 3 on modality x m 1 (similarly, g 1 , g 2 , and g 3 on modality x m 2 ).
Revised: In uni-modal approaches, we learn features f 1 , f 2 and f 3 on modality x m 1 (similarly, g 1 , g 2 , and g 3 on modality x m 2 ).

Original: Therefore, we learn features f 1 , f 2 , f 3 , g 1 , g 2 , g 3 in uni-modal ensemble.
Revised: Therefore, we learn features f 1 , f 2 , f 3 , g 1 , g 2 , g 3 in uni-modal ensemble.

Original: In multi-modal training approaches without paired feature, we can only learn three features f 1 , f 2 , gdue to the training priority f 1 > g 1 > f 2 > g 2 > f 3 > g 3 (decreasing order in p ).
Revised: In multi-modal training approaches without paired feature, we can only learn three features f 1 , f 2 , gdue to the training priority f 1 > g 1 > f 2 > g 2 > f 3 > g 3 (decreasing order in p ).

Original: This phenomenon is caused by modality laziness.
Revised: This phenomenon is caused by modality laziness.

Original: We next consider another paired feature h with probability p = 0 .
Revised: We next consider another paired feature h with probability p = 0 .

Original: Under the case, multi-modal training approaches only learn two features h and f 1 .
Revised: Under the case, multi-modal training approaches only learn two features h and f 1 .

Original: Therefore, when h is not powerful enough, uni-modal ensemble outperforms multi-modal training approaches.
Revised: Therefore, when h is not powerful enough, uni-modal ensemble outperforms multi-modal training approaches.

Original: Example B.5 .
Revised: Example B.5 .

Original: We follow the notations and dataset in Example B.4.
Revised: We follow the notations and dataset in Example B.4.

Original: By applying the pushing force, assume that each probability of uni-modal feature boosts 0.15, which changes the training priority to f 1 > g 1 > h > f 2 > g 2 > f 3 > g 3 (decreasing order in p ′ ).
Revised: By applying the pushing force, assume that each probability of uni-modal feature boosts 0.15, which changes the training priority to f 1 > g 1 > h > f 2 > g 2 > f 3 > g 3 (decreasing order in p ′ ).

Original: Therefore, multi-modal training approaches (with pushing force) learns f 1 , f 2 , h .
Revised: Therefore, multi-modal training approaches (with pushing force) learns f 1 , f 2 , h .

Original: As a comparison, multi-modal training approaches (without pushing force) can only learn f 1 , h .
Revised: As a comparison, multi-modal training approaches (without pushing force) can only learn f 1 , h .

Original: Therefore, pushing force helps learn more features.
Revised: Therefore, pushing force helps learn more features.

Original: We additionally remark that we only consider the training error in this example, and there might be other penalties in practice ( e.g. , distillation loss).
Revised: We additionally remark that we only consider the training error in this example, and there might be other penalties in practice ( e.g. , distillation loss).