Original: M ODALITY C OMPLEMENTARINESS : T OWARDS U NDERSTANDING M ULTI - MODAL R OBUSTNESS
Revised: M ODALITY C OMPLEMENTARITY : T OWARDS U NDERSTANDING M ULTI - MODAL R OBUSTNESS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Along with the success of multi-modal learning, the robustness of multi-modal learning is receiving attention due to real-world safety concerns.
Revised: Along with the success of multi-modal learning, the robustness of multi-modal learning is receiving attention due to real-world safety concerns.

Original: Multi-modal models are anticipated to be more robust due to the possible redundancy between modalities.
Revised: Multi-modal models are anticipated to be more robust due to the possible redundancy between modalities.

Original: However, some empirical results have offered contradictory conclusions.
Revised: However, some empirical results have offered contradictory conclusions.

Original: In this paper, we point out an essential factor that causes this discrepancy:
Revised: In this paper, we point out an essential factor that causes this discrepancy:

Original: The difference in the amount of modality-wise complementary information.
Revised: The difference in the amount of modality-wise complementary information.

Original: We provide an information-theoretical analysis of how the modality complementariness affects the multi-modal robustness.
Revised: We provide an information-theoretical analysis of how the modality complementarity affects the multi-modal robustness.

Original: Based on the analysis, we design a metric for quantifying how complementary the modalities are to others and propose an effective pipeline to calculate our metric.
Revised: Based on the analysis, we design a metric for quantifying how complementary the modalities are to others and propose an effective pipeline to calculate our metric.

Original: Experiments on carefully-designed synthetic data verify our theory.
Revised: Experiments on carefully-designed synthetic data verify our theory.

Original: Further, we apply our metric to real-world multi-modal datasets and reveal their property.
Revised: Further, we apply our metric to real-world multi-modal datasets and reveal their property.

Original: To our best knowledge, we are the first to identify modality complementariness as an important factor affecting multi-modal robustness.
Revised: To our best knowledge, we are the first to identify modality complementarity as an important factor affecting multi-modal robustness.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, deep neural networks have proved successful in various areas, such as image recognition (He et al., 2015; Krizhevsky et al., 2012), speech recognition (Chorowski et al., 2015) and neural machine translation (Wu et al., 2016).
Revised: Recently, deep neural networks have proved successful in various areas, such as image recognition (He et al., 2015; Krizhevsky et al., 2012), speech recognition (Chorowski et al., 2015) and neural machine translation (Wu et al., 2016).

Original: The revolution is also happening in multi-modal research, e.g. RGB-D semantic segmentation (Wang et al., 2016), audio-visual learning (Zhao et al., 2018), and visual question answering (Antol et al., 2015).
Revised: The revolution is also happening in multi-modal research, e.g. RGB-D semantic segmentation (Wang et al., 2016), audio-visual learning (Zhao et al., 2018), and visual question answering (Antol et al., 2015).

Original: Intuitively, multi-modal models are anticipated to be more robust due to the potential redundancy between modalities.
Revised: Intuitively, multi-modal models are anticipated to be more robust due to the potential redundancy between modalities.

Original: When one of the modalities is corrupted, others can compensate for the loss.
Revised: When one of the modalities is corrupted, others can compensate for the loss.

Original: This intuition is supported by both psychological studies of the human perception system (Sumby & Pollack, 1954) and deep learning practices (Zhang et al., 2019b;
Revised: This intuition is supported by both psychological studies of the human perception system (Sumby & Pollack, 1954) and deep learning practices (Zhang et al., 2019b;

Original: Qian et al., 2021; Wang et al., 2020).
Revised: Qian et al., 2021; Wang et al., 2020).

Original: However, some recent studies cast doubt on this belief.
Revised: However, some recent studies cast doubt on this belief.

Original: From a theoretical perspective, the multimodal models usually have a larger input dimension than uni-modal models, and the increase of input dimensions significantly degrades model robustness (Ford et al., 2019;
Revised: From a theoretical perspective, the multimodal models usually have a larger input dimension than uni-modal models, and the increase of input dimensions significantly degrades model robustness (Ford et al., 2019;

Original: Simon-Gabriel et al., 2019).
Revised: Simon-Gabriel et al., 2019).

Original: From an empirical view, some experiments suggest that multi-modal integration may be more vulnerable to attacks or corruptions than uni-modal models (Yu et al., 2020; Tian & Xu, 2021; Ma et al., 2022).
Revised: From an empirical view, some experiments suggest that multi-modal integration may be more vulnerable to attacks or corruptions than uni-modal models (Yu et al., 2020; Tian & Xu, 2021; Ma et al., 2022).

Original: What causes this contradiction in multi-modal robustness?
Revised: What causes this contradiction in multi-modal robustness?

Original: We notice that the conclusions above are drawn under assorted multi-modal task settings ranging from action classification to question answering, which vary in the presence and type of modality interconnections (Liang et al., 2022).
Revised: We notice that the conclusions above are drawn under assorted multi-modal task settings ranging from action classification to question answering, which vary in the presence and type of modality interconnections (Liang et al., 2022).

Original: Therefore, a question arises naturally:
Revised: Therefore, a question arises naturally:

Original: What aspects of modality interconnection affect the multi-modal robustness?
Revised: What aspects of modality interconnection affect the multi-modal robustness?

Original: We hypothesize that the complementariness of modalities plays an essential role.
Revised: We hypothesize that the complementarity of modalities plays an essential role.

Original: If the complementary part of each modality is negligible, the corruption of one modality would not severely damage the model performance.
Revised: If the complementary part of each modality is negligible, the corruption of one modality would not severely damage the model performance.

Original: Otherwise, the multi-modal model could perform even worse than a uni-modal model.
Revised: Otherwise, the multi-modal model could perform even worse than a uni-modal model.

Original: For the visual question answering task, the two modalities are highly complementary: Only perceiving either the question or the image could not lead to an ideal answer (Agrawal et al., 2018).
Revised: For the visual question answering task, the two modalities are highly complementary: Only perceiving either the question or the image could not lead to an ideal answer (Agrawal et al., 2018).

Original: For the action classification task, the RGB and optical flow are less complementary since each of them can suggest a roughly correct answer (Feichtenhofer et al., 2016b).
Revised: For the action classification task, the RGB and optical flow are less complementary since each of them can suggest a roughly correct answer (Feichtenhofer et al., 2016b).

Original: To validate the above hypothesis, we first demonstrate the key role of modality complementariness to model robustness through theoretical analysis.
Revised: To validate the above hypothesis, we first demonstrate the key role of modality complementarity to model robustness through theoretical analysis.

Original: Following previous work (Tsai et al., 2020; Sun et al., 2020; Sridharan & Kakade, 2008; Tosh et al., 2021), we use an information-theoretical framework for multi-modal learning and study how the complementary information affects robustness under missing and noisy modality settings.
Revised: Following previous work (Tsai et al., 2020; Sun et al., 2020; Sridharan & Kakade, 2008; Tosh et al., 2021), we use an information-theoretical framework for multi-modal learning and study how the complementary information affects robustness under missing and noisy modality settings.

Original: Based on the analysis, we design a novel metric and a practical calculation pipeline built on Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018) to quantify the complementariness of modalities in multi-modal datasets.
Revised: Based on the analysis, we design a novel metric and a practical calculation pipeline built on Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018) to quantify the complementarity of modalities in multi-modal datasets.

Original: With the specially designed metric and pipeline on hand, we verify our theory and the effectiveness of our proposed metric on synthetic data and a carefully-designed toy dataset AAV-MNIST.
Revised: With the specially designed metric and pipeline on hand, we verify our theory and the effectiveness of our proposed metric on synthetic data and a carefully-designed toy dataset AAV-MNIST.

Original: The results are consistent with the model robustness in modality missing, noisy modality, and adversarial attack settings on the datasets we test on.
Revised: The results are consistent with the model robustness in modality missing, noisy modality, and adversarial attack settings on the datasets we test on.

Original: Then we apply our metric to real-world multi-modal datasets to further investigate the modality complementariness in different settings.
Revised: Then we apply our metric to real-world multi-modal datasets to further investigate the modality complementarity in different settings.

Original: To our best knowledge, we are the first to identify and prove the important role of modality complementariness in multi-modal robustness.
Revised: To our best knowledge, we are the first to identify and prove the important role of modality complementarity in multimodal robustness.

Original: Hence, for future research, we recommend that researchers consider the modality complementariness as a control variable for a fairer comparison of multi-modal robustness.
Revised: Hence, for future research, we recommend that researchers consider the modality complementarity as a control variable for a fairer comparison of multi-modal robustness.

Original: The main contributions are highlighted as follows:
Revised: The main contributions are highlighted as follows:

Original: • We point out the effect of modality complementariness on multi-modal model robustness through information-theoretical analysis.
Revised: • We point out the effect of modality complementarity on multi-modal model robustness through information-theoretical analysis.

Original: • We propose a dataset-wise metric to qualitatively evaluate how complementary the modalities are in each multi-modal dataset, and also design a pipeline for computing the metric in real-world datasets.
Revised: • We propose a dataset-wise metric to qualitatively evaluate how complementary the modalities are in each multi-modal dataset, and also design a pipeline for computing the metric in real-world datasets.

Original: • We create a synthetic dataset and a toy dataset (AAV-MNIST) to test our metric and pipeline.
Revised: • We create a synthetic dataset and a toy dataset (AAV-MNIST) to test our metric and pipeline.

Original: These datasets cover various complementary situations of different modalities and are used to verify the effectiveness of our pipeline.
Revised: These datasets cover various complementary situations of different modalities and are used to verify the effectiveness of our pipeline.

Original: • We further reveal the modality complementariness and its relationship with model robustness in real-world multi-modal
Revised: • We further reveal the modality complementarity and its relationship with model robustness in real-world multi-modal

Original: datasets, which could lead to a less biased comparison for multimodal robustness.
Revised: datasets, which could lead to a less biased comparison for multimodal robustness.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-modal learning.
Revised: Multi-modal learning.

Original: Various multi-modal learning tasks and models are proposed in recent years (Baltrusaitis et al., 2017; Liang et al., 2021), such as multi-modal reasoning (Yi et al., 2019;
Revised: Various multi-modal learning tasks and models are proposed in recent years (Baltrusaitis et al., 2017; Liang et al., 2021), such as multi-modal reasoning (Yi et al., 2019;

Original: Johnson et al., 2016), cross-modal retrieval (Gu et al., 2017; Radford et al., 2021), and cross-modal translation (Ramesh et al., 2021).
Revised: Johnson et al., 2016), cross-modal retrieval (Gu et al., 2017; Radford et al., 2021), and cross-modal translation (Ramesh et al., 2021).

Original: Among these settings, we mainly focus on the supervised multimodal classification setting.
Revised: Among these settings, we mainly focus on the supervised multimodal classification setting.

Original: The theoretical understanding of multi-modal learning is relatively under-explored, with (Huang et al., 2021) deriving generalization error bounds and (Sun et al., 2020) comparing with the Bayesian posterior classifiers.
Revised: The theoretical understanding of multi-modal learning is relatively under-explored, with (Huang et al., 2021) deriving generalization error bounds and (Sun et al., 2020) comparing with the Bayesian posterior classifiers.

Original: A concept close to multi-modal learning is the multi-view learning (Xu et al., 2013).
Revised: A concept close to multi-modal learning is the multi-view learning (Xu et al., 2013).

Original: The theory of multi-view learning has long been studied both theoretically (Kakade & Foster, 2007; Sridharan & Kakade, 2008; Zhang et al., 2019a;Tosh et al., 2021;
Revised: The theory of multi-view learning has long been studied both theoretically (Zhang et al., 2019a;

Original: Tsai et al., 2020) and empirically (Sindhwani et al., 2005; Ding et al., 2021; Amini et al., 2009; Tian et al., 2019).
Revised: Tosh et al., 2021) and empirically (Sindhwani et al., 2005; Ding et al., 2021; Amini et al., 2009; Tian et al., 2019).

Original: Earlier work (Kakade & Foster, 2007; Sridharan & Kakade, 2008) proposes thewidely-adopted multi-view assumption: Each modality suffices to predict the label.
Revised: Earlier work (Kakade & Foster, 2007; Sridharan & Kakade, 2008) proposes the multi-view assumption: Each modality suffices to predict the label.

Revised: Recently, many multi-view analyses adopted this assumption (Han et al., 2021;

Revised: Tsai et al., 2020; Lin et al., 2021; Federici et al., 2020; Lin et al., 2022).

Original: However, as pointed out by (Huang et al., 2021; 2022), this might not hold in the multi-modal learning setting.
Revised: However, as pointed out by (Huang et al., 2021; 2022), this might not hold in the multi-modal learning setting.

Original: Model robustness.
Revised: Model robustness.

Original: Model robustness under data missing (Ramoni & Sebastiani, 2001), random corruption (Hendrycks & Dietterich, 2019), and adversarial attacks (Madry et al., 2017) is constantly been concerned in consideration of real-world safety issues.
Revised: Model robustness under data missing (Ramoni & Sebastiani, 2001), random corruption (Hendrycks & Dietterich, 2019), and adversarial attacks (Madry et al., 2017) is constantly been concerned in consideration of real-world safety issues.

Original: For uni-modal models, several methods are proposed to strengthen model robustness (Papernot et al., 2015; Huang et al., 2015; Meng &
Revised: For uni-modal models, several methods are proposed to strengthen model robustness (Papernot et al., 2015; Huang et al., 2015; Meng &

Original: Chen, 2017).
Revised: Chen, 2017).

Original: For multi-modal models, some papers regard the use of multi-modality as a way to improve robustness (Zhang et al., 2019b;
Revised: For multi-modal models, some papers regard the use of multi-modality as a way to improve robustness (Zhang et al., 2019b;

Original: Qian et al., 2021; Wang et al., 2020), while others continue to improve multi-modal models’ robustness by designing new network architectures and fusion methods (Kim & Ghosh, 2019a; Tsai et al., 2018; Yang et al., 2021) and training routines (Eitel et al., 2015; Ma et al., 2021).
Revised: Qian et al., 2021; Wang et al., 2020), while others continue to improve multi-modal models’ robustness by designing new network architectures and fusion methods (Kim & Ghosh, 2019a; Tsai et al., 2018; Yang et al., 2021) and training routines (Eitel et al., 2015; Ma et al., 2021).

Original: When dealing with known missing patterns, researchers explore additional ways: data imputation through available modalities or views (Tran et al., 2017), or training different models for different availability of modalities (Yuan et al., 2012).
Revised: When dealing with known missing patterns, researchers explore additional ways: data imputation through available modalities or views (Tran et al., 2017; Lin et al., 2021), or training different models for different availability of modalities (Yuan et al., 2012).

Original: Apart from improving robustness, another line of work aims to analyze or estimate the robustness of existing uni-modal methods (Cohen
Revised: Our analysis

Original: [Figure]
Revised: [Figure]

Revised: points out the upper bound of these methods.

Revised: Apart from improving robustness, another line of work aims to analyze or estimate the robustness of existing uni-modal methods (Cohen et al., 2019;

Original: et al., 2019; Carlini et al., 2019; Mahmood et al., 2021) and multi-modal methods (Yu et al., 2020; 
Revised: Carlini et al., 2019; Mahmood et al., 2021) and multi-modal methods (Yu et al., 2020; Tian & Xu, 2021;

Original: Tian & Xu, 2021; Ma et al., 2022; Rosenberg et al., 2021).
Revised: Ma et al., 2022; Rosenberg et al., 2021; Li et al., 2020).

Original: We in this work analyze one factor of multi-modal model robustness both theoretically and empirically.
Revised: We in this work analyze one factor of multi robustness both theoretically and empirically.

Original: Mutual information in deep learning.
Revised: Mutual information in deep learning.

Original: Mutual information is tightly related to deep learning through multiple ways, including information bottleneck method (Tishby et al., 2000), analysis of learning methods (Wu & Verdu, 2012; Tsai et al., 2020; Shwartz-Ziv & Tishby, 2017), and new learning methods based on mutual information (Hjelm et al., 2018; Bachman et al., 2019; Sun et al., 2020).
Revised: Mutual information is tightly related to deep learning through multiple ways, including information bottleneck method (Tishby et al., 2000), analysis of learning methods (Wu & Verdu, 2012; Tsai et al., 2020; Shwartz-Ziv & Tishby, 2017), and new learning methods based on mutual information (Hjelm et al., 2018; Bachman et al., 2019; Sun et al., 2020).

Original: On the other hand, learning methods help to estimate the amount of mutual information.
Revised: On the other hand, learning methods help to estimate the amount of mutual information.

Original: Representative work includes Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018), CPC (van den
Revised: Representative work includes Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018), CPC (van den

Original: Oord et al., 2018), DIM (Hjelm et al., 2018), and DoE estimator (McAllester & Stratos, 2018).
Revised: Oord et al., 2018), DIM (Hjelm et al., 2018), and DoE estimator (McAllester & Stratos, 2018).

Original: We apply the MINE to our calculation pipeline for its simplicity and effectiveness.
Revised: We apply the MINE to our calculation pipeline for its simplicity and effectiveness.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we first build an information-theoretical framework for multi-modal learning and show the impact of complementary information to model robustness in modality missing and single noisy source cases, which are commonly studied in previous work (Kim & Ghosh, 2019a; Tian & Xu, 2021) and widely encountered in practice, e.g., some sensors are broken, prone to noise (e.g. cameras in foggy environments), or expensive to use(e.g.
Revised: In this section, we first build an information-theoretical framework for multi-modal learning and show the impact of complementary information to model robustness in modality missing and single noisy source cases, which are commonly studied in previous work (Kim & Ghosh, 2019a; Tian & Xu, 2021) and widely encountered in practice, e.g., some sensors are broken, prone to noise (e.g. cameras in foggy environments), or expensive to use(e.g.

Original: X-ray data for medical analysis).
Revised: X-ray data for medical analysis).

Original: An illustration of these cases is plotted in Figure 1.
Revised: An illustration of these cases is plotted in Figure 1.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: Notations.
Revised: Notations.

Original: We use H ( A ) to represent the entropy of a random variable A , H
Revised: We use H ( A ) to represent the entropy of a random variable A , H

Original: ( A | B ) for the conditional entropy given another variable B , I ( A ; B ) for the mutual information between random variable A and B , I ( A ; B | C ) for the conditional mutual information conditioned on random variable C , and
Revised: ( A | B ) for the conditional entropy given another variable B , I ( A ; B ) for the mutual information between random variable A and B , I ( A ; B | C ) for the conditional mutual information conditioned on random variable C , and

Original: I ( A ; B ; C ) for the interaction information (i.e., mutual information of three variables, possibly a negative value).
Revised: I ( A ; B ; C ) for the interaction information (i.e., mutual information of three variables, possibly a negative value).

Original: Multi-modal learning formulation.
Revised: Multi-modal learning formulation.

Original: We adopt the formulation for multi-modal learning problem proposed in (Huang et al., 2021)
Revised: We adopt the formulation for multi-modal learning problem proposed in (Huang et al., 2021)

Original: Denote the M -modality input space as X =
Revised: Denote the M -modality input space as X =

Original: X 1 ×
Revised: X 1 ×

Original: X 2 × . . .
Revised: X 2 × . . .

Original: X M and the target space as Y .
Revised: X M and the target space as Y .

Original: Each data point ( X 1 , X 2 , . . .
Revised: Each data point ( X 1 , X 2 , . . .

Original: , X M , Y ) is sampled from an unknown distribution on X × Y .
Revised: , X M , Y ) is sampled from an unknown distribution on X × Y .

Original: Our goal is that, based on the random input variables X 1 , X 2 , . . .
Revised: Our goal is that, based on the random input variables X 1 , X 2 , . . .

Original: , X M from
Revised: , X M from

Original: M modalities, we would like to infer the target Y .
Revised: M modalities, we would like to infer the target Y .

Original: In classification tasks, Y is a discrete random variable, while in regression tasks Y is continuous.
Revised: In classification tasks, Y is a discrete random variable, while in regression tasks Y is continuous.

Original: For instance, considering audio-visual action recognition (Gao et al., 2019; Feichtenhofer et al., 2016a), let X 1 be the audio part and X 2 be the frames of a video clip, and we want to infer the label Y , i.e., what kind of action is performed in the clip.
Revised: For instance, considering audio-visual action recognition (Gao et al., 2019; Feichtenhofer et al., 2016a), let X 1 be the audio part and X 2 be the frames of a video clip, and we want to infer the label Y , i.e., what kind of action is performed in the clip.

Original: In the subsequent analysis, we will focus on the common case M = 2 (Feichtenhofer et al., 2016a) for simplicity, and we denote the two modalities as X ∈ X and Z ∈ Z respectively.
Revised: In the subsequent analysis, we will focus on the common case M = 2 (Feichtenhofer et al., 2016a) for simplicity, and we denote the two modalities as X ∈ X and Z ∈ Z respectively.

Original: Notice that our analysis and results can be extended to cases with more than two modalities at the expense of notations.
Revised: Notice that our analysis and results can be extended to cases with more than two modalities at the expense of notations.

Original: Complementary information.
Revised: Complementary information.

Original: Now we define the complementary information in the following, which is essential through our theoretical analysis.
Revised: Now we define the complementary information in the following, which is essential through our theoretical analysis.

Original: Definition 1 (complementary information) .
Revised: Definition 1 (complementary information) .

Original: For input variables X , Z and the target Y , define the complementary information provided by X , Z as follows
Revised: For input variables X , Z and the target Y , define the complementary information provided by X , Z as follows

Original: [Equation]
Revised: [Equation]

Original: When the target is clear from the context, we omit the Y in the subscript.
Revised: When the target is clear from the context, we omit the Y in the subscript.

Original: Mathematically, I ( X ; Y | Z ) represents the information in the target Y that is accessible for X
Revised: Mathematically, I ( X ; Y | Z ) represents the information in the target Y that is accessible for X

Original: but not predictable for Z .
Revised: but not predictable for Z .

Original: Thus Γ X can characterize the unique label information owned by modality X , and similarly for Γ Z .
Revised: Thus Γ X can characterize the unique label information owned by modality X , and similarly for Γ Z .

Original: Hence Γ X together with Γ Z can determine the complementariness of modality X and Z .
Revised: Hence Γ X together with Γ Z can determine the complementarity of modality X and Z .

Original: Clearly, larger Γ X and Γ Z imply higher complementary information content.
Revised: Clearly, larger Γ X and Γ Z imply higher complementary information content.

Original: From the standard derivation in information theory, we can obtain the following relation:
Revised: From the standard derivation in information theory, we can obtain the following relation:

Original: [Equation]
Revised: [Equation]

Original: Previous theoretical analyses of multi-view learning Sridharan & Kakade (2008); Xu et al.
Revised: Previous theoretical analyses of multi-view learning Sridharan & Kakade (2008); Xu et al.

Original: Tosh et al.
Revised: Tosh et al.

Original: (2021) usually adopt the multi-view assumption that each view is redundant in terms of predicting the target, i.e. Γ X and Γ Z are both small.
Revised: (2021) usually adopt the multi-view assumption that each view is redundant in terms of predicting the target, i.e. Γ X and Γ Z are both small.

Original: However, this assumption does not always hold in the multi-modal learning setting Antol et al.
Revised: However, this assumption does not always hold in the multi-modal learning setting Antol et al.

Original: In the following subsections, we will show how the complementary information Γ X and Γ Z affect the model robustness in missing modality and noise settings.
Revised: In the following subsections, we will show how the complementary information Γ X and Γ Z affect the model robustness in missing modality and noise settings.

Original: Motivated by this theoretical observation, we will propose a metric to evaluate the modality complementariness and a pipeline for calculation in Section 4.
Revised: Motivated by this theoretical observation, we will propose a metric to evaluate the modality complementarity and a pipeline for calculation in Section 4.

Original: Bayes error rate.
Revised: Bayes error rate.

Original: We introduce the Bayes error rate Fukunaga & Hummels (1987) to measure the model performance, which is the lowest possible error for any arbitrary classifier or predictor from the multiple modalities to infer the target.
Revised: We introduce the Bayes error rate Fukunaga & Hummels (1987) to measure the model performance, which is the lowest possible error for any arbitrary classifier or predictor from the multiple modalities to infer the target.

Original: Formally, given two modalities X and Z , the multi-modal
Revised: Formally, given two modalities X and Z , the multi-modal

Original: Bayes errors for classification P e c and regression P e r are defined as follows:
Revised: Bayes errors for classification P e c and regression P e r are defined as follows:

Original: [Equation]
Revised: [Equation]

Original: The Bayes error rate helps us focus on the interconnection among modalities X , Z and target Y in each multi-modal task and omit other factors’ effects on model robustness, e.g. dataset size, training routines, and network architectures.
Revised: The Bayes error rate helps us focus on the interconnection among modalities X , Z and target Y in each multi-modal task and omit other factors’ effects on model robustness, e.g. dataset size, training routines, and network architectures.

Original: 3. M ISSING M ODALITY
Revised: 3. M ISSING M ODALITY

Original: We first consider the missing modality scenario and assume modality Z is missing w.l.o.g..
Revised: We first consider the missing modality scenario and assume modality Z is missing w.l.o.g..

Original: Then the
Revised: Then the

Original: Bayes error rates for missing modality, denoted as P Miss e c and P Miss e r become
Revised: Bayes error rates for missing modality, denoted as P Miss e c and P Miss e r become

Original: [Equation]
Revised: [Equation]

Original: Now we establish the following theoretical guarantees to quantify differences between the Bayes error rate of multi-modal and missing-modality.
Revised: Now we establish the following theoretical guarantees to quantify differences between the Bayes error rate of multi-modal and missing-modality.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For random variables X, Z and discrete random variable Y , we have
Revised: For random variables X, Z and discrete random variable Y , we have

Original: [Equation]
Revised: [Equation]

Original: For continuous random variable Y , if we further assume that Y takes value in [ − 1 , 1] , then we have
Revised: For continuous random variable Y , if we further assume that Y takes value in [ − 1 , 1] , then we have

Original: [Equation]
Revised: [Equation]

Original: Remark 1 .
Revised: Remark 1 .

Original: The gap between P Miss e c (best model performance in modality missing setting) and P e c (best model performance in normal setting) reflects the best model robustness against modality missing.
Revised: The gap between P Miss e c (best model performance in modality missing setting) and P e c (best model performance in normal setting) reflects the best model robustness against modality missing.

Original: For the classification task, when Γ Z = 0 , i.e., there is no complementary information of
Revised: For the classification task, when Γ Z = 0 , i.e., there is no complementary information of

Original: Z , the information from Z can be covered by the information from X for predicting Y .
Revised: Z , the information from Z can be covered by the information from X for predicting Y .

Original: In this case, the P Miss e c shares the same lower and upper bound with P e c , so the performance of the best model would not be affected by modality missing.
Revised: In this case, the P Miss e c shares the same lower and upper bound with P e c , so the performance of the best model would not be affected by modality missing.

Original: As the Γ Z increases, the bounds for P Miss e c rise, while the bounds for P e c is fixed, indicating that the best model performance drops under modality missing, i.e. the robustness decays.
Revised: As the Γ Z increases, the bounds for P Miss e c rise, while the bounds for P e c is fixed, indicating that the best model performance drops under modality missing, i.e. the robustness decays.

Original: Considering the extreme case when Γ Z is large enough, the lower bound of
Revised: Considering the extreme case when Γ Z is large enough, the lower bound of

Original: P Miss e c is greater than the upper bound of P e c , so the missing modality performance is provably worse than normal performance.
Revised: P Miss e c is greater than the upper bound of P e c , so the missing modality performance is provably worse than normal performance.

Original: Remark 2 .
Revised: Remark 2 .

Original: For the regression task, the closer P Miss e r and P e r are, the robust the best model is.
Revised: For the regression task, the closer P Miss e r and P e r are, the robust the best model is.

Original: From the result above, the gap between two Bayes optimal predictors is bounded above by the complementary information, hence increased by Γ Z .
Revised: From the result above, the gap between two Bayes optimal predictors is bounded above by the complementary information, hence increased by Γ Z .

Original: So the model robustness under modality missing is worsened along with the increase of Γ Z .
Revised: So the model robustness under modality missing is worsened along with the increase of Γ Z .

Original: 3. S INGLE N OISY M ODALITY
Revised: 3. S INGLE N OISY M ODALITY

Original: The modality corrupted by noise is another situation that we often encounter in practice, e.g., the foggy weather results in noisy RGB images in autonomous driving.
Revised: The modality corrupted by noise is another situation that we often encounter in practice, e.g., the foggy weather results in noisy RGB images in autonomous driving.

Original: In this section, we study the case that one of the modalities has additional noise, which can be easily extended to the case that all modalities are noisy at the expense of notations.
Revised: In this section, we study the case that one of the modalities has additional noise, which can be easily extended to the case that all modalities are noisy at the expense of notations.

Original: Formally, we consider that Gaussian noise N is added to the input modality
Revised: Formally, we consider that Gaussian noise N is added to the input modality

Original: Z (Zheng et al., 2016; Kim & Ghosh, 2019b).
Revised: Z (Zheng et al., 2016; Kim & Ghosh, 2019b).

Original: We use R N ( Z ) =
Revised: We use R N ( Z ) =

Original: Z + N to denote the modality Z after adding Gaussian noise.
Revised: Z + N to denote the modality Z after adding Gaussian noise.

Original: By (Cover, 1999) we can obtain the following characterization for the mutual information between Z and R N ( Z )
Revised: By (Cover, 1999) we can obtain the following characterization for the mutual information between Z and R N ( Z )

Original: Proposition 1.
Revised: Proposition 1.

Original: If Z, N ∈ R , assuming that 0 < E [ Z 2 ] ≤ p Z , N ∼ N (0 , σ ) , and N is independent of Z , then we have
Revised: If Z, N ∈ R , assuming that 0 < E [ Z 2 ] ≤ p Z , N ∼ N (0 , σ ) , and N is independent of Z , then we have

Original: [Equation]
Revised: [Equation]

Original: Remark 3 .
Revised: Remark 3 .

Original: When the noise is heavy, i.e., the σ is large, the upper bound of I ( Z ; R N ( Z )) decays, indicating that it is harder to recover Z from R N ( Z ) and thus harder to infer Y from R N ( Z ) .
Revised: When the noise is heavy, i.e., the σ is large, the upper bound of I ( Z ; R N ( Z )) decays, indicating that it is harder to recover Z from R N ( Z ) and thus harder to infer Y from R N ( Z ) .

Original: When the noise becomes very heavy, I ( Z ; R N ( Z )) will be near zero and R N ( Z ) is close to pure Gaussian noise, as if the modality Z is missing, which suits our intuition.
Revised: When the noise becomes very heavy, I ( Z ; R N ( Z )) will be near zero and R N ( Z ) is close to pure Gaussian noise, as if the modality Z is missing, which suits our intuition.

Original: In this extreme case, we can refer to the analysis in section 3.2.
Revised: In this extreme case, we can refer to the analysis in section 3.2.

Original: In this setting, the Bayes error rate for classification denoted as P No e c can be written as:
Revised: In this setting, the Bayes error rate for classification denoted as P No e c can be written as:

Original: [Equation]
Revised: [Equation]

Original: Then we can provide the lower bound for P
Revised: Then we can provide the lower bound for P

Original: No e c .
Revised: No e c .

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: For random variables X, Y, Z, N , if E [ Z 2 ] ≤ p Z , N ∼ N (0 , σ ) , then
Revised: For random variables X, Y, Z, N , if E [ Z 2 ] ≤ p Z , N ∼ N [Equation] 0 , σ ) , then

Original: [Equation]
Revised: [Equation]

Original: Remark 4 .
Revised: Remark 4 .

Original: Similar to the analysis in modality missing setting, the gap between P No e c (best model performance in noisy modality setting) and P e c (best model performance in normal setting) reflects the best model robustness against noisy modality.
Revised: Similar to the analysis in modality missing setting, the gap between P No e c (best model performance in noisy modality setting) and P e c (best model performance in normal setting) reflects the best model robustness against noisy modality.

Original: For the classification task, the lower bound of P No e c increases as Γ Z or σ increases.
Revised: For the classification task, the lower bound of P No e c increases as Γ Z or σ increases.

Original: Since the bounds of P e c are fixed, the gap between P No e c and P e c becomes larger, and the model robustness under noisy setting is worse.
Revised: Since the bounds of P e c are fixed, the gap between P No e c and P e c becomes larger, and the model robustness under noisy setting is worse.

Original: Therefore, if the Γ Z is larger, the best predictor become more vulnerable to the added noise.
Revised: Therefore, if the Γ Z is larger, the best predictor become more vulnerable to the added noise.

Original:  M ETRIC
Revised:  M ETRIC

Original: In this section, we propose a dataset-wise metric based on the complementary information to quantify the modality complementariness.
Revised: In this section, we propose a dataset-wise metric based on the complementary information to quantify the modality complementarity.

Original: We also bring our metric to practical use by leveraging the existing mutual information estimator, Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018).
Revised: We also bring our metric to practical use by leveraging the existing mutual information estimator, Mutual Information Neural Estimator (MINE) (Belghazi et al., 2018).

Original: Modality Modality
Revised: Modality Modality

Original: Feature Extractor Feature Extractor
Revised: Feature Extractor Feature Extractor

Original: Pre-trained
Revised: Pre-trained

Original: Network
Revised: Network

Original: Pre-trained
Revised: Pre-trained

Original: Network
Revised: Network

Original: Feature
Revised: Feature

Original: Mutual Information Estimator
Revised: Mutual Information Estimator

Original: One-hot Label 𝑻 𝜽
Revised: One-hot Label 𝑻 𝜽

Original: Update by 𝑳 ( 𝜽 )
Revised: Update by 𝑳 ( 𝜽 )

Original: MLP
Revised: MLP

Original: Middle-fusion
Revised: Middle-fusion

Original: Feature
Revised: Feature

Original: 4. M ETRIC D ESIGN
Revised: 4. M ETRIC D ESIGN

Original: From the above analysis, it is natural to use Γ X + Γ Z as the metric since they represent how much complementary information the modalities X
Revised: From the above analysis, it is natural to use Γ X + Γ Z as the metric since they represent how much complementary information the modalities X

Original: and Z can provide exclusively about the target Y .
Revised: and Z can provide exclusively about the target Y .

Original: However, Γ X + Γ Z is not enough for comparing among datasets.
Revised: However, Γ X + Γ Z is not enough for comparing among datasets.

Original: According to equation 1, the same amount of Γ X + Γ Z could indicate different situations if the total information I ( X, Z ; Y ) is different.
Revised: According to equation 1, the same amount of Γ X + Γ Z could indicate different situations if the total information I ( X, Z ; Y ) is different.

Original: Therefore, to make the metric comparable among datasets, we need to perform normalization by dividing it with I ( X, Z ; Y ) , written as
Revised: Therefore, to make the metric comparable among datasets, we need to perform normalization by dividing it with I ( X, Z ; Y ) , written as

Original: [Equation]
Revised: [Equation]

Original: Now, our metric becomes the “proportion” of Γ X + Γ Z in I ( X, Z ; Y ) .
Revised: Now, our metric becomes the “proportion” of Γ X + Γ Z in I ( X, Z ; Y ) .

Original: When our metric is large, the modalities are more complementary to each other and more indispensable for the task.
Revised: When our metric is large, the modalities are more complementary to each other and more indispensable for the task.

Original: Note that this “proportion” could be greater than 1 because I ( X ; Y ; Z )
Revised: Note that this “proportion” could be greater than 1 because I ( X ; Y ; Z )

Original: =
Revised: =

Original: I ( X, Z ; Y ) − Γ X − Γ Z may be negative.
Revised: I ( X, Z ; Y ) − Γ X − Γ Z may be negative.

Original: This happens when Z (or X ) greatly increases the correlation strength between X (or Z ) and Y .
Revised: This happens when Z (or X ) greatly increases the correlation strength between X (or Z ) and Y .

Original: Without Z (or X ), the other modality becomes nearly uncorrelated with the target Y .
Revised: Without Z (or X ), the other modality becomes nearly uncorrelated with the target Y .

Original: Hence, when the metric is greater than 1, it can still reflect the modality complementariness and reveals more about the interconnection between the modalities and the target.
Revised: Hence, when the metric is greater than 1, it can still reflect the modality complementarity and reveals more about the interconnection between the modalities and the target.

Original: 4. C ALCULATION
Revised: 4. C ALCULATION

Original: Now we consider how to calculate our metric.
Revised: Now we consider how to calculate our metric.

Original: Γ X and Γ Z are in the form of conditional mutual information and could not be computed directly.
Revised: Γ X and Γ Z are in the form of conditional mutual information and could not be computed directly.

Original: We notice that
Revised: We notice that

Original: [Equation]
Revised: [Equation]

Original: So we transform the metric into
Revised: So we transform the metric into

Original: [Equation]
Revised: [Equation]

Original: Additionally, considering that most real-world datasets roughly satisfy the realizability assumption, i.e., there exists a function in the hypothesis space that can predict Y given X and Z with zero population risk, we could approximate I ( X, Z ; Y ) =
Revised: Additionally, considering that most real-world datasets roughly satisfy the realizability assumption, i.e., there exists a function in the hypothesis space that can predict Y given X and Z with zero population risk, we could approximate I ( X, Z ; Y ) =

Original: H ( Y ) − H
Revised: H ( Y ) − H

Original: ( Y | X, Z ) with H ( Y ) because the second term is close to zero.
Revised: ( Y | X, Z ) with H ( Y ) because the second term is close to zero.

Original: H ( Y ) is easier to compute given the distribution of Y , especially when we focus on the classification task with discrete labels.
Revised: H ( Y ) is easier to compute given the distribution of Y , especially when we focus on the classification task with discrete labels.

Original: For each mutual information term with the form I ( A ; B ) , we design a two-phase pipeline for computation (See Figure 2):
Revised: For each mutual information term with the form I ( A ; B ) , we design a two-phase pipeline for computation (See Figure 2):

Original: • In the first phase, we reduce the dimension of the high-dimensional input A and B to accelerate the computation by pre-trained feature extractors.
Revised: • In the first phase, we reduce the dimension of the high-dimensional input A and B to accelerate the computation by pre-trained feature extractors.

Original: The pre-trained models are shared among the calculation of all three terms.
Revised: The pre-trained models are shared among the calculation of all three terms.

Original: • In the second phase, we use the extracted features as inputs for MINE (Belghazi et al., 2018) to compute the mutual information.
Revised: • In the second phase, we use the extracted features as inputs for MINE (Belghazi et al., 2018) to compute the mutual information.

Original: Specifically, we calculate the value through optimization converging to a lower bound of the mutual information.
Revised: Specifically, we calculate the value through optimization converging to a lower bound of the mutual information.

Original: For each iteration, we sample an m-sample batch { ( a ( i ) , b ( i ) ) } mi =1 from the joint distribution P ( A, B ) and an m-sample batch { b ′ ( i ) } mi =1 from the marginal distributions P ( B ) .
Revised: For each iteration, we sample an m-sample batch { ( a ( i ) , b ( i ) ) } mi =1 from the joint distribution P ( A, B ) and an m-sample batch { b ′ ( i ) } mi =1 from the marginal distributions P ( B ) .

Original: Denote the estimator network as T and its parameters as θ .
Revised: Denote the estimator network as T and its parameters as θ .

Original: We evaluate the lower bound L as follows and the moving average of gradients of L ( θ ) for updating the network parameters.
Revised: We evaluate the lower bound L as follows and the moving average of gradients of L ( θ ) for updating the network parameters.

Original: [Equation]
Revised: [Equation]

Original: We adjust the original MINE by adding the following trick: The calculation of I ( X ; Y, Z ) and I ( Z ; Y, X ) involve the target Y , so we concatenate the one-hot encoding label with the extracted feature in a middle-fusion fashion and ensure that the estimator network T could combine the two information sources.
Revised: We adjust the original MINE by adding the following trick: The calculation of I ( X ; Y, Z ) and I ( Z ; Y, X ) involve the target Y , so we concatenate the one-hot encoding label with the extracted feature in a middle-fusion fashion and ensure that the estimator network T could combine the two information sources.

Original: For more details, please see the supplementary material.
Revised: For more details, please see the supplementary material.

Original: We believe that modality complementariness is crucial to the analysis of multi-modal robustness.
Revised: We believe that modality complementarity is crucial to the analysis of multi-modal robustness.

Original: Without controlling this factor, we cannot fairly compare experimental results on various multimodal datasets, and thus we cannot derive a universal conclusion on multi-modal robustness.
Revised: Without controlling this factor, we cannot fairly compare experimental results on various multimodal datasets, and thus we cannot derive a universal conclusion on multi-modal robustness.

Original: By calculating our metric on multi-modal datasets, we will better understand their difference in modality complementariness, leading to less biased comparisons and conclusions.
Revised: By calculating our metric on multi-modal datasets, we will better understand their difference in modality complementarity, leading to less biased comparisons and conclusions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct experiments to verify the validity of our analysis and the effectiveness of our pipeline.
Revised: We conduct experiments to verify the validity of our analysis and the effectiveness of our pipeline.

Original: We first introduce the training and testing settings and then show the results on the synthetic dataset, Additive AV-MNIST dataset, and real-world datasets.
Revised: We first introduce the training and testing settings and then show the results on the synthetic dataset, Additive AV-MNIST dataset, and real-world datasets.

Original: Unless otherwise specified, the missing/noise/adversarial robustness mentioned in the following subsections refers to the average accuracy under two sources of missing/noise/adversarial attack, divided by the model accuracy in the clean setting.
Revised: Unless otherwise specified, the missing/noise/adversarial robustness mentioned in the following subsections refers to the average accuracy under two sources of missing/noise/adversarial attack, divided by the model accuracy in the clean setting.

Original: For more detailed settings and results, please see the supplementary materials.
Revised: For more detailed settings and results, please see the Appendix B and C.

Original: Training setting.
Revised: Training setting.

Original: We use an MLP as the estimator.
Revised: We use an MLP as the estimator.

Original: For different datasets, the structure of the MLP varies to match the input size.
Revised: For different datasets, the structure of the MLP varies to match the input size.

Original: We train the estimator on the training set since in reality we only have access to it, and we assume that the validation set is i.i.d.
Revised: We train the estimator on the training set since in reality we only have access to it, and we assume that the validation set is i.i.d.

Original: sampled from the same distribution as the training set.
Revised: sampled from the same distribution as the training set.

Original: For the pre-trained feature extractors, we use Resnet18 (He et al., 2015) and
Revised: For the pre-trained feature extractors, we use Resnet18 (He et al., 2015) and

Original: AudioNet (1-D CNN) (Tian & Xu, 2021) for Kinetics-Sounds and AVE, LeNet5 (LeCun et al., 1998) and an 2-D CNN for AAV-MNIST, and Resnet

Revised: AudioNet (1-D CNN) (Tian & Xu, 2021) for Kinetics-Sounds and AVE, LeNet5 (LeCun et al., 1998) and an 2-D CNN for AAV-MNIST, and Resnet

Original: (He et al., 2015) and BERT (Devlin et al., 2019) for Hateful-Meme dataset.
Revised: (He et al., 2015) and BERT (Devlin et al., 2019) for Hateful-Meme dataset.

Original: For the models tested for robustness, we use late fusion models for
Revised: For the models tested for robustness, we use late fusion models for

Original: AVE, Kinetics-Sounds, and AV-MNIST, and we apply MMBT (Kiela et al., 2019) for Hateful-Meme dataset.
Revised: AVE, Kinetics-Sounds, and AV-MNIST, and we apply MMBT (Kiela et al., 2019) for Hateful-Meme dataset.

Original: Test setting.
Revised: Test setting.

Original: We test the model robustness under two settings discussed above: missing modality and single noisy modality.
Revised: We test the model robustness under two settings discussed above: missing modality and single noisy modality.

Original: We also explore the model robustness under adversarial attack.
Revised: We also explore the model robustness under adversarial attack.

Original: For the missing image or audio, we substitute them with the average of all inputs in the training set.
Revised: For the missing image or audio, we substitute them with the average of all inputs in the training set.

Original: For the missing text, we use a blank sentence < SOS >< EOS > as the input.
Revised: For the missing text, we use a blank sentence < SOS >< EOS > as the input.

Original: Note that the inputs are all scaled to the range [ − 1 , 1] (spectrogram) or [0 , 1] (image).
Revised: Note that the inputs are all scaled to the range [ − 1 , 1] (spectrogram) or [0 , 1] (image).

Original: For noisy image and audio, we add a Gaussian noise
Revised: For noisy image and audio, we add a Gaussian noise

Original: N ∼ N (0 , 0 . 5)
Revised: N ∼ N (0 , 0 . 5)

Original: to each dimension.
Revised: to each dimension.

Original: For noisy text, we replace each word by a random word with a probability 0 .
Revised: For noisy text, we replace each word by a random word with a probability 0 .

Original: For adversarial attack on image and audio, we use FGSM (Goodfellow et al., 2014) with step size ϵ = 0 .
Revised: For adversarial attack on image and audio, we use FGSM (Goodfellow et al., 2014) with step size ϵ = 0 .

Original: We use the results of missing text for adversarial text.
Revised: We use the results of missing text for adversarial text.

Original: 5. S YNTHETIC D ATASET
Revised: 5. S YNTHETIC D ATASET

Original: We first test our analysis on a well-designed synthetic dataset since we can adjust the degree of its modality complementariness.
Revised: We first test our analysis on a well-designed synthetic dataset since we can adjust the degree of its modality complementarity.

Original: Inspired by previous work (Hessel & Lee, 2020) and (Huang et al., 2021), we generate a set of synthetic data ( x, z, y ) .
Revised: Inspired by previous work (Hessel & Lee, 2020) and (Huang et al., 2021), we generate a set of synthetic data ( x, z, y ) .

Original: First, we sample random projection P X ∈ R d 1 × d ,
Revised: First, we sample random projection P X ∈ R d 1 × d ,

Revised: P Z ∈ R d 2 × d , and P ∈ R d × d from a uniform distribution U ( − 0 .

Revised: Then we repeat following steps:

Revised: We sample 5000 data points with 80/20 train/val split and ⟨ d, d 1 , d 2 , δ ⟩ = ⟨ 50 , 200 , 100 , 0 .

Original: O u r m e t r i c our metric 2.
Revised: O u r m e t r i c our metric 2.

Original: our metric 0.
Revised: our metric 0.

Original: 1.0 missing noise adversarial 0.
Revised: 1.0 missing noise adversarial 0.

Original: R obu s t ne ss missing noise adversarial
Revised: R obu s t ne ss missing noise adversarial

Original: As the overlap of two modalities becomes larger, they are less complementary and our metric correspondingly goes down.
Revised: As the overlap of two modalities becomes larger, they are less complementary and our metric correspondingly goes down.

Original: Meanwhile, the tested robustness increases in all three settings.
Revised: Meanwhile, the tested robustness increases in all three settings.

Original: The variance and trend of  mutual  term can be found in 
Revised: The variance and trend of each mutual information term can be found in the

Original: P Z ∈ R d 2 × d , and P ∈ R d × d from a uniform distribution U ( − 0 .

Original: Then we repeat following steps:

Original: Step 1.
Revised: Step 1.

Original: Sample x, z ∈ R d ∼ N (0 , 1) .
Revised: Sample x, z ∈ R d ∼ N (0 , 1) .

Original: Step 2.
Revised: Step 2.

Original: Set z ← (1 − α ) z
Revised: Set z ← (1 − α ) z

Original: + αx ; Then do projection z ← Pz .
Revised: + αx ; Then do projection z ← Pz .

Original: Step 3.
Revised: Step 3.

Original: Normalize x, z to unit length; if | x · z | ≤ δ , return to the Step 2.
Revised: Normalize x, z to unit length; if | x · z | ≤ δ , return to the Step 2.

Original: Step 4.
Revised: Step 4.

Original: Generate the label y :
Revised: Generate the label y :

Original: If x · z > 0 , then y = 1 ; else y = 0 .
Revised: If x · z > 0 , then y = 1 ; else y = 0 .

Original: Return the tuple ( P X x, P Z z, y ) .
Revised: Return the tuple ( P X x, P Z z, y ) .

Original: The α used in data generation controls the overlap between the two modalities X, Z .
Revised: The α used in data generation controls the overlap between the two modalities X, Z .

Original: When α = 0 , the modalities are independent and complementary in predicting the label Y .
Revised: When α = 0 , the modalities are independent and complementary in predicting the label Y .

Original: When α = 1 , they are redundant for prediction.
Revised: When α = 1 , they are redundant for prediction.

Original: Viewing the synthetic dataset with different α as different datasets, we calculate our metric using the pipeline in section 4 and test the robustness of simple two-layer perceptron neural networks 2 trained on these datasets.
Revised: Viewing the synthetic dataset with different α as different datasets, we calculate our metric using the pipeline in section 4 and test the robustness of simple two-layer perceptron neural networks 2 trained on these datasets.

Original: The results are shown in the plot 3.
Revised: The results are shown in the plot 3.

Original: In each dataset, our pipeline can estimate the proposed metric and quantify the complementariness of the two modalities.
Revised: In each dataset, our pipeline can estimate the proposed metric and quantify the complementarity of the two modalities.

Original: Further, the model robustness decreases as the complementariness increases, which verifies our analysis.
Revised: Further, the model robustness decreases as the complementarity increases, which verifies our analysis.

Original: 5. A DDITIVE AV-MNIST
Revised: 5. A DDITIVE AV-MNIST

Original: To show that our pipeline can effectively estimate the modality complementariness of more complex datasets, we further design a toy dataset named Additive AV-MNIST (AAV-MNIST) adapted from the AV-MNIST dataset (Vielzeuf et al., 2018).
Revised: To show that our pipeline can effectively estimate the modality complementarity of more complex datasets, we further design a toy dataset named Additive AV-MNIST (AAV-MNIST) adapted from the AV-MNIST dataset (Vielzeuf et al., 2018).

Original: The modality complementariness can be controlled by a parameter σ in the data generation process.
Revised: The modality complementarity can be controlled by a parameter σ in the data generation process.

Original: Below, we show how to generate AAV-MNIST dataset from the original AV-MNIST dataset.
Revised: Below, we show how to generate AAV-MNIST dataset from the original AV-MNIST dataset.

Original: The following steps are repeated for every image i in
Revised: The following steps are repeated for every image i in

Original: AV-MNIST:
Revised: AV-MNIST:

Original: Step 1.
Revised: Step 1.

Original: Let x be the label of i .
Revised: Let x be the label of i .

Original: Sample δ ∈ R ∼ N (0 , σ ) and round δ to the nearest integer.
Revised: Sample δ ∈ R ∼ N (0 , σ ) and round δ to the nearest integer.

Original: Step 2.
Revised: Step 2.

Original: Set y ← ( x + δ ) mod 10 .
Revised: Set y ← ( x + δ ) mod 10 .

Original: Uniformly sample a spectrogram s from all spectrograms in AV-MNIST with label y .
Revised: Uniformly sample a spectrogram s from all spectrograms in AV-MNIST with label y .

Original: Step 3.
Revised: Step 3.

Original: Calculate the new label t ← ( x + y ) / 2 .
Revised: Calculate the new label t ← ( x + y ) / 2 .

Original: Round t to the nearest integer.
Revised: Round t to the nearest integer.

Original: Return the tuple ( i, s, t ) .
Revised: Return the tuple ( i, s, t ) .

Original: The AAV-MNIST dataset is an extension of AV-MNIST dataset.
Revised: The AAV-MNIST dataset is an extension of AV-MNIST dataset.

Original: When σ = 0 , AAV-MNIST dataset is equivalent to AV-MNIST dataset where each image and its paired spectrogram represent the sameEach [Table] F instead number.
Revised: When σ = 0 , AAV-MNIST dataset is equivalent to AV-MNIST dataset where each image and its paired spectrogram represent the same number.

Original: As σ increases, each image become less correlated with its paired spectrogram, so they become more complementary for predicting the label t 
Revised: As σ increases, each image become less correlated with its paired spectrogram, so they become more complementary for predicting the label t .

Original: We show in the plot 3 that our metric reflects the complementariness of the AAV-MNIST dataset with different σ , indicating that our pipeline is effective in more complex settings beyond the synthetic dataset.
Revised: We show in the plot 3 that our metric reflects the complementarity of the AAV-MNIST dataset with different σ , indicating that our pipeline is effective in more complex settings beyond the synthetic validation sets. network structure more training settings [Table]

Original:  Further, the robustness in the three settings verifies our conclusion that with other conditions unchanged, the more complementary the modalities are, the less robust the best model will be.
Revised: dataset. Further, the robustness in the three settings verifies our conclusion that with other conditions unchanged, the more complementary the modalities are, the less robust the best model will be.

Original: 5. R EAL - WORLD D ATASETS
Revised: 5. R EAL - WORLD D ATASETS

Original: Now we apply our pipeline to real-world datasets to investigate their modality complementariness.
Revised: Now we apply our pipeline to real-world datasets to investigate their modality complementarity.

Original: Our results on AVE (Tian et al., 2018), Kinetics-Sounds (Carreira & Zisserman, 2017; Arandjelovic & Zisserman, 2017), and Hateful-Meme dataset (Kiela et al., 2020a) are shown in the table 1.
Revised: Our results on AVE (Tian et al., 2018), Kinetics-Sounds (Carreira & Zisserman, 2017; Arandjelovic & Zisserman, 2017), and Hateful-Meme dataset (Kiela et al., 2020a) are shown in the table 1.

Original:  We also list results on the AV-MNIST dataset and AAV-MNIST ( σ = 2 . 0 ) for reference.
Revised: The details of these datasets are described in the Appendix B. We also list results on the AV-MNIST dataset and AAV-MNIST ( σ = 2 . 0 ) for reference.

Original: The low value in our metric of AVE, Kinetics-Sounds, and AV-MNIST indicates that they possess relatively little modality complementariness, revealing the heavy redundancy between the two modalities.
Revised: The low value in our metric of AVE, Kinetics-Sounds, and AV-MNIST indicates that they possess relatively little modality complementarity, revealing the heavy redundancy between the two modalities.

Original: On the contrary, the modalities in the Hateful-Meme dataset are more complementary.
Revised: On the contrary, the modalities in the Hateful-Meme dataset are more complementary.

Original: This finding suits our intuition: In the Hateful-Meme dataset, altering the paired text of an image probably changes the label (Kiela et al., 2020b).
Revised: This finding suits our intuition: In the Hateful-Meme dataset, altering the paired text of an image probably changes the label (Kiela et al., 2020b).

Original: Hence, only perceiving the image would not derive the right answer.
Revised: Hence, only perceiving the image would not derive the right answer.

Original: For the event classification task defined by Kinetics-Sounds or AVE, the audio and frames both lead to a rough answer.
Revised: For the event classification task defined by Kinetics-Sounds or AVE, the audio and frames both lead to a rough answer.

Original: The tested robustness demonstrates how the modality complementariness affects model robustness.
Revised: The tested robustness demonstrates how the modality complementarity affects model robustness.

Original: The missing case affects AAV-MNIST( σ = 2 . 0 ) and Hateful-Meme far more than the other three datasets.
Revised: The missing case affects AAV-MNIST( σ = 2 . 0 ) and Hateful-Meme far more than the other three datasets.

Original: They are also more vulnerable in single source noisy case than other datasets.
Revised: They are also more vulnerable in single source noisy case than other datasets.

Original: Hence, to compare model robustness among these datasets, we should take modality complementariness into account.
Revised: Hence, to compare model robustness among these datasets, we should take modality complementarity into account.

Original: For instance, we only compare robustness among datasets with a similar degree of modality complementariness, or we can normalize the results by our metric.
Revised: For instance, we only compare robustness among datasets with a similar degree of modality complementarity, or we can normalize the results by our metric.

Revised: We show an analysis of the existing measure of modality missing robustness by applying our metric in the Appendix C.3.

Original: Furthermore, the model robustness, especially the adversarial robustness, is also affected by factors other than modality complementariness.
Revised: Furthermore, the model robustness, especially the adversarial robustness, is also affected by factors other than modality complementarity.

Original: For instance, the model adversarial robustness of AVE and
Revised: For instance, the model adversarial robustness of AVE and

Original: Kinetics-Sounds dataset is significantly lower than that of AV-MNIST dataset.
Revised: Kinetics-Sounds dataset is significantly lower than that of AV-MNIST dataset.

Original: We conjecture that this is related to the number of robust features in each modality of the datasets, which requires future work to confirm.
Revised: We conjecture that this is related to the number of robust features in each modality of the datasets, which requires future work to confirm.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we partly explain the contradiction in previous conclusions on multi-modal robustness by pointing out the importance of the modality complementariness through information-theoretical analysis and carefully-designed experiments.
Revised: In this work, we partly explain the contradiction in previous conclusions on multi-modal robustness by pointing out the importance of the modality complementarity through information-theoretical analysis and carefully-designed experiments.

Original: As a reflection of modality interconnection, our proposed metric provides a basis for better understanding various multi-modal datasets/tasks and can be used beyond analyzing multi-modal robustness.
Revised: As a reflection of modality interconnection, our proposed metric provides a basis for better understanding various multi-modal datasets/tasks and can be used beyond analyzing multi-modal robustness.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide the source code and configuration for the key experiments, including instructions on generating data, training the models, and evaluating the robustness.
Revised: We provide the source code and configuration for the key experiments, including instructions on generating data, training the models, and evaluating the robustness.

Original: We thoroughly checked the code implementations and empirically verified the effectiveness of our method.
Revised: We thoroughly checked the code implementations and empirically verified the effectiveness of our method.

Original: All proofs are stated in the appendix with explanations and underlying assumptions
Revised: All proofs are stated in the appendix with explanations and underlying assumptions.