Original: R ECOGNIZING AND OVERCOMING THE GREEDY NATURE OF LEARNING IN MULTI - MODAL DEEP NEURAL NETWORKS
Revised: R ECOGNIZING AND OVERCOMING THE GREEDY NATURE OF LEARNING IN MULTI - MODAL DEEP NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks (DNNs), these models tend to rely on just one modality while under-utilizing the other modalities.
Revised: We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend to rely on just one modality while under-utilizing the other modalities.

Original: We observe empirically that such behavior hurts its overall generalization.
Revised: We observe empirically that such behavior hurts their overall generalization.

Original: We validate our hypothesis by estimating the gain on the accuracy when the model has access to an additional modality. We refer to this gain as the conditional utilization rate of the  modality.
Revised: To estimate the model’s modality-wise dependency, we compute the gain on the accuracy when the model has access to an additional modality and refer to this gain as the conditional utilization rate of the added modality.

Original: In the experiments, we consistently observe an imbalance in conditional utilization rate between modalities, across multiple tasks and architectures.
Revised: In the experiments, we consistently observe an imbalance in conditional utilization rates between modalities, across multiple tasks and architectures.

Original: Since conditional utilization rate cannot be computed efﬁciently during training, we introduce anefﬁcient proxy  based on the pace at which a DNN learns from each modality, which we refer to as conditional learning speed.
Revised: Since conditional utilization rate cannot be computed efﬁciently during training, we introduce an proxy for it based on the pace at which the model learns from each modality, which we refer to as conditional learning speed .

Original: Wethus propose a training algorithm, balanced multi-modal learning , and demonstrate that it indeed addresses the issue of greedy learning.
Revised: We propose an algorithm to balance the conditional learning speed between modalities during training and demonstrate that it indeed addresses the issue of greedy learning.

Original: The proposed algorithm is found to improve the model’s generalization on three datasets: Colored MNIST (Kim et al., 2019),
Revised: The proposed algorithm improves the model’s generalization on three datasets: Colored MNIST (Kim et al., 2019),

Original: Princeton ModelNet40 (Wu et al., 2015), and NVIDIA Dynamic Hand Gesture Dataset (Molchanov et al., 2016).
Revised: Princeton ModelNet40 (Wu et al., 2015), and NVIDIA Dynamic Hand Gesture Dataset (Molchanov et al., 2016).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In real-world problems, each instance frequently has multiple modalities associated with it.
Revised: In real-world problems, each instance frequently has multiple modalities associated with it.

Original: For example, we detect cancer in both X-ray and ultrasound images.
Revised: For example, we detect cancer in both X-ray and ultrasound images.

Original: We seek clues from images to answer questions given in text.
Revised: We seek clues from images to answer questions given in text.

Original: We are naturally interested in training deep neural networks (DNNs) end-to-end to learn from all available input modalities.
Revised: We are naturally interested in training deep neural networks (DNNs) end-to-end to learn from all available input modalities.

Original: We refer to such a training regimen as a multi-modal learning process and DNNs resulting from this process as multi-modal DNNs .
Revised: We refer to such a training regime as a multi-modal learning process and DNNs resulting from this process as multi-modal DNNs .

Original: Several recent studies have reported unsatisfactory performance of multimodal DNN’s in various tasks (Wang et al., 2020a;
Revised: Several recent studies have reported unsatisfactory performance of multi-modal DNNs in various tasks (Wang et al., 2020a;

Original: Wu et al., 2020; Gat et al., 2020; Cadene et al., 2019; Agrawal et al., 2016;
Revised: Wu et al., 2020; Gat et al., 2020; Cadene et al., 2019; Agrawal et al., 2016;

Original: Hessel & Lee, 2020; Han et al., 2021).
Revised: Hessel & Lee, 2020; Han et al., 2021).

Original: Such as in visual question answering, multi-modal DNNs were found to ignore the visual modality (presented as an image) and exploit statistical regularities shared between the textual modality (presented as a question) and the answer alone, resulting in poor generalization (Cadene et al., 2019;
Revised: For example, in Visual Question Answering, multi-modal DNNs were found to ignore the visual modality (presented as an image) and exploit statistical regularities shared between the textual modality (presented as a question) and the answer alone, resulting in poor generalization (Cadene et al., 2019;

Original: Gat et al., 2020; Agrawal et al., 2016).
Revised: Gat et al., 2020; Agrawal et al., 2016).

Original: In human action detection task, multi-modal DNNs trained on images and audio were observed to under-perform uni-modal DNNs trained on images only (Wang et al., 2020a).
Revised: Similarly, in Human Action Recognition, multi-modal DNNs trained on images and audio were observed to under-perform uni-modal DNNs trained on images only (Wang et al., 2020a).

Original: These earlier negative ﬁndings compel us to ask, what prevents multi-modal DNNs from achieving better performance?
Revised: These earlier negative ﬁndings compel us to ask, what prevents multi-modal DNNs from achieving better performance?

Original: In order to answer this question, we ﬁrst introduce a metric, conditional utilization rate .
Revised: In order to answer this question, we ﬁrst diagnose these DNNs as lacking utilization of all modalities by analyzing their conditional utilization rates.

Original: For a multi-modal DNN trained with two modalities, m 0 and m 1 , the conditional utilization rate of m 1 given m 0 , denoted by u ( m 1 | m 0 ) , measures how important it is to use m 1 , given the presence of m 0 .
Revised: For a multi-modal DNN trained with two modalities, m 0 and m 1 , the conditional utilization rate of m 1 given m 0 , denoted by u ( m 1 | m 0 ) , measures how important it is to use m 1 , given the presence of m 0 .

Revised: It is computed as the relative difference in accuracy between two derived models from the DNN, one using both modalities and the other using only one modality.

Original: We experiment on several multi-modal learning tasks and we consistently observe a signiﬁcant difference of conditional utilization rates between modalities.
Revised: In several multi-modal learning tasks, we consistently observe a signiﬁcant imbalance in conditional utilization rates between modalities.

Original: For example, we observe u RGB | depth = 0 .
Revised: For example, we observe u (RGB | depth) = 0 .

Original: 01 and u depth | RGB = 0 .
Revised: 01 and u (depth | RGB) = 0 .

Original: 63 for a model trained to identify gestures in videos using the NVIDIA Dynamic Hand Gesture Dataset (NVGesture) (Molchanov et al., 2016).
Revised: 63 for a DNN trained to identify gestures in videos using the NVIDIA Dynamic Hand Gesture Dataset (NVGesture) (Molchanov et al., 2016).

Original: It indicates that the model relies on the depth modality to make predictions and does not pay attention to the RGB modality.
Revised: It indicates that the DNN relies on the depth modality to make predictions and does not pay attention to the RGB modality.

Original: This observation leads to a conjecture that the multi-modal learning process often results in models that under-utilize some of the input modalities.
Revised: These observations lead to a conjecture that the multi-modal learning process often results in models that under-utilize some of the input modalities.

Original: As the focal point of this work, we put forward the greedy learner hypothesis.
Revised: As the focal point of this work, we put forward the greedy learner hypothesis .

Original: Under the greedy learner hypothesis, a multi-modal DNN learns to rely on one of the input modalities, based on which it could learn faster, and does not continue to learn to use the other modalities.
Revised: The greedy learner hypothesis states that a multi-modal DNN learns to rely on one of the input modalities, based on which it could learn faster, and does not continue to learn to use the other modalities.

Original: This greediness leads to  imbalance in conditional utilization rate between modalities.
Revised: This greediness leads to an imbalance in conditional utilization rates between modalities.

Original: In other words, it prevents a multi-modal DNN from learning to exploit all available modalities and often results in worse generalization.
Revised: In other words, it prevents a multi-modal DNN from learning to exploit all available modalities and often results in worse generalization.

Original: It explains the above observed phenomenon, answers the challenges in training multi-modal DNNs, and motivates us to design a better multi-modal learning algorithm.
Revised: It explains the challenge in training multi-modal DNNs and motivates us to design a better multi-modal learning algorithm.

Original: According to the greedy learner hypothesis, it is the  speed at which a multi-modal DNN learns from each modality that leads to  imbalance in conditional utilization rate.
Revised: According to the greedy learner hypothesis, it is the diverged speed at which a multi-modal DNN learns from different modalities that leads to an imbalance in conditional utilization rate.

Original: If weappropriately intervene in the learning process to adjust these speeds, we may be able to prevent the hurtful imbalance across input modalities.
Revised: If we intervene in the training process to adjust these speeds, we may be able to prevent the hurtful imbalance across input modalities.

Original: We characterize such learning speed by introducing a metric, called conditional learning speed .
Revised: We analyze the learning dynamics of model components and propose a metric named by conditional learning speed using the gradient norm and weight norm of models’ parameters. It measures the relative learning speed at which the model learns from one modality against the other modality.

Original: We empirically show that it is a reasonable proxy for conditional utilization rate.
Revised: We empirically show that it is a reasonable proxy for conditional utilization rate.

Original: We introduce a training algorithm, balanced multi-modal learning which accelerates the model to learn  from either modality alternatively according to their conditional learning speeds.
Revised: We introduce a training algorithm, balanced multi-modal learning which guides the model to learn intentionally from one of the modalities according to the conditional learning speeds.

Original: We show thatit produces models that learn to use all modalities appropriately and achieve stronger generalization on three multi-modal datasets: Colored MNIST dataset (Kim et al., 2019), ModelNet40 dataset of 3D objects (Su et al., 2015) and NVGesture Dataset (Molchanov et al., 2015).
Revised: We show that models trained with this algorithm learn to use all modalities appropriately and achieve stronger generalization on three multi-modal datasets: Colored MNIST dataset (Kim et al., 2019), ModelNet40 dataset of 3D objects (Su et al., 2015) and NVGesture Dataset (Molchanov et al., 2015).

Original:  P ROBLEM SETUP
Revised:  P ROBLEM SETUP

Original: We consider two input modalities, referred to as m 0 and m 1 , without loss of generality.
Revised: We consider two input modalities, referred to as m 0 and m 1 , without loss of generality towards tasks with more than two modalities.

Original: We denote a multi-modal dataset by D = (cid:8) ( x i , y i ) (cid:9) N i =1 , where x = ( x m 0 , x m 1 ) .
Revised: We denote a multi-modal dataset by D = (cid:8) ( x i , y i ) (cid:9) N i =1 , where x = ( x m 0 , x m 1 ) .

Original: We partition the dataset D into training, validation and test sets, denoted by D train , D val , and D test , respectively.
Revised: We partition the dataset D into training, validation and test sets, denoted by D train , D val , and D test , respectively.

Original: The goal is then to  train a multi-modal DNN on this datset. [Equation]
Revised: The goal is then to use this data set to train a multi-modal DNN such that it accurately predicts y from x .

Original: [Figure]
Revised: [Figure]

Original: Multi-modal DNNs We use a multi-modal DNN that has two uni-modal branches , each taking one modality as input.
Revised: Multi-modal DNNs We use a multi-modal DNN that has two uni-modal branches, each taking one modality as input.

Original: The two uni-modal branches are interconnected by layer-wise fusion modules.
Revised: The two uni-modal branches are interconnected by layer-wise fusion modules.

Original: These fusion modules enable information ﬂow from one branch to another.
Revised: These fusion modules enable information ﬂow from one branch to another.

Original: This network architecture employs intermediate fusion, according the categorization of multi-modal fusion strategy in the deep learning literature (Ngiam et al., 2011; Atrey et al., 2010; Baltrušaitis et al., 2018).
Revised: According to categorization of multi-modal fusion strategies in the deep learning literature, this is intermediate fusion (Ngiam et al., 2011; Atrey et al., 2010; Baltrušaitis et al., 2018).

Original: It has demonstrated competitive performance against multi-modal DNNs with late fusion in many tasks (Perez et al., 2018; 
Revised: It has demonstrated competitive performance against multi-modal DNNs with late fusion in many tasks (Perez et al., 2018; Joze et al., 2020;

Original: Joze et al., 2020; Anderson et al., 2018; Wang et al., 2020b).
Revised: Anderson et al., 2018; Wang et al., 2020b).

Original: We implement the fusion module as a multi-modal transfer module (MMTM) (Joze et al., 2020).
Revised: We implement every fusion module by a multi-modal transfer module (MMTM) (Joze et al., 2020).

Original: The ﬁrst step in MMTM is to squeeze feature maps from each uni-modal branch to vector representations via global average pooling  over spatial dimensions.
Revised: Each MMTM connects two layers from the two uni-modal branches. There is ﬁrst the global average pooling applied over spatial dimensions to transform feature maps into a vector.

Original: Next we concatenate these representations and applya linear transformation to obtain cross-modal context representation.
Revised: We concatenate the two vectors and apply linear transformation. We refer to its output as context representation.

Original: We predict channel-wise weights for each modality based this context representation through two independent fully-connected layers. Finally, these weights are used tore-calibrate the channel-wise features per modality.
Revised: Next, for each uni-modal branch, we implement a fully connected layer on the context representation and get a vector with a dimension of the number of feature maps. Feature maps are re-scaled by this vector before passing to the next layer of the uni-modal branch.

Original: We train a multi-modal DNN f on D train .
Revised: We train a multi-modal DNN f on D train .

Original: Let ˆ y be the prediction of f for an input x : ˆ y = f
Revised: Let ˆ y be the prediction of f for an input x : ˆ y = f

Original: ( x m 0 , x m 1 ) .
Revised: ( x m 0 , x m 1 ) .

Original: As shown in Figure 1, ˆ y = 12 (ˆ y 

Revised: As shown in Figure 1, ˆ y = 12 (ˆ y

Original: + ˆ y 1 ) , where ˆ y 0 and ˆ y 1 are the outputs of the two uni-modal branches.
Revised: + ˆ y 1 ) , where ˆ y 0 and ˆ y 1 are the outputs of the two uni-modal branches.

Original: During training, the parameters of the multi-modal DNN are updated by stochastic gradient descent (SGD) to minimize the loss: L = CE( y, ˆ y 0 )
Revised: During training, the parameters of the multi-modal DNN are updated by stochastic gradient descent (SGD) to minimize the loss: L = CE( y, ˆ y 0 )

Original: + CE( y, ˆ y 1 ) , where CE stands for cross-entropy.
Revised: + CE( y, ˆ y 1 ) , where CE stands for cross-entropy.

Original: We refer to each of the cross-entropy losses as a modality-speciﬁc loss.
Revised: We refer to each of the cross-entropy losses as a modality-speciﬁc loss.

Original: We train the model until  the highest accuracy on D val is reached.
Revised: We train the model until ˆ y = y for all samples in D train and take the checkpoint of it when ˆ y reaches the highest accuracy on D val .

Original: During training, the uni-modal branch largely focuses on the associated modality.
Revised: During training, each uni-modal branch largely focuses on its associate input modality.

Original: The fusion modules generatecross-modal context information from the uni-modal branches and pass it back to them.
Revised: The fusion modules generate context representation using all modalities and feed such information to the unimodal branches.

Original: Both ˆ y 0 and ˆ y 1 depend on information from both modalities.
Revised: Both ˆ y 0 and ˆ y 1 depend on information from both modalities.

Original: We end up with two functions, f 0 and f 1 , corresponding to the two uni-modal branches:
Revised: We end up with two functions, f 0 and f 1 , corresponding to the two uni-modal branches:

Original: [Equation]
Revised: [Equation]

Original: By shutting down information sharing between branches, we derive f (cid:48) 0 and f (cid:48) 1 , each taking only one of the input modalities as input.
Revised: To evaluate the multi-modal DNN’s ability in using a single modality, we disable information sharing between branches ad derive f (cid:48) 0 and f (cid:48) 1 , each taking only one of the input modalities as input.

Original: We describe how we derive them in detail in Appendix A.1.
Revised: We describe how we derive them in detail in Appendix A.1.

Original: We use ˆ y (cid:48) 

Revised: We use ˆ y (cid:48)

Original: and ˆ y (cid:48) 1 to refer to their outputs:
Revised: and ˆ y (cid:48) 1 to refer to their outputs:

Original: [Equation]
Revised: [Equation]

Original: In other words, we derive four models from f . We can now deﬁne conditional utilization rate using these four derived models.
Revised: In summary, we derive four models from f and we deﬁne the ability of each model as its accuracy

Original: We ﬁrst deﬁne the ability of each model as its accuracy A on D test .A ( f ) is the accuracy reached by the original multi-modal DNN f and is the multi-modal DNN’s performance.
Revised: A on D test .

Original: Based on the symmetry between two modalities in our multi-modal DNN, we group the four accuracies into two pairs:  A ( f 0 ) and A ( f (cid:48) 0 ) , and  A ( f 1 ) and A ( f (cid:48) 1 ) .
Revised: We group the four accuracies into two pairs: ( A ( f 0 ) , A ( f (cid:48) 0 )) , and ( A ( f 1 ) , A ( f (cid:48) 1 )) and deﬁne the conditional utilization rates as bellow:

Original: The conditional utilization rate is then the relative change of the accuracy between the two within each pair:

Original: Deﬁnition 2.1 (Conditional utilization rate) For a multi-modal DNN, f , taking two modalities mand m 1 as inputs, its conditional utilization rates for m 1 and m  are deﬁned 
Revised: Deﬁnition 2.1 (Conditional utilization rate) For a multi-modal DNN, f , taking two modalities mand m 1 as inputs, its conditional utilization rates for m 1 and m 0 are deﬁned as u ( m 1 | m 0 )

Original: as
Revised: =

Original: u ( m 1 | m 0 ) =

Original: A ( f 0 ) − A ( f (cid:48) 
Revised: A ( f 0 ) − A ( f (cid:48) 0 )

Original: ) A ( f 0 ) and u ( m 0 | m 1 ) =
Revised: A ( f 0 ) and u ( m 0 | m 1 ) =

Original: A ( f 1 ) − A ( f (cid:48) 1 ) A ( f 1 ) , respectively.
Revised: A ( f 1 ) − A ( f (cid:48) 1 ) A ( f 1 ) , respectively.

Revised: The conditional utilization rate is the relative change in accuracy between the two models within each pair.

Original: The conditional utilization rate of m 0 , u ( m 0 | m 1 ) , measures how important it is for f to use mtogether with m 1 .
Revised: The conditional utilization rate of m 0 , u ( m 0 | m 1 ) , measures how important it is for f to use mtogether with m 1 .

Original: In other words, it estimates how much m 0 contribute to ˆ y 1  .
Revised: In other words, it estimates the marginal contribution that m 0 has in increasing the accuracy of ˆ y 1 depending on m 1 .

Original: A multi-modal learning process Given the model architecture and the dataset, described above, a multi-modal learning process includes the following steps: (1) sample a value of learning rate randomly from a given range, (2) initialize the DNN’s parameters randomly, (3) update them to ﬁt
Revised: A multi-modal learning process Given a DNN’s architecture and a dataset, every time we run a multi-modal learning process, we perform the following steps: (1) sample a value of learning rate randomly from a given range, (2) initialize the DNN’s parameters randomly, (3) update them to ﬁt

Original: D train with SGD, and (4) select the model checkpoint with the highest accuracy on D val .
Revised: D train with SGD, and (4) select the model checkpoint with the highest accuracy on D val .

Original:  T HE GREEDY LEARNER HYPOTHESIS
Revised:  T HE GREEDY LEARNER HYPOTHESIS

Original: In this section, we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs.
Revised: In this section, we introduce the greedy learner hypothesis to explain challenges observed in training multi-modal DNNs.

Original: Before describing our hypothesis, we start by discussing some assumptions on the multi-modal data and preliminary observations made in theliterature on multi-modal learning.
Revised: Before describing our hypothesis, we discuss some assumptions on the multimodal data and preliminary observations made in the multi-modal learning literature.

Original: 3. A SSUMPTIONS IN MULTI - MODAL LEARNING
Revised: 3. A SSUMPTIONS IN MULTI - MODAL LEARNING

Original: First, in multi-modal learning literature (Sridharan & Kakade, 2008), both modalities are assumed to be predictive of the target, i.e., I ( Y , X m 0 ) > 

Revised: First, in multi-modal learning literature (Blum & Mitchell, 1998; Sridharan & Kakade, 2008), both modalities are assumed to be predictive of the target, i.e., I ( Y , X m 0 ) >

Original: and I ( Y , X m 1 ) > 0 , where I denotes the mutual information.
Revised: and I ( Y , X m 1 ) > 0 , where I denotes the mutual information.

Original: In order to minimize one of the modality-speciﬁc losses, e.g. CE( y, ˆ y 0 ) , one can either update the parameters of the fusion layers that pass information from m 1 to ˆ y 0 , or the parameters of the uni-modal branch taking m 0 as input, or both.
Revised: In order to minimize one of the modality-speciﬁc losses, e.g. CE( y, ˆ y 0 ) , one can either update the parameters of the fusion layers that pass information from mto ˆ y 0 , or the parameters of the uni-modal branch taking m 0 as input, or both.

Original: Second,the modalities are predictive of the target at different degrees.
Revised: Second, modalities are predictive of the target at different degrees.

Original: For example, we have observed that when training DNNs on each modality separately, they usually do not reach the same performance (Wang et al., 2020b; Joze et al., 2020).
Revised: For example, it has been observed that when training DNNs on each modality separately, they usually do not reach the same performance (Wang et al., 2020b; Joze et al., 2020).

Original: In addition, multi-modal DNNs exhibit varying performance when being trained on different combinations of modalities present for the task (Weng et al., 2021; Pérez-Rúa et al., 2019; Liu et al., 2018).
Revised: In addition, multi-modal DNNs exhibit varying performance when being trained on different subsets of modalities present for the task (Weng et al., 2021; Pérez-Rúa et al., 2019; Liu et al., 2018).

Original: Third,DNNs trained with only one of the modalities learn at different speeds.
Revised: Third, modalities are learned by the model at different speeds.

Original: Multi-modal DNNs have been observed to learn from different modalities at different speeds as well (Wang et al., 2020a; Wu et al., 2020).
Revised: This has been observed in both uni-modal DNNs and Multi-modal DNNs (Wang et al., 2020a; Wu et al., 2020).

Original: 3. M ULTI - MODAL LEARNING PROCESS IS GREEDY
Revised: 3. M ULTI - MODAL LEARNING PROCESS IS GREEDY

Original: Before formalizing our  hypothesis and verifying it experimentally, we ﬁrst demonstrate that we can tell whether the model utilizes all modalities by analyzing its conditional utilization rates.
Revised: Before formalizing our greedy learner hypothesis that explains the behaviour of multi-modal DNNs, we ﬁrst demonstrate that we can tell whether the model utilizes all modalities by analyzing its conditional utilization rates.

Original: Given a multi-modal DNN, f , with two input modalities, m 0 and m 1 , we compute the difference between conditional utilization rates, u ( m 1 | m 0 ) and u ( m 0 | m 1 ) as
Revised: Given a multi-modal DNN, f , with two input modalities, m 0 and m 1 , we compute the difference between conditional utilization rates, u ( m 1 | m 0 ) and u ( m 0 | m 1 ) as

Original: [Equation]
Revised: [Equation]

Original: If we observe high | d util ( f ) | , we say that f exhibits imbalance in utilization between modalities .
Revised: If we observe high | d util ( f ) | , we say that f exhibits imbalance in utilization between modalities.

Original: The difference between conditional utilization rates is bounded between − 1 and 1 .
Revised: The difference between conditional utilization rates is bounded between − 1 and 1 .

Original: When it is close to − 1 or 1 , the model beneﬁts only from one of the modalities given the other but not vice versa.
Revised: When it is close to − 1 or 1 , the model beneﬁts only from one of the modalities given the other but not vice versa.

Original: This implies that the model’s ability to predict ˆ y 0 and ˆ y 1 comes only from one of the modalities.
Revised: This implies that the model’s ability to predict ˆ y 0 and ˆ y 1 comes only from one of the modalities.

Original: We now introduce the greedy learner hypothesis :
Revised: We now introduce the greedy learner hypothesis :

Original: We call a multi-modal learning process greedy , when it trains models to rely on only one of the available modalities.
Revised: We call a multi-modal learning process greedy , when it trains models to rely on only one of the available modalities.

Original: A greedy multi-modal learning process cannot avoid producing models that exhibit a high degree of imbalance in utilization between modalities.
Revised: A greedy multi-modal learning process cannot avoid producing models that exhibit a high degree of imbalance in utilization between modalities.

Original: We hypothesize that a multi-modal learning process, in which a multi-modal DNN is trained to minimize the sum of modality-speciﬁc losses, is greedy.
Revised: We hypothesize that a multi-modal learning process, in which a multi-modal DNN is trained to minimize the sum of modality-speciﬁc losses, is greedy.

Original: Let E [ (cid:98) d
Revised: Let E [ (cid:98) d

Original: util ] denote the expectation of (cid:98) d util over the empirical distribution of models resulting from a multi-modal learning process (as deﬁned in §2).
Revised: util ] denote the expectation of (cid:98) d util over the empirical distribution of models resulting from a multi-modal learning process (as deﬁned in §2).

Original: The absolute value of E [ (cid:98) d util ] is associated with the greediness of the process.
Revised: The absolute value of E [ (cid:98) d util ] is associated with the greediness of the process.

Original: The higher the | E [ (cid:98) d util ] | is, the greedier the learning process is.
Revised: The higher the | E [ (cid:98) d util ] | is, the greedier the learning process is.

Original: We propose the following conjectures consistent with the greedy learner hypothesis:
Revised: We propose the following conjectures consistent with the greedy learner hypothesis:

Original: There exists an (cid:15) > 0 , s.t.
Revised: There exists an (cid:15) > 0 , s.t.

Original: | E [ (cid:98) d util ] | > (cid:15) , as longas the multi-modal dataset has the last two properties from §3.1.
Revised: | E [ (cid:98) d util ] | > (cid:15) , as long the multi-modal dataset has the last two properties from §3.1.

Original: Otherwise, (cid:98) d util is distributed symmetrically around zero and E [ (cid:98) d util ] = 0 .
Revised: Otherwise, (cid:98) d util is distributed symmetrically around zero and E [ (cid:98) d util ] = 0 .

Original: The stronger the regularization we apply  in training, the higher the | E [d util ] | is.
Revised: The stronger the regularization we apply to the DNNs’ parameters in training, the higher the | E [ (cid:98) d util ] | is.

Original: We expect the ﬁrst scenario in the ﬁrst conjecture to be true for all real-world multi-modallearning tasks.
Revised: We expect the ﬁrst scenario in the ﬁrst conjecture to be true for all real-world multi-modal tasks.

Original: We construct datasets by duplicating one of the input modalities to test the second scenario in the ﬁrst conjecture.
Revised: To test the second scenario in the ﬁrst conjecture, we construct datasets of two identical modalities.

Revised: We suspect that the implicit regularization of DNNs is a factor impacting the greediness of learning in multi-modal DNNs.

Original: Thesecond conjecture is motivated by the discussion on the implicit regularization of DNNs (VallePerez et al., 2019; Zhang et al., 2017; 2021; Smith et al., 2021). It refers to a consensus that DNNs have the tendency to lean towards simplicity during training and it explains their strong generalization.
Revised: The implicit regularization of DNNs commonly refers to a consensus that DNNs have the tendency to lean towards simplicity during training and it explains their strong generalization (Valle-Perez et al., 2019; Zhang et al., 2017; 2021; Smith et al., 2021).

Original: We suspect that the regularization of DNNs is a strong factor impacting the greediness of the multi-modal learning process.
Revised: Since it is difﬁcult to precisely measure and control such implicit regularization, alternatively, we construct the second conjecture using explicit regularization applied to the DNNs’ parameters.

Original: In order to validate theconjectures and the hypothesis above  empirically, we conduct experiments on several multi-modal datasets using different network architectures (cf. §5.1).
Revised: In order to validate the above hypothesis and conjectures empirically, we conduct experiments on several multi-modal datasets using different network architectures (cf. §5.1).

Original: We present ourempirical observations in §5.2 and §5.3.
Revised: We present our observations in §5.2 and §5.3.

Revised: We show that the multi-modal learning process cannot avoid producing models that exhibit a high degree of imbalance in utilization between modalities.

Revised: Further, in the next section, we propose a metric for measuring the speeds at which the model learns from different modalities and we show the imbalance in such speeds is correlated with the imbalance in utilization.

Original:  M AKING MULTI - MODAL LEARNING LESS GREEDY
Revised:  M AKING MULTI - MODAL LEARNING LESS GREEDY

Original: According tothe greedy learner hypothesis, we can make multi-modal learning less greedy by controlling the speed at which a multi-modal DNN learns to rely on each modality.
Revised: We aim to make multi-modal learning less greedy by controlling the speed at which a multi-modal DNN learns to rely on each modality.

Original: To this end, we derive conditional learning speed to measure the speed at which the DNN learns from one modality.
Revised: To this end, we deﬁne conditional learning speed to measure the speed at which the DNN learns from one modality.

Original: It serves as an efﬁcient proxy to the conditional utilization rate of the corresponding modality.
Revised: It serves as an efﬁcient proxy to the conditional utilization rate of the corresponding modality, as shown empirically in §5.2 and §5.3.

Original: We then propose the balanced multi-modal learning algorithm, which controls model’s conditional learning speed between modalities in order to prevent it from being greedy.
Revised: We then propose the balanced multi-modal learning algorithm, which controls the difference in conditional learning speed between modalities that the model exhibits during training.

Original: 4. C ONDITIONAL LEARNING SPEED
Revised: 4. C ONDITIONAL LEARNING SPEED

Original: As demonstrated in §3.2, the imbalance in conditional utilization rates is a sign of the model exploiting the connection between the target and only one of the input modalities, ignoring crossmodal information.
Revised: As demonstrated in §3.2, the imbalance in conditional utilization rates is a sign of the model exploiting the connection between the target and only one of the input modalities, ignoring crossmodal information.

Original: Conditional utilization rate was however can be measured once training was done, making it difﬁcult to use it in real-time during training.
Revised: However, conditional utilization rates are measured after training is done, making it expensive to use them in real-time during training.

Original: We instead derive a proxy metric, called conditional learning speed , that captures relative learning speed between modalities during training.
Revised: We instead derive a proxy metric, called conditional learning speed , that captures relative learning speed between modalities during training.

Original: Let us revisit the architecture of the multi-modal DNN.
Revised: Let us revisit the architecture of the multi-modal DNN.

Original: We denote the parameters of each uni-modal branch taking m 0 and m 1 as inputs, by θ 0 and θ 1 .
Revised: We denote the parameters of each uni-modal branch taking m 0 and m 1 as inputs, by θ 0 and θ 1 .

Original: Layers of the fusion modules marked with green and green/blue in Figure 1, are part of the function mapping x to ˆ y 0 .
Revised: Layers of the fusion modules marked with green and green/blue in Figure 1, are part of the function mapping x to ˆ y 0 .

Original: Let θ (cid:48) 0 refer to their parameters.
Revised: Let θ (cid:48) 0 refer to their parameters.

Original: Accordingly, we have the layers marked with green and green/red in Figure 1that are part of the function mapping x to ˆ y 1 .
Revised: Analogously, the layers marked with green and green/red in Figure 1 are part of the function mapping x to ˆ y 1 .

Original: We denote their parameters by θ (cid:48) 1 .
Revised: We denote their parameters by θ (cid:48) 1 .

Original: In summary, we divide the model’s parameters into two pairs: ( θ 0 , θ (cid:48) 0 ) and ( θ 1 , θ (cid:48) 1 ) . 
Revised: In this way, we divide the model’s parameters into two pairs: ( θ 0 , θ (cid:48) 0 ) and ( θ 1 , θ (cid:48) 1 ) . For parameters θ , let θ

Revised: i − 1) and θ ( i ) denote its value before and after the gradient descent step i , and

Original:  We deﬁne the model’s conditional learning speed based on these two pairs of parameters:
Revised: G = ∂L∂ θ | θ ( i − 1) . We deﬁne the model’s conditional learning speed as:

Original: Deﬁnition 4.1 (Conditional learning speed)
Revised: Deﬁnition 4.1 (Conditional learning speed)

Original: Given a multi-modal DNN, f , with two input modalities, m 0 and m 1 , the conditional learning speeds of these modalities after t training steps, are
Revised: Given a multi-modal DNN, f , with two input modalities, m 0 and m 1 , the conditional learning speeds of these modalities after t training steps, are

Original: [Equation]
Revised: [Equation]

Original: where µ ( θ ; · ) =|| G ||
Revised: where µ ( θ ; i ) =

Original: ||
Revised: ||

Original: θ
Revised: G ||

Original:  || 22 quantiﬁes the impact on the model f that comes from the change of θ at one update step and G = ∂L/∂ θ .
Revised: 22 / || θ ( i ) || 22 quantiﬁes the change of f that comes from updating θ at the i th step, which can be also interpreted as the effective update on θ .

Original: This deﬁnition of µ ( θ ; · ) is inspired by discussion on the effective update of parameters in the literature (Van Laarhoven, 2017;
Revised: This deﬁnition of µ ( θ ; i ) is inspired by discussion on the effective update of parameters in the literature (Van Laarhoven, 2017;

Original: Zhang et al., 2019; Brock et al., 2021).
Revised: Zhang et al., 2019; Brock et al., 2021; Hoffer et al., 2018).

Original: Previous studies found that when normalization techniques are applied, such as batch normalization (Ioffe & Szegedy, 2015),  the update of the direction of  θ ,  i.e., θ / ||
Revised: When normalization techniques, such as batch normalization (Ioffe & Szegedy, 2015), are applied to the DNNs, the key property of the weight vector, θ , is its direction, i.e., θ / ||

Original: θ
Revised: θ

Original: || 22 , reﬂects how much the update on θ changes the model f in order to ﬁt the batch of samples.
Revised: || 22 . Thus, we measure the update on θ using the norm of its gradient normalized by its norm.

Original: The conditional learning speed, s ( m 1 | m 0 ; t ) (and analogously s ( m 0 | m 1 ; t ) ), is the log-ratio between the learning speed of θ (cid:48) 0 and that of θ 0 .
Revised: The conditional learning speed, s ( m 1 | m 0 ; t ) (and analogously s ( m 0 | m 1 ; t ) ), is the log-ratio between the learning speed of θ (cid:48) 0 and that of θ 0 .

Original: Because θ (cid:48) 0 carries information from m 1 to ˆ y 0  , s ( m 1 | m 0 ; t ) reﬂects how fast the model learns from m 1 relative to m 0 learned through the corresponding unimodal branch, after the ﬁrst t steps.
Revised: Because θ (cid:48) 0 carries information from m 1 to ˆ y 0 and information from m 0 to ˆ y 0 is carried by θ 0 , s ( m 1 | m 0 ; t ) reﬂects how fast the model learns from m 1 relative to m 0 , after the ﬁrst t steps.

Original: We further compute the difference between s ( m 1 | m 0 ; t ) and s ( m 0 | m 1 ; t ) as:
Revised: We further compute the difference between s ( m 1 | m 0 ; t ) and s ( m 0 | m 1 ; t ) as:

Original: [Equation]
Revised: [Equation]

Original: analogous to d util  .
Revised: analogous to d util ( f ) .

Revised: For each model, we report d speed ( f ; T ) as d speed ( f ) where the model takes

Revised: T steps until reaching the highest accuracy on D val .

Original: If the greedy learner hypothesis is true, the conditional learning speed would serve as a proxy for the conditional utilization rate. d speed ( f ; t ) would predict d util ( f ) at the end of training.
Revised: We anticipate that the conditional learning speed would serve as a proxy for the conditional utilization rate and we say d speed ( f ; t ) would predict d util ( f ) at the end of training.

Original: In §5.2 and §5.3, we show the distributions of (cid:98)
Revised: In §5.2 and §5.3, we show the distributions of (cid:98)

Original: d speed and (cid:98) d util side-by-side to validate this.
Revised: d speed and (cid:98) d util side-by-side to validate this.

Original: For each model, we report d speed ( f ; T ) as (cid:98) d speed ( f ) where the model takes T steps until reaching the highest accuracy on D val .

Original: 4. B ALANCED MULTI - MODAL LEARNING
Revised: 4. B ALANCED MULTI - MODAL LEARNING

Original: Because d speed ( f, t ) is predictive of the imbalanced utilization between modalities, we can take advantage of d speed ( f, t ) to balance conditional utilization on-the-ﬂy.
Revised: Because d speed ( f, t ) is predictive of the imbalanced utilization between modalities, we can take advantage of d speed ( f, t ) to balance conditional utilization on-the-ﬂy.

Original: In addition to training the network normally with both modalities, we accelerate the model to learn from either modality alternatively to balance the conditional learning speeds of them.
Revised: In addition to training the network normally with both modalities, we accelerate the model to learn from either modality alternately to balance the conditional learning speeds of them.

Original: See Algorithm 1 for an overall description of the algorithm.
Revised: See Algorithm 1 for an overall description of the algorithm.

Original: We refer to the training steps at which we perform forward and backward passes normally as regular steps .
Revised: We refer to the training steps at which we perform forward and backward passes normally as regular steps .

Original: We introduce re-balancing steps at which weforce the model to updateonly one of the unimodal branches  in order to accelerate learning from the associated input modality.
Revised: We introduce re-balancing steps at which we update one of the uni-modal branches intentionally in order to accelerate the model to learn from the corresponding modality.

Original: See Appendix A. for the full explanation of the re-balancing step.
Revised: See Appendix A.2 for the full explanation of the re-balancing step.

Original: To warm-up the model, we perform  regular steps in the ﬁrst  epoch.
Revised: To warm-up the model, we perform only regular steps in the ﬁrst training epoch.

Original: We switch from regular steps to re-balancing steps if | d speed ( t ) | > α , where α is  the imbalance  parameter .
Revised: Then we switch from regular steps to re-balancing steps if | d speed ( t ) | > α , where α is a hyperparameter, referred to as the imbalance tolerance parameter .

Original: The training takes Q re-balancing steps before returning to regular mode.
Revised: The training takes Q re-balancing steps before returning to regular mode.

Original: We refer to  Q as the re-balancing window size .
Revised: We refer to the hyperparameter Q as the re-balancing window size .

Revised: Algorithm 1: Balanced multi-modal learning

Original: Input: Q , re-balancing window size; α , imbalance  parameter
Revised: Input: Q , re-balancing window size; α , imbalance tolerance parameter

Original: T , # of training steps;
Revised: T , # of training steps;

Original: T 1 , # of steps in the 1 th training epoch “‘Let M denote the accumulated effective update  in regular steps”’
Revised: T 1 , # of steps in the 1 th training epoch Let M denote the accumulated effective update ( µ ) in regular steps

Original: M θ 0 , M θ (cid:48) 

Revised: M θ 0 , M θ (cid:48)

Original: = 0 , 0 ;
Revised: = 0 , 0 ;

Original: M θ 1 , M θ (cid:48) 1 = 0 , 0 ; for t ← 1 to T 1 do
Revised: M θ 1 , M θ (cid:48) 1 = 0 , 0 ; for t ← 1 to T 1 do

Original: Take a regular step;
Revised: Take a regular step;

Original: Update M θ 0 , M θ (cid:48) 0 , M θ 1 , M θ (cid:48)end regular_mode =
Revised: Update M θ 0 , M θ (cid:48) 0 , M θ 1 , M θ (cid:48)end regular_mode =

Original: True for t ← T 1 to T do if regular_mode then
Revised: True for t ← T 1 to T do if regular_mode then

Original: Take a regular step;
Revised: Take a regular step;

Original: Update M θ 0 , M θ (cid:48) 0 , M θ 1 , M θ (cid:48) 1 ;
Revised: Update M θ 0 , M θ (cid:48) 0 , M θ 1 , M θ (cid:48) 1 ;

Original: Compute d speed ; if | d speed | > α then q = 0 ; regular_mode =
Revised: Compute d speed ; if | d speed | > α then q = 0 ; regular_mode =

Original: False end else q ← q + 1 ; if q = Q then regular_mode =
Revised: False end else q ← q + 1 ; if q = Q then regular_mode =

Original: True;
Revised: True;

Original: Take a re-balancing step to accelerate learning from m [-
 If-] d speed > 0 Else m 1 ; end end
Revised: Take a re-balancing step to accelerate learning from m [-
 If-]0 if d speed > 0 else m 1 ; end end

Original:  E XPERIMENTS AND RESULTS 5. D ATASETS AND TASKS 
Revised:  E XPERIMENTS AND RESULTS 5. D ATASETS , TASKS AND BASELINES

Original: Colored-and-gray-MNIST (Kim et al., 2019) is a synthetic dataset based on MNIST (LeCun et al., 1998).
Revised: Colored-and-gray-MNIST (Kim et al., 2019) is a synthetic dataset based on MNIST (LeCun et al., 1998).

Original: In the training set of 60,000 examples, each example has two images, a gray-scale image and a monochromatic image, with color strongly correlated with its digit label.
Revised: In the training set of 60,000 examples, each example has two images, a gray-scale image and a monochromatic image, with color strongly correlated with its digit label.

Original: For the validation set of 10,000 examples, each example also has a gray-scale  and a corresponding monochromatic image however with a low correlation between the color and its label.
Revised: For the validation set of 10,000 examples, each example also has a gray-scale image and a corresponding monochromatic image however with a low correlation between the color and its label.

Original: We consider the monochromatic image as the ﬁrst modality m 0 and the gray-scale one as the second modality m 1 .
Revised: We consider the monochromatic image as the ﬁrst modality m 0 and the gray-scale one as the second modality m 1 .

Original: We use a neural network with four convolutional layers as the uni-modal branch and employ three MMTMs(Joze et al., 2020) to connect them.
Revised: We use a neural network with four convolutional layers as the uni-modal branch and employ three MMTMs to connect them.

Original: The corresponding uni-modal DNNs trained on the monochromatic images and 
Revised: The corresponding uni-modal DNNs trained on the monochromatic images and the gray-scale images achieve the validation accuracies of 43% and 98%, respectively.

Original: We use this synthetic dataset mainly to show that the proposed greedy learner hypothesis happens in practice.
Revised: We use this synthetic dataset mainly to demonstrate the proposed greedy learner hypothesis.

Original: ModelNet40 is one of the Princeton ModelNet datasets (Wu et al., 2015) with 3D objects ofcategories (9,483 training samples and 2,468 test samples).
Revised: ModelNet40 is one of the Princeton ModelNet datasets (Wu et al., 2015) with 3D objects ofcategories (9,483 training samples and 2,468 test samples).

Original: We use the task in which the goal is to classify a 3D object based on the 2D views of its front and back (Su et al., 2015).
Revised: We use the task that is to classify a 3D object based on the 2D views of its front and back (Su et al., 2015).

Original: Each example is a 3D object rendered as 2D images of 224 × 224 pixels.
Revised: Each example is a 3D object rendered as 2D images of 224 × 224 pixels.

Original: For the uni-modal branches, we use ResNet18 (He et al., 2016) and apply MMTMs in the three ﬁnal residual blocks.
Revised: For the uni-modal branches, we use ResNet18 (He et al., 2016) and apply MMTMs in the three ﬁnal residual blocks.

Original: The uni-modal DNNs achieve 91.6% and 93.1% accuracy, when learning from the front view ( m 0 ) and the rear view ( m 1 ), respectively.
Revised: The uni-modal DNNs achieve 91.6% and 93.1% accuracy, when learning from the front view ( m 0 ) and the rear view ( m 1 ), respectively.

Original: NVIDIA Dynamic Hand Gesture Dataset, or NVGesturein short (Molchanov et al., 2015), consists of 1,532 video clips (1,050 training and 482 test ones) of hand gestures in 25 classes.
Revised: NVIDIA Dynamic Hand Gesture Dataset (or NVGesture (Molchanov et al., 2015)), consists of 1, video clips (1,050 training and 482 test ones) of hand gestures in 25 classes.

Original: We sample 20% training samples as the validation set and use depth and RGB as  two modalities.
Revised: We sample 20% training examples as the validation set and use depth and RGB as the two modalities.

Original: We adopt the conﬁguration by Joze et al.
Revised: We adopt the data preparation steps used in Joze et al.

Original: (2020) for data preparation and use  I3D Carreira & Zisserman (2017) as uni-modal branches and MMTMs as fusion modules in the six ﬁnal inception modules.
Revised: and use the I3D architecture (Carreira & Zisserman, 2017) as uni-modal branches and MMTMs as fusion modules in the six ﬁnal inception modules.

Original: We provide examples of each dataset and details on data preprocessing in Appendix A.
Revised: We provide examples of each dataset and details on data preprocessing in § A.4 in Appendix.

Original:  V ALIDATING THE GREEDY LEARNER HYPOTHESIS
Revised: 5. V ALIDATING THE GREEDY LEARNER HYPOTHESIS

Original: In this section, we test the conventional multi-modal learning process on seven tasks with different input and output pairs to demonstrate the ﬁrst conjecture in §3.2 experimentally andvalidate the proposedgreedy learner hypothesis.
Revised: In this section, we run the conventional multi-modal learning process on seven tasks with different input and output pairs to validate the ﬁrst conjecture from §3.2 experimentally and the proposed

Original: Study design For each task introduced in §5.1,  we construct a dataset where we duplicate one of the modalities as the two  input modalities, in addition to the original dataset.
Revised: Study design For each task introduced in §5.1, in addition to the original dataset, we construct a dataset with two identical input modalities by copying one of the modalities.

Original: For example,  we predict the digit class using two identical gray-scale images in the case of MNIST.
Revised: For example, when using the colored-and-gray-MNIST dataset, we predict the digit class using two identical gray-scale images.

Original: We train a multi-modal DNN on each datasetof each task as explained below:
Revised: We train a multi-modal DNN on these dataset as explained below for each task:

Original: • Colored-and-gray-MNIST: we train multi-modal DNNs using SGD with the momentum coefﬁcient of 0.9 and a batch size of 128.
Revised: • Colored-and-gray-MNIST: we train multi-modal DNNs using SGD with the momentum coefﬁcient of 0.9 and a batch size of 128.

Original: We sampleuniformly 20 learning rates at random from the interval [10 − 5 , 1] on a logarithmic scale.
Revised: We sample 20 learning rates at random from the interval [10 − 5 , 1] on a logarithmic scale.

Original: We train each model four times while varying random seeds.
Revised: We train the model four times using each of the learning rate and random initialization of the parameters.

Original: In total, we train 80 models.
Revised: In total, we train 80 models.

Original: ModelNet40: we use SGD without momentum and use minibatches of size 8.
Revised: ModelNet40: we use SGD without momentum and use minibatches of eight examples.

Original: Werandomly select nine learning rates from 10 − 3 to 1 .
Revised: We select nine learning rates from 10 − 3 to 1 and train model using each learning rate for three times.

Original: We train each model three times and end up with 27 models.
Revised: This ends up with 27 models.

Original: • NVGesture: we use a batch size of four, SGD with momentum of 0.9, and uniformly samplelearning ratesat random from the interval [10 − 4 , 10 − 1 .
Revised: • NVGesture: we use a batch size of four, SGD with momentum of 0.9, and uniformly samplelearning rates from the interval [10 − 4 , 10 − 1 .

Original: 5 ] on a logarithmic scale.
Revised: 5 ] on a logarithmic scale.

Original: We train each model three times, resulting in 60 models in total.
Revised: We train the model three times using each learning rate, resulting in 60 models in total.

Original: [Figure]
Revised: [Figure]

Original: lines.

Original: Many models have large (cid:98) d util .

Original: We see (cid:98) d util is distributed asymmetrically around zero when using two different input modalities.

Original: When using two identical input modalities, it is distributed symmetrically around zero.

Original: (cid:98)

Original: d speed exhibits similar distributions as (cid:98) d util .

Original: Results We present the results for the experiments on ModelNet40 and NVGesture in Figure 2 and on Colored-and-gray-MNIST inthe Appendix Figure 2.
Revised: Results We present the results for the experiments on ModelNet40 and NVGesture in Figure 2 and on Colored-and-gray-MNIST in Figure 2 in Appendix.

Original: First, many models have high | d util | , especially when trained to use two distinct modalities.
Revised: First, many models have high | d util | .

Original: This conﬁrms that thestandard multi-modal learning process encourages the model to rely on one modality and ignore the other one, which is consistent with our hypothesis.
Revised: This conﬁrms that the multi-modal learning process encourages the model to rely on one modality and ignore the other one, which is consistent with our hypothesis.

Original: We make this observation across all tasks, conﬁrming that the conventional multi-modal learning process is greedy regardless of network architectures and tasks.
Revised: We make this observation across all tasks, conﬁrming that the conventional multi-modal learning process is greedy regardless of network architectures and tasks.

Original: Second, (cid:98) d util is distributed symmetrically centered at zero, and| E [ (cid:98) d util ]| is approximately 0 . 0 , for all the experiments using two identical input modalities.
Revised: Second, (cid:98) d util is distributed symmetrically around zero, and E [ (cid:98) d util ] is approximately 0 . 0 , for all the experiments using two identical input modalities.

Original: On the other hand, if we use two distinct modalities, (cid:98) d util is distributed asymmetrically, and we observepositive | E [ (cid:98) d util ] |’s of approximately 0.3, 0.1 and 0.4 for colored-and-gray-MNIST, ModelNet40 and NVGesture, respectively.
Revised: On the other hand, if we use two distinct modalities, (cid:98) d util is distributed asymmetrically, and we observe | E [ (cid:98) d util ] | of approximately 0.3, 0. and 0.4 for colored-and-gray-MNIST, ModelNet40 and NVGesture, respectively.

Original: This validates  ﬁrst conjecture in §3.2.
Revised: This validates the ﬁrst conjecture in §3.2.

Original: Third, we draw the same set of conclusions even when we consider conditional learning speed (cid:98) d speed instead of conditional utilization rate.
Revised: Third, by observing conditional learning speed (cid:98) d speed , we can draw the same conclusions.

Original: In fact, the distributions of (cid:98) d speed largely replicate theobserved distributions of (cid:98) d util .
Revised: In fact, the distributions of (cid:98) d speed largely replicate the distributions of (cid:98) d util .

Original: It validates our greedy learner hypothesis which blames the varying rate at which the learner learns to rely on each modality.
Revised: It validates our greedy learner hypothesis which blames the varying rate at which the learner learns to rely on each modality.

Original: It moreover conﬁrms d speed is an appropriate proxy to use to re-balance multi-modal learning.
Revised: It moreover conﬁrms d speed is an appropriate proxy to use to re-balance multi-modal learning.

Revised: We analyze the model’s behavior when using different learning rates.

Revised: Interestingly, we see relatively balanced conditional utilization rates when using high learning rates, as shown in Figure 3 in

Revised: Appendix.

Revised: This observation indicates that high learning rates implicitly calibrate the learning pace between modalities.

Revised: We leave this for future analysis.

Original: 5. S TRONG REGULARIZATION ENCOURAGES GREEDINESS
Revised: 5. S TRONG REGULARIZATION ENCOURAGES GREEDINESS

Original: Regularization is known to impact the learning speed when training a deep neural net with SGD. Here, we investigate the effect of regularization on multi-modal learning. We demonstrate that, aspredicted in the last conjecture 3.2, strong regularization encourages the multi-modal learning process to be greedy.
Revised: We investigate L1 regularization’s impact on multi-modal DNNs and demonstrate that, as the second conjecture in §3.2 says, strong regularization encourages greediness in multi-modal learning.

Original: Study design We apply L1 regularization to the multi-modal DNNs.
Revised: Study design We apply L1 regularization to the multi-modal DNNs.

Original: That is, we train the networks to optimize the loss L (cid:48)
Revised: That is, we train the networks to optimize the loss L (cid:48)

Original: =
Revised: =

Original: L + λ ||
Revised: L + λ ||

Original: θ
Revised: θ

Original: || 1 , where L is the classiﬁcation loss, θ stands for all model parameters and λ is the weight on the regularizer.
Revised: || 1 , where L is the classiﬁcation loss in §2, θ stands for all model parameters and λ is the weight on the regularizer.

Original: We measure the effect of λ on the network f by computing the fraction of its parameters smaller than 10 − 7 .
Revised: We measure the effect of λ on the network f by computing the fraction of its parameters smaller than 10 − 7 .

Original: We denote this  by R ( f ) .
Revised: We denote this quantity by R ( f ) .

Original: Since L1 regularization encourages sparsity of the network’s parameters, as shown in Appendix Figure 3, the larger the λ is, the higher the R ( f ) .
Revised: Since L1 regularization encourages sparsity of the network’s parameters, as shown in Appendix Figure 4, the larger the λ we use, the higher the R ( f ) we observe.

Original: We conduct this study with ModelNet40, using the front and rear views.
Revised: We conduct this study with ModelNet40, using the front and rear views.

Original: We choose ten values from an interval [10 − 9 , 10 − 3 ] as λ .
Revised: We choose ten values from an interval [10 − 9 , 10 − 3 ] as λ .

Original: We use SGD without momentum, set the learning rate to 0.1 and batch size to eight.
Revised: We use SGD without momentum, set the learning rate to 0.1 and batch size to eight.

Original: For each combination of hyperparameters, we train three model repetitions.
Revised: Using each combination of hyperparameters, we repeat training for three times with random initialization and get three models.

Original: Results As shown in Figure 3, | d util ( f ) | increases along λ , especially when log( λ ) ≥ − 5 and | d util ( f ) | is positively correlated with R ( f ) .
Revised: Results As shown in Figure 3, | d util ( f ) | increases along λ , especially when log( λ ) ≥ − 5 . We also see that | d util ( f ) | is positively correlated with R ( f ) .

Original: In other words, the stronger the regularization, the larger the imbalance in utilization between modalities we obverse.
Revised: In other words, the stronger the regularization is, the larger the imbalance in utilization between modalities we observe.

Original: It conﬁrms the second conjecture in §3.2, which states that the stronger regularization we apply, the greedier the multi-modal learning process becomes.

Original: We see  | d speed | follows the same trend as | d util | .
Revised: We see that | d speed | follows the same trend as | d util | .

Original: Again, it supports our choice of using the conditional learning speed to predict the conditional utilization rate.
Revised: Again, it supports our choice of using the conditional learning speed to predict the conditional utilization rate.

Original: [Figure]
Revised: [Figure]

Revised: 3 R ( f ) 0.

Original:  d |
Revised: | d s p ee d |

Original: 5. B ALANCED MULTI - MODAL LEARNING
Revised: 5. B ALANCED MULTI - MODAL LEARNING

Original: Besides the proposed training algorithm (cf. §4.2, referred to as guided )), we introduce a variant of it in Appendix A.3, referred to as random .
Revised: Besides the proposed training algorithm (cf. §4.2, referred to as guided ), we introduce a variant of it in Appendix A.3, referred to as random .

Original: This algorithm is motivated by Modality Dropout (Neverova et al., 2015) but better suited to the multi-modal DNNs with intermediate fusion.
Revised: This algorithm is motivated by Modality Dropout (Neverova et al., 2015) but better suited to the multi-modal DNNs with intermediate fusion.

Original: We consider it a stronger baseline which can also balance learning from  modalities.
Revised: We consider it a stronger baseline that can also balance learning from inputs of different modalities.

Original: In this section we compare the guided, the random and the standard training algorithm (referred to as vanilla ) on the ModelNet dataset (front and rear views), the NVGesture dataset (RGB and depth), and the Colored-and-grayMNIST dataset (colored and gray-scale images).

Original: Calibrated modality utilization We train multi-modal DNNs as described in §5.2, using each training algorithm.
Revised: Calibrated modality utilization We train multi-modal DNNs as described in §5.2, using the guided, the random, and the conventional training algorithm (referred to as vanilla ).

Original: We set the imbalance  parameter α to 0.01 and the re-balancing window size Q to 5 for ModelNet40, and α to 0.1 and Q to 5 for NVGesture.
Revised: For ModelNet40, we set the imbalance tolerance parameter α to 0.01 and the re-balancing window size Q to 5. For NVGesture, we use α of 0.1 and Q of 5.

Original:  Results are shown in Figure 4.
Revised: 2 Results are shown in Figure 4.

Original: Models trained with the guided algorithm have lower | (cid:98) d util | compared to the vanilla algorithm.
Revised: Models trained with the guided algorithm have lower | (cid:98) d util | compared to the vanilla algorithm.

Original: On 
Revised: On NVGesture, we obtain

Original: NVGesture, we obtain E ( (cid:98) d util ) of approximately 0.31 and 0.40 for models trained with the guided and the vanilla algorithm. On ModelNet40, we obtain E ( (cid:98) d util ) of approximately -0.02, and -0.12 for models trained with the guided and the vanilla algorithm.
Revised: E ( (cid:98) d util ) of approximately 0.3 and 0.4 for models trained with the guided and the vanilla algorithm. On ModelNet40, we obtain E ( (cid:98) d util ) of approximately -0.0, and -0.1 for models trained with the guided and the vanilla algorithm.

Original: We would expect greater changes from the guided algorithm if we tune α and Q to ﬁnd the optimal values for each task.

Original: The random algorithm also calibrates modality utilization effectively, as shown in Appendix Figure 4.
Revised: The random algorithm also calibrates modality utilization effectively (see Figure 7 in Appendix).

Revised: We will see in the next section that it helps less on generalization compared to the guided algorithm.

Original: Improved generalization performance We compare the generalization ability of  the three algorithms (guided, random and vanilla).
Revised: Improved generalization performance We compare the generalization ability of multi-modal DNNs trained by the three algorithms (guided, random and vanilla) and the RUBi learning strategy (Cadene et al., 2019).

Original: For each algorithm, we trainthree repetitions of each model using the same learning rate: 0.01, 0.1 and 0.01  for Colored-and-gray-MNIST, ModelNet40 and
Revised: For each algorithm, we train each model three times with the same learning rate. We use 0.01, 0.1 and 0.01 as learning rate for Colored-and-gray-MNIST, ModelNet40 and

Original: NVGesture respectively, given results of the preliminary experiments.
Revised: NVGesture respectively.

Original: We use α of 0.1 for Coloredand-gray-MNIST and NVGesture andα of 0.01 for ModelNet40.
Revised: We use α of 0.1 for Colored-and-gray-MNIST and NVGesture and 0.01 for ModelNet40.

Original: We set Q of 5 for all three datasets.
Revised: We set Q of 5 for all three datasets.

Original: For NVGesture, we add one experiment where we initialize the model with parameters pre-trained 
Revised: For NVGesture, we add one experiment where we initialize the model with parameters pre-trained using the Kinetics dataset (Carreira & Zisserman, 2017) in addition to random initialization. We refer to this setting as “NVGesture-pretrained” and to the other one as “NVGesture-scratch”.

Original: [Figure]
Revised: [Figure]

Original: obtained models trained using the guided and the vanilla algorithm. We  mark zero and E [ (cid:98) d dashed lines.
Revised: We use dashed lines to mark zero and E [ (cid:98) d util ] over the set of models trained with each of the algorithm.

Original:  guided algorithm is less greedy than the vanilla one using the Kinetics dataset (Carreira & Zisserman, 2017) in addition to random initialization. We refer to this setting as “NVGesture-pretrained” and to the other one as “NVGesture-scratch”.
Revised: The guided algorithm is less greedy than the vanilla one.

Original: We report means and standard deviations of the models’ test accuracy in Table 1.[-
-]
Revised: We report means and standard deviations of the models’ test accuracies in Table 1.[-
-]

Original:  The guided algorithm improves the models’ generalization performance over the vanilla algorithm in all four cases.It also outperforms the random algorithm, with the exception of ModelNet40, where their performances are very close.
Revised: 3 RUBi does not show consistent improvement across tasks compared to the vanilla algorithm. The guided algorithm improves the models’ generalization performance over all three other methods in all four cases.

Original: [Table]
Revised: [Table]

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Previous works on multi-modal learning focus more on architectural designs of the DNNs (Ngiam et al., 2011; Tran et al., 2015; Lazaridou et al., 2015; Wang et al., 2020b;
Revised: Previous works on multi-modal learning focus more on architectural designs of the DNNs (Ngiam et al., 2011; Tran et al., 2015; Lazaridou et al., 2015; Wang et al., 2020b;

Original: Pérez-Rúa et al., 2019).
Revised: Pérez-Rúa et al., 2019).

Original: Recently, unbalanced learning dynamics and modality-wise utilization in the end-to-end trained multi-modal classiﬁers were discussed (Goyal et al., 2017; Han et al., 2021;
Revised: Recently, unbalanced learning dynamics and modality-wise utilization in the end-to-end trained multi-modal classiﬁers were discussed (Goyal et al., 2017; Han et al., 2021;

Original: Wang et al., 2020a;
Revised: Wang et al., 2020a;

Original: Gat et al., 2020; Hessel & Lee, 2020; Collell & Moens, 2018).
Revised: Gat et al., 2020; Hessel & Lee, 2020; Collell & Moens, 2018; Winterbottom et al., 2020; Sun et al., 2021).

Original: For example, Wang et al.
Revised: For example, Wang et al.

Original: (2020b) design a parameter-free multi-modal fusion framework that dynamically exchanges channels between the uni-modal branches.
Revised: (2020b) design a parameter-free multi-modal fusion framework that dynamically exchanges channels between the uni-modal branches.

Original: Wang et al.
Revised: Wang et al.

Original: (2020a) estimate the uni-modal branches’ generalization and overﬁtting speeds and calibrate the learning through loss re-weighting.
Revised: (2020a) estimate the uni-modal branches’ generalization and overﬁtting speeds and calibrate the learning through loss re-weighting.

Original: Their weight estimation slows down training and relies on a representative subset of the training data.
Revised: Their weight estimation slows down training and relies on a representative subset of the training data.

Original: Finally Gat et al.
Revised: Finally Gat et al.

Original: (2020) point out the bias in the multi-modal data as the cause of the issue and propose to supply inputs with
Revised: (2020) point out the bias in the multi-modal data as the cause of the issue.

Original:  Gaussian perturbations to the model in order to regularize it by maximizing functional entropies.
Revised: They propose to supply inputs with Gaussian perturbations to the model and regularize it by maximizing functional entropies.

Original: In this work, we explain the phenomenon by the greedy nature of learning in multi-modal DNNs and propose a calibration algorithm based on the training dynamics of model.
Revised: In this work, we explain this phenomenon by the greedy nature of learning in multi-modal DNNs and propose an algorithm based on the training dynamics of model.

Original: Our proposed method does not introduce any additional computation and only relies on the norm of the gradients and weights computed during training.
Revised: Our proposed method does not introduce any additional computation and only relies on the norm of the gradients and weights computed during training.

Original: It does not require architectural changes of the multi-modal DNNs and can be easily modiﬁed to accommodate different fusion modules.
Revised: It does not require architectural changes of the multi-modal DNNs and can be easily modiﬁed to accommodate different fusion modules.

Original: In addition to the multi-modal classiﬁers, the unbalanced modality-wise utilization has been identiﬁed in the multi-modal pre-trained models (Li et al., 2020; Cao et al., 2020).
Revised: In addition to the multi-modal classiﬁers, the unbalanced modality-wise utilization has been identiﬁed in the multi-modal pre-trained models (Li et al., 2020; Cao et al., 2020).

Original: We expect that the insights we provide in this work are applicable to the self-supervised multi-modal training framework too.
Revised: We expect that the insights we provide in this work are applicable to the self-supervised multi-modal training framework too.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Our work demonstratesthe fact that the end-to-end trained multi-modal DNNs rely on one of the input modalities to make predictions while leaving the other modalities underutilized.
Revised: Our work demonstrates that the end-to-end trained multi-modal DNNs rely on one of the input modalities to make predictions while leaving the other modalities underutilized.

Original: We explained this through the greedy learner hypothesis, and validated our statements experimentally on three multi-modal datasets.
Revised: We hypothesize that it is due to the greedy nature of learning in multi-modal DNNs, and validated our statements experimentally on three multi-modal datasets.

Original: We illustrated that using the proposed algorithm to balance the models’ learning from different modalities, enhances generalization.
Revised: We illustrated that using the proposed algorithm to balance the models’ learning from different modalities enhances generalization.

Original: This result emphasizes that the adequate modality utilization is a desired property that a model should achieve in multi-modal learning.
Revised: This result emphasizes that the adequate modality utilization is a desired property that a model should achieve in multi-modal learning.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This work focuses on analyzing the challenges of training multi-modal deep neural networks.
Revised: This work focuses on analyzing the challenges of training multi-modal deep neural networks.

Original: It does not involve any human subjects.
Revised: It does not involve any human subjects.

Original: We do not anticipate any potentially harmful consequences for society due to the publication of this work.
Revised: We do not anticipate any potentially harmful consequences for society due to the publication of this work.

Original: As we use publicly available datasets in our study, we do not anticipate privacy and security issues.
Revised: As we use publicly available datasets in our study, we do not anticipate privacy and security issues.

Original: We claim no conﬂicts of interest.
Revised: We claim no conﬂicts of interest.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide complete descriptions of the used datasets and the applied processing steps in §5.1 and in
Revised: We provide complete descriptions of the used datasets and the applied processing steps in §5.1 and in

Original: Appendix §A.4.
Revised: Appendix §A.4.

Original: We state our experimental conﬁgurations for each study in §5.1, §5. and §5.3.
Revised: We state our experimental conﬁgurations for each study in §5.1, §5. and §5.3.

Original: We plan to publish the source code once the paper is released
Revised: We plan to publish the source code once the paper is released

Original: A A PPENDIX A. D ERIVING f (cid:48) 0 AND f (cid:48) 1 FROM THE MULTI - MODAL DNN S
Revised: A A PPENDIX A. D ERIVING f (cid:48) 0 AND f (cid:48) 1 FROM THE MULTI - MODAL DNN S

Original: Given an input x , let h m 0 k ( x ) and h m 1 k ( x ) denote the feature vectors from theuni-modal branches. Speciﬁcally, these are the feature vectors from the layer which is followed by the k th fusion modules, denoted by g k .
Revised: We denote the k th fusion module by g k . For an input x , let h m 0 k ( x ) and h m 1 k ( x ) denote the feature vectors from the layer in the uni-modal branches where g k is implemented.

Original: To derive f (cid:48) {+
+} ( x m 0 ) and f (cid:48)  ( x m 1 ) , we force each uni-modal branch to make predictions based on the single modality.
Revised: To derive f (cid:48) {+
+} ( x m 0 ) and f (cid:48) 1 ( x m 1 ) , we force each uni-modal branch to make predictions based on the single modality.

Original: Precisely, for each uni-modal branch, we aggregate its feature vectors over the whole training set and compute their average:
Revised: Precisely, for each uni-modal branch, we aggregate its feature vectors over the whole training set and compute their average:

Original: [Equation]
Revised: [Equation]

Original: Let w m 0 k ( x ) and w m 1 k ( x ) denote the context information g k gives:
Revised: Let w m 0 k ( x ) and w m 1 k ( x ) denote the context information g k gives:

Original: [Equation]
Revised: [Equation]

Original: In this way, we cut off the cross-modality information sharing between the uni-modal branches while limiting the distribution shift for both the fusion networks and the layers in the uni-modal branches.
Revised: In this way, we cut off the cross-modal information sharing between the uni-modal branches while limiting the distribution shift for both the fusion modules and the layers in the uni-modal branches.

Original: In our multi-modal DNN, we employ the fusion module as a Multi-Modal Transfer Module (MMTM).
Revised: In our multi-modal DNN, we employ the fusion module as a Multi-Modal Transfer Module (MMTM).

Original: We adopt the annotation used by Joze et al.
Revised: We adopt the annotation used by Joze et al.

Original: A MMTM takes two sets of tensors, A ∈
Revised: A MMTM takes two sets of tensors, A ∈

Original: R N 1 ×···× N K × C and B ∈ R M 1 ×···× M L × C (cid:48) , each as feature maps of a convolutional layer of a unimodal branch.
Revised: R N 1 ×···× N K × C and B ∈ R M 1 ×···× M L × C (cid:48) , each as feature maps of a convolutional layer of a unimodal branch.

Original: Here, N i and M i represent the spatial dimensions of each feature map, and C and C (cid:48) represent the number of feature maps.
Revised: Here, N i and M i represent the spatial dimensions of each feature map, and C and C (cid:48) represent the number of feature maps.

Original: In order to derive f (cid:48) {+
+} ( x m 0 ) and f (cid:48)  ( x m 1 ) , we have operations in a MMTM written as:
Revised: In order to derive f (cid:48) {+
+} ( x m 0 ) and f (cid:48) 1 ( x m 1 ) , we have operations in a MMTM written as:

Original: [Equation]
Revised: [Equation]

Original: where [ · , · ] represents the concatenation operation; W i and b i are parameters of the fully connected layers; (cid:12) is the channel-wise product operation; and σ ( · ) is the sigmoid function.
Revised: where [ · , · ] represents the concatenation operation; W i and b i are parameters of the fully connected layers; (cid:12) is the channel-wise product operation; and σ ( · ) is the sigmoid function.

Original: A. T HE RE - BALANCING STEP
Revised: A. T HE RE - BALANCING STEP

Original: Suppose that we have K fusion modules and let w m 

Revised: Suppose that we have K fusion modules and let w m

Original: k ( x ) and w m 1 k ( x ) denote the output of the k th fusion module.
Revised: k ( x ) and w m 1 k ( x ) denote the output of the k th fusion module.

Original: At the t training step in the balanced multi-modal learning process, we have
Revised: At the t training step in the balanced multi-modal learning process, we have

Original: [Equation]
Revised: [Equation]

Original: where D t passed denotes all samples appeared in the previous regular training steps until now.
Revised: where D t passed denotes all samples appeared in the previous regular training steps until now.

Original: For example, in a re-balancing step on m 0 , we aim to accelerate the model’s learning from m 0 , and we have
Revised: For example, in a re-balancing step on m 0 , we aim to accelerate the model’s learning from m 0 , and we have

Original: [Equation]
Revised: [Equation]

Original:  where x = ( x m 0 , x m 1 ) is the input at this step, and φ 0 and φ 1 denote the uni-modal branch take x m 0 and x m 1 as inputs.
Revised: w m 1 1 ( x ) , · · · , w m 1 K ( x )) , where x = ( x m 0 , x m 1 ) is the input at this step, and φ 0 and φ 1 denote the uni-modal branch take mand m 1 as inputs.

Original: A. R ANDOM VERSION OF THE BALANCED MULTI - MODAL LEARNING ALGORITHM
Revised: A. R ANDOM VERSION OF THE BALANCED MULTI - MODAL LEARNING ALGORITHM

Original: We present a random version of the balanced multi-modal learning algorithm in Algorithm 1.
Revised: We present a random version of the balanced multi-modal learning algorithm in Algorithm 1.

Original: Similar to the guided version, we perform regular steps in the ﬁrst epoch.
Revised: Similar to the guided version, we perform regular steps in the ﬁrst epoch.

Original: Afterwards, at each time, we let the model take the step that is randomly sampled from: regular step, re-balancing step on m 0 , and re-balancing step on m 1 .
Revised: Afterwards, at each time, we let the model take the step that is randomly sampled from: regular step, re-balancing step on m 0 , and re-balancing step on m 1 .

Original: Algorithm

Original: Input: T , # of training steps;

Original: T 1 , # of steps in the 1 th training epoch; Ω step = { regular step, re-balancing step to accelerate learning from m 0 , re-balancing step to accelerate learning from m 1 } for t ← 1 to T 1 do

Original: Take a regular step; endfor t ← T 1 to T do

Original: Randomly sample a step from Ω step ;

Original: Take the sampled step; end

Original: A. D ATA PREPARATION
Revised: A. D ATA PREPARATION

Original: We used three datasets in the paper:
Revised: We used three datasets in the paper:

Original: Colored MNIST dataset (Kim et al., 2019), ModelNet dataset (Su et al., 2015) and NVGesture dataset (Molchanov et al., 2015), as illustrated in Figure 1.
Revised: Colored MNIST dataset (Kim et al., 2019), ModelNet dataset (Su et al., 2015) and NVGesture dataset (Molchanov et al., 2015), as illustrated in Figure 1.

Original: For Colored MNISTdataset and ModelNet40 dataset, we did not perform any extra data preprocessing steps on the original datasets.
Revised: For Colored MNIST and ModelNet40, we did not perform any extra data pre-processing steps on the original datasets.

Original: For the NVGesture dataset, each video has a resolution of 240 × 320 and a duration of 80 frames from action starting to ending.
Revised: For the NVGesture dataset, each video has a resolution of 240 × 320 and a duration of 80 frames from action starting to ending.

Original: There areonly three videos with unmatched starting indices between RGB and depth and we adopt the starting frame indice of RGB for all modalities.
Revised: There are three videos with unmatched starting indices between RGB and depth. We adopt the starting frame indice of RGB for all modalities.

Original: We randomly select 64 consecutive frames from the videos in the dataset and if the video has less than 64 frames, we zero-pad on both sides of it to obtain 64 frames.
Revised: We randomly select 64 consecutive frames from the videos in the dataset and if the video has less than 64 frames, we zero-pad on both sides of it to obtain 64 frames.

Original: Frames are resized as 256 × 256 and are cropped into 224 × 224 as inputs (we use static cropping where we crop from the same location across times and modalities).
Revised: Frames are resized as 256 × 256 and are cropped into 224 × 224 as inputs (we use static cropping where we crop from the same location across times and modalities).

Original: During training, we perform spatial augmentation on the video, including ﬂipping and random cropping.
Revised: During training, we perform spatial augmentation on the video, including ﬂipping and random cropping.

Original: During inference on validation or test, we perform center cropping on the video.
Revised: During inference on validation or test set, we perform center cropping on the video.

Original: [Figure]
Revised: [Figure]

Revised: of the object

Revised: The NVGesture dataset

Revised: Molchanov et al.,

Original: A. M ORE RESULTS
Revised: A. M ORE RESULTS

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] f ) for three model repetitions obtained by training the model with λ as the weight on the L1 regularization. Generally, the larger the λ is, the higher the

Revised: R ( f ) is, i.e., the sparser the model’s parameters are.

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]