Original: Improving Multimodal fusion via Mutual Dependency Maximisation Anonymous  submission
Revised: Improving Multimodal fusion via Mutual Dependency Maximisation Anonymous EMNLP submission

Original: Abstract
Revised: Abstract

Original: Multimodal sentiment analysis is a trending area of research, and the multimodal fusion is one of is most active topic.
Revised: Multimodal sentiment analysis is a trendingarea of research, and the multimodal fusionis one of its most active topic.

Original: Acknowledging humans communicate through a variety of channels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating these different unimodal representations into one synthetic representation.
Revised: Acknowledging humans communicate through a variety ofchannels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating differentunimodal representations into a synthetic one.

Original: So far, a consequent effort has been made on developing complex architectures allowing the fusion of these modalities.
Revised: So far, a consequent effort has been made ondeveloping complex architectures allowing thefusion of these modalities.

Original: However, such systems are mainly trained by minimising simple losses such as L 1 or cross-entropy.
Revised: However, such systems are mainly trained by minimising simple losses such as L 1 or cross-entropy.

Original: In this work, we investigate unexplored penalties and propose a set of new objectives that measure the dependency between modalities.
Revised: Inthis work, we investigate unexplored penaltiesand propose a set of new objectives that measure the dependency between modalities.

Original: We demonstrate that our new penalties lead to a consistent improvement (up to 4 . 3 on accuracy) across a large variety of state-of-the-art models on two well-known sentiment analysis datasets: CMU-MOSI and CMU-MOSEI .
Revised: Wedemonstrate that our new penalties lead to aconsistent improvement (up to 4 . 3 on accuracy) across a large variety of state-of-the-artmodels on two well-known sentiment analysisdatasets: CMU-MOSI and CMU-MOSEI .

Original: Our method not only achieves a new SOTA on both datasets but also produces representations that are more robust to modality drops.
Revised: Ourmethod not only achieves a new SOTA on bothdatasets but also produces representations thatare more robust to modality drops.

Original: Finally, a by-product of our methods includes a statistical network which can be used to interpret the high dimensional representations learnt by the model.
Revised: Finally, aby-product of our methods includes a statistical network which can be used to interpret thehigh dimensional representations learnt by themodel.

Original:  Introduction
Revised:  Introduction

Original: Humans employ three different modalities to communicate in a coordinated manner: the language modality with the use of words and sentences, the vision modality with gestures, poses and facial expressions and the acoustic modality through change in vocal tones.
Revised: Humans employ three different modalities to communicate in a coordinated manner: the languagemodality with the use of words and sentences, thevision modality with gestures, poses and facial expressions and the acoustic modality through changein vocal tones.

Original: Multimodal representation learning has shown great progress in a large variety of tasks including emotion recognition, sentiment analysis (Soleymani et al., 2017), speaker trait analysis (Park et al., 2014) and ﬁne-grained opinion mining (Garcia et al., 2019a).
Revised: Multimodal representation learninghas shown great progress in a large variety of tasksincluding emotion recognition, sentiment analysis (Soleymani et al., 2017), speaker trait analysis(Park et al., 2014) and ﬁne-grained opinion mining (Garcia et al., 2019a).

Original: Learning from different modalities is an efﬁcient way to improve performance on the target tasks (Xu et al., 2013).
Revised: Learning from differentmodalities is an efﬁcient way to improve performance on the target tasks (Xu et al., 2013).

Original: Nevertheless, heterogeneities across modalities increase the difﬁculty of learning multimodal representations and raise speciﬁc challenges.
Revised: Nevertheless, heterogeneities across modalities increasethe difﬁculty of learning multimodal representations and raise speciﬁc challenges.

Original: Baltruˇsaitis et al.
Revised: Baltruˇsaitis et al.

Original: (2018) identiﬁes fusion as one of the ﬁve core challenges in multimodal representation learning, the four other being: representation, modality alignment, translation and co-learning.
Revised: (2018) identiﬁes fusion as one of the ﬁve core challenges in multimodal representation learning, the four other being: representation, modality alignment, translation and co-learning.

Original: Fusion aims at integrating the different unimodal representations into one common synthetic representation.
Revised: Fusion aims at integrating the different unimodal representations into one common synthetic representation.

Original: Effective fusion is still an open problem: the best multimodal models in sentiment analysis (Rahman et al., 2020) improve over their unimodal counterparts, relying on text modality only, by less than 1.5% on accuracy.
Revised: Effective fusion is still an open problem: the best multimodal models in sentiment analysis (Rahman et al., 2020) improve over their unimodal counterparts, relying on text modality only, by less than 1.5% on accuracy.

Original: Additionally, the fusion should not only improve accuracy but also make representations more robust to missing modalities.
Revised: Additionally, the fusion should not only improve accuracy but also make representations more robust to missing modalities.

Original: Multimodal fusion can be divided into early and late fusion techniques: early fusion takes place at the feature level (Ye et al., 2017), while late fusion takes place at the decision or scoring level (Khan et al., 2012).
Revised: Multimodal fusion can be divided into early and late fusion techniques: early fusion takes place at the feature level (Ye et al., 2017), while late fusion takes place at the decision or scoring level (Khan et al., 2012).

Original: Current research in multimodal sentiment analysis mainly focuses on developing new fusion mechanisms relying on deep architectures ( e.g TFN (Zadeh et al., 2017),
Revised: Current research in multimodal sentiment analysis mainly focuses on developing new fusion mechanisms relying on deep architectures ( e.g TFN (Zadeh et al., 2017),

Original: LFN (Liu et al., 2018), MARNE (Zadeh et al., 2018b), MISA (Hazarika et al., 2020), MCTN (Pham et al., 2019), HFNN (Mai et al., 2019), ICCN (Sun et al., 2020)).
Revised: LFN (Liu et al., 2018), MARN (Zadeh et al., 2018b), MISA (Hazarika et al., 2020), MCTN (Pham et al., 2019), HFNN (Mai et al., 2019), ICCN (Sun et al., 2020)).

Original: Theses models are evaluated on several multimodal sentiment analysis benchmark such as 
Revised: Theses models are evaluated on several multimodal sentiment analysis benchmark such as IEMOCAP (Busso et al., 2008), MOSI (W¨ollmer et al., 2013),

Original: IEMOCAP (Busso et al., 2008), MOSI (W¨ollmer et al., 2013), MOSEI (Zadeh et al., 2018c) and POM (Garcia et al., 2019b; Park et al., 2014).
Revised: MOSEI (Zadeh et al., 2018c) and POM (Garcia et al., 2019b; Park et al., 2014).

Original: Current state-of-the-art on these datasets uses architectures based on pre-trained transformers (Tsai et al., 2019; Siriwardhana et al., 2020) such as MultiModal Bert ( MAGBERT ) or MultiModal XLNET ( MAGXLNET ) (Rahman et al., 2020).
Revised: Current state-ofthe-art on these datasets uses architectures based on pre-trained transformers (Tsai et al., 2019; Siriwardhana et al., 2020) such as MultiModal Bert ( MAGBERT ) or MultiModal XLNET ( MAGXLNET ) (Rahman et al., 2020).

Original: The aforementioned architectures are trained by minimising either a L 1 loss or a Cross-Entropy loss between the predictions and the ground-truth labels.
Revised: The aforementioned architectures are trained by minimising either a L 1 loss or a Cross-Entropy loss between the predictions and the ground-truth labels.

Original: To the best of our knowledge, few efforts have been dedicated to exploring alternative losses.
Revised: To the best of our knowledge, few efforts have been dedicated to exploring alternative losses.

Original: In this work, we propose a set of new objectives to perform and improve over existing fusion mechanisms.
Revised: In this work, we propose a set of new objectivesto perform and improve over existing fusion mechanisms.

Original: These improvements are inspired by the InfoMax principle (Linsker, 1988): choose a representation maximising the mutual information (MI) between two possibly overlapping views of the input.The
Revised: These improvements are inspired by the InfoMax principle (Linsker, 1988), i.e. choosingthe representation maximising the mutual information (MI) between two possibly overlapping viewsof the input.

Original:  MI quantiﬁes the dependence of two random variables; contrarily to correlation, MI also captures non-linear dependencies between the considered variables.
Revised: The MI quantiﬁes the dependenceof two random variables; contrarily to correlation, MI also captures non-linear dependencies betweenthe considered variables.

Original: Different from previous work, which mainly focuses on comparing two modalities, our learning problem involves multiple modalities ( e.g text, audio, video).
Revised: Different from previouswork, which mainly focuses on comparing twomodalities, our learning problem involves multiplemodalities ( e.g text, audio, video).

Original: Our proposed method, which induces no architectural changes, relies on jointly optimising the target loss with an additional penalty term measuring the mutual dependency between different modalities.
Revised: Our proposedmethod, which induces no architectural changes,relies on jointly optimising the target loss with anadditional penalty term measuring the mutual dependency between different modalities.

Original: Our Contributions We study new objectives to buildbetter and more  robust multi-modal representations through an enhanced fusion mechanism, and evaluate them on multimodal sentiment analysis. 
Revised: We study new objectives to build more performantand robust multimodal representations through anenhanced fusion mechanism and evaluate them onmultimodal sentiment analysis. Our method alsoallows us to explain the learnt high dimensionalmultimodal embeddings.

Original: The paper contributions can be summarised as follows:
Revised: The paper contributionscan be summarised as follows:

Original: • A set of novel objectives to learn better representations .
Revised: A set of novel objectives using multivariate de- pendency measures .

Original: We introduce three new trainable surrogates to maximise the mutual dependencies between the three modalities ( i.e audio, language and video).
Revised: We introduce three new trainable surrogates to maximise the mutual de- pendencies between the three modalities ( i.e audio, language and video).

Original: We provide a general algorithm inspired by MINE (Belghazi et al., 2018), which was developed in a bi-variate setting for estimating the MI.
Revised: We provide a general algo- rithm inspired by MINE (Belghazi et al., 2018) which was developed in a bi-variate setting for esti- mating the MI.

Revised: Our new method enriches MINE by

Revised: extending the procedure to a multivariate setting

Original: Our new method enriches MINE by extending the procedure to a multivariate setting that allows us to maximise different Mutual Dependency Measures: the Total Correlation (Watanabe, 1960), the f-Total Correlation and the Multivariate Wasserstein Dependency Measure (Ozair et al.,2019).
Revised: that allows us to maximise different Mutual Depen- dency Measures: the Total Correlation (Watanabe, 1960), the f-Total Correlation and the Multivari- ate Wasserstein Dependency Measure (Ozair et al.,

Original: • Applications and numerical results .
Revised: Applications and numerical results .

Revised: We apply

Original: We apply our new set of objectives to ﬁve different architectures relying on LSTM cells (Huang et al., 2015) ( e.g EF-LSTM , LFN , MFN ) or transformer layers ( e.g MAGBERT , MAG-XLNET ).
Revised: our new set of objectives to ﬁve different architectures relying on LSTM cells (Huang et al.,2015) ( e.g EF-LSTM , LFN , MFN ) or transformerlayers ( e.g MAGBERT , MAG-XLNET ).

Original: Our proposed method (1) brings a substantial improvement on two different multimodal sentiment analysis datasets ( i.e MOSI and MOSEI , sec. 5.1), (2) makes the encoder more robust to missing modalities ( i.e when predicting without language, audio or video the observed performance drop is smaller, sec. 5.3), (3) provides an explanation of the decision taken by the neural architecture (sec. 5.4).
Revised: Our proposed method (1) brings a substantial improvementon two different multimodal sentiment analysisdatasets ( i.e MOSI and MOSEI ,sec. 5.1), (2) makesthe encoder more robust to missing modalities ( i.ewhen predicting without language, audio or videothe observed performance drop is smaller, sec. 5.3),(3) provides an explanation of the decision taken by the neural architecture (sec. 5.4).

Original:  Problem formulation & related work
Revised:  Problem formulation & related work

Original: In this section we formulate the problem of learning multi-modal representation (sec. 2.1) and we review both existing measures of mutual dependency (see sec. 2.2) and estimation methods (sec. 2.3).
Revised: In this section, we formulate the problem of learning multi-modal representation (sec. 2.1) and we review both existing measures of mutual dependency (see sec. 2.2) and estimation methods (sec. 2.3).

Original: In the rest of the paper we will focus on learning from three modalities ( i.e language, audio and video), however our approach can be generalised to any arbitrary number of modalities.
Revised: In the rest of the paper, we will focus on learning from three modalities ( i.e language, audio and video), however our approach can be generalised to any arbitrary number of modalities.

Original: 2. Learning multimodal representations
Revised: 2. Learning multimodal representations

Original: Plethora of neural architectures have been proposed to learn multimodal representations for sentiment classiﬁcation.
Revised: Plethora of neural architectures have been proposed to learn multimodal representations for sentiment classiﬁcation.

Original: Models often rely on a fusion mechanism ( e.g multi-layer perceptron (Khan et al., 2012), tensor factorisation (Liu et al., 2018;
Revised: Models often rely on a fusion mechanism ( e.g multi-layer perceptron (Khan et al., 2012), tensor factorisation (Liu et al., 2018;

Original: Zadeh et al., 2019) or complex attention mechanisms (Zadeh et al., 2018a)) that is fed with modalityspeciﬁc representations.
Revised: Zadeh et al., 2019) or complex attention mechanisms (Zadeh et al., 2018a)) that is fed with modalityspeciﬁc representations.

Original: The fusion problem boils down to learning a model M f : X a × X v × X l →
Revised: The fusion problem boils down to learning a model M f : X a × X v × X l →

Original: R d .
Revised: R d .

Original: M f is fed with uni-modal representations of the inputs X a,v,l = ( X a , X v , X l ) obtained through three embedding networks f a , f v and f l .
Revised: M f is fed with uni-modal representations of the inputs X a,v,l = ( X a , X v , X l ) obtained through three embedding networks f a , f v and f l .

Original: M f has to retain both modality-speciﬁc interactions ( i.e interactions that involve only one modality) and cross-view interactions ( i.e more complex, they span across both views).
Revised: M f has to retain both modality-speciﬁc interactions ( i.e interactions that involve only one modality) and cross-view interactions ( i.e more complex, they span across both views).

Original: Overall, the learning of
Revised: Overall, the learning of

Original: M f involves both the minimisation of the downstream task loss and the maximisation of the mutual dependency between the different modalities.
Revised: M f involves both the minimisation of the downstream task loss and the maximisation of the mutual dependency between the different modalities.

Original: 2. Mutual dependency maximisation
Revised: 2. Mutual dependency maximisation

Original: Mutual information as mutual dependency measure: the core ideas we rely on to better learn cross-view interactions are not new.
Revised: Mutual information as mutual dependency measure: the core ideas we rely on to better learncross-view interactions are not new.

Original: They consist of mutual information maximisation (Linsker, 1988), and deep representation learning.
Revised: They consist of mutual information maximisation (Linsker, 1988), and deep representation learning.

Original: Thus, one of the most natural choices is to use the MI that measures the dependence of two random variables, including high-order statistical dependencies (Kinney and Atwal, 2014).
Revised: Thus, one of the most natural choices is to use the MI that measures the dependence between two random variables, including high-order statistical dependencies (Kinney and Atwal, 2014).

Original: Given two random variables X and Y , the MI is deﬁned by
Revised: Given two random variables X and Y , the MI is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where p XY is the joint probability density function (pdf) of the random variables ( X, Y ) , and p X , p Y  represent the marginal pdfs.
Revised: where p XY is the joint probability density function (pdf) of the random variables ( X, Y ) , and p X , p Y  represent the marginal pdfs.

Original: Mutual Information is a measure of the difference between the joint probability p ( x, y ) and the marginal probabilities p ( x ) and p ( y ) , and it can  be written as:
Revised: MI can also be deﬁnedwith a the KL divergence:

Original: [Equation]
Revised: [Equation]

Original: Extension of mutual dependency to different metrics: KL divergence seems to be limited when used for estimating MI (McAllester and Stratos, 2020).
Revised: Extension of mutual dependency to differentmetrics: the KL divergence seems to be limitedwhen used for estimating MI (McAllester and Stratos, 2020).

Original: A natural step is to replace the KL divergence in Eq.
Revised: A natural step is to replace the KL divergence in Eq.

Original: 2 with different divergences such as the f-divergences or distances such as the Wasserstein distance.
Revised: 2 with different divergencessuch as the f-divergences or distances such as the Wasserstein distance.

Original: Hence, we introduce new mutual dependency measures (MDM): the f-Mutual Information (Belghazi et al., 2018), denoted I f and the Wasserstein Measures (Ozair et al., 2019), denoted I W .
Revised: Hence, we introduce new mutual dependency measures (MDM): the f-Mutual Information (Belghazi et al., 2018), denoted I f andthe Wasserstein Measures (Ozair et al., 2019), denoted I W .

Original: As previously, p XY denotes the joint pdf, and p X , p Y denote the marginal pdfs.
Revised: As previously, p XY denotes the jointpdf, and p X , p Y denote the marginal pdfs.

Original: The new measures are deﬁned as follows:
Revised: The newmeasures are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where D f denotes any f − divergences and
Revised: where D f denotes any f -divergences and

Original: [Equation]
Revised: [Equation]

Original: where W denotes the Wasserstein distance.
Revised: where W denotes the Wasserstein distance (Peyr´eet al., 2019).

Original: 2. Estimating mutual dependency measures
Revised: 2. Estimating mutual dependency measures

Original: The computation of MI and other mutual dependency measures can be difﬁcult without knowing the marginal and joint probability distributions, thus it is popular to maximise lower bounds to obtain better representations of different modalities including image (Tian et al., 2019; Hjelm et al., 2018), audio (Dilpazir et al., 2016) and text (Kong et al., 2019) data.
Revised: The computation of MI and other mutual dependency measures can be difﬁcult without knowingthe marginal and joint probability distributions,thus it is popular to maximise lower bounds toobtain better representations of different modalities including image (Tian et al., 2019; Hjelmet al., 2018), audio (Dilpazir et al., 2016) and text(Kong et al., 2019) data.

Original: Several estimators have been proposed: MINE (Belghazi et al., 2018) uses the Donsker-Varadhan representation (Donsker and
Revised: Several estimators havebeen proposed: MINE (Belghazi et al., 2018) usesthe Donsker-Varadhan representation (Donsker and

Original: Varadhan, 1985) to derive a parametric lower bound holds, (Nguyen et al., 2017, 2010) uses variational characterisation of f-divergence and a multi-sample version of the density ratio (also known as noise contrastive estimation (Oord et al., 2018; Ozair et al., 2019)).
Revised: Varadhan, 1985) to derive a parametric lower boundholds, Nguyen et al. (2017, 2010) uses variationalcharacterisation of f-divergence and a multi-sampleversion of the density ratio (also known as noisecontrastive estimation (Oord et al., 2018; Ozairet al., 2019)).

Original: These methods have mostly been developed and studied in a bi-variate setting.
Revised: These methods have mostly beendeveloped and studied in a bi-variate setting.

Original: Illustration of neural dependency measures on a bivariate case .
Revised: Illustration of neural dependency measures ona bivariate case .

Original: In Fig.
Revised: In Fig.

Original: 1 we can see the aforementioned dependency measures ( i.e see Eq.
Revised: 1 we can see the aforementioned dependency measures ( i.e see Eq.

Original: 2, Eq. 4,
Revised: 2, Eq. 4,

Original: [Figure] Eq. 3) when estimated with MINE (Belghazi et al., 2018) for multivariate Gaussian random variables,
Revised: Eq. 3) when estimated with MINE (Belghazi et al.,2018) for multivariate Gaussian random variables,

Original: X a and X b .
Revised: X a and X b .

Original: The component wise correlation for the considered multivariate Gaussian is deﬁned as  follow: corr ( X i , X k )
Revised: The component wise correlation forthe considered multivariate Gaussian is deﬁned as [Figure] follow: corr ( X i , X k )

Original: =
Revised: =

Original: δ i,k ρ , where ρ ∈ ( − 1 , 1) and δ i,k is Kronecker’s delta.
Revised: δ i,k ρ , where ρ ∈ ( − 1 , 1) and δ i,k is Kronecker’s delta.

Original: We observe that the dependency measure based on Wasserstein distance is different from the one based on the divergences and thus will lead to different gradients.
Revised: We observe that the dependency measure based on Wasserstein distance is different from the one based on the divergences and thus will lead to different gradients.

Original: Although theoretical studies have been done on the use of different metrics for dependency estimations, it remains an open question to know which one is the best suited.
Revised: Although theoretical studies have been done on the use of different metrics for dependency estimations, it remains an open question to know which one is the best suited.

Original: In this work, we will provide an experimental response in a speciﬁc case.
Revised: In this work, we will provide an experimental response in a speciﬁc case.

Original:  Model and training objective
Revised:  Model and training objective

Original: In this section, we introduce our new set of losses to improve fusion.
Revised: In this section, we introduce our new set of losses to improve fusion.

Original: In sec. 3.1, we ﬁrst extend widely used bi-variate dependency measures to multivariate dependencies (James and Crutchﬁeld, 2017) measures (MDM).
Revised: In sec. 3.1, we ﬁrst extend widely used bi-variate dependency measures to multivariate dependencies (James and Crutchﬁeld, 2017) measures (MDM).

Original: We then introduce variational bounds on the MDM, and in sec.
Revised: We then introduce variational bounds on the MDM, and in sec.

Original: 3.2, we describe our method to minimise the proposed variational bounds.
Revised: 3.2, we describe our method to minimise the proposed variational bounds.

Original: Notations We consider X a , X v , X l as the multimodal data from the audio,video and language modality respectively with joint probability distribution P X a ,X v ,X l .
Revised: Notations We consider X a , X v , X l as the multimodal data from the audio,video and language modality respectively with joint probability distribution p X a X v X l .

Original: We denote as P X j the marginal distribution of X j with j ∈ { a, v, l } corresponding to the j th modality.
Revised: We denote as p X j the marginal distribution of X j with j ∈ { a, v, l } corresponding to the j th modality.

Original: General loss As previously mentioned, we rely on the InfoMax principle (Linsker, 1988) and aim at jointly maximising the MDM between the different modalities and minimising the task loss; hence, we are in a multi-task setting (Argyriou et al., 2007;
Revised: General loss As previously mentioned, we rely on the InfoMax principle (Linsker, 1988) and aim at jointly maximising the MDM between the different modalities and minimising the task loss; hence, we are in a multi-task setting (Argyriou et al., 2007;

Original: Ruder, 2017) and the objective of interest can be deﬁned as:
Revised: Ruder, 2017) and the objective of interest can be deﬁned as:

Original: L ≡

Original: [Equation]
Revised: [Equation]

Original: L down.
Revised: L down.

Original: represents a downstream speciﬁc (target task) loss i.e a binary cross-entropy or a L 1 loss, λ is a meta-parameter and L MDM is the multivariate dependencies measures (see sec. 3.2).
Revised: represents a downstream speciﬁc (targettask) loss i.e a binary cross-entropy or a L 1 loss, λis a meta-parameter and L MDM is the multivariatedependencies measures (see sec. 3.2).

Original: Minimisation of our newly deﬁned objectives requires to derive lower bounds on the L MDM terms, and then to obtain trainable surrogates.
Revised: Minimisation of our newly deﬁned objectives requires toderive lower bounds on the L MDM terms, and thento obtain trainable surrogates.

Original: 3. From bivariate to multivariate dependencies
Revised: 3. From bivariate to multivariatedependencies

Original: In our setting, we aim at maximising cross-view interactions involving three modalities, thus we need to generalise bivariate dependency measures to multivariate dependency measures.
Revised: In our setting, we aim at maximising cross-viewinteractions involving three modalities, thus weneed to generalise bivariate dependency measuresto multivariate dependency measures.

Original: Deﬁnition 3.1 (Multivariate Dependencies Measures) .
Revised: Deﬁnition 3.1 (Multivariate Dependencies Measures) .

Original: Let X 1 ,. . .
Revised: Let X a , X v ,

Original: X M be a set of random variables with joint pdf p X 1 ,...,X M and respective marginal pdf p

Original: X
Revised: X

Original: i .
Revised: l be a set of random variables with joint pdf p X a X v X l and respectivemarginal pdf p X j with j ∈ { a, v, l } .

Original: Then we deﬁned the multivariate mutual information I  , also refered as total correlation (Watanabe, 1960) or multi-information (Studen`y and Vejnarov´a, 1998): 
Revised: Then we deﬁned the multivariate mutual information I kl , alsorefered as total correlation (Watanabe, 1960) ormulti-information (Studen`y and Vejnarov´a, 1998): [Equation]

Original: [Equation]
Revised: [Equation]

Original: Simarly for any f-divergence we deﬁne the multivariate f-mutual information I f as:
Revised: Simarly for any f-divergence we deﬁne the multivariate f-mutual information I f as:

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: [Equation]

Original: Finally, we also extend Eq.
Revised: Finally, we also extend Eq.

Original: 3 to obtain the multivariate Wasserstein dependency measure
Revised: 3 to obtain the multivariate Wasserstein dependency measure

Original: I W :
Revised: I W :

Original: [Equation]
Revised: [Equation]

Revised: p X j ( x j ))

Original: where W denotes the Wasserstein distance.
Revised: .where W denotes the Wasserstein distance.3.

Original: 3. From theoretical bounds to trainable surrogates
Revised: From theoretical bounds to trainablesurrogates

Original: To train our neural architecture we need to estimate the previously deﬁned multivariate dependency measures.
Revised: To train our neural architecture we need to estimate the previously deﬁned multivariate dependency measures.

Original: We rely on neural estimators that are given in Th.
Revised: We rely on neural estimators thatare given in Th.

Original: Theorem 1.
Revised: Theorem 1.

Original: Multivariate Neural Dependency
Revised: Multivariate Neural Dependency

Original: Measures Let the family of functions T ( θ ) :
Revised: Measures Let the family of functions T ( θ ) :

Original: X a ×
Revised: X a ×

Original: X v × X
Revised: X v × X

Original: l
Revised: l

Original: → R parametrized by a deep neural network with learnable parameters θ ∈ Θ .
Revised: → R parametrized by a deep neural network with learnable parameters θ ∈ Θ .

Original: The multivariate mutual information measure I  is deﬁned as follows:
Revised: The multivariate mutual information measure I kl is deﬁnedas:

Original: [Equation]
Revised: [Equation]

Original: The neural multivariate f-mutual information measure I f is deﬁned as follows:
Revised: The neural multivariate f-mutual information measure I f is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: The neural multivariate Wasserstein dependency measure I W is deﬁned as follows:
Revised: The neural multivariate Wasserstein dependency measure I W is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Where L is the set of all 1-Lipschitz functions from
Revised: Where L is the set of all 1-Lipschitz functions from

Original: R d → R Sketch of proofs: Eq. 6 is a direct application of the Donsker-Varadhan representation of the KL divergence (we assume that the integrability constraints are satisﬁed).
Revised: R [Equation] Sketch of proofs: Eq. 6 is a direct application of the Donsker-Varadhan representation of the KL divergence (we assume that the integrability constraints are satisﬁed).

Original: Eq.
Revised: Eq.

Original: 7 comes from the work of (Nguyen et al., 2017).
Revised: 7 comes from the work of (Nguyen et al., 2017).

Original: Eq. 8 comes from the
Revised: Eq. 8 comes from the

Original: Kantorovich-Rubenstein: we refer the reader to (Villani, 2008; Peyr´e et al., 2019) for a rigorous and exhaustive treatment.
Revised: Kantorovich-Rubenstein: we refer the reader to (Villani, 2008; Peyr´e et al., 2019) for a rigorous and exhaustive treatment.

Original: Practical estimate of the variational bounds.
Revised: Practical estimate of the variational bounds.

Original: The empirical estimator that we derive from Th.can be used in practical way: the expectations in
Revised: The empirical estimator that we derive from Th. can be used in practical way: the expectations in

Original: Eq. 6, Eq. 7 and Eq. 8 are estimated using empirical samples from the joint distribution P X a ,X v ,X l .
Revised: Eq. 6, Eq. 7 and Eq. 8 are estimated using empirical samples from the joint distribution p X a X v X l .

Original: The empirical samples from P X a P X v P X l are obtained by shufﬂing the samples from the joint distribution in a batch.
Revised: The empirical samples from (cid:81) j ∈{ a,v,l } p X j are obtained by shufﬂing the samples from the joint distribution in a batch.

Original: We integrate this into minimising a multi-task objective (5) by using minus the estimator.
Revised: We integrate this into minimising a multi-task objective (5) by using minus the estimator.

Original: We refer to the losses obtained with the penalty based on the estimators described in Eq. 6, 
Revised: We refer to the losses obtained with the penalty based on the estimators described in Eq. 6, Eq.

Original: Eq.and Eq.
Revised: and Eq.

Original: as L kl , L f and L W respectively.
Revised: as L kl , L f and L W respectively.

Original: Details on the practical minimisation of our variational bounds are provided inthe Algorithm 1.
Revised: Details on the practical minimisation of our variational bounds are provided in Algorithm 1.

Original: Experimental setting
Revised: Remark.

Original: In this section, we present our experimental settings including the neural architectures we compare, the datasets, the metrics and our methodology, which includes the hyper-parameter selection.
Revised: In this work we choose to generalise MINE to compute multivariate dependencies. Comparing our proposed algorithm to other alternatives mentioned in sec.

Original: 4. Datasets
Revised: 2 is left for future work. This choice is driven by two main reasons: (1) our framework allows the use of different types of contrast measures (e.g Wasserstein distance, fdivergences); (2) the critic network T θ can be used for interpretability purposes as shown in sec.  Experimental setting

Original: We empirically evaluate our methods on two english datasets: CMU-MOSI and CMU-MOSEI . Both datasets have been frequently used to assess Algorithm 1 Two- to multivariate dependency measures.
Revised: In this section, we present our experimental settings including the neural architectures we compare, the datasets, the metrics and our methodology, which includes the hyper-parameter selection.

Original: INPUT:multimodal training dataset D n = { ( x ja , x jv , x jl ) , ∀ j ∈ [1 , n ] } , batch size m , permutation σ a , σ v , σ l  ,  θ c weights of the deep classiﬁer, θ weights of the statistical network T θ .
Revised: INPUT: D n = { ( x ja , x jv , x jl ) , ∀ j ∈ [1 , n ] } multimodal training dataset, m batch size, σ a , σ v , σ l : [1 , m ] → [1 , m ] three permutations, θ c weights of the deep classiﬁer, θ weights of the statistical network T θ .

Original: Initialization: parameters θ and θ c
Revised: Initialization: parameters θ and θ c

Original: Build Negative Dataset: ¯
Revised: Build Negative Dataset: ¯

Original: D n = { ( x σ a ( j ) a , x σ v ( j ) v , x σ l ( j ) l ) , ∀ j ∈ [1 , n ] }
Revised: D n = { ( x σ a ( j ) a , x σ v ( j ) v , x σ l ( j ) l ) , ∀ j ∈ [1 , n ] }

Original: Optimization: while ( θ, θ c ) not converged do for i ∈ [1 , Unroll ] do
Revised: Optimization: while ( θ, θ c ) not converged do for i ∈ [1 , Unroll ] do

Original: Sample from D n , B ∼ P X a ,X v ,X l
Revised: Sample from D n , B ∼ p X a X v X l

Original: Sample from ¯ D n , ¯ B ∼ P X a P X v P X l
Revised: Sample from ¯ D n , ¯ B ∼ (cid:81) j ∈{ a,v,l } p X j

Original: Update θ based on the empirical version of Eq.
Revised: Update θ based on the empirical version of Eq.

Original: 6 or Eq.
Revised: 6 or Eq.

Original: 7 or Eq. 8.
Revised: 7 or Eq. 8.

Original: end for
Revised: end for

Original: Sample a batch B from D
Revised: Sample a batch B from D

Original: Update θ c with B using Eq. 5.
Revised: Update θ c with B using Eq. 5.

Original: end while
Revised: end while

Original: OUTPUT: Classiﬁers weights θ c model performance in human multimodal affection recognition.
Revised: OUTPUT: Classiﬁers weights θ c 4. Datasets We empirically evaluate our methods on twoenglish datasets: CMU-MOSI and CMU-MOSEI . Both datasets have been frequently used to assessmodel performance in human multimodal sentiment and emotion recognition.

Original: CMU-MOSI :
Revised: CMU -MOSI :

Original: Multimodal Opinion Sentiment Intensity (W¨ollmer et al., 2013) is a sentiment annotated dataset gathering 2 , 199 short monologue video clips.
Revised: Multimodal Opinion Sentiment In- tensity (W¨ollmer et al., 2013) is a sentiment an- notated dataset gathering 2 , 199 short monologue video clips.

Original: CMU-MOSEI : CMU
Revised: CMU-MOSEI : CMU-Multimodal Opinion Senti-

Original: Multimodal Opinion Sentiment and Emotion Intensity (Zadeh et al., 2018c) is an emotion and sentiment annotated corpus consisting of 23 , 454 movie review videos taken from
Revised: ment and Emotion Intensity (Zadeh et al., 2018c) is an emotion and sentiment annotated corpus con- sisting of 23 , 454 movie review videos taken from

Original: YouTube.
Revised: YouTube.

Original: Both CMU-MOSI and CMU-MOSEI are labelled by humans with a sentiment score in [ − 3 , 3] .
Revised: Both CMU-MOSI and CMU-MOSEIare labelled by humans with a sentiment score in[ − 3 , 3] .

Original: For each dataset, three modalities are available; we follow prior work (Zadeh et al., 2018b, 2017; Rahman et al., 2020) and the features that have been obtained as follows 1 :
Revised: For each dataset, three modalities are available; we follow prior work (Zadeh et al., 2018b,2017; Rahman et al., 2020) and the features thathave been obtained as follows 1 :

Original: Language : Video transcripts are converted to word embeddings using either Glove (Pennington et al., 2014), BERT or XLNET contextualised embeddings.
Revised: Language : Video transcripts are converted to wordembeddings using either Glove (Pennington et al., , BERT or XLNET contextualised embeddings.

Original: For Glove, the embeddings are of dimension 300 , where for BERT and XLNET this dimension is 768 .
Revised: For Glove, the embeddings are of dimension 300 , where for BERT and XLNET this dimension is 768 .

Original: Vision : Vision features are extracted using Facet which results into facial action units corresponding to facial muscle movement.
Revised: Vision : Vision features are extracted using Facet which results into facial action units corresponding to facial muscle movement.

Original: For CMU-MOSEI , the video vectors are composed of 47 units, and for
Revised: For CMU-MOSEI , the video vectors are composed of 47 units, and for

Original: CMU-MOSI they are composed of 35 .
Revised: CMU-MOSI they are composed of 35 .

Original: Audio : Audio features are extracted using COVAREP (Degottex et al., 2014). 
Revised: Audio : Audio features are extracted using CO- VAREP (Degottex et al., 2014). This results into

Original: This results into a vector of dimension 74 which includes 12 Melfrequency cepstral coefﬁcients (MFCCs), as well as pitch tracking and voiced/unvoiced segmenting features, peak slope parameters, maxima dispersion quotients and glottal source parameters.
Revised: a vector of dimension 74 which includes 12 Melfrequency cepstral coefﬁcients (MFCCs), as well as pitch tracking and voiced/ unvoiced segmenting features , peak slope parameters, maxima disper-

Original:  Video and audio are aligned on text-based following the convention introduced in (Chen et al., 2017) and the forced alignment described in (Yuan and
Revised: sion quotients and glottal source parameters. Video and audio are aligned on text-based following the convention introduced in (Chen et al., 2017) and the forced alignment described in (Yuan and

Original: Liberman, 2008).
Revised: Liberman, 2008).

Original: 4. Evaluation metrics
Revised: 4. Evaluation metrics

Original: Multimodal Opinion Sentiment Intensity prediction is treated as a regression problem.
Revised: Multimodal Opinion Sentiment Intensity prediction is treated as a regression problem.

Original: Thus, we report both the Mean Absolute Error (MAE) and the correlation of model predictions with true labels.
Revised: Thus, we report both the Mean Absolute Error (MAE) and the correlation of model predictions with true labels.

Original: In the literature, the regression task is also turned into a binary classiﬁcation task for polarity prediction.
Revised: In the literature, the regression task is also turned into a binary classiﬁcation task for polarity prediction.

Original: We follow standard practices (Rahman et al., 2020) and report the Accuracy 2 ( Acc 7 denotes accuracy on 7 classes and Acc 2 the binary accuracy) of our best performing models.
Revised: We follow standard practices (Rahman et al., 2020) and report the Accuracy 2 ( Acc 7 denotes accuracy on 7 classes and Acc 2 the binary accuracy) of our best performing models.

Original: 4. Neural architectures
Revised: 4. Neural architectures

Original: In our experiments, we choose to modify the loss function of the different models that have been introduced for multi-modal sentiment analysis on both CMU-MOSI and CMU-MOSEI : Memory Fusion Network ( MFN (Zadeh et al., 2018a)), Lowrank Multimodal Fusion ( LFN (Liu et al., 2018)) and two state-of-the-art transformers based models (Rahman et al., 2020) for fusion rely on BERT (Devlin et al., 2018)
Revised: In our experiments, we choose to modify the loss function of the different models that have been introduced for multi-modal sentiment analysis on both CMU-MOSI and CMU-MOSEI : Memory Fusion Network ( MFN (Zadeh et al., 2018a)), Lowrank Multimodal Fusion ( LFN (Liu et al., 2018)) and two state-of-the-art transformers based models (Rahman et al., 2020) for fusion rely on BERT (Devlin et al., 2018)

Original: ( MAG-BERT ) and XLNET
Revised: ( MAG-BERT ) and XLNET

Original: (Yang et al., 2019) ( MAG-XLNT ).
Revised: (Yang et al., 2019) ( MAG-XLNT ).

Original: To assess the validity of the proposed losses, we also apply our method to a simple early fusion LSTM ( EF-LSTM ) as a baseline model.
Revised: To assess the validity of the proposed losses, we also apply our method to a simple early fusion LSTM ( EF-LSTM ) as a baseline model.

Original: Model overview: Aforementioned models can be seen as a multi-modal encoder f θ e providing a representation Z avl containing information and dependencies between modalities X l , X a , X v namely:
Revised: Model overview: Aforementioned models can be seen as a multi-modal encoder f θ e providing a representation Z avl containing information and dependencies between modalities X l , X a , X v namely:f θ e ( X a , X v , X l ) =

Original: [Equation]
Revised: Z avl .

Original: As a ﬁnal step, a linear transformation A θ p is applied to Z avl to perform the regression.
Revised: As a ﬁnal step, a linear transformation A θ p is applied to Z avl to perform the regression.

Original: EF-LSTM : is the most basic architecture used in the current multimodal analysis where each sequence view is encoded separately with LSTM channels then a fusion function is applied to each representation.
Revised: EF-LSTM : is the most basic architecture used inthe current multimodal analysis where each sequence view is encoded separately with LSTMchannels. Then, a fusion function is applied toall representations.

Original: TFN : computes a representation of each view,  fusion operator is then applied.
Revised: TFN : computes a representation of each view, andthen applies a fusion operator.

Original: Acoustic and visual views are ﬁrst mean-pooled then encoded through a 2-layers perceptron.
Revised: Acoustic and visualviews are ﬁrst mean-pooled then encoded througha 2-layers perceptron.

Original: Linguistic features are computed with a LSTM channel as described below and Z l = (cid:76) t h tl .
Revised: Linguistic features are computed with a LSTM channel.

Original: Here, the fusion function is a cross-modal product capturing unimodal, bimodal and trimodal interactions across modalities.
Revised: Here, the fusion function is a cross-modal product capturing unimodal,bimodal and trimodal interactions across modalities.

Original: MFN enriches the previous EF-LSTM architecture with an attention module that computes a crossview representation at each time step.
Revised: MFN enriches the previous EF-LSTM architecturewith an attention module that computes a crossview representation at each time step.

Original: They are then gathered and a ﬁnal representation is computed by a gated multi-view memory (Zadeh et al., 2018a).
Revised: They arethen gathered and a ﬁnal representation is computed by a gated multi-view memory (Zadeh et al.,2018a).

Original: MAG-BERT and MAG-XLNT are based on pretrained transformer architecture (Devlin et al., 2018;
Revised: MAG-BERT and MAG-XLNT are based on pretrained transformer architectures (Devlin et al.

Original:  Yang et al., 2019) allowingmultimodal inputs on the i -th transformer unit with a special gate inspired by Wang et al.
Revised: ,2018; Yang et al., 2019) allowing inputs on each ofthe transformer units to be multimodal, thanks toa special gate inspired by Wang et al.

Original: The Z lav is the [ CLS ] representation provided by the last transformer head.
Revised: The Z avl is the [ CLS ] representation provided by thelast transformer head.

Original: For each architecture, we use the optimal architecture hyperparameters provided by the associated papers (see Appendix A).
Revised: For each architecture, we usethe optimal architecture hyperparameters providedby the associated papers (see Appendix A).

Original:  Numerical results
Revised:  Numerical results

Original: We present and discuss here the results obtained using the experimental setting described in sec.
Revised: We present and discuss here the results obtained using the experimental setting described in sec.

Original: To better understand the impact of our new methods, we propose to investigate the following points:
Revised: Tobetter understand the impact of our new methods,we propose to investigate the following points:

Original: • Efﬁciency of the L MDM : to gain understanding of the usefulness of our new objectives, we study the impact of adding the mutual dependency term on the basic multimodal neural model, namely, EF-LSTM .
Revised: Efﬁciency of the L MDM : to gain understandingof the usefulness of our new objectives, we studythe impact of adding the mutual dependency termon the basic multimodal neural model EF-LSTM .

Original: Improving model performance and comparing multivariate dependency measures: the choice of the most suitable dependency measure for a given task is still an open problem (see sec. 3).
Revised: Improving model performance and comparingmultivariate dependency measures: the choiceof the most suitable dependency measure for agiven task is still an open problem (see sec. 3).

Original: Thus, we compare the performance – on both multimodal sentiment and emotion prediction tasks– of the different dependency measures.
Revised: Thus, we compare the performance – on both multimodal sentiment and emotion prediction tasks– ofthe different dependency measures.

Original: The compared measures are combined with different models using various fusion mechanisms.
Revised: The compared measures are combined with different models using various fusion mechanisms.

Original: • Improving the robustness to modality drop: a desirable quality of multimodal representation is the robustness to a missing modality.
Revised: Improving the robustness to modality drop: a desirable quality of multimodal representations is the robustness to a missing modality.

Original: We study how the maximisation of mutual dependency measures during training affect the robustness of the representation when a modality is missing.
Revised: We study how the maximisation of mutual dependency measures during training affects the robustness of the representation when a modality becomes missing.

Original: • Towards explainable representations: the statistical network T θ allows us to compute a dependency measure between the three considered modalities.
Revised: Towards explainable representations: the statistical network T θ allows us to compute a dependency measure between the three considered modalities.

Original: We carry out a qualitative analysis in order to investigate if a high dependency can be explained with complementariness acrossmodalities and low dependencies can be explained with contradictoriness across several modalities.
Revised: We carry out a qualitative analysis in order to investigate if a high dependency can be explained by complementariness across modalities.

Original: 5. Efﬁciency of the MDM penalty
Revised: 5. Efﬁciency of the MDM penalty

Original: For aLSTM based multi-modal architecture with a simple MLP as a fusion mechanism, we study the improvement induced by addition of our MDM penalty.
Revised: For a simple EF-LSTM , we study the improvement induced by addition of our MDM penalty.

Original: The resultsfor the EF-LSTM are presented in Tab. 1, where a EF-LSTM trained with no mutual dependency term is denoted with L ∅ .
Revised: The results are presented in Tab. 1, where a EF-LSTM trained with no mutual dependency term is denoted with L ∅ .

Original: On both studied datasets, we observe that the addition of a MDM penalty has a positive impact on both Acc h 2 and Acc h 7 .
Revised: On both studied datasets, we observe that the addition of a MDM penalty leads to stronger performances on all metrics.

Original: For both datasets, we observe that the best performing models are obtained by training with an additional mutual dependency measure term.
Revised: For both datasets, we observe that the best performing models are obtained by training with an additional mutual dependency measure term.

Original: Keeping in mind the  Fig.
Revised: Keeping in mind the example shown in Fig.

Original: 1, we can draw a ﬁrst comparison between the different dependency measures.
Revised: 1, we can draw a ﬁrst comparison between the different dependency measures.

Original: Although in a simple case L f and L kl estimate a similar quantity (see Fig. 1), in more complex practical applications they do not achieve the same performance.
Revised: Although in a simple case L f and L kl estimate a similar quantity (see Fig. 1), in more complex practical applications they do not achieve the same performance.

Original: Even though, the Donsker-Varadhan bound used for L kl is strongerthan the one used to estimate L f ; for a simple model the stronger bound does not lead to better results.
Revised: Even though, the Donsker-Varadhan bound used for L kl is stronger 3 than the one used to estimate L f ; for a simple model the stronger bound does not lead to better results.

Original: It is possible that most of the differences in performance observed come from the optimisation process during training 4 .
Revised: It is possible that most of the differences in performance observed come from the optimisation process during training 4 .

Original: [Table]
Revised: Takeaways:

Original: [Table]
Revised: On the simple case of EF-LSTM adding MDM penalty improves the performance on the downstream tasks.

Original: 5. Improving models and multivariate dependency measures comparison
Revised: 5. Improving models and comparing

Revised: multivariate dependency measures

Original: In this experiment, we apply the different penalties with more advanced architectures using various fusion mechanisms. 
Revised: In this experiment, we apply the different penalties to more advanced architectures, using various fusion mechanisms. [Table]

Original: General analysis .
Revised: General analysis .

Original: Tab. 2 shows the performance of various neural architecture trained with and without MDM penalty.
Revised: Tab. 2 shows the performanceof various neural architectures trained with andwithout MDM penalty.

Original: Presented results are coherent with the previous experiment: we observe that jointly maximising a mutual dependency measure leads to better result on the downstream task. A MFN on CMU-MOSI trained with L f outperforms by 4 . 3% (on Acc h 7 ) and 3 .
Revised: Results are coherent withthe previous experiment: we observe that jointlymaximising a mutual dependency measure leadsto better results on the downstream task: for example, a MFN on CMU-MOSI trained with L Woutperforms by 4 . 6 points on Acc h 7 the modeltrained without the mutual dependency term.

Original: 2% (on Acc h 2 ) the model trained without the mutual dependency term.

Original: On
Revised: On

Original: CMU-MOSEI we also see an improvement for the same MFN trained with L f compared to L ∅ .
Revised: CMU-MOSEI we also obtain subsequent improvements while training with MMD.

Original: On

Original: CMU-MOSI the TFN also  beneﬁts from the mutual dependency term with an absolute improvement of 3 .2% (on Acc h 7 ) and
Revised: On CMU-MOSIthe TFN also strongly beneﬁts from the mutual dependency term with an absolute improvement of3 .

Original: 1 . 2% (on Acc h 2 ) with L W compared to L ∅ .
Revised: 7% (on Acc h 7 ) with L W compared to L ∅ .

Original: Tab. 2 shows that our methods not only perform well on recurrent architectures but also on 
Revised: Tab. 2shows that our methods not only perform wellon recurrent architectures but also on pretrained

Original: Transformer-based models that achieve higher results due to a superior capacity to model contextual dependencies (see (Rahman et al., 2020)).
Revised: Transformer-based models, that achieve higher results due to a superior capacity to model contextualdependencies (see (Rahman et al., 2020)).

Original: Improving state-of-the-art models .
Revised: Improving state-of-the-art models .

Original: MAGBERT and MAGXLNET are state-of-the art models on both
Revised: MAGBERTand MAGXLNET are state-of-the art models on both

Original: CMU-MOSI and CMU-MOSEI .
Revised: CMU-MOSI and CMU-MOSEI .

Original: From Tab. 2, we observe that our methods can improve the performance of both models.
Revised: From Tab. 2, weobserve that our methods can improve the performance of both models.

Original: It is worth noting that, in both cases, L W combined with pre-trained transformers achieves good results.
Revised: It is worth noting that, inboth cases, L W combined with pre-trained transformers achieves good results.

Original: This performance
Revised: This performancegain suggests that our method is able to capturedependencies that are not learnt during either pretraining of the language model ( i.e BERT or XL-

Original: [Table]
Revised: [Table]

Original: improvement suggests that our method is able to capture dependencies that are not learnt during either pretraining of the language model ( i.e BERT or XLNET) or by the Multimodal Adaptation Gate used to perform the fusion.
Revised: L W 35.6 77. 0.57 MAGBERT L ∅ 40.2 84. L kl 42.0 85.

Original: Comparing dependency measures .
Revised: L f 41.7 85. L W 41.8 85. 0.79 MAGXLNET L ∅ 43.0 86.

Revised: L kl 44.5 86.

Revised: L f 43.9 86.

Revised: L W 44.4 86.

Revised: NET) or by the Multimodal Adaptation Gate used to perform the fusion.

Revised: Comparing dependency measures .

Original: Tab.
Revised: Tab.

Original: 2 shows that there is no dependency measure that achieves the best results in all cases.
Revised: 2 shows that there is no dependency measure that achieves the best results in all cases.

Original: This result tends to conﬁrm that the optimisation process during training plays an important role (see hypothesis in sec. 5.1).
Revised: This result tends to conﬁrm that the optimisation process during training plays an important role (see hypothesis in sec. 5.1).

Original: However, we can observe thatin general optimising the multivariate Wasserstein dependency measure is usually a good choice and achieves state of the art results in many conﬁgurations.
Revised: However, we can observe that optimising the multivariate Wasserstein dependency measure is usually a good choice, since it achieves state of the art results in many conﬁgurations.

Original: It is worth noting that several pieces of research point the limitations of mutual information estimators (McAllester and
Revised: It is worth noting that several pieces of research point the limitations of mutual information estimators (McAllester and

Original: Stratos, 2020; Song and Ermon, 2019).
Revised: Stratos, 2020; Song and Ermon, 2019).

Revised: Takeaways: The addition of MMD not only beneﬁts simple models ( e.g EF-LSTM ) but also improves performance when combined with both complex fusion mechanisms and pretrained models.

Revised: For practical applications, the Wasserstein distance is a good choice of contrast function.

Original: 5. Improved robustness to modality drop
Revised: 5. Improved robustness to modality drop

Original: Although fusion with visual and acoustic modalities provided a performance improvement (Wang et al., 2018), the performance of Multimodal systems on sentiment prediction tasks are mainly carried by the linguistic modality (Zadeh et al., 2018a, 2017).
Revised: Although fusion with visual and acoustic modalities provided a performance improvement (Wang et al., 2018), the performance of Multimodal systems on sentiment prediction tasks is mainly carried by the linguistic modality (Zadeh et al., 2018a, 2017).

Original: Thus it is interesting to study how a multimodal system behaves when either language is missing because it gives insights on the robustness of the representation.
Revised: Thus it is interesting to study how a multimodal system behaves when the text modality is missing because it gives insights on the robustness of the representation.

Original: Experiment description .
Revised: Experiment description .

Original: In this experiment, we focus on the MAGBERT and MAGXLNET since they are the best performing models. Because of space constraints results corresponding to MAGXLNET
Revised: In this experiment, we Spoken Transcripts

Original: Acoustic and visual behaviour
Revised: Acoustic and visual behaviour

Original: T θ um the story was all right low energy monotonous voice + headshake
Revised: T θ um the story was all right low energy monotonous voice + headshake

Original: L i mean its a Nicholas Sparks book it must be good disappointed tone + neutral facial expression
Revised: L i mean its a Nicholas Sparks book it must be good disappointed tone + neutral facial expression

Original: L the action is fucking awesome head nod + excited voice
Revised: L the action is fucking awesome head nod + excited voice

Original: H it was cute
Revised: H it was cute

Original: you know the actors did a great job bringing the smurfs to life such as joe george lopez neil patrick harris katy perry and a fourth multiple smiles
Revised: you know the actors did a great job bringing the smurfs to life such as joe george lopez neil patrick harris katy perry and a fourth multiple smiles

Original: H
Revised: H

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] A cc c o rr u p t /A
Revised: p t /

Original: [Figure]
Revised: [Figure]

Original: are reported in Appendix A.
Revised: MAGBERT f kl focus on the MAGBERT and MAGXLNET since theyare the best performing models.

Original:  As previously the considered models are trained using the losses described in sec.
Revised: 5 As before, theconsidered models are trained using the losses described in sec.

Original: 3 and all modalities are kept during training time.
Revised: 3 and all modalities are kept duringtraining time.

Original: During inference, we either keep only one modality (namely Audio or Video) or both.
Revised: During inference, we either keeponly one modality (Audio or Video) or both.

Original: Language is always dropped.
Revised: Textmodality is always dropped.

Original: Results .
Revised: Results .

Original: Results of thedescribed experiments conducted on CMU-MOSI are shown in Fig.
Revised: Results of the experiments conducted on CMU-MOSI are shown in Fig.

Original: 2 in term of ratioAcc corrupt/
Revised: 2, giving values forthe ratio

Original: Acc  2 where Acc corruptis the obtained binary accuracy in the corrupted conﬁguration and Acc 2 the one obtained when all modalities are considered.
Revised: Acc corrupt/Acc 2 where Acc corruptis thebinary accuracy in the corrupted conﬁguration and Acc 2 the accuracy obtained when all modalities areconsidered.

Original: We observe that models trained with MDM penalty (either L kl , L f or L W ) resists better to the perturbation compared to L ∅ .
Revised: We observe that models trained withan MDM penalty (either L kl , L f or L W ) resist better to missing modalities than those trained with

Original:  For example, when trained with L kl , L f the drop is limited to ≈ 25% in any settings.
Revised: L ∅ . For example, when trained with L kl or L f , thedrop in performance is limited to ≈ 25% in anysetting.

Original: Interestingly, for MAGBERT L W and L KL achieve comparable results;the L KL is more resistant to  language drop thus could be preferred in practical applications.
Revised: Interestingly, for MAGBERT L W and L KLachieve comparable results; L KL is more resistant to dropping the language modality, and thus, could be preferred in practical applications.

Original: Overall this experiment showcases the  information transfer allowed by the maximisation of the multivariate dependency measure.
Revised: Takeaway: Maximising the MMD allows an information transfer between modalities.

Original: 5. Towards explainable representations
Revised: 5. Towards explainable representations

Original: In this section, we propose a qualitative experiment  to explain the deep neural classiﬁer.
Revised: In this section, we propose a qualitative experiment allowing us to interpret the predictions made by the deep neural classiﬁer.

Original: During training the statistical network is used to estimate the mutual dependency measures, using the surrogates introduced in Th.
Revised: During training, T θ estimates the mutual dependency measure, using the surrogates introduced in Th.

Original: However, the inference process only involves the classiﬁer, and the statistical network is unused.
Revised: However, the inference process only involves the classiﬁer, and T θ is unused.

Original: When looking at Eq. 6,

Original: Eq.
Revised: Eq.

Revised: 6, Eq.

Original: 7, Eq.
Revised: 7, Eq.

Original: 8 one can observe that T θ is trained to discriminate between valid representations (coming from the joint distribution) and corrupted representations (coming from the product of the marginals).
Revised: 8 show that T θ is trained to discriminate between valid representations (coming from the joint distribution) and corrupted representations (coming from the product of the marginals).

Original: Thus T θ measures the mutual dependency of the representations learnt by the network.
Revised: Thus, T θ can be used, at inference time, to measure the mutual dependency of the representations used by the neural model.

Original: In Tab.we report examples of low and high discrepancy measures for MAGBERT on CMU-MOSI .
Revised: In Tab. we report examples of low and high discrepancy measures for MAGBERT on CMU-MOSI .

Original: We can observe that high values correspond to video clips where audio, text and video are complementary ( e.g use of head node (McClave, 2000)) and low values correspond to the case where there exists contradictions across several modalities.
Revised: We can observe that high values correspond to video clips where audio, text and video are complementary ( e.g use of head node (McClave, 2000)) and low values correspond to the case where there exists contradictions across several modalities.

Original: Results on MAGXLET can be found in sec.
Revised: Results on MAGXLET can be found in sec.

Original: A.3.
Revised: A.3.

Revised: Takeaways: T θ used to estimate the MDM provides a mean to interpret representations learnt by the encoder.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper we introduced three new losses based on Multivariate Mutual Dependency Measure.
Revised: In this paper, we introduced three new losses based on MDM.

Original: Through extensive set of experiments on CMU-MOSI and CMU-MOSEI , we have shown that for a large set of state-of-the-art architectures our approach improves both the accuracy and the robustness of the models.
Revised: Through extensive set of experiments on CMU-MOSI and CMU-MOSEI , we have shown that SOTA architectures can beneﬁt from these innovations with little modiﬁcations.

Original: A by-product of our method involve a statistical network that is a powerful tool to explain the learnt high dimensional multi-modal representations.
Revised: A by-product of our method involves a statistical network that is a useful tool to explain the learnt high dimensional multi-modal representations.

Original: This work leverages recent advances in mutual dependency estimations, it shows that SOTA architecture can beneﬁts from these innovations with little modiﬁcations.

Original: This work pave the way for using and developing new methods to estimate mutual dependency in a multivariate setting.
Revised: This work paves the way for using and developing new methods to estimate mutual dependency in a multivariate setting.

Original: A SupplementaryA. Training details
Revised: A Supplementary

Revised: A. Training details

Original: In this section, we both present a comprehensive illustration of the Algorithm 1 and state the details of experimental hyperparameters selection as well as and the architectures used for the statistic network
Revised: In this section, we both present a comprehensive illustration of the Algorithm 1 and state the details ofexperimental hyperparameters selection as well asand the architectures used for the statistic network

Original: T θ . A.1. Illustration of Algorithm 
Revised: T θ . A.1. Illustration of Algorithm 1

Original: Fig.
Revised: Fig.

Original: 3 describes the Algorithm 1.
Revised: 3 describes the Algorithm 1.

Original: As can be seen in the ﬁgure, to compute the mutual dependency measure the statistic network T θ takes the two embeddings of the different batch B and ¯ B .
Revised: As can be seenin the ﬁgure, to compute the mutual dependencymeasure the statistic network T θ takes the two embeddings of the different batch B and ¯ B .

Original: f ✓ c <latexit sha1_base64="PM3jgCJ6O15jTYfYVvsRMSDmJd0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0mkoMeiF48V7Ac0oWy2k3bpZhN2J0IJ/RtePCji1T [Table] sha
Revised: f ✓ c <latexit sha1_base64="PM3jgCJ6O15jTYfYVvsRMSDmJd0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0mkoMeiF48V7Ac0oWy2k3bpZhN2J0IJ/RtePCji1T/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTHNo8kYnuhcyAFAraKFBCL9XA4lBCN5zczf3uE2gjEvWI0xSCmI2UiARnaCU/GuQ+jgHZgM8G1Zpbdxeg68QrSI0UaA2qX/4w4VkMCrlkxvQ9N8UgZxoFlzCr+JmBlPEJG0HfUsViMEG+uHlGL6wypFGibSmkC/X3RM5iY6ZxaDtjhmOz6s3F/7x+htFNkAuVZgiKLxdFmaSY0HkAdCg0cJRTSxjXwt5K+ZhpxtHGVLEheKsvr5POVd1r1BsPjVrztoijTM7IObkkHrkmTXJPWqRNOEnJM3klb07mvDjvzseyteQUM6fkD5zPH2GYkeo=</latexit>

Original: [Table]
Revised: [Table]

Original: < [Table] /vEqalxWvWqneV8u1mzyOApzCGVyAB1dQgzuoQwMoPMEzvMKbM3ZenHfnY9G65uQzJ/
Revised: AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokU9Fj04rGC/cA2lMl20y7dbOLuplBCf4cXD4p49cd489+4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6jhVlDVoLGLVDlAzwSVrGG4EayeKYRQI1gpGtzO/NWZK81g+mEnC/AgHkoecorGS3w1QZY/TXiZwPO2Vym7FnYOsEi8nZchR75W+uv2YphGThgrUuuO5ifEzVIZTwabFbqpZgnSEA9axVGLEtJ/Nj56Sc6v0SRgrW9KQufp7IsNI60kU2M4IzVAvezPxP6+TmvDaz7hMUsMkXSwKU0FMTGYJkD5XjBoxsQSp4vZWQoeokBqbU9GG4C2/vEqalxWvWqneV8u1mzyOApzCGVyAB1dQgzuoQwMoPMEzvMKbM3ZenHfnY9G65uQzJ/

Original: AHzucPX66SgA==</latexit>
Revised: AHzucPX66SgA==</latexit>

Original: A.1. Hyperparameters selection
Revised: A.1. Hyperparameters selection

Original: We use dropout (Srivastava et al., 2014) and optimise the global loss Eq.
Revised: We use dropout (Srivastava et al., 2014) and optimise the global loss Eq.

Original: 5 by gradient descent using
Revised: 5 by gradient descent using

Original: AdamW (Loshchilov and Hutter, 2017; Kingma and Ba, 2014) optimiser.
Revised: AdamW (Loshchilov and Hutter, 2017; Kingmaand Ba, 2014) optimiser.

Original: The best learning rate is found in the grid { 0 . 002 , 0 .001 , 0 . 0005 , 0 . 0001 } .
Revised: The best learning rate isfound in the grid { 0 . 002 , 0 .

Original: The best model is selected using the lowest MAE on the validation set.
Revised: The best model is selected using the lowest MAEon the validation set.

Revised: We Unroll to 10.

Original: A.1. Architectures of Tθ
Revised: A.1. Architectures of T

Revised: θ

Original: Across the different experiment we use a statistic network with an architecture as describes in
Revised: Across the different experiment we use a statistic network with an architecture as describes in

Original: Tab. 4.
Revised: Tab. 4.

Original: We follow (Belghazi et al., 2018) and use
Revised: We follow (Belghazi et al., 2018) and use

Original: LeakyRELU (Agarap, 2018; Xu et al., 2015) as activation function.
Revised: LeakyRELU (Agarap, 2018; Xu et al., 2015) asactivation function.

Original: Statistic Network
Revised: [Table]

Original: Layer
Revised: Layer

Original: Number of outputs Activation function [ Z lav ,Z lav ] d in ,d inDense layer d in / LeakyReLU Dropout 0.
Revised: [Table]

Original: Dense layer d in /

Original: LeakyReLU

Original: Dropout 0.

Original: Dense layer d in /

Original: LeakyReLU Dropout 0. Dense layer Sigmo¨ıd
Revised: [Table]

Original: A. Additional experiments for robustness tomodality drop
Revised: A. Additional experiments for robustness to

Revised: modality drop

Original: Fig.
Revised: Fig.

Original: 4 shows the results of the robustness text on
Revised: 4 shows the results of the robustness text on

Original: MAGXLNET .
Revised: MAGXLNET .

Original: Similarly to Fig.
Revised: Similarly to Fig.

Original: 2 we observe more robust representation to modality drop when jointly maximising the L W and L kl with the target loss.
Revised: 2 we observe more robust representation to modality drop when jointly maximising the L W and L kl with the target loss.

Original: Fig.
Revised: Fig.

Original: 4 shows no improvement when training with
Revised: 4 shows no improvement when training with

Original: L f .
Revised: L f .

Original: This can also be linked to Tab. 2 which similarly shows no improvement in this very speciﬁc conﬁguration. 
Revised: This can also be linked to Tab. 2 which similarly shows no improvement in this very speciﬁc conﬁguration. [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: A cc c o rr u p t / A cc

Original: MAGXLNET kl f
Revised: MAGXLNET kl f

Original: A. Additional qualitative examples
Revised: A. Additional qualitative examples

Original: Tab. 5 illustrates the use of T θ to explain the representations learnt by the model.
Revised: Tab. 5 illustrates the use of T θ to explain the representations learnt by the model.

Original: Similarly to Tab.we observe that high values correspond to complementarity across modalities and low values are related to contradictoriness across modalities.
Revised: Similarly to Tab.  we observe that high values correspond to complementarity across modalities and low values arerelated to contradictoriness across modalities.

Original: Spoken Transcripts
Revised: Spoken Transcripts

Original: Acoustic and visual behaviour
Revised: [Table] behaviour

Original: T θ
Revised: T θ

Original: but the m the script is corny high energy voice + headshake + (many) smiles
Revised: but the m the script is corny [Table] + [Table]

Revised: many

Revised: [Table]

Original:  L as for gi joe was it was just like laughing its the the plot the the acting is terrible high enery voice + laughts + smiles
Revised: smiles L as for gi joe was it was just like laughing its the the plot the the acting is terrible high [Table] smiles

Original: L
Revised: L

Original: but i think this one did beat scream 2 now headshake + long sigh
Revised: but i think this one did beat scream 2 now headshake [Table] sigh

Original: L the xxx sequence is really well done static head +low energy monotonous voice
Revised: L the xxx sequence is really well done static [Table] +

Original:  L you know of course i was waithing for the princess and the frog smiles + high energy voice + + high pitch
Revised: [Table] monotonous voice L you know of course i was waithing for the princess and the frog smiles + [Table]

Original:  H dennis quaid i think had a lot of fun smiles +high energy voice
Revised: energy voice + + high pitch H dennis quaid i think had a lot of fun smiles +

Original:  H it was very very very boring low energy voice + frown eyebrows
Revised: [Table] energy voice H it was very very very boring [Table]

Original:  H i do not wanna see any more of this angry voice + angry facial expression
Revised: eyebrows H i do not wanna see any more of this [Table] expression

Original: H
Revised: H