Original: Improving Multimodal fusion via Mutual Dependency Maximisation
Revised: Improving Multimodal fusion via Mutual Dependency Maximisation

Original: Anonymous EMNLP submission

Original: Abstract
Revised: Abstract

Original: Multimodal sentiment analysis is a trendingarea of research, and the multimodal fusionis one of its most active topic.
Revised: Multimodal sentiment analysis is a trending area of research, and the multimodal fusion is one of its most active topic.

Original: Acknowledging humans communicate through a variety ofchannels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating differentunimodal representations into a synthetic one.
Revised: Acknowledging humans communicate through a variety of channels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating diﬀerent unimodal representations into a synthetic one.

Original: So far, a consequent effort has been made ondeveloping complex architectures allowing thefusion of these modalities.
Revised: So far, a consequent eﬀort has been made on developing complex architectures allowing the fusion of these modalities.

Original: However, such systems are mainly trained by minimising simple losses such as L 1 or cross-entropy.
Revised: However, such systems are mainly trained by minimising simple losses such as L 1 or crossentropy.

Original: Inthis work, we investigate unexplored penaltiesand propose a set of new objectives that measure the dependency between modalities.
Revised: In this work, we investigate unexplored penalties and propose a set of new objectives that measure the dependency between modalities.

Original: Wedemonstrate that our new penalties lead to aconsistent improvement (up to 4 . 3 on accuracy) across a large variety of state-of-the-artmodels on two well-known sentiment analysisdatasets: CMU-MOSI and CMU-MOSEI .
Revised: We demonstrate that our new penalties lead to a consistent improvement (up to 4 . 3 on accuracy) across a large variety of state-of-the-art models on two well-known sentiment analysis datasets: CMU-MOSI and CMU-MOSEI .

Original: Ourmethod not only achieves a new SOTA on bothdatasets but also produces representations thatare more robust to modality drops.
Revised: Our method not only achieves a new SOTA on both datasets but also produces representations that are more robust to modality drops.

Original: Finally, aby-product of our methods includes a statistical network which can be used to interpret thehigh dimensional representations learnt by themodel.
Revised: Finally, a by-product of our methods includes a statistical network which can be used to interpret the high dimensional representations learnt by the model.

Original:  Introduction
Revised:  Introduction

Original: Humans employ three different modalities to communicate in a coordinated manner: the languagemodality with the use of words and sentences, thevision modality with gestures, poses and facial expressions and the acoustic modality through changein vocal tones.
Revised: Humans employ three diﬀerent modalities to communicate in a coordinated manner: the language modality with the use of words and sentences, the vision modality with gestures, poses and facial expressions and the acoustic modality through change in vocal tones.

Original: Multimodal representation learninghas shown great progress in a large variety of tasksincluding emotion recognition, sentiment analysis (Soleymani et al., 2017), speaker trait analysis(Park et al., 2014) and ﬁne-grained opinion mining (Garcia et al., 2019a).
Revised: Multimodal representation learning has shown great progress in a large variety of tasks including emotion recognition, sentiment analysis (Soleymani et al., 2017), speaker trait analysis (Park et al., 2014) and ﬁne-grained opinion mining (Garcia et al., 2019a).

Original: Learning from differentmodalities is an efﬁcient way to improve performance on the target tasks (Xu et al., 2013).
Revised: Learning from diﬀerent modalities is an eﬃcient way to improve performance on the target tasks (Xu et al., 2013).

Original: Nevertheless, heterogeneities across modalities increasethe difﬁculty of learning multimodal representations and raise speciﬁc challenges.
Revised: Nevertheless, heterogeneities across modalities increase the diﬃculty of learning multimodal representations and raise speciﬁc challenges.

Original: Baltruˇsaitis et al.
Revised: Baltruˇsaitis et al.

Original: (2018) identiﬁes fusion as one of the ﬁve core challenges in multimodal representation learning, the four other being: representation, modality alignment, translation and co-learning.
Revised: (2018) identiﬁes fusion as one of the ﬁve core challenges in multimodal representation learning, the four other being: representation, modality alignment, translation and co-learning.

Original: Fusion aims at integrating the different unimodal representations into one common synthetic representation.
Revised: Fusion aims at integrating the diﬀerent unimodal representations into one common synthetic representation.

Original: Effective fusion is still an open problem: the best multimodal models in sentiment analysis (Rahman et al., 2020) improve over their unimodal counterparts, relying on text modality only, by less than 1.5% on accuracy.
Revised: Eﬀective fusion is still an open problem: the best multimodal models in sentiment analysis (Rahman et al., 2020) improve over their unimodal counterparts, relying on text modality only, by less than 1.5% on accuracy.

Original: Additionally, the fusion should not only improve accuracy but also make representations more robust to missing modalities.
Revised: Additionally, the fusion should not only improve accuracy but also make representations more robust to missing modalities.

Original: Multimodal fusion can be divided into early and late fusion techniques: early fusion takes place at the feature level (Ye et al., 2017), while late fusion takes place at the decision or scoring level (Khan et al., 2012).
Revised: Multimodal fusion can be divided into early and late fusion techniques: early fusion takes place at the feature level (Ye et al., 2017), while late fusion takes place at the decision or scoring level (Khan et al., 2012).

Original: Current research in multimodal sentiment analysis mainly focuses on developing new fusion mechanisms relying on deep architectures ( e.gTFN (Zadeh et al., 2017),
Revised: Current research in multimodal sentiment analysis mainly focuses on developing new fusion mechanisms relying on deep architectures ( e.g

Original:  LFN (Liu et al., 2018), MARN (Zadeh et al., 2018b), MISA (Hazarika et al., 2020), MCTN (Pham et al., 2019), HFNN (Mai et al., 2019), ICCN (Sun et al., 2020)).
Revised: TFN (Zadeh et al., 2017), LFN (Liu et al., 2018), MARN (Zadeh et al., 2018b), MISA (Hazarika et al., 2020), MCTN (Pham et al., 2019), HFNN (Mai et al., 2019), ICCN (Sun et al., 2020)).

Original: Theses models are evaluated on several multimodal sentiment analysis benchmark such as IEMOCAP (Busso et al., 2008), MOSI (W¨ollmer et al., 2013), 
Revised: Theses models are evaluated on several multimodal sentiment analysis benchmark such as IEMOCAP (Busso et al., 2008), MOSI (W¨ollmer et al., 2013), MOSEI (Zadeh et al., 2018c) and

Original: MOSEI (Zadeh et al., 2018c) and POM (Garcia et al., 2019b; Park et al., 2014).
Revised: POM (Garcia et al., 2019b; Park et al., 2014).

Original: Current state-ofthe-art on these datasets uses architectures based on pre-trained transformers (Tsai et al., 2019; Siriwardhana et al., 2020) such as MultiModal Bert ( MAGBERT ) or MultiModal XLNET ( MAGXLNET ) (Rahman et al., 2020).
Revised: Current state-of-the-art on these datasets uses architectures based on pre-trained transformers (Tsai et al., 2019; Siriwardhana et al., 2020) such as MultiModal Bert ( MAGBERT ) or Mul- tiModal XLNET ( MAGXLNET ) (Rahman et al., 2020).

Original: The aforementioned architectures are trained by minimising either a L 1 loss or a Cross-Entropy loss between the predictions and the ground-truth labels.
Revised: The aforementioned architectures are trained by minimising either a L 1 loss or a CrossEntropy loss between the predictions and the ground-truth labels.

Original: To the best of our knowledge, few efforts have been dedicated to exploring alternative losses.
Revised: To the best of our knowledge, few eﬀorts have been dedicated to exploring alternative losses.

Original: In this work, we propose a set of new objectivesto perform and improve over existing fusion mechanisms.
Revised: In this work, we propose a set of new objectives to perform and improve over existing fusion mechanisms.

Original: These improvements are inspired by the InfoMax principle (Linsker, 1988), i.e. choosingthe representation maximising the mutual information (MI) between two possibly overlapping viewsof the input.
Revised: These improvements are inspired by the InfoMax principle (Linsker, 1988), i.e. choosing the representation maximising the mutual information (MI) between two possibly overlapping views of the input.

Original: The MI quantiﬁes the dependenceof two random variables; contrarily to correlation, MI also captures non-linear dependencies betweenthe considered variables.
Revised: The MI quantiﬁes the dependence of two random variables; contrarily to correlation, MI also captures non-linear dependencies between the considered variables.

Original: Different from previouswork, which mainly focuses on comparing twomodalities, our learning problem involves multiplemodalities ( e.g text, audio, video).
Revised: Diﬀerent from previous work, which mainly focuses on comparing two modalities, our learning problem involves multiple modalities ( e.g text, audio, video).

Original: Our proposedmethod, which induces no architectural changes,relies on jointly optimising the target loss with anadditional penalty term measuring the mutual dependency between different modalities.
Revised: Our proposed method, which induces no architectural changes, relies on jointly optimising the target loss with an additional penalty term measuring the mutual dependency between diﬀerent modalities.

Original: 1. Our Contributions
Revised: 1. Our Contributions

Original: We study new objectives to build more performantand robust multimodal representations through anenhanced fusion mechanism and evaluate them onmultimodal sentiment analysis.
Revised: We study new objectives to build more performant and robust multimodal representations through an enhanced fusion mechanism and evaluate them on multimodal sentiment analysis.

Original: Our method alsoallows us to explain the learnt high dimensionalmultimodal embeddings.
Revised: Our method also allows us to explain the learnt high dimensional multimodal embeddings.

Original: The paper contributionscan be summarised as follows:
Revised: The paper contributions can be summarised as follows:

Original: A set of novel objectives using multivariate de- pendency measures .
Revised: A set of novel objectives using multivariate dependency measures .

Original: We introduce three new trainable surrogates to maximise the mutual de- pendencies between the three modalities ( i.e audio, language and video).
Revised: We introduce three new trainable surrogates to maximise the mutual dependencies between the three modalities ( i.e audio, language and video).

Original: We provide a general algo- rithm inspired by MINE (Belghazi et al., 2018) which was developed in a bi-variate setting for esti- mating the MI.
Revised: We provide a general algorithm inspired by MINE (Belghazi et al., 2018), which was developed in a bi-variate setting for estimating the MI.

Original: Our new method enriches MINE by extending the procedure to a multivariate setting that allows us to maximise different Mutual Depen- dency Measures: the Total Correlation (Watanabe, 
Revised: Our new method enriches MINE by extending the procedure to a multivariate setting that allows us to maximise diﬀerent Mutual Dependency Measures: the Total Correlation (Watanabe, 1960), the f-Total Correlation and the Multivariate Wasserstein

Original: 1960), the f-Total Correlation and the Multivari-

Original: ate Wasserstein Dependency Measure (Ozair et al., 
Revised: Dependency Measure (Ozair et al., 2019).

Original: Applications and numerical results .
Revised: Applications and numerical results .

Original: We apply

Original:  our new set of objectives to ﬁve different architectures relying on LSTM cells (Huang et al.,2015) ( e.g EF-LSTM , LFN , MFN ) or transformerlayers ( e.g MAGBERT , MAG-XLNET ).
Revised: We apply our new set of objectives to ﬁve different architectures relying on LSTM cells (Huang et al., 2015) ( e.g EF-LSTM , LFN , MFN ) or transformer layers ( e.g MAGBERT , MAG-XLNET ).

Original: Our proposed method (1) brings a substantial improvementon two different multimodal sentiment analysisdatasets ( i.e MOSI and MOSEI ,sec. 5.1), (2) makesthe encoder more robust to missing modalities ( i.ewhen predicting without language, audio or videothe observed performance drop is smaller, sec. 5.3),(3) provides an explanation of the decision taken by the neural architecture (sec. 5.4).
Revised: Our proposed method (1) brings a substantial improvement on two diﬀerent multimodal sentiment analysis datasets ( i.e MOSI and MOSEI ,sec. 5.1), (2) makes the encoder more robust to missing modalities ( i.e when predicting without language, audio or video the observed performance drop is smaller, sec. 5.3), (3) provides an explanation of the decision taken by the neural architecture (sec. 5.4).

Original:  Problem formulation & related work
Revised:  Problem formulation & related work

Original: In this section, we formulate the problem of learning multi-modal representation (sec. 2.1) and we review both existing measures of mutual dependency (see sec. 2.2) and estimation methods (sec. 2.3).
Revised: In this section, we formulate the problem of learning multi-modal representation (sec. 2.1) and we review both existing measures of mutual dependency (see sec. 2.2) and estimation methods (sec. 2.3).

Original: In the rest of the paper, we will focus on learning from three modalities ( i.e language, audio and video), however our approach can be generalised to any arbitrary number of modalities.
Revised: In the rest of the paper, we will focus on learning from three modalities ( i.e language, audio and video), however our approach can be generalised to any arbitrary number of modalities.

Original: 2. Learning multimodal representations
Revised: 2. Learning multimodal representations

Original: Plethora of neural architectures have been proposed to learn multimodal representations for sentiment classiﬁcation.
Revised: Plethora of neural architectures have been proposed to learn multimodal representations for sentiment classiﬁcation.

Original: Models often rely on a fusion mechanism ( e.g multi-layer perceptron (Khan et al., 2012), tensor factorisation (Liu et al., 2018;
Revised: Models often rely on a fusion mechanism ( e.g multi-layer perceptron (Khan et al., 2012), tensor factorisation (Liu et al., 2018;

Original: Zadeh et al., 2019) or complex attention mechanisms (Zadeh et al., 2018a)) that is fed with modalityspeciﬁc representations.
Revised: Zadeh et al., 2019) or complex attention mechanisms (Zadeh et al., 2018a)) that is fed with modality-speciﬁc representations.

Original: The fusion problem boils down to learning a model M f : X a × X v × X l → R d .
Revised: The fusion problem boils down to learning a model M f : X a × X v × X l → R d .

Original: M f is fed with uni-modal representations of the inputs X a,v,l = ( X a , X v , X l ) obtained through three embedding networks f a , f v and f l .
Revised: M f is fed with uni-modal representations of the inputs [Equation] obtained through three embedding networks f a , f v and f l .

Original: M f has to retain both modality-speciﬁc interactions ( i.e interactions that involve only one modality) and cross-view interactions ( i.e more complex, they span across both views).
Revised: M f has to retain both modality-speciﬁc interactions ( i.e interactions that involve only one modality) and cross-view interactions ( i.e more complex, they span across both views).

Original: Overall, the learning of M f involves both the minimisation of the downstream task loss and the maximisation of the mutual dependency between the different modalities.
Revised: Overall, the learning of M f involves both the minimisation of the downstream task loss and the maximisation of the mutual dependency between the diﬀerent modalities.

Original: 2. Mutual dependency maximisation
Revised: 2. Mutual dependency maximisation

Original: Mutual information as mutual dependency
Revised: Mutual information as mutual depen

Original:  measure: the core ideas we rely on to better learncross-view interactions are not new.
Revised: - dency measure: the core ideas we rely on to better learn cross-view interactions are not new.

Original: They consist of mutual information maximisation (Linsker, 1988), and deep representation learning.
Revised: They consist of mutual information maximisation (Linsker, 1988), and deep representation learning.

Original: Thus, one of the most natural choices is to use the MI that measures the dependence between two random variables, including high-order statistical dependencies (Kinney and Atwal, 2014).
Revised: Thus, one of the most natural choices is to use the MI that measures the dependence between two random variables, including high-order statistical dependencies (Kinney and Atwal, 2014).

Original: Given two random variables X and Y , the MI is deﬁned by
Revised: Given two random variables X and Y , the MI is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where p XY is the joint probability density function (pdf) of the random variables ( X, Y ) , and p X , p Y represent the marginal pdfs.
Revised: where p XY is the joint probability density function (pdf) of the random variables ( X, Y ), and p X , p Y represent the marginal pdfs.

Original: MI can also be deﬁnedwith a the KL divergence:
Revised: MI can also be deﬁned with a the KL divergence:

Original: [Equation]
Revised: [Equation]

Original: Extension of mutual dependency to differentmetrics: the KL divergence seems to be limitedwhen used for estimating MI (McAllester and Stratos, 2020).
Revised: Extension of mutual dependency to different metrics: the KL divergence seems to be limited when used for estimating MI (McAllester and Stratos, 2020).

Original: A natural step is to replace the KL divergence in Eq.
Revised: A natural step is to replace the KL divergence in Eq.

Original: 2 with different divergencessuch as the f-divergences or distances such as the Wasserstein distance.
Revised: 2 with diﬀerent divergences such as the f-divergences or distances such as the Wasserstein distance.

Original: Hence, we introduce new mutual dependency measures (MDM): the f-Mutual Information (Belghazi et al., 2018), denoted I f andthe Wasserstein Measures (Ozair et al., 2019), denoted I W .
Revised: Hence, we introduce new mutual dependency measures (MDM): the f-Mutual Information (Belghazi et al., 2018), denoted I f and the Wasserstein Measures (Ozair et al., 2019), denoted I W .

Original: As previously, p XY denotes the jointpdf, and p X , p Y denote the marginal pdfs.
Revised: As previously, p XY denotes the joint pdf, and p X , p Y denote the marginal pdfs.

Original: The newmeasures are deﬁned as follows:
Revised: The new measures are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where D f denotes any f -divergences and
Revised: where D f denotes any f -divergences and

Original: [Equation]
Revised: [Equation]

Original: where W denotes the Wasserstein distance (Peyr´eet al., 2019).
Revised: where W denotes the Wasserstein distance (Peyr´e et al., 2019).

Original: 2. Estimating mutual dependency measures
Revised: 2. Estimating mutual dependency measures

Original: The computation of MI and other mutual dependency measures can be difﬁcult without knowingthe marginal and joint probability distributions,thus it is popular to maximise lower bounds toobtain better representations  of different modalities including image (Tian et al., 2019; Hjelmet{+
 ; Hjelm et+} al., 2018), audio (Dilpazir et al., 2016) and text(Kong et al., 2019) data.
Revised: The computation of MI and other mutual dependency measures can be diﬃcult without knowing the marginal and joint probability distributions, thus it is popular to maximise lower bounds to obtain better representations [Figure] of diﬀerent modalities including image (Tian et al., {+
 ; Hjelm et+} al., 2018), audio (Dilpazir et al., 2016) and text (Kong et al., 2019) data.

Original: Several estimators havebeen proposed:MINE (Belghazi et al., 2018)
Revised: Several estimators have been proposed:

Original: usesthe Donsker-Varadhan representation (Donsker and Varadhan, 1985) to derive a parametric lower boundholds, Nguyen et al.
Revised: MINE (Belghazi et al., 2018) uses the DonskerVaradhan representation (Donsker and Varadhan, 1985) to derive a parametric lower bound holds, Nguyen et al.

Original: (2017, 2010) uses variationalcharacterisation of f-divergence and a multi-sampleversion of the density ratio (also known as noisecontrastive estimation (Oord et al., 2018; Ozairet al., 2019)).
Revised: (2017, 2010) uses variational characterisation of f-divergence and a multi-sample version of the density ratio (also known as noise contrastive estimation (Oord et al., 2018; Ozair et al., 2019)).

Original: These methods have mostly beendeveloped and studied in a bi-variate setting.
Revised: These methods have mostly been developed and studied in a bi-variate setting.

Original: Illustration of neural dependency measures ona bivariate case .
Revised: Illustration of neural dependency measures on a bivariate case .

Original: In Fig.
Revised: In Fig.

Original: 1 we can see the aforementioned dependency measures ( i.e see Eq.
Revised: 1 we can see the aforementioned dependency measures ( i.e see Eq.

Original: 2, Eq. 4, Eq. 3) when estimated with MINE (Belghazi et al.,2018) for multivariate Gaussian random variables, X a and X b .
Revised: 2, Eq. 4, Eq. 3) when estimated with MINE (Belghazi et al., 2018) for multivariate Gaussian random variables, X a and X b .

Original: The component wise correlation forthe considered multivariate Gaussian is deﬁned as[Figure] follow: corr ( X i , X k )
Revised: The component wise correlation for the considered multivariate Gaussian is deﬁned as follow: corr ( X i , X k )

Original: =
Revised: =

Original: δ i,k ρ , where ρ ∈ ( − 1 , 1) and δ i,k is Kronecker’s delta.
Revised: δ i,k ρ , where ρ ∈ ( − 1 , 1) and δ i,k is Kronecker’s delta.

Original: We observe that the dependency measure based on Wasserstein distance is different from the one based on the divergences and thus will lead to different gradients.
Revised: We observe that the dependency measure based on Wasserstein distance is diﬀerent from the one based on the divergences and thus will lead to diﬀerent gradients.

Original: Although theoretical studies have been done on the use of different metrics for dependency estimations, it remains an open question to know which one is the best suited.
Revised: Although theoretical studies have been done on the use of diﬀerent metrics for dependency estimations, it remains an open question to know which one is the best suited.

Original: In this work, we will provide an experimental response in a speciﬁc case.
Revised: In this work, we will provide an experimental response in a speciﬁc case.

Original:  Model and training objective
Revised:  Model and training objective

Original: In this section, we introduce our new set of losses to improve fusion.
Revised: In this section, we introduce our new set of losses to improve fusion.

Original: In sec. 3.1, we ﬁrst extend widely used bi-variate dependency measures to multivariate dependencies (James and Crutchﬁeld, 2017) measures (MDM).
Revised: In sec. 3.1, we ﬁrst extend widely used bi-variate dependency measures to multivariate dependencies (James and Crutchﬁeld, 2017) measures (MDM).

Original: We then introduce variational bounds on the MDM, and in sec.
Revised: We then introduce variational bounds on the MDM, and in sec.

Original: 3.2, we describe our method to minimise the proposed variational bounds.
Revised: 3.2, we describe our method to minimise the proposed variational bounds.

Original: Notations We consider X a , X v , X l as the multimodal data from the audio,video and language modality respectively with joint probability distribution p X a X
Revised: Notations We consider X a , X v , X l as the multimodal data from the audio,video and language modality respectively with joint probability distribution p X a X

Original: v X l .
Revised: v X l .

Original: We denote as p X j the marginal distribution of X j with j ∈ { a, v, l } corresponding to the j th modality.
Revised: We denote as p X j the marginal distribution of X j with j ∈ { a, v, l } corresponding to the j th modality.

Original: General loss As previously mentioned, we rely on the InfoMax principle (Linsker, 1988) and aim at jointly maximising the MDM between the different modalities and minimising the task loss; hence, we are in a multi-task setting (Argyriou et al., 2007; Ruder, 2017) and the objective of interest can be deﬁned as:
Revised: General loss As previously mentioned, we rely on the InfoMax principle (Linsker, 1988) and aim at jointly maximising the MDM between the diﬀerent modalities and minimising the task loss; hence, we are in a multi-task setting (Argyriou et al., 2007; Ruder, 2017) and the objective of interest can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: L down.
Revised: L down.

Original: represents a downstream speciﬁc (targettask) loss i.e a binary cross-entropy or a L 1 loss, λis a meta-parameter and L MDM is the multivariatedependencies measures (see sec. 3.2).
Revised: represents a downstream speciﬁc (target task) loss i.e a binary cross-entropy or a L 1 loss, λ is a meta-parameter and L MDM is the multivariate dependencies measures (see sec. 3.2).

Original: Minimisation of our newly deﬁned objectives requires toderive lower bounds on the L MDM terms, and thento obtain trainable surrogates.
Revised: Minimisation of our newly deﬁned objectives requires to derive lower bounds on the L MDM terms, and then to obtain trainable surrogates.

Original: 3. From bivariate to multivariatedependencies
Revised: 3. From bivariate to multivariate dependencies

Original: In our setting, we aim at maximising cross-viewinteractions involving three modalities, thus weneed to generalise bivariate dependency measuresto multivariate dependency measures.
Revised: In our setting, we aim at maximising crossview interactions involving three modalities, thus we need to generalise bivariate dependency measures to multivariate dependency measures.

Original: Deﬁnition 3.1 (Multivariate Dependencies Measures) .
Revised: Deﬁnition 3.1 (Multivariate Dependencies Measures) .

Original: Let X a , X v ,
Revised: Let X a , X v ,

Original: X
Revised: X

Original: l be a set of random variables with joint pdf p X a X v X l and respectivemarginal pdf p X j with j ∈ { a, v, l } .
Revised: l be a set of random variables with joint pdf p X a X v X l and respective marginal pdf p X j with j ∈ { a, v, l } .

Original: Then we deﬁned the multivariate mutual information I kl , alsorefered as total correlation (Watanabe, 1960) ormulti-information (Studen`y and Vejnarov´a, 1998):
Revised: Then we deﬁned the multivariate mutual information I kl , also refered as total correlation (Watanabe, 1960) or multi-information (Studen`y and Vejnarov´a, 1998):

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Simarly for any f-divergence we deﬁne the multivariate f-mutual information I f as:
Revised: Simarly for any f-divergence we deﬁne the multivariate f-mutual information I f as:

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: Finally, we also extend Eq.
Revised: Finally, we also extend Eq.

Original: 3 to obtain the multivariate Wasserstein dependency measure
Revised: 3 to obtain the multivariate Wasserstein dependency measure

Original: I W :
Revised: I W :

Original: [Equation]
Revised: [Equation]

Original: p X j ( x j ))

Original: .where W denotes the Wasserstein distance.3.
Revised: where W denotes the Wasserstein distance.

Original:  From theoretical bounds to trainablesurrogates
Revised: 3. From theoretical bounds to trainable surrogates

Original: To train our neural architecture we need to estimate the previously deﬁned multivariate dependency measures.
Revised: To train our neural architecture we need to estimate the previously deﬁned multivariate dependency measures.

Original: We rely on neural estimators thatare given in Th.
Revised: We rely on neural estimators that are given in Th.

Original: Theorem 1.
Revised: Theorem 1.

Original: Multivariate Neural Dependency Measures Let the family of functions T ( θ ) :
Revised: Multivariate Neural Dependency Measures Let the family of functions T ( θ ) :

Original: X a × X v × X l
Revised: X a × X v × X l

Original: → R parametrized by a deep neural network with learnable parameters θ ∈ Θ .
Revised: → R parametrized by a deep neural network with learnable parameters θ ∈ Θ .

Original: The multivariate mutual information measure I kl is deﬁnedas:
Revised: The multivariate mutual information measure I kl is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: The neural multivariate f-mutual information measure I f is deﬁned as follows:
Revised: The neural multivariate f-mutual information measure I f is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: The neural multivariate Wasserstein dependency measure I W is deﬁned as follows:
Revised: The neural multivariate Wasserstein dependency measure I W is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Where L is the set of all 1-Lipschitz functions from R 
Revised: Where L is the set of all 1-Lipschitz functions from R d → R

Original: [Equation] Sketch of proofs: Eq. 6 is a direct application of the Donsker-Varadhan representation of the KL divergence (we assume that the integrability constraints are satisﬁed).
Revised: Sketch of proofs: Eq. 6 is a direct application of the Donsker-Varadhan representation of the KL divergence (we assume that the integrability constraints are satisﬁed).

Original: Eq.
Revised: Eq.

Original: 7 comes from the work of (Nguyen et al., 2017).
Revised: 7 comes from the work of (Nguyen et al., 2017).

Original: Eq. 8 comes from the Kantorovich-Rubenstein: we refer the reader to (Villani, 2008; Peyr´e et al., 2019) for a rigorous and exhaustive treatment.
Revised: Eq.comes from the Kantorovich-Rubenstein: we refer the reader to (Villani, 2008; Peyr´e et al., 2019) for a rigorous and exhaustive treatment.

Original: Practical estimate of the variational bounds.
Revised: Practical estimate of the variational bounds.

Original: The empirical estimator that we derive from Th.
Revised: The empirical estimator that we derive from Th.

Original:  can be used in practical way:the expectations in
Revised: 1 can be used in practical way:

Original: Eq. 6, Eq. 7 and Eq. 8 are estimated using empirical samples from the joint distribution p X a X v X l . The empirical samples from (cid:81) j ∈{ a,v,l } p X j are obtained by shufﬂing the samples from the joint distribution in a batch. We integrate this into minimising a multi-task objective (5) by using minus the estimator.
Revised: INPUT: D n = { ( x ja , x

Original: We refer to the losses obtained with the penalty based on the estimators described in Eq. 6, Eq. and Eq. as L kl , L f and L W respectively. Details on the practical minimisation of our variational bounds are provided in Algorithm 1.
Revised: [Equation]

Original: Remark.
Revised: [Equation]

Original: In this work we choose to generalise MINE to compute multivariate dependencies. Comparing our proposed algorithm to other alternatives mentioned in sec.
Revised: ) , ∀ j ∈ [1 , n

Original: 2 is left for future work. This choice is driven by two main reasons: (1) our framework allows the use of different types of contrast measures (e.g Wasserstein distance, fdivergences); (2) the critic network T θ can be used for interpretability purposes as shown in sec.  Experimental setting In this section, we present our experimental settings including the neural architectures we compare, the datasets, the metrics and our methodology, which includes the hyper-parameter selection.
Revised: [Equation]

Original: INPUT: D n = { ( x ja , x jv , x jl ) , ∀ j ∈ [1 , n ] } multimodal training dataset, m batch size, σ a , σ v , σ l : [1 , m ] → [1 , m ] three permutations, θ c weights of the deep classiﬁer, θ weights of the statistical network T θ .
Revised: } multimodal training dataset, m batch size, σ a , σ v , σ l : [1 , m ] → [1 , m ] three permutations, θ c weights of the deep classiﬁer, θ weights of the statistical network T θ .

Original: Initialization: parameters θ and θ c
Revised: Initialization: parameters θ and θ c

Original: Build Negative Dataset: ¯
Revised: Build Negative Dataset: ¯

Original: D n = { ( x σ a ( j ) a , x σ v ( j ) v , x σ l ( j ) l ) , ∀ j ∈ [1 , n ] }
Revised: [Equation] [Equation] , n [Equation]

Original: Optimization: while ( θ, θ c ) not converged do for i ∈ [1 , Unroll ] do
Revised: Optimization: while ( θ, θ c ) not converged do for i ∈ [1 , Unroll ] do

Original: Sample from D n , B ∼ p X a X v X l
Revised: Sample from D n , B ∼ p X a X v X l

Original: Sample from ¯ D n , ¯ B ∼ (cid:81) j ∈{ a,v,l } p X j
Revised: Sample from ¯ D n , ¯ B ∼ (cid:81) j ∈{ a,v,l } p X j

Original: Update θ based on the empirical version of Eq.
Revised: Update θ based on the empirical version of Eq.

Original: 6 or Eq.
Revised: 6 or Eq.

Original: 7 or Eq. 8.
Revised: 7 or Eq. 8.

Original: end for
Revised: end for

Original: Sample a batch B from D
Revised: Sample a batch B from D

Original: Update θ c with B using Eq. 5.
Revised: Update θ c with B using Eq. 5.

Original: end while
Revised: end while

Original: OUTPUT: Classiﬁers weights θ c 
Revised: OUTPUT: Classiﬁers weights θ c the expectations in Eq.

Original: 4. Datasets
Revised: 6, Eq. and Eq.are estimated using empirical samples from the joint distribution p X a X v X l . The empirical samples from (cid:81) j ∈{ a,v,l } p X j are obtained by shufﬂing the samples from the joint distribution in a batch. We integrate this into minimising a multi-task objective (5) by using minus the estimator.

Revised: We refer to the losses obtained with the penalty based on the estimators described in Eq.

Revised: 6, Eq.

Revised: and Eq.

Revised: as L kl , L f and L W respectively.

Revised: Details on the practical minimisation of our variational bounds are provided in Algorithm 1.

Revised: Remark.

Revised: In this work we choose to generalise MINE to compute multivariate dependencies.

Revised: Comparing our proposed algorithm to other alternatives mentioned in sec.

Revised: 2 is left for future work.

Revised: This choice is driven by two main reasons: (1) our framework allows the use of diﬀerent types of contrast measures (e.g

Revised: Wasserstein distance, f -divergences); (2) the critic network T θ can be used for interpretability purposes as shown in sec.

Revised:  Experimental setting

Revised: In this section, we present our experimental settings including the neural architectures we compare, the datasets, the metrics and our methodology, which includes the hyper-parameter selection.

Revised: 4. Datasets

Original: We empirically evaluate our methods on twoenglish datasets: CMU-MOSI and CMU-MOSEI .
Revised: We empirically evaluate our methods on two english datasets: CMU-MOSI and CMU-MOSEI .

Original: Both datasets have been frequently used to assessmodel performance in human multimodal sentiment and emotion recognition.
Revised: Both datasets have been frequently used to assess model performance in human multimodal sentiment and emotion recognition.

Original: CMU -MOSI :
Revised: CMU-MOSI :

Original: Multimodal Opinion SentimentIn-
Revised: Multimodal Opinion Sentiment

Original: tensity (W¨ollmer et al., 2013) is a sentiment an- notated dataset gathering 2 , 199 short monologue video clips.
Revised: Intensity (W¨ollmer et al., 2013) is a sentiment annotated dataset gathering 2 , 199 short monologue video clips.

Original: CMU-MOSEI : CMU-Multimodal OpinionSenti-
Revised: CMU-MOSEI : CMU-Multimodal Opinion

Original: ment and Emotion Intensity (Zadeh et al., 2018c) is an emotion and sentiment annotated corpus con- sisting of 23 , 454 movie review videos taken from YouTube.
Revised: Sentiment and Emotion Intensity (Zadeh et al., 2018c) is an emotion and sentiment annotated corpus consisting of 23 , 454 movie review videos taken from YouTube.

Original: Both CMU-MOSI and CMU-MOSEIare labelled by humans with a sentiment score in[ − 3 , 3] .
Revised: Both CMU-MOSI and CMU-MOSEI are labelled by humans with a sentiment score in [ − 3 , 3].

Original: For each dataset, three modalities are available; we follow prior work (Zadeh et al., 2018b,2017; Rahman et al., 2020) and the features thathave been obtained as follows 1 :
Revised: For each dataset, three modalities are available; we follow prior work (Zadeh et al., 2018b, 2017; Rahman et al., 2020) and the features that have been obtained as follows 1 :

Original: Language : Video transcripts are converted to wordembeddings using either Glove (Pennington et al., , BERT or XLNET contextualised embeddings.
Revised: Language : Video transcripts are converted to word embeddings using either Glove (Pennington et al., 2014), BERT or XLNET contextualised embeddings.

Original: For Glove, the embeddings are of dimension 300 , where for BERT and XLNET this dimension is 768 .
Revised: For Glove, the embeddings are of dimension 300, where for BERT and XLNET this dimension is 768.

Original: Vision : Vision features are extracted usingFacet
Revised: Vision : Vision features are extracted using

Original:  which results into facial action units corresponding to facial muscle movement.
Revised: Facet which results into facial action units corresponding to facial muscle movement.

Original: For CMU-MOSEI , the video vectors are composed of 47 units, and for CMU-MOSI they are composed of 35 .
Revised: For CMU-MOSEI , the video vectors are composed of 47 units, and for CMU-MOSI they are composed of 35.

Original: Audio : Audio features are extracted usingCO-
Revised: Audio : Audio features are extracted using

Original: VAREP (Degottex et al., 2014).This
Revised: COVAREP (Degottex et al., 2014).

Original: results into

Original:  a vector of dimension 74 which includes 12 Melfrequency cepstral coefﬁcients (MFCCs), as well as pitch tracking and voiced/ unvoiced segmenting features , peak slope parameters, maxima disper-
Revised: This results into a vector of dimension 74 which includes 12 Mel-frequency cepstral coeﬃcients (MFCCs), as well as pitch tracking and voiced/unvoiced segmenting features, peak slope parameters, maxima dispersion quotients and glottal source parameters.

Original: sion quotients and glottal source parameters. Video and audio are aligned on text-based following the convention introduced in (Chen et al., 2017) and the forced alignment described in (Yuan and Liberman, 2008).
Revised: Video and audio are aligned on text-based following the convention introduced in (Chen et al., 2017) and the forced alignment described in (Yuan and Liberman, 2008).

Original: 4. Evaluation metrics
Revised: 4. Evaluation metrics

Original: Multimodal Opinion Sentiment Intensity prediction is treated as a regression problem.
Revised: Multimodal Opinion Sentiment Intensity prediction is treated as a regression problem.

Original: Thus, we report both the Mean Absolute Error (MAE) and the correlation of model predictions with true labels.
Revised: Thus, we report both the Mean Absolute Error (MAE) and the correlation of model predictions with true labels.

Original: In the literature, the regression task is also turned into a binary classiﬁcation task for polarity prediction.
Revised: In the literature, the regression task is also turned into a binary classiﬁcation task for polarity prediction.

Original: We follow standard practices (Rahman et al., 2020) and report the Accuracy 2 ( Acc 7 denotes accuracy on 7 classes and Acc 2 the binary accuracy) of our best performing models.
Revised: We follow standard practices (Rahman et al., 2020) and report the Accuracy 2 ( Acc 7 denotes accuracy on 7 classes and Acc 2 the binary accuracy) of our best performing models.

Original: 4. Neural architectures
Revised: 4. Neural architectures

Original: In our experiments, we choose to modify the loss function of the different models that have been introduced for multi-modal sentiment analysis on both CMU-MOSI and CMU-MOSEI : Memory Fusion Network ( MFN (Zadeh et al., 2018a)), Lowrank Multimodal Fusion ( LFN (Liu et al., 2018)) and two state-of-the-art transformers based models (Rahman et al., 2020) for fusion rely on BERT (Devlin et al., 2018)
Revised: In our experiments, we choose to modify the loss function of the diﬀerent models that have been introduced for multi-modal sentiment analysis on both CMU-MOSI and CMU-MOSEI : Memory Fusion Network ( MFN (Zadeh et al., 2018a)), Low-rank Multimodal Fusion ( LFN (Liu et al., 2018)) and two state-of-the-art transformers based models (Rahman et al., 2020) for fusion rely on BERT (Devlin et al., 2018)

Original: ( MAG-BERT ) and XLNET
Revised: ( MAG-BERT ) and XLNET

Original: (Yang et al., 2019) ( MAG-XLNT ).
Revised: (Yang et al., 2019) ( MAG-XLNT ).

Original: To assess the validity of the proposed losses, we also apply our method to a simple early fusion LSTM ( EF-LSTM ) as a baseline model.
Revised: To assess the validity of the proposed losses, we also apply our method to a simple early fusion LSTM ( EF-LSTM ) as a baseline model.

Original: Model overview: Aforementioned models can be seen as a multi-modal encoder f θ e providing a representation Z avl containing information and dependencies between modalities X l , X a , X v namely:f θ e ( X a , X v , X l ) =
Revised: Model overview: Aforementioned models can be seen as a multi-modal encoder f θ e providing a representation Z avl containing information and dependencies between modalities X l , X a , X v namely: f θ e ( X a , X v , X l ) =

Original: Z avl .
Revised: Z avl .

Original: As a ﬁnal step, a linear transformation A θ p is applied to Z avl to perform the regression.
Revised: As a ﬁnal step, a linear transformation A θ p is applied to Z avl to perform the regression.

Original: EF-LSTM : is the most basic architecture used inthe current multimodal analysis where each sequence view is encoded separately with LSTMchannels.
Revised: EF-LSTM : is the most basic architecture used in the current multimodal analysis where each sequence view is encoded separately with LSTM channels.

Original: Then, a fusion function is applied toall representations.
Revised: Then, a fusion function is applied to all representations.

Original: TFN : computes a representation of each view, andthen applies a fusion operator.
Revised: TFN : computes a representation of each view, and then applies a fusion operator.

Original: Acoustic and visualviews are ﬁrst mean-pooled then encoded througha 2-layers perceptron.
Revised: Acoustic and visual views are ﬁrst mean-pooled then encoded through a 2-layers perceptron.

Original: Linguistic features are computed with a LSTM channel.
Revised: Linguistic features are computed with a LSTM channel.

Original: Here, the fusion function is a cross-modal product capturing unimodal,bimodal and trimodal interactions across modalities.
Revised: Here, the fusion function is a crossmodal product capturing unimodal, bimodal and trimodal interactions across modalities.

Original: MFN enriches the previous EF-LSTM architecturewith an attention module that computes a crossview representation at each time step.
Revised: MFN enriches the previous EF-LSTM architecture with an attention module that computes a cross-view representation at each time step.

Original: They arethen gathered and a ﬁnal representation is computed by a gated multi-view memory (Zadeh et al.,2018a).
Revised: They are then gathered and a ﬁnal representation is computed by a gated multi-view memory (Zadeh et al., 2018a).

Original: MAG-BERT and MAG-XLNT are based on pretrained transformer architectures (Devlin et al.
Revised: MAG-BERT and MAG-XLNT are based on pretrained transformer architectures (Devlin et al., 2018;

Original: ,2018; Yang et al., 2019) allowing inputs on each ofthe transformer units to be multimodal, thanks toa special gate inspired by Wang et al.
Revised: Yang et al., 2019) allowing inputs on each of the transformer units to be multimodal, thanks to a special gate inspired by Wang et al.

Original: The Z avl is the [ CLS ] representation provided by thelast transformer head.
Revised: The Z avl is the [ CLS ] representation provided by the last transformer head.

Original: For each architecture, we usethe optimal architecture hyperparameters providedby the associated papers (see Appendix A).
Revised: For each architecture, we use the optimal architecture hyperparameters provided by the associated papers (see Appendix A).

Original:  Numerical results
Revised:  Numerical results

Original: We present and discuss here the results obtained using the experimental setting described in sec.
Revised: We present and discuss here the results obtained using the experimental setting described in sec.

Original: Tobetter understand the impact of our new methods,we propose to investigate the following points:
Revised: To better understand the impact of our new methods, we propose to investigate the following points:

Original: Efﬁciency of the L MDM : to gain understandingof the usefulness of our new objectives, we studythe impact of adding the mutual dependency termon the basic multimodal neural model EF-LSTM .
Revised: Eﬃciency of the L MDM : to gain understanding of the usefulness of our new objectives, we study the impact of adding the mutual dependency term on the basic multimodal neural model EF-LSTM .

Original: Improving model performance and comparingmultivariate dependency measures: the choiceof the most suitable dependency measure for agiven task is still an open problem (see sec. 3).
Revised: Improving model performance and comparing multivariate dependency measures: the choice of the most suitable dependency measure for a given task is still an open problem (see sec. 3).

Original: Thus, we compare the performance – on both multimodal sentiment and emotion prediction tasks– ofthe different dependency measures.
Revised: Thus, we compare the performance – on both multimodal sentiment and emotion prediction tasks– of the diﬀerent dependency measures.

Original: The compared measures are combined with different models using various fusion mechanisms.
Revised: The compared measures are combined with diﬀerent models using various fusion mechanisms.

Original: Improving the robustness to modality drop: a desirable quality of multimodal representations is the robustness to a missing modality.
Revised: drop: a desirable quality of multimodal representations is the robustness to a missing modality.

Original: We study how the maximisation of mutual dependency measures during training affects the robustness of the representation when a modality becomes missing.
Revised: We study how the maximisation of mutual dependency measures during training aﬀects the robustness of the representation when a modality becomes missing.

Original: Towards explainable representations: the statistical network T θ allows us to compute a dependency measure between the three considered modalities.
Revised: Towards explainable representations: the statistical network T θ allows us to compute a dependency measure between the three considered modalities.

Original: We carry out a qualitative analysis in order to investigate if a high dependency can be explained by complementariness across modalities.
Revised: We carry out a qualitative analysis in order to investigate if a high dependency can be explained by complementariness across modalities.

Original: 5. Efﬁciency of the MDM penalty
Revised: 5. Eﬃciency of the MDM penalty

Original: For a simple EF-LSTM , we study the improvement induced by addition of our MDM penalty.
Revised: For a simple EF-LSTM , we study the improvement induced by addition of our MDM penalty.

Original: The results are presented in Tab. 1, where a EF-LSTM trained with no mutual dependency term is denoted with L ∅ .
Revised: The results are presented in Tab. 1, where a EF-LSTM trained with no mutual dependency term is denoted with L ∅ .

Original: On both studied datasets, we observe that the addition of a MDM penalty leads to stronger performances on all metrics.
Revised: On both studied datasets, we observe that the addition of a MDM penalty leads to stronger performances on all metrics.

Original: For both datasets, we observe that the best performing models are obtained by training with an additional mutual dependency measure term.
Revised: For both datasets, we observe that the best performing models are obtained by training with an additional mutual dependency measure term.

Original: Keeping in mind the example shown in Fig.
Revised: Keeping in mind the example shown in Fig.

Original: 1, we can draw a ﬁrst comparison between the different dependency measures.
Revised: 1, we can draw a ﬁrst comparison between the diﬀerent dependency measures.

Original: Although in a simple case L f and L kl estimate a similar quantity (see Fig. 1), in more complex practical applications they do not achieve the same performance.
Revised: Although in a simple case L f and L kl estimate a similar quantity (see Fig. 1), in more complex practical applications they do not achieve the same performance.

Original: Even though, the Donsker-Varadhan bound used for L kl is stronger 3 than the one used to estimate L f ; for a simple model the stronger bound does not lead to better results.
Revised: Even though, the Donsker-Varadhan bound used for L kl is stronger 3 than the one used to estimate L f ; for a simple model the stronger bound does not lead to better results.

Original: It is possible that most of the differences in performance observed come from the optimisation process during training 4 .
Revised: It is possible that most of the diﬀerences in performance observed come from the optimisation process during training 4 .

Original: Takeaways:
Revised: Takeaways:

Original: On the simple case of EF-LSTM adding MDM penalty improves the performance on the downstream tasks.
Revised: On the simple case of EF-LSTM adding MDM penalty improves the performance on the downstream tasks.

Original: 5. Improving models and comparing 
Revised: 5. Improving models and comparing multivariate dependency measures

Original: multivariate dependency measures

Original: In this experiment, we apply the different penalties to more advanced architectures, using various fusion mechanisms.
Revised: In this experiment, we apply the diﬀerent penalties to more advanced architectures, using various fusion mechanisms.

Original: [Table]

Original: General analysis .
Revised: General analysis .

Original: Tab. 2 shows the performanceof various neural architectures trained with andwithout MDM penalty.
Revised: Tab. 2 shows the performance of various neural architectures trained with and without MDM penalty.

Original: Results are coherent withthe previous experiment: we  observe that jointlymaximising a mutual dependency measure leadsto better results on the downstream task: for example, a MFN on CMU-MOSI trained with L Woutperforms by 4 . 6 points on Acc h 7 the modeltrained without the mutual dependency term.On
Revised: Results are coherent with the previous experiment: we [Table] observe that jointly maximising a mutual dependency measure leads to better results on the downstream task: for example, a MFN on CMU-MOSI trained with L W outperforms by 4 . 6 points on Acc h 7 the model trained without the mutual dependency term.

Original:  CMU-MOSEI we also obtain subsequent improvements while training with MMD.
Revised: On CMU-MOSEI we also obtain subsequent improvements while training with MMD.

Original: On CMU-MOSIthe TFN also strongly beneﬁts from the mutual dependency term with an absolute improvement of3 .
Revised: On CMU-MOSI the TFN also strongly beneﬁts from the mutual dependency term with an absolute improvement of 3 .

Original: 7% (on Acc h 7 ) with L W compared to L ∅ .
Revised: 7% (on Acc h 7 ) with L W compared to L ∅ .

Original: Tab. 2shows that our methods not only perform wellon recurrent architectures but also on pretrained Transformer-based models, that achieve higher results due to a superior capacity to model contextualdependencies (see (Rahman et al., 2020)).
Revised: Tab.shows that our methods not only perform well on recurrent architectures but also on pretrained Transformer-based models, that achieve higher results due to a superior capacity to model contextual dependencies (see (Rahman et al., 2020)).

Original: Improving state-of-the-art models .
Revised: Improving state-of-the-art models .

Original: MAGBERTand MAGXLNET are state-of-the art models on both CMU-MOSI and CMU-MOSEI .
Revised: MAGBERT and MAGXLNET are state-of-the art models on both CMU-MOSI and CMU-MOSEI .

Original: From Tab. 2, weobserve that our methods can improve the performance of both models.
Revised: From Tab. 2, we observe that our methods can improve the performance of both models.

Original: It is worth noting that, inboth cases, L W combined with pre-trained transformers achieves good results.
Revised: It is worth noting that, in both cases, L W combined with pre-trained transformers achieves good results.

Original: This performancegain suggests that our method is able to capturedependencies that are not learnt during either pretraining of the language model ( i.e BERT or XL-
Revised: This performance gain suggests that our method is able to capture dependencies that are not learnt during either pretraining of the language model ( i.e BERT or XLNET) or

Original: [Table]
Revised: [Table]

Original: L W 35.6 77.

Original: 0.57 MAGBERT

Original: L ∅ 40.2 84.

Original: L kl 42.0 85.

Original: L f 41.7 85.

Original: L W 41.8 85.

Original: 0.79 MAGXLNET

Original: L ∅ 43.0 86.

Original: L kl 44.5 86.

Original: L f 43.9 86.
Revised: L f 43.9 86.

Original: L W 44.4 86.
Revised: L W 44.4 86.

Original: NET) or by the Multimodal Adaptation Gate used to perform the fusion.
Revised: by the Multimodal Adaptation Gate used to perform the fusion.

Original: Comparing dependency measures .
Revised: Comparing dependency measures .

Original: Tab. 2 shows that there is no dependency measure that achieves the best results in all cases.
Revised: Tab.shows that there is no dependency measure that achieves the best results in all cases.

Original: This result tends to conﬁrm that the optimisation process during training plays an important role (see hypothesis in sec. 5.1).
Revised: This result tends to conﬁrm that the optimisation process during training plays an important role (see hypothesis in sec. 5.1).

Original: However, we can observe that optimising the multivariate Wasserstein dependency measure is usually a good choice, since it achieves state of the art results in many conﬁgurations.
Revised: However, we can observe that optimising the multivariate Wasserstein dependency measure is usually a good choice, since it achieves state of the art results in many conﬁgurations.

Original: It is worth noting that several pieces of research point the limitations of mutual information estimators (McAllester and 
Revised: It is worth noting that several pieces of research point the limitations of mutual information estimators (McAllester and Stratos, 2020; Song and

Original: Stratos, 2020; Song and Ermon, 2019).
Revised: Ermon, 2019).

Original: Takeaways: The addition of MMD not only beneﬁts simple models ( e.g EF-LSTM ) but also improves performance when combined with both complex fusion mechanisms and pretrained models.
Revised: Takeaways: The addition of MMD not only beneﬁts simple models ( e.g EF-LSTM ) but also improves performance when combined with both complex fusion mechanisms and pretrained models.

Original: For practical applications, the Wasserstein distance is a good choice of contrast function.
Revised: For practical applications, the Wasserstein distance is a good choice of contrast function.

Original: 5. Improved robustness to modality drop
Revised: 5. Improved robustness to modality drop

Original: Although fusion with visual and acoustic modalities provided a performance improvement (Wang et al., 2018), the performance of Multimodal systems on sentiment prediction tasks is mainly carried by the linguistic modality (Zadeh et al., 2018a, 2017).
Revised: Although fusion with visual and acoustic modalities provided a performance improvement (Wang et al., 2018), the performance of Multimodal systems on sentiment prediction tasks is mainly carried by the linguistic modality (Zadeh et al., 2018a, 2017).

Original: Thus it isinteresting to study how a multimodal system behaves when the text modality is missing because it gives insights on the robustness of the representation. Experiment description . In this experiment, we Spoken Transcripts
Revised: Thus it is

Original: Acoustic and visual behaviour T θ um the story was all right low energy monotonous voice + headshake L i mean its a Nicholas Sparks book it must be good disappointed tone + neutral facial expression
Revised: [Figure]

Original: L the action is fucking awesome head nod + excited voice H it was cute you know the actors did a great job bringing the smurfs to life such as joe george lopez neil patrick harris katy perry and a fourth multiple smiles H
Revised: MAGBERT

Original: [Figure]
Revised: [Figure]

Original: p t /
Revised: interesting to study how a multimodal system behaves when the text modality is missing because it gives insights on the robustness of the representation.

Original: [Figure]
Revised: Experiment description .

Original: MAGBERT f kl focus on the MAGBERT and MAGXLNET since theyare the best performing models.
Revised: In this experiment, we focus on the MAGBERT and MAGXLNET since they are the best performing models.

Original: 5 As before, theconsidered models are trained using the losses described in sec.
Revised: 5 As before, the considered models are trained using the losses described in sec.

Original: 3 and all modalities are kept duringtraining time.
Revised: 3 and all modalities are kept during training time.

Original: During inference, we either keeponly one modality (Audio or Video) or both.
Revised: During inference, we either keep only one modality (Audio or Video) or both.

Original: Textmodality is always dropped.
Revised: Text modality is always dropped.

Original: Results .
Revised: Results .

Original: Results of the experiments conducted on CMU-MOSI are shown in Fig.
Revised: Results of the experiments conducted on CMU-MOSI are shown in Fig.

Original: 2, giving values forthe ratio
Revised: 2, giving values for the ratio

Original: Acc corrupt/Acc 2 where Acc corruptis thebinary accuracy in the corrupted conﬁguration and Acc 2 the accuracy obtained when all modalities areconsidered.
Revised: Acc corrupt/Acc 2 where Acc corruptis the binary accuracy in the corrupted conﬁguration and Acc 2 the accuracy obtained when all modalities are considered.

Original: We observe that models trained withan MDM penalty (either L kl , L f or L W ) resist better to missing modalities than those trained with
Revised: We observe that models trained with an MDM penalty (either L kl , L f or L W ) resist better to missing modalities than those trained with

Original: L ∅ .
Revised: L ∅ .

Original: For example, when trained with L kl or L f , thedrop in performance is limited to ≈ 25% in anysetting.
Revised: For example, when trained with L kl or L f , the drop in performance is limited to ≈ 25% in any setting.

Original: Interestingly, for MAGBERT L W and L KLachieve comparable results; L KL is more resistant to dropping the language modality, and thus, could be preferred in practical applications.
Revised: Interestingly, for MAGBERT L W and L KL achieve comparable results; L KL is more resistant to dropping the language modality, and thus, could be preferred in practical Spoken Transcripts Acoustic and visual behaviour

Revised: T θ um the story was all right low energy monotonous voice + headshake

Revised: L i mean its a Nicholas Sparks book it must be good disappointed tone + neutral facial expression

Revised: L the action is fucking awesome head nod + excited voice

Original:  Takeaway: Maximising the MMD allows an information transfer between modalities.
Revised: H it was cute you know the actors did a great job bringing the smurfs to life such as joe george lopez neil patrick harris katy perry and a fourth multiple smiles H applications. Takeaway: Maximising the MMD allows an information transfer between modalities.

Original: 5. Towards explainable representations
Revised: 5. Towards explainable representations

Original: In this section, we propose a qualitative experiment allowing us to interpret the predictions made by the deep neural classiﬁer.
Revised: In this section, we propose a qualitative experiment allowing us to interpret the predictions made by the deep neural classiﬁer.

Original: During training, T θ estimates the mutual dependency measure, using the surrogates introduced in Th.
Revised: During training, T θ estimates the mutual dependency measure, using the surrogates introduced in Th.

Original: However, the inference process only involves the classiﬁer, and T θ is unused.
Revised: However, the inference process only involves the classiﬁer, and T θ is unused.

Original: Eq. 6, Eq.
Revised: Eq. 6, Eq.

Original: 7, Eq.
Revised: 7, Eq.

Original: 8 show that T θ is trained to discriminate between valid representations (coming from the joint distribution) and corrupted representations (coming from the product of the marginals).
Revised: 8 show that T θ is trained to discriminate between valid representations (coming from the joint distribution) and corrupted representations (coming from the product of the marginals).

Original: Thus, T θ can be used, at inference time, to measure the mutual dependency of the representations used by the neural model.
Revised: Thus, T θ can be used, at inference time, to measure the mutual dependency of the representations used by the neural model.

Original: In Tab.
Revised: In Tab.

Original:  we report examples of low and high discrepancy measures for MAGBERT on CMU-MOSI .
Revised: 3 we report examples of low and high discrepancy measures for MAGBERT on CMU-MOSI .

Original: We can observe that high values correspond to video clips where audio, text and video are complementary ( e.g use of head node (McClave, 2000)) and low values correspond to the case where there exists contradictions across several modalities.
Revised: We can observe that high values correspond to video clips where audio, text and video are complementary ( e.g use of head node (McClave, 2000)) and low values correspond to the case where there exists contradictions across several modalities.

Original: Results on MAGXLET can be found in sec.
Revised: Results on MAGXLET can be found in sec.

Original: A.3.
Revised: A.3.

Original: Takeaways: T θ used to estimate the MDM provides a mean to interpret representations learnt by the encoder.
Revised: Takeaways: T θ used to estimate the MDM provides a mean to interpret representations learnt by the encoder.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we introduced three new losses based on MDM.
Revised: In this paper, we introduced three new losses based on MDM.

Original: Through extensive set of experiments on CMU-MOSI and CMU-MOSEI , we have shown that SOTA architectures can beneﬁt from these innovations with little modiﬁcations.
Revised: Through extensive set of experiments on CMU-MOSI and CMU-MOSEI , we have shown that SOTA architectures can beneﬁt from these innovations with little modiﬁcations.

Original: A by-product of our method involves a statistical network that is a useful tool to explain the learnt high dimensional multi-modal representations.
Revised: A by-product of our method involves a statistical network that is a useful tool to explain the learnt high dimensional multi-modal representations.

Original: This work paves the way for using and developing new methods to estimate mutual dependency in a multivariate setting.
Revised: This work paves the way for using and developing new methods to estimate mutual dependency in a multivariate setting.

Original: A Supplementary 
Revised: A Supplementary A. Training details

Original: A. Training details

Original: In this section, we both present a comprehensive illustration of the Algorithm 1 and state the details ofexperimental hyperparameters selection as well asand the architectures used for the statistic network T θ .
Revised: In this section, we both present a comprehensive illustration of the Algorithm 1 and state the details of experimental hyperparameters selection as well as and the architectures used for the statistic network T θ .

Original: A.1. Illustration of Algorithm1
Revised: A.1. Illustration of Algorithm

Original: Fig.
Revised: Fig.

Original: 3 describes the Algorithm 1.
Revised: 3 describes the Algorithm 1.

Original: As can be seenin the ﬁgure, to compute the mutual dependencymeasure the statistic network T θ takes the two embeddings of the different batch B and ¯ B .
Revised: As can be seen in the ﬁgure, to compute the mutual dependency measure the statistic network T θ takes the two embeddings of the diﬀerent batch B and ¯ B .

Original: f ✓ c <latexit sha1_base64="PM3jgCJ6O15jTYfYVvsRMSDmJd0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0mkoMeiF48V7Ac0oWy2k3bpZhN2J0IJ/RtePCji1T/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTHNo8kYnuhcyAFAraKFBCL9XA4lBCN5zczf3uE2gjEvWI0xSCmI2UiARnaCU/GuQ+jgHZgM8G1Zpbdxeg68QrSI0UaA2qX/4w4VkMCrlkxvQ9N8UgZxoFlzCr+JmBlPEJG0HfUsViMEG+uHlGL6wypFGibSmkC/X3RM5iY6ZxaDtjhmOz6s3F/7x+htFNkAuVZgiKLxdFmaSY0HkAdCg0cJRTSxjXwt5K+ZhpxtHGVLEheKsvr5POVd1r1BsPjVrztoijTM7IObkkHrkmTXJPWqRNOEnJM3klb07mvDjvzseyteQUM6fkD5zPH2GYkeo=</latexit> 
Revised: f ✓ c <latexit sha1_base64="PM3jgCJ6O15jTYfYVvsRMSDmJd0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0mkoMeiF48V7Ac0oWy2k3bpZhN2J0IJ/RtePCji1T/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTHNo8kYnuhcyAFAraKFBCL9XA4lBCN5zczf3uE2gjEvWI0xSCmI2UiARnaCU/GuQ+jgHZgM8G1Zpbdxeg68QrSI0UaA2qX/4w4VkMCrlkxvQ9N8UgZxoFlzCr+JmBlPEJG0HfUsViMEG+uHlGL6wypFGibSmkC/X3RM5iY6ZxaDtjhmOz6s3F/7x+htFNkAuVZgiKLxdFmaSY0HkAdCg0cJRTSxjXwt5K+ZhpxtHGVLEheKsvr5POVd1r1BsPjVrztoijTM7IObkkHrkmTXJPWqRNOEnJM3klb07mvDjvzseyteQUM6fkD5zPH2GYkeo=</latexit> [Table] <latexit sha1_base64="ILzysQzlOiYR8TzjKG7363K2UAM

Original: [Table]
Revised: [Table]

Original: AAAB9HicbVBNS8NAEJ34WetX1aOXxSJ4KokU9Fj04rGC/cA2lMl20y7dbOLuplBCf4cXD4p49cd489+4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6jhVlDVoLGLVDlAzwSVrGG4EayeKYRQI1gpGtzO/NWZK81g+mEnC/AgHkoecorGS3w1QZY/TXiZwPO2Vym7FnYOsEi8nZchR75W+uv2YphGThgrUuuO5ifEzVIZTwabFbqpZgnSEA9axVGLEtJ/Nj56Sc6v0SRgrW9KQufp7IsNI60kU2M4IzVAvezPxP6+TmvDaz7hMUsMkXSwKU0FMTGYJkD5XjBoxsQSp4vZWQoeokBqbU9GG4C2/vEqalxWvWqneV8u1mzyOApzCGVyAB1dQgzuoQwMoPMEzvMKbM3ZenHfnY9G65uQzJ/{+/cA2lMl20y7dbOLuplBCf4cXD4p49cd
 +4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6jhVlDVoLGLVDlAzwSVrGG4EayeKYRQI1gpGtzO/NWZK81g+mEnC/AgHkoecorGS3w1QZY/TXiZwPO2Vym7FnYOsEi8nZchR75W+uv2YphGThgrUuuO5ifEzVIZTwabFbqpZgnSEA9axVGLEtJ/Nj56Sc6v0SRgrW9KQufp7IsNI60kU2M4IzVAvezPxP6+TmvDaz7hMUsMkXSwKU0FMTGYJkD5XjBoxsQSp4vZWQoeokBqbU9GG4C2/vEqalxWvWqneV8u1mzyOApzCGVyAB1dQgzuoQwMoPMEzvMKbM3ZenHfnY9G65uQzJ/+}
Revised: {+/cA2lMl20y7dbOLuplBCf4cXD4p49cd
 +4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6jhVlDVoLGLVDlAzwSVrGG4EayeKYRQI1gpGtzO/NWZK81g+mEnC/AgHkoecorGS3w1QZY/TXiZwPO2Vym7FnYOsEi8nZchR75W+uv2YphGThgrUuuO5ifEzVIZTwabFbqpZgnSEA9axVGLEtJ/Nj56Sc6v0SRgrW9KQufp7IsNI60kU2M4IzVAvezPxP6+TmvDaz7hMUsMkXSwKU0FMTGYJkD5XjBoxsQSp4vZWQoeokBqbU9GG4C2/vEqalxWvWqneV8u1mzyOApzCGVyAB1dQgzuoQwMoPMEzvMKbM3ZenHfnY9G65uQzJ/+}

Original: AHzucPX66SgA==</latexit>
Revised: AHzucPX66SgA==</latexit>

Original: A.1. Hyperparameters selection
Revised: A.1. Hyperparameters selection

Original: We use dropout (Srivastava et al., 2014) and optimise the global loss Eq.
Revised: We use dropout (Srivastava et al., 2014) and optimise the global loss Eq.

Original: 5 by gradient descent using AdamW (Loshchilov and Hutter, 2017; Kingmaand Ba, 2014) optimiser.
Revised: 5 by gradient descent using AdamW (Loshchilov and Hutter, 2017; Kingma and Ba, 2014) optimiser.

Original: The best learning rate isfound in the grid { 0 . 002 , 0 . 
Revised: The best learning rate is found in the grid { 0 . 002 , 0 . 001 , 0 . 0005 , 0 . 0001 } .

Original: The best model is selected using the lowest MAEon the validation set.
Revised: The best model is selected using the lowest MAE on the validation set.

Original: We Unroll to 10.
Revised: We Unroll to 10.

Original: A.1. Architectures of T 
Revised: A.1. Architectures of T θ

Original: θ

Original: Across the different experiment we use a statistic network with an architecture as describes in Tab. 4.
Revised: Across the diﬀerent experiment we use a statistic network with an architecture as describes in Tab. 4.

Original: We follow (Belghazi et al., 2018) and use LeakyRELU (Agarap, 2018; Xu et al., 2015) asactivation function.
Revised: We follow (Belghazi et al., 2018) and use LeakyRELU (Agarap, 2018; Xu et al., 2015) as activation function.

Original: [Table]
Revised: Statistic Network

Original: Layer
Revised: Layer

Original: [Table]
Revised: Number of outputs Activation function [ Z lav ,Z lav ] d in ,d inDense layer d in / LeakyReLU Dropout 0. Dense layer d in

Revised: LeakyReLU

Revised: Dropout 0.

Revised: Dense layer d in

Revised: LeakyReLU

Revised: Dropout 0.

Revised: Dense layer d in /

Revised: LeakyReLU

Revised: Dropout 0.

Revised: Dense layer d in /

Original: [Table]
Revised: LeakyReLU Dropout 0. Dense layer Sigmo¨ıd

Original: A. Additional experiments for robustness to 
Revised: A. Additional experiments for robustness to modality drop

Original: modality drop

Original: Fig.
Revised: Fig.

Original: 4 shows the results of the robustness text on MAGXLNET .
Revised: 4 shows the results of the robustness text on MAGXLNET .

Original: Similarly to Fig.
Revised: Similarly to Fig.

Original: 2 we observe more robust representation to modality drop when jointly maximising the L W and L kl with the target loss.
Revised: 2 we observe more robust representation to modality drop when jointly maximising the L W and L kl with the target loss.

Original: Fig.
Revised: Fig.

Original: 4 shows no improvement when training with L f .
Revised: 4 shows no improvement when training with L f .

Original: This can also be linked to Tab. 2 which similarly shows no improvement in this very speciﬁc conﬁguration.
Revised: This can also be linked to Tab. 2 which similarly shows no improvement in this very speciﬁc conﬁguration.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: kl f is MAGXLNET on CMU-MOSI .

Original: [Figure] [Table] [Figure] A cc c o rr u p t / A cc
Revised: The ratio between the accuracy achieved with a corrupted linguistic modality Acc corruptand the accuracy Acc 2 without any corruption is reported on y -axis.

Original: MAGXLNET kl f
Revised: The preserved modalities during inference are reported on x -axis. A , V respectively stands for the acoustic and visual modality.

Original: A. Additional qualitative examples
Revised: A. Additional qualitative examples

Original: Tab. 5 illustrates the use of T θ to explain the representations learnt by the model.
Revised: Tab. 5 illustrates the use of T θ to explain the representations learnt by the model.

Original: Similarly to Tab.  we observe that high values correspond to complementarity across modalities and low values arerelated to contradictoriness across modalities.
Revised: Similarly to Tab. 4 we observe that high values correspond to complementarity across modalities and low values are related to contradictoriness across modalities.

Original: Spoken Transcripts
Revised: Spoken Transcripts

Original: [Table] behaviour
Revised: Acoustic and visual behaviour

Original: T θ
Revised: T θ

Original: but the m the script is corny [Table] + [Table]
Revised: but the m the script is corny high energy voice + headshake + (many) smiles

Original: many

Original: [Table]

Original: smiles L as for gi joe was it was just like laughing its the the plot the the acting is terrible high [Table] smiles
Revised: L as for gi joe was it was just like laughing its the the plot the the acting is terrible high enery voice + laughts + smiles

Original: L
Revised: L

Original: but i think this one did beat scream 2 now headshake [Table] sigh
Revised: but i think this one did beat scream 2 now headshake + long sigh

Original: L the xxx sequence is really well done static [Table] + [Table]
Revised: L the xxx sequence is really well done static head + low energy monotonous voice

Original: monotonous voice

Original: L you know of course i was waithing for the princess and the frog smiles + [Table] energy voice + + high pitch
Revised: L you know of course i was waithing for the princess and the frog smiles + high energy voice + + high pitch

Original: H dennis quaid i think had a lot of fun smiles +
Revised: H dennis quaid i think had a lot of fun smiles +

Original: [Table]
Revised: [Table]

Original: energy voice
Revised: energy voice

Original: H it was very very very boring 
Revised: H it was very very very boring low

Original: [Table]
Revised: [Table]

Original:  eyebrows
Revised: frown eyebrows

Original: H i do not wanna see any more of this
Revised: H i do not wanna see any more of this

Original: [Table]
Revised: [Table]

Original: expression
Revised: expression

Original: H
Revised: H