Original: M AXIMUM L IKELIHOOD E STIMATION FOR M ULTIMODAL L EARNING WITH M ISSING M ODALITY
Revised: M AXIMUM L IKELIHOOD E STIMATION FOR M ULTIMODAL L EARNING WITH M ISSING M ODALITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multimodal learning has achieved great successes in many scenarios.
Revised: Multimodal learning has achieved great successes in many scenarios.

Original: Compared with unimodal learning, it can effectively combine the information from different modalities to improve the performance of learning tasks.
Revised: Compared with unimodal learning, it can effectively combine the information from different modalities to improve the performance of learning tasks.

Original: In reality, the multimodal data may have missing modalities due to various reasons, such as sensor failure and data transmission error.
Revised: In reality, the multimodal data may have missing modalities due to various reasons, such as sensor failure and data transmission error.

Original: In previous works, the information of the modalitymissing data has not been well exploited.
Revised: In previous works, the information of the modalitymissing data has not been well exploited.

Original: To address this problem, we propose an efﬁcient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-missing data.
Revised: To address this problem , we propose an efﬁcient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-missing data.

Original: Speciﬁcally, we design a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data, which is theoretically optimal.
Revised: Speciﬁcally , we design a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data, which is theoretically optimal.

Original: Moreover, we develop a generalized form of the softmax function to effectively implement maximum likelihood estimation in an end-to-end manner.
Revised: Moreover we develop a generalized form of the softmax function to effectively implement maximum likelihood estimation in an end-to-end manner

Original: Such training strategy guarantees the computability of our algorithm capably.
Revised: Such training strategy guarantees the computability of our algorithm capably

Original: Finally, we conduct a series of experiments on real-world multimodal datasets.
Revised: Finally , we conduct a series of experiments on real- world multimodal datasets.

Original: Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality.
Revised: Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multimodal learning is an important research area, which builds models to process and relate information between different modalities (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2014;
Revised: Multimodal learning is an important research area, which builds models to process and relate information between different modalities (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2014;

Original: Baltrušaitis et al., 2018).
Revised: Baltrušaitis et al., 2018).

Original: Compared with unimodal learning, multimodal learning caneffectively utilize the multimodal data to achieve better performance.
Revised: Compared with unimodal learning, multimodal learning can achieve better performance by properly utilizing the multimodal data.

Original: It has been successfully used in many applications, such as multimodal emotion recognition (Soleymani et al., 2011; Mittal et al., 2020), multimedia event detection (Li et al., 2020), and visual question-answering (Yu et al., 2019).
Revised: It has been successfully used in many applications, such as multimodal emotion recognition (Soleymani et al., 2011; Mittal et al., 2020), multimedia event detection (Li et al., 2020), and visual question-answering (Yu et al., 2019).

Original: With the emergence of big data, multimodal learning becomes more and more important to combine the multimodal data from different sources.
Revised: With the emergence of big data, multimodal learning becomes more and more important to combine the multimodal data from different sources.

Original: A number of previous works (Tzirakis et al., 2017; Zhang et al., 2017; Elliott et al., 2017; Kim et al., 2020; Zhang et al., 2020) have achieved great successes based on complete observations during the training process.
Revised: A number of previous works (Tzirakis et al., 2017; Zhang et al., 2017; Elliott et al., 2017; Kim et al., 2020; Zhang et al., 2020) have achieved great successes based on complete observations during the training process.

Original: However, in practice, the multimodal data may have missing modalities (Du et al., 2018; Ma et al., 2021a;b).
Revised: However, in practice, the multimodal data may have missing modalities (Du et al., 2018; Ma et al., 2021a;b).

Original: This may be caused by various reasons.
Revised: This may be caused by various reasons.

Original: For instance, the sensor that collects the multimodal data is damaged or the network transmission fails.
Revised: For instance, the sensor that collects the multimodal data is damaged or the network transmission fails.

Original: Examples of the multimodal data are shown in Figure 1.
Revised: Examples of the multimodal data are shown in Figure 1.

Original: In the past years, different approaches have been proposed to deal with modality missing.
Revised: In the past years, different approaches have been proposed to deal with modality missing.

Original: A simple and typical way (Hastie et al., 2009) is to directly discard the data with missing modalities.
Revised: A simple and typical way (Hastie et al., 2009) is to directly discard the data with missing modalities.

Original: Since the information contained in the modality-missing data is neglected, such method often has limited performance.
Revised: Since the information contained in the modality-missing data is neglected, such method often has limited performance.

Original: In addition, researchers (Tran et al., 2017; Chen & Zhang, 2020; Liu et al., 2021; Ma et al., 2021b) have proposed approaches to heuristically combine the information of the modalitymissing data.
Revised: In addition, researchers (Tran et al., 2017; Chen & Zhang, 2020; Liu et al., 2021; Ma et al., 2021b) have proposed approaches to heuristically combine the information of the modalitymissing data.

Original: However, most of these works lack theoretical explanations, and these empirical methods are often implemented using multiple training stages rather than an end-to-end manner, which lead to the information of the modality-missing data not being well exploited.
Revised: However, most of these works lack theoretical explanations, and these empirical methods are often implemented using multiple training stages rather than an end-to-end manner, which lead to the information of the modality-missing data not being well exploited.

Original: To tackle above issues, we propose an efﬁcient approach based on maximum likelihood estimation to effectively utilize the modality-missing data.
Revised: To tackle above issues, we propose an efﬁcient approach based on maximum likelihood estimation to effectively utilize the modality-missing data.

Original: To be speciﬁc, we present a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data, which is theoretically optimal.
Revised: To be speciﬁc, we present a likelihood function to characterize the conditional distributions of the modality-complete data and the modality-missing data, which is theoretically optimal.

Original: Furthermore, we adopt a generalized form of the softmax
Revised: Furthermore, we adopt a generalized form of the softmax

Original: [Figure]
Revised: [Figure]

Original: function to efﬁciently implement our maximum likelihood estimation algorithm.
Revised: function to efﬁciently implement our maximum likelihood estimation algorithm.

Original: Such training strategy guarantees the computability of our framework in an end-to-end scheme.
Revised: Such training strategy guarantees the computability of our framework in an end-to-end scheme.

Original: In this way, our approach can effectively leverage the information of the modality-missing data during the training process, Finally, we perform several experiments on real-world multimodal datasets, including eNTERFACE’05 (Martin et al., 2006) and RAVDESS (Livingstone & Russo, 2018).
Revised: In this way, our approach can effectively leverage the information of the modality-missing data during the training process, Finally, we perform several experiments on real-world multimodal datasets, including eNTERFACE’05 (Martin et al., 2006) and RAVDESS (Livingstone & Russo, 2018).

Original: The results show the effectiveness of our approach in handling the problem of modality missing.
Revised: The results show the effectiveness of our approach in handling the problem of modality missing.

Original: To summarize, our contribution is three-fold:
Revised: To summarize, our contribution is three-fold:

Original: • We design a likelihood function to learn the conditional distributions of the modalitycomplete data and the modality-missing data, which is theoretically optimal.
Revised: • We design a likelihood function to learn the conditional distributions of the modalitycomplete data and the modality-missing data, which is theoretically optimal.

Original: • We develop a generalized form of the softmax function to implement our maximum likelihood estimation framework in an end-to-end manner, which is more effective than previous works.
Revised: • We develop a generalized form of the softmax function to implement our maximum likelihood estimation framework in an end-to-end manner, which is more effective than previous works.

Original: • We conduct a series of experiments on real-world multimodal datasets.
Revised: • We conduct a series of experiments on real-world multimodal datasets.

Original: The results validate the effectiveness of our approach, even when 95% of the training data has missing modality.
Revised: The results validate the effectiveness of our approach, even when 95% of the training data has missing modality.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Our goal is to deal with the problem of modality missing in multimodal learning based on maximum likelihood estimation.
Revised: Our goal is to deal with the problem of modality missing in multimodal learning based on maximum likelihood estimation.

Original: In the following, we ﬁrst show the problem formulation, and then describe the details of our framework.
Revised: In the following, we ﬁrst show the problem formulation, and then describe the details of our framework.

Original: 2. P ROBLEM F ORMULATION
Revised: 2. P ROBLEM F ORMULATION

Original: In this paper, we consider that the multimodal data has two modalities.
Revised: In this paper, we consider that the multimodal data has two modalities.

Original: Here, the random variables corresponding to these two modalities and their category labels are denoted as X , Y , and Z , respectively.
Revised: Here, the random variables corresponding to these two modalities and their category labels are denoted as X , Y , and Z , respectively.

Original: In the training process, we assume that there are two independently observed datasets: modality-complete and modality-missing.
Revised: In the training process, we assume that there are two independently observed datasets: modality-complete and modality-missing.

Original: We use D XY Z
Revised: We use D XY Z

Original: =
Revised: =

Original: (cid:8) ( x ( i ) c , y
Revised: (cid:8) ( x ( i ) c , y

Original: ( i ) c , z ( i ) c )
Revised: ( i ) c , z ( i ) c )

Original: | z ( i )
Revised: | z ( i )

Original: c ∈
Revised: c ∈

Original: Z = { 1 , 2 , · · · , |Z|} (cid:9) n c i =1 to represent the modality-complete dataset, where x ( i ) c and y ( i ) c represent the two modalities of the i -th sample of D XY Z respectively, z ( i ) c is their corresponding category label, and the size of D XY Z is n c .
Revised: Z = { 1 , 2 , · · · , |Z|} (cid:9) n c i =1 to represent the modality-complete dataset, where x ( i ) c and y ( i ) c represent the two modalities of the i -th sample of D XY Z respectively, z ( i ) c is their corresponding category label, and the size of D XY Z is n c .

Original: We then use D XZ = (cid:8) ( x ( i )m , z ( i )m )
Revised: We then use D XZ = (cid:8) ( x ( i )m , z ( i )m )

Original: | z ( i )m ∈
Revised: | z ( i )m ∈

Original: Z = { 1 , 2 , · · · , |Z|} (cid:9) n m i =1 to represent the modality-missing dataset, where the size of D XZ is n m .
Revised: Z = { 1 , 2 , · · · , |Z|} (cid:9) n m i =1 to represent the modality-missing dataset, where the size of D XZ is n m .

Original: In addition, we adopt [ D XY Z ] XY to represent
Revised: In addition, we adopt [ D XY Z ] XY to represent

Original: [Equation]
Revised: [Equation]

Original: [ D
Revised: [ D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , [ D XZ ] X , and [ D XZ ]
Revised: , [ D XZ ] X , and [ D XZ ]

Original: Z are expressed in the same way.
Revised: Z are expressed in the same way.

Original: The multimodal data of D XY Z and D XZ are assumed to be i.i.d.
Revised: The multimodal data of D XY Z and D XZ are assumed to be i.i.d.

Original: generated from an unknown underlying joint distribution.
Revised: generated from an unknown underlying joint distribution.

Original: By utilizing the knowledge of the modality-complete data and the modality-missing data, we hope our framework can predict the category labels correctly.
Revised: By utilizing the knowledge of the modality-complete data and the modality-missing data, we hope our framework can predict the category labels correctly.

Original: 2. M AXIMUM L IKELIHOOD E STIMATION FOR M ISSING M ODALITY
Revised: 2. M AXIMUM L IKELIHOOD E STIMATION FOR M ISSING M ODALITY

Original: In this section, we ﬁrst present how to design a likelihood function to learn the conditional distributions of the modality-complete data and the modality-missing data.
Revised: In this section, we ﬁrst present how to design a likelihood function to learn the conditional distributions of the modality-complete data and the modality-missing data.

Original: Then, we show that by adopting a generalized form of the softmax function, we design a training strategy to implement our algorithm.
Revised: Then, we show that by adopting a generalized form of the softmax function, we design a training strategy to implement our algorithm.

Original: [Table]
Revised: [Table]

Original: o
Revised: o

Original: [Table]
Revised: [Table]

Original: k
Revised: k

Original: [Table]
Revised: [Table]

Original: k
Revised: k

Original: [Table]
Revised: [Table]

Original: w o
Revised: w o

Original: [Table]
Revised: [Table]

Original: /oV3xhTUIjohyZlz7zkz994wjYTSnvdacObmFxaXissrq2vrG5ulre2mSjLJeIMlUSLbYaB4JGLe0EJHvJ1KHozCiLfC4ZmJt265VCKJL/U45d1RMIhFX7BAE1W/uy6VvYpnlzsL/ByUka9aUnrBFXpIwJBhBI4YmnCEAIqeDnx4SInrYkKcJCRsnOMeK6TNKItTRkDskL4D2nVyNqa98VRWzeiUiF5JShf7pEkoTxI2p7k2nllnw/7mPbGe5m5j+oe514hYjRti/9JNM/+rM7Vo9HFiaxBUU2oZUx3LXTLbFXNz90tVmhxS4gzuUVwSZlY57bNrNcrWbnob2PibzTSs2bM8N8O7uSUN2P85zlnQPKz4XsWvH5Wrp/moi9jFHg5onseo4gI1NKz3I57w7Jw7kaOc7DPVKeSaHXxbzsMHb9yPfw==</latexit>
Revised: /oV3xhTUIjohyZlz7zkz994wjYTSnvdacObmFxaXissrq2vrG5ulre2mSjLJeIMlUSLbYaB4JGLe0EJHvJ1KHozCiLfC4ZmJt265VCKJL/U45d1RMIhFX7BAE1W/uy6VvYpnlzsL/ByUka9aUnrBFXpIwJBhBI4YmnCEAIqeDnx4SInrYkKcJCRsnOMeK6TNKItTRkDskL4D2nVyNqa98VRWzeiUiF5JShf7pEkoTxI2p7k2nllnw/7mPbGe5m5j+oe514hYjRti/9JNM/+rM7Vo9HFiaxBUU2oZUx3LXTLbFXNz90tVmhxS4gzuUVwSZlY57bNrNcrWbnob2PibzTSs2bM8N8O7uSUN2P85zlnQPKz4XsWvH5Wrp/moi9jFHg5onseo4gI1NKz3I57w7Jw7kaOc7DPVKeSaHXxbzsMHb9yPfw==</latexit>

Original: 2.2. L IKELIHOOD F UNCTION A NALYSES
Revised: 2.2. L IKELIHOOD F UNCTION A NALYSES

Original: Maximum likelihood estimation is a statistical method of using the observed data to estimate the distribution by maximizing the likelihood function.
Revised: Maximum likelihood estimation is a statistical method of using the observed data to estimate the distribution by maximizing the likelihood function.

Original: The estimated distribution makes the observed data most likely (Myung, 2003).
Revised: The estimated distribution makes the observed data most likely (Myung, 2003).

Original: With this idea, we study the likelihood function on datasets D XY Z and D XZ .
Revised: With this idea, we study the likelihood function on datasets D XY Z and D XZ .

Original: For the classiﬁcation task, the conditional likelihood is commonly used.
Revised: For the classiﬁcation task, the conditional likelihood is commonly used.

Original: Inspired by this, we use a model Q XY Z to learn the underlying joint distribution of D XY Z and D XZ .
Revised: Inspired by this, we use a model Q XY Z to learn the underlying joint distribution of D XY Z and D XZ .

Original: The conditional likelihood can be represented as:
Revised: The conditional likelihood can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where the step a follows from the fact that datasets D XY Z and D XZ are observed independently, and the step b is due to that samples in each dataset are i.i.d.
Revised: where the step a follows from the fact that datasets D XY Z and D XZ are observed independently, and the step b is due to that samples in each dataset are i.i.d.

Original: Q Z | XY and Q Z | X are conditional distributions of Q XY Z .
Revised: Q Z | XY and Q Z | X are conditional distributions of Q XY Z .

Original: In this way, we show the likelihood function using the information of D XY Z and D XZ .
Revised: In this way, we show the likelihood function using the information of D XY Z and D XZ .

Original: Then, we use the negative log-likelihood as the loss function to train our deep learning model, i.e.,
Revised: Then, we use the negative log-likelihood as the loss function to train our deep learning model, i.e.,

Original: [Equation]
Revised: [Equation]

Original: It is worth noting that in (Daniels, 1961; Lehmann, 2004), maximum likelihood estimation is proved to be an asymptotically-efﬁcient strategy, which guarantees the theoretical optimality of our method to deal with modality missing.
Revised: It is worth noting that in (Daniels, 1961; Lehmann, 2004), maximum likelihood estimation is proved to be an asymptotically-efﬁcient strategy, which guarantees the theoretical optimality of our method to deal with modality missing.

Original: To optimize L , we use deep neural networks to extract the k -dimensional feature representations from the observation ( x, y, z ) , which are represented as f ( x ) =
Revised: To optimize L , we use deep neural networks to extract the k -dimensional feature representations from the observation ( x, y, z ) , which are represented as f ( x ) =

Original: [ f 1 ( x ) , f 2 ( x ) , · · · , f k ( x )] T , g ( y ) =
Revised: [ f 1 ( x ) , f 2 ( x ) , · · · , f k ( x )] T , g ( y ) =

Original: [ g 1 ( y ) , g 2 ( y ) , · · · , g k ( y )]
Revised: [ g 1 ( y ) , g 2 ( y ) , · · · , g k ( y )]

Original: T , and h ( z ) =
Revised: T , and h ( z ) =

Original: [ h 1 ( z ) , h 2 ( z ) , · · · , h k ( z )] T , respectively.
Revised: [ h 1 ( z ) , h 2 ( z ) , · · · , h k ( z )] T , respectively.

Original: We then utilize these features to learn Q Z | XY and Q Z | X in L .
Revised: We then utilize these features to learn Q Z | XY and Q Z | X in L .

Original: Our framework is shown in Figure 2.
Revised: Our framework is shown in Figure 2.

Original: In this way, we show the log-likelihood function L .
Revised: In this way, we show the log-likelihood function L .

Original: By characterizing the conditional distributions of the modality-complete data and the modality-missing data, it leverages the underlying structure information behind the multimodal data, which constitutes the theoretical basis of our framework.
Revised: By characterizing the conditional distributions of the modality-complete data and the modality-missing data, it leverages the underlying structure information behind the multimodal data, which constitutes the theoretical basis of our framework.

Original: 2.2. M AXIMUM L IKELIHOOD E STIMATION I MPLEMENTATION
Revised: 2.2. M AXIMUM L IKELIHOOD E STIMATION I MPLEMENTATION

Original: In fact, it is not easy to optimize the log-likelihood function L in Equation (2) by designing neural networks, which is mainly due to two reasons.
Revised: In fact, it is not easy to optimize the log-likelihood function L in Equation (2) by designing neural networks, which is mainly due to two reasons.

Original: Firstly, the representations of the high-dimensional data and the procedure to model them are complicated.
Revised: Firstly, the representations of the high-dimensional data and the procedure to model them are complicated.

Original: Secondly, since Q Z | XY and Q Z | X in L are
Revised: Secondly, since Q Z | XY and Q Z | X in L are

Original: [Table]
Revised: [Table]

Original: +SddK+jl1+gDjkcUWtrFL/TxEGWeooEret3jCM16UpnKn3CsP41Qlk2o28G0oj5+HhZrU</latexit><latexit sha1_base64="2Z6CrbYA0+t+x2MycOGIsaBOqJQ=">AAAC2XicjVHLSsNAFD2Nr1pf8bFzEyyCq5KIoMuCG5cV7APaUpLptIbmRTIRa+jCnbj1B9zqD4l/oH/hnTEFtYhOyMyZc+85M3euE3luIkzztaDNzS8sLhWXSyura+sb+uZWIwnTmPE6C70wbjl2wj034HXhCo+3opjbvuPxpjM6lfHmFY8TNwwuxDjiXd8eBu7AZbYgqqfvdAS/Jl2mVmeQDSeTUk8vmxVTDWMWWDkoIx+1UH9BB32EYEjhgyOAIOzBRkJfGxZMRMR1kREXE3JVnGOCEmlTyuKUYRM7onlIu3bOBrSXnolSMzrFoz8mpYF90oSUFxOWpxkqnipnyf7mnSlPebcxrU7u5RMrcEnsX7pp5n91shaBAU5UDS7VFClGVsdyl1S9iry58aUqQQ4RcRL3KR4TZko5fWdDaRJVu3xbW8XfVKZk5Z7luSne5S2pwdbPds6CxmHFMivWuVmuHuWtLmIXezigfh6jijPUUCfvGzziCc9aW7vV7rT7z1StkGu28W1oDx+ntJgq</latexit> related, how to build models to learn their relationships is difﬁcult.
Revised: +SddK+jl1+gDjkcUWtrFL/TxEGWeooEret3jCM16UpnKn3CsP41Qlk2o28G0oj5+HhZrU</latexit><latexit sha1_base64="2Z6CrbYA0+t+x2MycOGIsaBOqJQ=">AAAC2XicjVHLSsNAFD2Nr1pf8bFzEyyCq5KIoMuCG5cV7APaUpLptIbmRTIRa+jCnbj1B9zqD4l/oH/hnTEFtYhOyMyZc+85M3euE3luIkzztaDNzS8sLhWXSyura+sb+uZWIwnTmPE6C70wbjl2wj034HXhCo+3opjbvuPxpjM6lfHmFY8TNwwuxDjiXd8eBu7AZbYgqqfvdAS/Jl2mVmeQDSeTUk8vmxVTDWMWWDkoIx+1UH9BB32EYEjhgyOAIOzBRkJfGxZMRMR1kREXE3JVnGOCEmlTyuKUYRM7onlIu3bOBrSXnolSMzrFoz8mpYF90oSUFxOWpxkqnipnyf7mnSlPebcxrU7u5RMrcEnsX7pp5n91shaBAU5UDS7VFClGVsdyl1S9iry58aUqQQ4RcRL3KR4TZko5fWdDaRJVu3xbW8XfVKZk5Z7luSne5S2pwdbPds6CxmHFMivWuVmuHuWtLmIXezigfh6jijPUUCfvGzziCc9aW7vV7rT7z1StkGu28W1oDx+ntJgq</latexit> related, how to build models to learn their relationships is difﬁcult.

Original: To address these two issues, we develop a generalized form of the softmax function to describe Q XY Z as follows 1 :
Revised: To address these two issues, we develop a generalized form of the softmax function to describe Q XY Z as follows 1 :

Original: [Equation]
Revised: [Equation]

Original: where φ ( f , g ) represents the function to fuse features f and g .
Revised: where φ ( f , g ) represents the function to fuse features f and g .

Original: We study three forms of φ to investigate its effect in our framework, as shown in Figure 3.
Revised: We study three forms of φ to investigate its effect in our framework, as shown in Figure 3.

Original: R X , R Y , and R Z represent the underlying marginal distributions of the variables X , Y , and Z , respectively.
Revised: R X , R Y , and R Z represent the underlying marginal distributions of the variables X , Y , and Z , respectively.

Original: Their use makes the denominator of Equation (3) expressed in the form of the mean over R X , R Y , and R Z , which serves as the normalization to make Q XY Z a valid distribution and is helpful for our further derivation.
Revised: Their use makes the denominator of Equation (3) expressed in the form of the mean over R X , R Y , and R Z , which serves as the normalization to make Q XY Z a valid distribution and is helpful for our further derivation.

Original: In the following, we will show that we use empirical distributions to implement these underlying marginal distributions in our algorithm.

Original: In addition, the generalized softmax function we propose can be regarded as a generalization of softmax learning in (Xu et al., 2018) from unimodal learning to multimodal learning.
Revised: In addition, the generalized softmax function we propose can be regarded as a generalization of softmax learning in (Xu et al., 2018) from unimodal learning to multimodal learning.

Original: In this way, we show the distribution Q XY Z by adopting a generalized form of the softmax function, which has the following two beneﬁts.
Revised: In this way, we show the distribution Q XY Z by adopting a generalized form of the softmax function, which has the following two beneﬁts.

Original: Firstly, by depicting the representation of Q XY Z , we can further derive Q Z | XY and Q Z | X .
Revised: Firstly, by depicting the representation of Q XY Z , we can further derive Q Z | XY and Q Z | X .

Original: It makes our approach a uniﬁed framework to combine the information of the modality-complete data and the modality-missing data.
Revised: It makes our approach a uniﬁed framework to combine the information of the modality-complete data and the modality-missing data.

Original: Secondly, it avoids modeling the relationship between Q Z | XY and Q Z | X .
Revised: Secondly, it avoids modeling the relationship between Q Z | XY and Q Z | X .

Original: In fact, the correlation between the high-dimensional data can be rather complex.
Revised: In fact, the correlation between the high-dimensional data can be rather complex.

Original: Then, we derive conditional distributions Q Z | XY and Q Z | X from Equation (3):
Revised: Then, we derive conditional distributions Q Z | XY and Q Z | X from Equation (3):

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: Q Z

Original: [Equation]
Revised: [Equation]

Original: We can observe that by introducing R X , R Y , and R Z into Q XY Z , the derived Q Z | XY and Q Z | X are expressed in the form of the mean over R Y and R Z .
Revised: We can observe that by introducing R X , R Y , and R Z into Q XY Z , the derived Q Z | XY and Q Z | X are expressed in the form of the mean over R Y and R Z .

Original: In practice, we can use the empirical mean as an estimation.
Revised: In practice, we can use the empirical mean as an estimation.

Original: Correspondingly, by plugging Equations (4) and (5) into Equation (2), we can summarize the detailed steps to compute our objective function L , as shown in Algorithm 1.
Revised: Correspondingly, by plugging Equations (4) and (5) into Equation (2), we can summarize the detailed steps to compute our objective function L , as shown in Algorithm 1.

Original: It is worth pointing out that when we compute Q Z | X , we need to use the information of the modality y .
Revised: It is worth pointing out that when we compute Q Z | X , we need to use the information of the modality y .

Original: Since in the training process, the modality y of the dataset D XZ is missing, we utilize samples of the modality y of the dataset D XY Z to compute Q Z | X .
Revised: Since in the training process, the modality y of the dataset D XZ is missing, we utilize samples of the modality y of the dataset D XY Z to compute Q Z | X .

Original: Finally, we utilize neural networks to extract features f , g , and h from the modality-complete data and the modality-missing data to optimize our log-likelihood function L .
Revised: Finally, we utilize neural networks to extract features f , g , and h from the modality-complete data and the modality-missing data to optimize our log-likelihood function L .

Original: It performs classiﬁcation directly, 
Revised: It performs classiﬁcation directly, which does not need to explicitly complement the modality-missing data before the classiﬁcation task.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: 4: Compute our empirical objective L :
Revised: 4: Compute our empirical objective L :

Original: [Equation]
Revised: [Equation]

Original: which does not need to explicitly complement the modality-missing data before the classiﬁcation task.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrst describe the real-world multimodal datasets used in our experiment, then explain the experimental settings and baseline methods, and ﬁnally give the experimental results to show the effectiveness of our approach.
Revised: In this section, we ﬁrst describe the real-world multimodal datasets used in our experiment, then explain the experimental settings and baseline methods, and ﬁnally give the experimental results to show the effectiveness of our approach.

Original: 3. D ATASETS
Revised: 3. D ATASETS

Original: We perform experiments on two public real-world multimodal datasets: eNTERFACE’05 (Martin et al., 2006) and RAVDESS (Livingstone & Russo, 2018).
Revised: We perform experiments on two public real-world multimodal datasets: eNTERFACE’05 (Martin et al., 2006) and RAVDESS (Livingstone & Russo, 2018).

Original: eNTERFACE’05 is an audio-visual emotion database in English.
Revised: eNTERFACE’05 is an audio-visual emotion database in English.

Original: It contains 42 subjects eliciting the six basic emotions: anger, disgust, fear, happiness, sadness, and surprise.
Revised: It contains 42 subjects eliciting the six basic emotions: anger, disgust, fear, happiness, sadness, and surprise.

Original: There are 213 videos for happiness, and 216 videos for each of the remaining emotions.
Revised: There are 213 videos for happiness, and 216 videos for each of the remaining emotions.

Original: Following (Ma et al., 2020), we extract 30 segment samples from each video and then obtain a processed dataset with 38,790 samples.
Revised: Following (Ma et al., 2020), we extract 30 segment samples from each video and then obtain a processed dataset with 38,790 samples.

Original: RAVDESS is a multimodal database of emotional speech and song, which consists of 24 professional actors in a neutral North American accent.
Revised: RAVDESS is a multimodal database of emotional speech and song, which consists of 24 professional actors in a neutral North American accent.

Original: Here, we use the speech part, which includes calm, happy, sad, angry, fearful, surprise, and disgust expressions.
Revised: Here, we use the speech part, which includes calm, happy, sad, angry, fearful, surprise, and disgust expressions.

Original: Each recording is also in the video form.
Revised: Each recording is also in the video form.

Original: Similar to the eNTERFACE’05 dataset, we only consider six basic emotions, each of which has 5, segment samples.
Revised: Similar to the eNTERFACE’05 dataset, we only consider six basic emotions, each of which has 5, segment samples.

Original: 3. E XPERIMENTAL S ETTINGS
Revised: 3. E XPERIMENTAL S ETTINGS

Original: We perform experiments on the processed eNTERFACE’05 and RAVDESS datasets.
Revised: We perform experiments on the processed eNTERFACE’05 and RAVDESS datasets.

Original: For each segmentsample of these two datasets, we take the central frame  as the visualmodality and extract the log Mel-spectrogram as the audio modality.
Revised: Each segment of these two datasets has a duration of 0.5 seconds. As shown in ((Ma et al., 2020)), consecutive frames within 0.5 seconds usually contain the same emotion in a similar way, which inspired us to choose the central frame of each segment as the visual modality.

Original: The spectrum representation is similar to the RGB image.
Revised: This technique not only makes that the visual data contains enough emotional information, but also avoids the redundancy in multiple frames. Besides, the log Mel-spectrogram is extracted from each segment as the audio modality, which is similar to the RGB image.

Original: We then feed these data into our framework to obtain the classiﬁcation result.
Revised: We then feed these data into our framework to obtain the classiﬁcation result.

Original: ResNet-50 (He et al., 2016) is used as the backbone of visual network f and audio network g to extract features from visual and audio modalities, respectively.
Revised: ResNet-50 (He et al., 2016) is used as the backbone of visual network f and audio network g to extract features from visual and audio modalities, respectively.

Original: In addition, we transform the corresponding label into the one-hot form and then extract the label feature using label network h with a fully connected layer.
Revised: In addition, we transform the corresponding label into the one-hot form and then extract the label feature using label network h with a fully connected layer.

Original: f , g , and h are trained together.
Revised: f , g , and h are trained together.

Original: On each processed dataset, we split all data into three parts: training set, validation set, and test set.
Revised: On each processed dataset, we split all data into three parts: training set, validation set, and test set.

Original: Their proportions are 70%, 15%, and 15%.
Revised: Their proportions are 70%, 15%, and 15%.

Original: In practice, modality missing often occurs with a high
Revised: In practice, modality missing often occurs with a high

Original: [Table]
Revised: [Table]

Original: missing rate (Suo et al., 2019; Ma et al., 2021b).
Revised: missing rate (Suo et al., 2019; Ma et al., 2021b).

Original: Here, in the training stage, we study three kinds of missing rates: 80%, 90%, and 95%.
Revised: Here, in the training stage, we study three kinds of missing rates: 80%, 90%, and 95%.

Original: The case where the audio modality is missing and the case where the visual modality is missing are investigated respectively.
Revised: The case where the audio modality is missing and the case where the visual modality is missing are investigated respectively.

Original: Following (Suo et al., 2019; Yu et al., 2020; Chen & Zhang, 2020; Du et al., 2021), we set modality missing arising during the training phase to show that a large amount of unimodal data can assist the training of our multimodal learning framework.
Revised: Following (Yu et al., 2020; Chen & Zhang, 2020; Du et al., 2021), we set modality missing arising during the training phase to show that a large amount of unimodal data can assist the training of our multimodal learning framework.

Original: In the inference phase, we use Equation (4) to predict the class label of the given test data.
Revised: In the inference phase, we use Equation (4) to predict the class label of the given test data.

Original: Finally, we run each experiment ﬁve times and report average test accuracies to evaluate the performance of our approach and baseline methods.
Revised: Finally, we run each experiment ﬁve times and report average test accuracies to evaluate the performance of our approach and baseline methods.

Original: Adam optimizer (Kingma & Ba, 2015) is used to train neural networks with the learning rate of 0.0001.
Revised: Adam optimizer (Kingma & Ba, 2015) is used to train neural networks with the learning rate of 0.0001.

Original: Both the size of modality-complete batch and the size of modality-missing batch are set to 90.
Revised: Both the size of modality-complete batch and the size of modality-missing batch are set to 90.

Original: The number of epochs is set to 100.
Revised: The number of epochs is set to 100.

Original: All experiments are implemented by Pytorch (Paszke et al., 2019) on a NVIDIA TITAN V GPU card.
Revised: All experiments are implemented by Pytorch (Paszke et al., 2019) on a NVIDIA TITAN V GPU card.

Original: 3. B ASELINE M ETHODS
Revised: 3. B ASELINE M ETHODS

Original: To show the effectiveness of our method, we compare our approach with the following methods which can also handle modality missing to some extent.
Revised: To show the effectiveness of our method, we compare our approach with the following methods which can also handle modality missing to some extent

Original: • Discarding Modality-incomplete Data (Lower Bound): One simple strategy to handle modality missing is to directly discard the modality-incomplete data, and then only use the modality-complete data for the classiﬁcation task.
Revised: • Discarding Modality-incomplete Data (Lower Bound): One simple strategy to handle modality missing is to directly discard the modality-incomplete data, and then only use the modality-complete data for the classiﬁcation task.

Original: This method does not use the information of the data with missing modalities.
Revised: This method does not use the information of the data with missing modalities.

Original: In our maximum likelihood estimation model, this is equivalent to calculating Q Z | XY without calculating Q Z | X .
Revised: In our maximum likelihood estimation model, this is equivalent to calculating Q Z | XY without calculating Q Z | X .

Original: Therefore, this method can also be used as the ablation study of our approach.
Revised: Therefore, this method can also be used as the ablation study of our approach.

Original: • Hirschfeld-Gebelein-Renyi Maximal Correlation (Hirschfeld, 1935; Gebelein, 1941; Rényi, 1959)
Revised: • Hirschfeld-Gebelein-Renyi Maximal Correlation (Hirschfeld, 1935; Gebelein, 1941; Rényi, 1959)

Original: (HGR MC): HGR MC is a statistical measure which calculates the dependence between different random variables.
Revised: (HGR MC): HGR MC is a statistical measure which calculates the dependence between different random variables.

Original: It has been successfully used for multimodal learning (Ma et al., 2021a; 2020; Wang et al., 2019; Xu & Huang, 2020).
Revised: It has been successfully used for multimodal learning (Ma et al., 2021a; 2020; Wang et al., 2019; Xu & Huang, 2020).

Original: Here, we use it further to deal with modality missing.
Revised: Here, we use it further to deal with modality missing.

Original: For the modality-complete data, we learn the maximal correlation between x , y , and z .
Revised: For the modality-complete data, we learn the maximal correlation between x , y , and z .

Original: For the modality-missing data, we learn the maximal correlation between x and z .
Revised: For the modality-missing data, we learn the maximal correlation between x and z .

Original: • Zero Padding (ZP):
Revised: • Zero Padding (ZP):

Original: Padding the feature representation of the missing modality with zero is another widely used way to cope with incomplete modalities (Jo et al., 2019;
Revised: Padding the feature representation of the missing modality with zero is another widely used way to cope with incomplete modalities (Jo et al., 2019;

Original: Chen et al., 2020; Shen et al., 2020).
Revised: Chen et al., 2020; Shen et al., 2020).

Original: For ZP, we consider two forms of φ to fuse features f
Revised: For ZP, we consider two forms of φ to fuse features f

Original: and g : addition and concatenation.
Revised: and g : addition and concatenation.

Original: The reason why the form of outer product is not studied here is that if the feature of one modality is zero, the outer product of it and the non-zero feature of another modality is also zero, which makes the modality-missing data useless.[Table]
Revised: The reason why the form of outer product is not studied here is that if the feature of one modality is zero, the outer product of it and the non-zero feature of another modality is also zero, which makes the modality-missing data useless.

Original: • Autoencoder (AE):
Revised: • Autoencoder (AE):

Original: An autoencoder is a neural network framework used to learn the representation from the training data.
Revised: An autoencoder is a neural network framework used to learn the representation from the training data.

Original: Some previous approaches apply autoencoders to complementthe missing modality (Liu et al., 2021;
Revised: Some previous approaches apply autoencoders to complement

Original: Jaques et
Revised: [Table]

Original: al ., 2017; Pereira et al. , 2020; Shi et al., 
Revised: the missing modality (Liu et al., 2021; Jaques et al., 2017; Pereira et al., 2020; Shi et al., 2019).

Original: Following these works, for the modality-complete data, we use modality x as the input of the autoencoder to reconstruct modality y .
Revised: Following these works, for the modality-complete data, we use modality x as the input of the autoencoder to reconstruct modality y .

Original: Then we use the trained autoencoder to predict the modality y to impute the modality-missing data.
Revised: Then we use the trained autoencoder to predict the modality y to impute the modality-missing data.

Original: Then we use the imputed data to perform the classiﬁcation task.
Revised: Then we use the imputed data to perform the classiﬁcation task.

Original: It is noted that the AE method has several stages while our method is end-to-end.
Revised: It is noted that the AE method has several stages while our method is end-to-end.

Original: For a fair comparison, we make that each method has the same network architecture and training strategy, and report the classiﬁcation results of each method after the same number of repeated experiments.
Revised: For a fair comparison, we make that each method has the same network architecture and training strategy, and report the classiﬁcation results of each method after the same number of repeated experiments.

Original: 3. E XPERIMENTAL R ESULTS
Revised: 3. E XPERIMENTAL R ESULTS

Original: We ﬁrst conduct classiﬁcation experiments on the eNTERFACE’05 and RAVDESS datasets by comparing our framework with other methods.
Revised: We ﬁrst conduct classiﬁcation experiments on the eNTERFACE’05 and RAVDESS datasets by comparing our framework with other methods.

Original: The experimental setting is shown in Section 3.2.
Revised: The experimental setting is shown in Section 3.2.

Original: We report the classiﬁcation accuracy of each method in each setting.
Revised: We report the classiﬁcation accuracy of each method in each setting.

Original: The results are shown in Tableand Table 2.
Revised: The results are shown in Tableand Table 2.

Original: We have the following summarizations from Table 1 and Table 2: (1) The methods of AE, HGR MC,
Revised: We have the following summarizations from Table 1 and Table 2: (1) The methods of AE, HGR MC,

Original: ZP, and ours can improve the classiﬁcation accuracy compared to the Lower Bound method which only uses the modality-complete data.
Revised: ZP, and ours can improve the classiﬁcation accuracy compared to the Lower Bound method which only uses the modality-complete data.

Original: Our method achieves the highest classiﬁcation performance among all methods under different settings.
Revised: Our method achieves the highest classiﬁcation performance among all methods under different settings.

Original: The higher the missing rate, the more obvious the gap between other methods and our method.
Revised: The higher the missing rate, the more obvious the gap between other methods and our method.

Original: These show that our maximum likelihood estimation approach are more effective to tackle modality missing compared with other methods.
Revised: These show that our maximum likelihood estimation approach are more effective to tackle modality missing compared with other methods.

Original: (2) Different forms of φ will affect the classiﬁcation performance.
Revised: (2) Different forms of φ will affect the classiﬁcation performance.

Original: For example, for our approach, addition and outer product perform better than concatenation on the eNTERFACE’05 dataset.
Revised: For example, for our approach, addition and outer product perform better than concatenation on the eNTERFACE’05 dataset.

Original: However, on the
Revised: However, on the

Original: RAVDESS dataset, the concatenation form of φ achieves higher classiﬁcation performance than the addition and outer product forms under some settings.
Revised: RAVDESS dataset, the concatenation form of φ achieves higher classiﬁcation performance than the addition and outer product forms under some settings.

Original: This indicates that in different settings, the discrimination ability of the learned feature representations is different.
Revised: This indicates that in different settings, the discrimination ability of the learned feature representations is different.

Original: We need to design the appropriate form of φ to fuse features of the multimodal data.
Revised: We need to design the appropriate form of φ to fuse features of the multimodal data.

Original: (3) When the visual modality is missing, the classiﬁcation accuracy is lower than that when the audio modality is missing, indicating that the visual modality has a more signiﬁcant contribution to the classiﬁcation performance, which is consistent with previous works (Zhang et al., 2017; Ma et al., 2020).
Revised: (3) When the visual modality is missing, the classiﬁcation accuracy is lower than that when the audio modality is missing, indicating that the visual modality has a more signiﬁcant contribution to the classiﬁcation performance, which is consistent with previous works (Zhang et al., 2017; Ma et al., 2020).

Original: In addition, we show the classiﬁcation confusion matrices using the methods of AE, HGR MC, ZP, and ours when the missing rate of visual modality reaches 95% on the eNTERFACE’05 dataset, as shown in Figure 4.
Revised: In addition, we show the classiﬁcation confusion matrices using the methods of AE, HGR MC, ZP, and ours when the missing rate of visual modality reaches 95% on the eNTERFACE’05 dataset, as shown in Figure 4.

Original: It can be seen that the classiﬁcation accuracy of each emotion using AE or HGR
Revised: It can be seen that the classiﬁcation accuracy of each emotion using AE or HGR

Original: MC is not high, which indicates that they can only deal with modality missing to a certain extent.
Revised: MC is not high, which indicates that they can only deal with modality missing to a certain extent.

Revised: The overall classiﬁcation performance of ZP is lower than ours, but the classiﬁcation accuracy of

Original: [Table]
Revised: [Table]

Original: ZP
Revised: ZP

Original: [Figure]
Revised: [Figure]

Original: [Figure] The overall classiﬁcation performance of ZP is lower than ours, but the classiﬁcation accuracy of “happiness” is slightly higher than ours.
Revised: “happiness” is slightly higher than ours.

Original: This shows that our method is more efﬁcient to exploit the information from most categories for the classiﬁcation task.
Revised: This shows that different emotions have different clues for the classiﬁcation task.

Original: We then investigate the effect of the backbone in coping with modality missing.
Revised: We then investigate the effect of the backbone in coping with modality missing.

Original: In the above experiments, we use ResNet-50 as the backbone of different methods to extract feature representations.
Revised: In the above experiments, we use ResNet-50 as the backbone of different methods to extract feature representations.

Original: Here, we replace ResNet-50 with ResNet-34 (He et al., 2016) and VGG-16 (Simonyan & Zisserman, 2015) respectively, and conduct experiments to compare the performance of different backbones when 95% of the training data has missing visual modality on the RAVDESS dataset, as shown in
Revised: Here, we replace ResNet-50 with ResNet-34 (He et al., 2016) and VGG-16 (Simonyan & Zisserman, 2015) respectively, and conduct experiments to compare the performance of different backbones when 95% of the training data has missing visual modality on the RAVDESS dataset, as shown in

Original: Figure 5.
Revised: Figure 5.

Original: We can observe that compared with VGG-16 and ResNet-34, ResNet-50 achieves the highest performance.
Revised: We can observe that compared with VGG-16 and ResNet-34, ResNet-50 achieves the highest performance.

Original: In addition, no matter what kind of backbone is based on, the classiﬁcation accuracy using our method is the highest, followed by using AE, ZP and HGR MC, and the lowest using Lower Bound, which shows that our approach can take effect for different backbones.
Revised: In addition, no matter what kind of backbone is based on, the classiﬁcation accuracy using our method is the highest, followed by using AE, ZP and HGR MC, and the lowest using Lower Bound, which shows that our approach can take effect for different backbones.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Multimodal learning has achieved great successes in many applications.
Revised: Multimodal learning has achieved great successes in many applications.

Original: An important topic in this ﬁeld is multimodal representations (Baltrušaitis et al., 2018; Zhu et al., 2020), which learn feature representations from the multimodal data by using the information of different modalities.
Revised: An important topic in this ﬁeld is multimodal representations (Baltrušaitis et al., 2018; Zhu et al., 2020), which learn feature representations from the multimodal data by using the information of different modalities.

Original: How to learn good representations is investigated in (Ngiam et al., 2011;
Revised: How to learn good representations is investigated in (Ngiam et al., 2011;

Original: Wu et al., 2014; Pan et al., 2016; Xu et al., 2015).
Revised: Wu et al., 2014; Pan et al., 2016; Xu et al., 2015).

Original: Another important topic is multimodal fusion (Atrey et al., 2010; Poria et al., 2017), which combines the information from different modalities to make predictions.
Revised: Another important topic is multimodal fusion (Atrey et al., 2010; Poria et al., 2017), which combines the information from different modalities to make predictions.

Original: Feature-based fusion is one of the most common types of multimodal fusion.
Revised: Feature-based fusion is one of the most common types of multimodal fusion.

Original: It concatenates the feature representations extracted from different modalities.
Revised: It concatenates the feature representations extracted from different modalities.

Original: This fusion approach is adopted by previous works (Tzirakis 
Revised: This fusion approach is adopted by previous works (Tzirakis et al., 2017; Zhang et al., 2017;

Original: [Figure]
Revised: Castellano et al., 2008; Zhang et al., 2016). Modality missing is a key challenge in applying multimodal learning to the real world.

Original: [Figure]
Revised: [Figure]

Original: AE

Original: HGR MC

Original: ZP
Revised: ZP

Original: [Figure]
Revised: [Figure]

Original: (b) ResNetLower
Revised: (b) ResNet

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: AE
Revised: AE

Original: HGR MC
Revised: HGR MC

Original: ZP
Revised: ZP

Original: Ours (c) ResNet-et al., 2017; Zhang et al., 2017; Castellano et al., 2008; Zhang et al., 2016). Modality missing is a key challenge in applying multimodal learning to the real world.
Revised: [Figure] (c) ResNet-

Original: To cope with the problem of modality missing, a few methods have been proposed.
Revised: To cope with the problem of modality missing, a few methods have been proposed.

Original: For example,
Revised: For example,

Original: Ma et al.
Revised: Ma et al.

Original: (2021b) propose a Bayesian meta learning framework to perturb the latent feature space so that embeddings of single modality can approximate embeddings of full modality.
Revised: (2021b) propose a Bayesian meta learning framework to perturb the latent feature space so that embeddings of single modality can approximate embeddings of full modality.

Original: Tran et al.
Revised: Tran et al.

Original: (2017) propose a cascaded residual autoencoder for imputation with missing modalities, which is composed of a set of stacked residual autoencoders that iteratively model the residuals.
Revised: (2017) propose a cascaded residual autoencoder for imputation with missing modalities, which is composed of a set of stacked residual autoencoders that iteratively model the residuals.

Original: Chen & Zhang (2020) propose a heterogeneous graph-based multimodal fusion approach to enable multimodal fusion of incomplete data within a heterogeneous graph structure.
Revised: Chen & Zhang (2020) propose a heterogeneous graph-based multimodal fusion approach to enable multimodal fusion of incomplete data within a heterogeneous graph structure.

Original: Liu et al.
Revised: Liu et al.

Original: (2021) propose an autoencoder framework to complement the missing data in the kernel space while taking into account the structural information of data and the inherent association between multiple views.
Revised: (2021) propose an autoencoder framework to complement the missing data in the kernel space while taking into account the structural information of data and the inherent association between multiple views.

Original: The above approaches can combine the information of the modality-missing data to some extent.
Revised: The above approaches can combine the information of the modality-missing data to some extent.

Original: Our work is signiﬁcantly different from them.
Revised: Our work is signiﬁcantly different from them.

Original: The reason lies in the following two facts.
Revised: The reason lies in the following two facts.

Original: Firstly, by exploiting the likelihood function to learn the conditional distributions of the modality-complete data and the modality-missing data, our method has a theoretical guarantee, which is skipped by previous works.
Revised: Firstly, by exploiting the likelihood function to learn the conditional distributions of the modality-complete data and the modality-missing data, our method has a theoretical guarantee, which is skipped by previous works.

Original: Secondly, the training process of our approach is in an end-to-end manner, while the training processes of most above methods are relatively cumbersome.
Revised: Secondly, the training process of our approach is in an end-to-end manner, while the training processes of most above methods are relatively cumbersome.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Multimodal learning is a hot topic in the academic and industry communities, of which a key challenge is modality missing.
Revised: Multimodal learning is a hot topic in the academic and industry communities, of which a key challenge is modality missing.

Original: In practice, the multimodal data may not be complete due to various reasons.
Revised: In practice, the multimodal data may not be complete due to various reasons.

Original: Most previous works cannot effectively utilize the modality-missing data for the learning task.
Revised: Most previous works cannot effectively utilize the modality-missing data for the learning task.

Original: To address this problem, we propose an efﬁcient approach to leverage the knowledge in the modalitymissing data during the training stage.
Revised: To address this problem, we propose an efﬁcient approach to leverage the knowledge in the modalitymissing data during the training stage.

Original: Speciﬁcally, we present a framework based on maximum likelihood estimation to characterize the conditional distributions of the modality-complete data and the modality-missing data, which has a theoretical guarantee.
Revised: Speciﬁcally, we present a framework based on maximum likelihood estimation to characterize the conditional distributions of the modality-complete data and the modality-missing data, which has a theoretical guarantee.

Original: Furthermore, we develop a generalized form of the softmax function to effectively implement our maximum likelihood estimation framework in an end-to-end way.
Revised: Furthermore, we develop a generalized form of the softmax function to effectively implement our maximum likelihood estimation framework in an end-to-end way.

Original: We conduct experiments on the eNTERFACE’05 dataset and the RAVDESS dataset for multimodal learning to demonstrate the effectiveness of our approach.
Revised: We conduct experiments on the eNTERFACE’05 dataset and the RAVDESS dataset for multimodal learning to demonstrate the effectiveness of our approach.

Original: In the future, we can further extend our framework to other multimodal learning domains.
Revised: In the future, we can further extend our framework to other multimodal learning domains.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide our code in “ supplement.zip ”.
Revised: We provide our code in “ supplement.zip ”.

Original: In this folder, “ eNTERFACE_preprocess.py ” and “ RAVDESS_preprocess.py ” extract segment samples from the original videos of the eNTERFACE’05 dataset and the RAVDESS dataset, respectively.
Revised: In this folder, “ eNTERFACE_preprocess.py ” and “ RAVDESS_preprocess.py ” extract segment samples from the original videos of the eNTERFACE’05 dataset and the RAVDESS dataset, respectively.

Original: “ mle.py ” shows the function to compute our maximum likelihood estimation algorithm.
Revised: “ mle.py ” shows the function to compute our maximum likelihood estimation algorithm.