MM -Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences (2022)

Abstract: Existing multimodal tasks mostly target at the complete input modality setting, i.e., each modality is either complete or completely missing in both training and test sets. However, the randomly missing situations have still been underexplored. In this paper, we present a novel approach named MM-Align to address the missing-modality inference problem. Concretely, we propose 1) an alignment dynamics learning module based on the theory of optimal transport (OT) for missing data imputation; 2) a denoising training algorithm to enhance the quality of imputation as well as the accuracy of model predictions. Compared with previous generative methods which devote to restoring the missing inputs, MM-Align learns to capture and imitate the alignment dynamics between modality sequences. Results of comprehensive experiments on two multimodal tasks empirically demonstrate that our method can perform more accurate and faster inference and alleviate the overfitting issue under different missing conditions.

## introduction the topic of multimodal learning has grown unprecedentedly prevalent in recent years @xcite , ranging from a variety of machine learning tasks such as computer vision @xcite , natural langauge processing @xcite , autonomous driving @xcite and medical care @xcite , etc. despite the promising achievements in these fields, most of existent approaches assume a complete input modality setting of training data, in which every modality is either complete or completely missing (at inference time) in both training and test sets @xcite @xcite , as shown in fig. 1a and 1b . such synergies between train and test sets in the modality input patterns are usually far from the realistic scenario where there is a certain portion of data without parallel modality sequences, probably due to noise pollution during collecting and preprocessing time. in other words, data from each modality are more probable to be missing at random (fig. 1c and 1d ) than completely present or missing (fig. 1a and 1b ) @xcite @xcite . based on the complete input modality setting, a family of popular routines regarding the missing-modality inference is to design intricate generative modules attached to the main network and train the model under full supervision with complete modality data. by minimizing a customized reconstruction loss, the data restoration (a.k.a. missing data imputation (van buuren, 2018)) capability of the generative modules is enhanced @xcite @xcite so that the model can be tested in the missing situations (fig. 1b ). however, we notice that (i) if modality-complete data in the training set is scarce, a severe overfitting issue may occur, especially when the generative model is large @xcite @xcite ; (ii) global attention-based (i.e., attention over the whole sequence) imputation may bring unexpected noise since true correspondence mainly exists between temporally adjacent parallel signals @xcite . @xcite proposed to leverage unit-length sequential representation to represent the missing modality from the seen complete modality from the input for training. nevertheless, such kinds of methods inevitably overlook the temporal correlation between modality sequences and only acquire fair performance on the downstream tasks. to mitigate these issues, in this paper we present mm-align, a novel framework for fast and effective multimodal learning on randomly missing multimodal sequences. the core idea behind the framework is to imitate some indirect but informative clues for the paired modality sequences instead of learning to restore the missing modality directly. the framework consists of three essential functional units: 1) a backbone network that handles the main task; 2) an alignment matrix solver based on the optimal transport algorithm to produce contextwindow style solutions only part of whose values are non-zero and an associated meta-learner to imitate the dynamics and perform imputation in the modality-invariant hidden spaces; 3) a denoising training algorithm that optimizes and coalesces the backbone network and the learner so that they can work robustly on the main task in missing-modality scenarios. to empirically study the advantages of our models over current imputation approaches, we test on two settings of the random missing conditions, as shown in fig. 1c and fig. 1d , for all possible modality pair combinations. to the best of our knowledge, it is the first work that applies optimal transport and denoising training to the problem of inference on missing modality sequences. in a nutshell, the contribution of this work is threefold: ## analysis impact of the window size to further explore the impact of window size, we run our models by increasing window size from 4 to 256 which exceeds the lengths of all sentences so that all timestamps are enclosed by the window. the variation of mae and f1 in this process is depicted in fig. 4 . there is a dropping trend (mae increment or f1 decrement) towards both sides of the optimal size. we argue that it is because when the window expands, it is more probable for the newly included frame to add noise rather than provide valuable alignment information. in the beginning, the marginal benefit is huge so the performance almost keeps climbing. the optimal size is reached when the marginal benefit decreases to zero. to explain this claim, we randomly select a raw example from the cmu-mosi dataset. as shown in fig. 3 , the textual expression does not advance in a uniform speed. from the second to the third word 1.80 seconds elapses, while the last eight words are covered in only 2.53 seconds. intuitively we can assume all the frames in the video that span across the pronunciation of a word are causally correlated with that word so that the representation mappings from the word to these frames are necessary and can benefit the downstream tasks. for example, for the word "i" present at @xmath0, it can benefit the timestamps until at least @xmath1. note that we may overlook some potential advantages that could not be easily justified in this way and possess different effect scope, but we deem that those advantages would like-wisely disappear as the window size keeps growing. ## conclusion in this paper, we propose mm-align, a fast and efficient framework for the problem of missing modality inference. it applies the theory of optimal transport to learn the alignment dynamics between temporal modality sequences for the inference in the case of missing modality sequences. experiments on three datasets of demonstrate that mm-align can achieve much better performance and thus reveal the higher robustness of our method. we hope that our work can inspire other research works in this field.