Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! (2020)

Abstract: Modeling expressive cross-modal interactions seems crucial in multimodal tasks, such as visual question answering. However, sometimes high-performing black-box algorithms turn out to be mostly exploiting unimodal signals in the data. We propose a new diagnostic tool, empirical multimodally-additive function projection (EMAP), for isolating whether or not cross-modal interactions improve performance for a given model on a given task. This function projection modifies model predictions so that cross-modal interactions are eliminated, isolating the additive, unimodal structure. For seven image+text classification tasks (on each of which we set new state-of-the-art benchmarks), we find that, in many cases, removing cross-modal interactions results in little to no performance degradation. Surprisingly, this holds even when expressive models, with capacity to consider interactions, otherwise outperform less expressive models; thus, performance improvements, even when present, often cannot be attributed to consideration of cross-modal feature interactions. We hence recommend that researchers in multimodal machine learning report the performance not only of unimodal baselines, but also the EMAP of their best-performing model.

## introduction given the presumed importance of reasoning across modalities in multimodal machine learning tasks, we should evaluate a model's ability to leverage cross-modal interactions. but such evaluation is not straightforward; for example, an early visual question-answering (vqa) challenge was later "broken" by a high-performing method that ignored the image entirely (jabri et al., 2016). one response is to create multimodal-reasoning datasets that are specifically and cleverly balanced to resist language-only or visual-only models; examples are vqa 2.0 (goyal et al., 2017), nlvr2 ## emap background. we consider models f that assign scores to textual-visual pairs (t, v), where t is a piece of text (e.g., a sentence), and v is an image. @xcite in multi-class classification settings, values f (t, v) ∈ r d typically serve as per-class scores. in ranking settings f (t 1 , v 1 ) may be compared to f (t 2 , v 2 ) via a ranking objective. we are particularly interested in the types of compositionality that f uses over its visual and textual inputs to produce scores. specifically, we distinguish between additive models (hastie and tibshirani, 1987) and interactive models (friedman, 2001; friedman and popescu, 2008). @xcite a function f of a representation z of n input features is additive in i if it decomposes as in our case, because features are the union of textual and visual predictors, we say that a model is multimodally additive if it decomposes as additive multimodal models are simply ensembles of unimodal classifiers and as such, may be considered underwhelming to multiple communities. a recent acl paper, for example, refers to such ensembles as "naive." on the semiotics side, the conditionality implied by meaning multiplication (barthes, 1988) -that the joint semantics of an image/caption depends non-linearly on its accompaniment -cannot be modeled additively: multimodally additive models posit that each image, independent of its text pairing, contributes a fixed score to per-class logits (and vice versa). in contrast, multimodally interactive models are the set of functions that cannot be decomposed as in equation 1 -that is, f 's output conditionally depends on its inputs in a non-additive fashion. machine learning models. one canonical multimodally additive model is a linear model trained over a concatenation of textual and visual features [t; v], i.e., we later detail several multimodally interactive models, including multi-layer neural networks, polynomial kernel svms, pretrained transformer attention-based models, etc. however, even though interactive models are theoretically capable of modeling a more expressive set of relationships, it's not clear that they will learn to exploit this additional expressivity, particularly when simpler patterns suffice. empirical multimodally-additive projections: emap. given a fixed, trained model f (usually one theoretically capable of modeling visualtextual interactions) and a set of n labelled datapoints {(v i , t i , y i )} n @xmath0, we aim to answer: does f utilize cross-modal feature interactions to produce more accurate predictions for these datapoints ? hooker (2004) gives a general method for computing the projection of a function f onto a set of functions with a specified anova structure (see also sobol (2001); liu and owen (2006)): our algorithmic contributions are to extend the method to multimodal models with @xmath1, and to prove that the multimodal empirical approximation is optimal. specifically: we are algorithm 1 empirical multimodally-additive projection (emap); worked example in appendix g. input: a trained model f that outputs logits; a set of text/visual pairs @xmath2, vi)} n @xmath3 output: the predictions of f , the empirical projection of f onto the set of multimodally-additive functions, on d. in appendix a, we prove that the (meancentered) sum of empirical partial dependence functions is optimal with respect to squared error, that is: claim. subject to the constraint that f is multimodally additive, algorithm 1 produces a unique and optimal solution to arg min in §5.1, we will see that emap provides a very strong baseline for "unbalanced" multimodal classification tasks. but first, we first seek to verify that emap degrades model performance in cases that are designed to require cross-modal interactions. ## conclusion and future work the last question on our faq list in §6 leaves us with the following conundrum: 1) additive models are incapable of most cross-modal reasoning; but 2) for most of the unbalanced tasks we consider, emap finds an additive approximation that makes nearly identical predictions to the full, interactive model. we postulate the following potential explanations, pointing towards future work: concluding thoughts. our hope is that future work on multimodal classification tasks report not only the predictive performance of their best model + baselines, but also the emap of that model. emap (and related algorithms) has practical implications beyond image+text classification: there are straightforward extensions to nonvisual/non-textual modalities, to classifiers using more than 2 modalities as input, and to singlemodal cases where one wants to check for feature interactions between two groups of features, e.g., premise/hypothesis in nli. focused research award. any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsors.