Greedy Modality Selection via Approximate Submodular Maximization
Abstract
Multimodal learning considers learning from multimodality data, aiming to fuse heterogeneous sources of information.
However, it is not always feasible to leverage all available modalities due to memory constraints.
Further, training on all the modalities may be inefﬁcient when redundant information exists within data, such as different subsets of modalities providing similar performance.
In light of these challenges, we study modality selection , intending to efﬁciently select the most informative and complementary modalities under certain computational constraints.
We formulate a theoretical framework for optimizing modality selection in multimodal learning and introduce a utility measure to quantify the beneﬁt of selecting a modality.
For this optimization problem, we present efﬁcient algorithms when the utility measure exhibits monotonicity and approximate submodularity.
We also connect the utility measure with existing Shapley-value-based feature importance scores.
Last, we demonstrate the efﬁcacy of our algorithm on synthetic (Patch-MNIST) and real-world (PEMS-SF, CMU-MOSI) datasets.
INTRODUCTION
Multimodal learning considers learning with data from multiple modalities (e g., images, text, speech, etc) to improve generalization of the learned models by using complementary information from different modalities.
In many realworld applications, multimodal learning has shown superior performance
[Bapna et al., 2022, Wu et al., 2021]
and has demonstrated a stronger capability over learning from a single modality.
The advantages of multimodal learning have also been studied from a theoretical standpoint.
Prior work showed that learning with more modalities achieves a smaller population risk [Huang et al., 2021], or utilizing cross-modal information can provably improve prediction in multiview learning [Zhang et al., 2019] or semi-supervised learning [Sun et al., 2020].
With the recent advances in training large-scale neural network models from multiple modalities [Devlin et al., 2018, Brown et al., 2020], one emerging challenge lies in the modality selection problem.
From the modeling perspective, it might be tempting to use all the modalities available.
However, it is inefﬁcient or even infeasible to learn from all modalities as the total number of input modalities increases.
A modality often consists of highdimensional data.
And model complexity can scale linearly or exponentially with the number of input modalities [Zadeh et al., 2017, Liu et al., 2018], resulting in large consumption of computational and energy resources.
The marginal beneﬁt from the new modalities may also decrease as more modalities have been included.
In some cases, learning from fewer modalities is sufﬁcient to achieve the desirable outcome, due to the potential overlap in the information provided by these modalities.
Furthermore, proactively selecting the modalities most informative towards prediction reduces the cost of collecting the inferior ones.
For example, in sensor placement problems where each sensor can be treated as a modality, ﬁnding the optimal subset of sensors for a learning objective (e.g., temperature or trafﬁc prediction) eliminates the cost of maintaining extra sensors [Krause et al., 2011].
In light of the aforementioned challenges, in this paper, we study the optimization problem of modality selection in multimodal learning: given a set of input modalities and a ﬁxed budget on the number of selected modalities, how to select a subset that optimizes prediction performance?
Note that in general this problem is of combinatorial nature, since one may have to enumerate all the potential subsets of modalities in order to ﬁnd the best one.
Hence, without further assumptions on the structure of the underlying prediction problem, it is intractable to solve this modality selection problem exactly and efﬁciently.
To approach these challenges, we propose a utility function that conveniently quantiﬁes the beneﬁt of any set of modalities towards prediction in most typical learning settings.
We then identify a proper assumption that is suitable for multimodal/multiview learning, which allows us to develop efﬁcient approximate algorithms for modality selection.
We assume that the input modalities are approximately conditionally independent given the target.
Since the strength of conditional independence is now parameterized, our results are generalizable to problems on multimodal data with different levels of conditional independence.
We show that our deﬁnition of utility for a modality naturally manifests as the Shannon mutual information between the modality and the prediction target, in the setting of binary classiﬁcation with cross-entropy loss.
Under approximate conditional independence, mutual information is monotone and approximately submodular.
These properties intrinsically describe the empirical advantages of learning with more modalities, and allow us to formulate modality selection as a submodular optimization problem.
In this context, we can have efﬁcient selection algorithms with provable performance guarantee on the selected subset.
For example, we show a performance guarantee of the greedy maximization algorithm from Nemhauser et al.
[1978] under approximate submodularity.
Further, we connect modality selection to marginal-contribution-based feature importance scores in feature selection.
We examine the Shapley value and
Marginal Contribution Feature Importance (MCI) [Catav et al., 2021] for ranking modality importance.
We show that these scores, although are originally intractable, can be solved efﬁciently under assumptions in the context of modality selection.
Lastly, we evaluate our theoretical results on three classiﬁcation datasets.
The experiment results conﬁrm both the utility and the diversity of the selected modalities.
To summarize, we contributes the following in this paper:
• Propose a general measure of modality utility, and identify a proper assumption that is suitable for multimodal learning and helpful for developing efﬁcient approximate algorithms for modality selection.
• Demonstrate algorithm with performance guarantee on the selected modalities for prediction theoretically and empirically in classiﬁcation problems with crossentropy loss.
• Establish theoretical connections between modality selection and feature importance scores, i.e., Shapley value and Marginal Contribution Feature Importance.
PRELIMINARIES
In this section, we ﬁrst describe our notation and problem setup, and we then provide a brief introduction to submodular function maximization and feature importance scores.
2.1 NOTATION AND SETUP
We use X and Y to denote the random variables that take values in input space X and output space Y , respectively.
The instantiation of X and Y is denoted by x and y .
We use
H to denote the hypothesis class of predictors from input to output space, and ˆ Y to denote the predicted variable.
Let X be multimodal, i.e., X = X 1 × ...
×X k .
Each X
i is the input from the i -th modality.
We use X i to denote the random variable that takes value in X i , and V to denote the full set of all input modalities, i.e., V = { X 1 , ..., X k } .
Throughout the paper, we often use S and S (cid:48) to denote arbitrary subsets of V .
Lastly, we use I ( · , · ) to mean the Shannon mutual information, H ( · ) for entropy, (cid:96) ce
( Y, ˆ Y ) for the cross-entropy loss 1 ( Y = 1) log ˆ Y + 1 ( Y = 0) log(1 − ˆ Y ) , and (cid:96)
( Y, ˆ Y ) for zero-one loss 1 ( Y (cid:54) =
ˆ Y ) .
For the simplicity of discussion, we primarily focus on the setting of binary classiﬁcation with cross-entropy loss 2 .
In this setting, a subset of input modalities S ⊆ V and output
Y ∈ { 0 , 1 } are observed.
The predictor aims to make prediction ˆ Y ∈ [0 , 1] which minimizes the cross-entropy loss between Y and ˆ Y .
The goal of modality selection is to select the subset of input modalities to this loss minimization goal under certain constraints.
Our results rely on the following assumption to hold.
Assumption 2.1 ( (cid:15) -Approximate
Conditional Independence) .
There exists a positive constant (cid:15) ≥ 0 such that, ∀ S, S (cid:48) ⊆
V, S ∩ S (cid:48)
= ∅ , we have I ( S ; S (cid:48) | Y ) ≤ (cid:15) .
Note that when (cid:15) = 0 , Assumption 2.1 reduces to strict conditional independence between disjoint modalities given the target variable.
In fact, this is a common assumption used in prior work in multimodal learning [White et al., 2012,
Wu and Goodman, 2018, Sun et al., 2020].
In practice, however, strict conditional independence is often difﬁcult to be satisﬁed.
Thus, we use a more general assumption above, in which input modalities are approximately conditionally independent.
In this assumption, the strength of the conditional independence relationship is controlled by a positive constant (cid:15) , which is the upper bound of the conditional mutual information between modalities given the target.
Connection to feature selection.
It is worth mentioning that modality selection shares a natural correspondence to the problem of feature selection.
Without loss of generality, a modality could be considered as a group of features; theoretically, the group could even contain a single feature in some settings.
But a distinction between these two problems lies in the feasibility of conditional independence.
In multimodal learning where input data is often heterogeneous, the (approximate) independence assumption more likely to hold among input modalities.
Whereas in the feature level, such an assumption is quite restrictive [Zhang et al., 2012], as it boils down to asking the data to approximately satisfy the Naive Bayes assumption.
2.2 SUBMODULAR OPTIMIZATION
Submodularity is a property of set functions that has many theoretical implications and applications in computer science.
A deﬁnition of submodularity is as follows, where 2 V denotes the power set of V , and the set function f assigns each subset S ⊆ V to a value f ( S ) ∈ R .
Deﬁnition 2.1 (Nemhauser et al.
Given a ﬁnite set V , a function f : 2 V → R is submodular if for any
A ⊆ B ⊆ V , and e ∈ V \ B , we have f
( A ∪{ e } ) − f ( A ) ≥ f
( B ∪ { e } ) − f ( B ) .
In other words, adding new elements to a larger set does not yield larger marginal beneﬁt comparing to adding new elements to its subset.
One common type of optimization on submodular function is submodular function maximization with cardinality constraints.
It asks to ﬁnd a subset
S ⊆ V that maximizes f ( S ) subject to | S | ≤ q .
Finding the optimal solution to this problem is NP -hard.
However,
Nemhauser et al.
[1978] propose that a greedy maximization algorithm can provide a solution with approximate guarantee to the optimal solution in polynomial time.
We provide the pseudocode of this greedy algorithm below.
Data: Full set V =
{ X 1 , ..., X k } , constraint q ∈ Z + .
Input: f : 2 V → R , and p ∈
Z
+ , where p ≤ q ≤ | V |
Output: Subset S p
S 0 = ∅ for i = 0 , 1 , ..., p − 1 do
X i = arg max X j ∈ V \ S i ( f ( S i ∪ { X j } ) − f ( S i ))
S i +
=
S i ∪ { X i }
In this algorithm, V is the full set to select elements from, f is the submodular function to be maximized, p is the number of iterations for the algorithm to run, and q is the cardinality constraint.
It starts with an empty set S 0 , and subsequently adds to the current set S i the element X i that maximizes the marginal gain f
( S i ∪ { X j } ) − f ( S i ) at each iteration i .
Algorithm 1 runs in pseudo-polynomial time
O
( p | V | ) , and has an approximation guarantee as follows.
Theorem 2.1 (Nemhauser et al.
Let q ∈ Z
+ , S p be the solution from Algorithm 1 at iteration p , and e is the
Euler’s number, we have:
[Equation]
max S : | S |≤ q f ( S ) is the optimal value from the optimal subset whose cardinality is at most q .
If f is monotone, arg max S : | S |≤ q f ( S ) has cardinality exactly q .
By running
Algorithm 1 for exactly q iterations, we obtain a greedilyobtained value that is at least 1 − 1 e of the optimal value.
2.3 FEATURE IMPORTANCE SCORES
The feature importance domain in machine learning studies scoring methods that measure the contribution of individual features.
A common setting of these feature importance scoring methods is to treat each feature as a participant in a coalitional game, in which all of them contribute to an overall gain.
Then a scoring method assigns each feature a importance score by evaluating their individual contributions.
Many notable feature importance scores are adapted from the Shapley value, deﬁned as follows:
Deﬁnition 2.2 (Shapley [1952]) .
Given a set of all players
F in a coalitional game v : 2 F → R , the Shapley value of player i deﬁned by v is:
[Equation]
The Shapley value of a player i is the average of its marginal contribution in game v in each possible player subsets excluding i .
The game v is a set function that computes the gain of a set of players.
Computing the exact Shapley value of a player is (cid:93) P -hard, and its complexity is exponential to the number of players in the coalitional game – there are O (2 | F | ) unique subsets, and each subset S could have a unique (cid:52) v ( i | S ) value [Roth, 1988, Winter, 2002].
Nonetheless, in certain game settings, there are approximation methods to Shapley value, such as Monte Carlo simulation [Faigle and Kern, 1992].
When Shapley value is adapted to the feature importance domain, each input feature is a player, and v is also called the evaluation function .
But v is not unique – it can be a prediction model or utility measure; different v may induce different properties to the
Shapley value.
We also examine another feature importance score from
Catav et al.
[2021], known as the Marginal Contribution
Feature Importance (MCI).
Kumar et al.
[2020] has shown that Shapley-value-based feature importance scores [Shapley, 1952, Lundberg and Lee, 2017, Covert et al., 2020] could underestimate the importance of correlated features by assigning them lower scores if these features present together in the full set.
In light of this, MCI is proposed to overcome this issue.
In Deﬁnition 2.3, MCI of a feature i is the maximum marginal contribution in v over all possible feature subsets.
The complexity of computing the exact MCI of a feature is also exponential to the number of features.
Deﬁnition 2.3 (Catav et al.
Given a set of all features F , and a non-decreasing set function v : 2 F → R , the
MCI of feature i evaluated on v is:
[Equation]
MODALITY SELECTION
This section presents our theoretical results.
In Section 3.1, we introduce the utility function to measure the prediction beneﬁt of modalities, and present its subsequent properties.
In Section 3.2, we present theoretical guarantees of greedy modality selection via maximizing an approximately submodular function.
In Section 3.3, we show computational advantages of feature importance scores (i.e., Shapley value and MCI) in the context of modality selection.
Due to space limit, proofs are deferred to supplementary materials.
3.1 UTILITY FUNCTION
In order to compare the beneﬁt of different sets of input modalities in multimodal learning, we motivate a general deﬁnition of utility function that can quantify the impact of a set of input modalities towards prediction.
Deﬁnition 3.1.
Let c be some constant in the output space, and (cid:96) ( · , · ) be a loss function.
For a set of input modalities
S ⊆ V , the utility of S given by the utility function f u : 2 V → R is deﬁned to be:
[Equation]
In other words, the utility of a set of modalities f u ( S ) is the reduction of the minimum expected loss in predicting Y by observing S comparing to observing some constant value c .
The intuition is based on the phenomena that multimodal input tends to reduce prediction loss in practice.
Deﬁnition 3.
can be easily interpretable in different loss functions and learning settings.
Note that it is also used in feature selection to measure the unversial predictive power of a given feature [Covert et al., 2020].
Under the binary classiﬁcation setting with cross-entropy loss, f u is the Shannon mutual information between the output and multimodal input.
Proposition 3.1.
Given Y ∈ { 0 , 1 } and (cid:96) ( Y, ˆ Y ) :
= 1 ( Y = 1) log ˆ Y +
( Y = 0) log(1 − ˆ Y ) , f u ( S )
=
I ( S ; Y ) .
The result above is well-known, and has also been proven in [Grünwald and Dawid, 2004, Farnia and Tse, 2016].
We further can show that I ( S ; Y ) is monotonically nondecreasing on the set of input modalities S .
Proposition 3.2.
∀ M ⊆ N ⊆ V , I ( N ; Y )
− I ( M ; Y ) =
I ( N \ M ; Y | M ) ≥ 0 .
A combination of Proposition 3.1 and Proposition 3.2 implies that using more modalities as input leads to equivalent or better prediction.
It also shows that Deﬁnition 3.1 can quantitatively capture the extra prediction beneﬁt from the additional modalities in closed-form (e.g., I ( N \ M ; Y |
M ) ).
And this extra beneﬁt is the most apparent when test loss reaches convergence (e.g., inf ).
This monotonicity property is also a key indication that Deﬁnition 3.1 can intrinsically characterize the advantage of multimodal learning over unimodal learning.
Comparison to previous results.
Previous work [Amini et al., 2009, Huang et al., 2021] have discovered similar conclusions that more views/modalities will not lead to worse optimal population error in the context of multiview and multimodal learning, respectively.
They obtained this observation through analysis to the excess risks of learning from multiple and single modalities, and show that the excess risk of learning from multiple modalities cannot be larger than that of single modality.
Instead, our work adopts an information-theoretic characterization, which leads to an easy-to-interpret measure on the beneﬁts of additional modalities.
Furthermore, using well-developed entropy estimators, it is relatively straightforward to estimate these measures in practice.
As a comparison, excess risks are hard to estimate in practice, since they depend on the Bayes optimal errors, which limits their uses in many applications.
Next, we show that f u ( S ) =
I ( S ; Y ) is approximately submodular under Assumption 2.1.
Previously, Krause and
Guestrin [2012] has shown mutual information to be submodular under strict conditional independence.
Here we provide a more ﬂexible notion of submodularity for mutual information.
There are also other generalizations of submodularity such as weak submodularity [Khanna et al., 2017] or adaptive submodularity [Golovin and Krause, 2011].
Our deﬁnition of approximate submodularity is more speciﬁc to the case of mutual information and Assumption 2.1.
Proposition 3.3.
Under Assumption 2.1, I ( S ; Y ) is (cid:15) - approximately submodular, i.e., ∀ A ⊆ B ⊆ V , e ∈ V \ B ,
I ( A ∪ { e } ; Y ) − I ( A ; Y ) + (cid:15) ≥ I ( B ∪ { e } ; Y ) − I ( B ; Y ) .
The above proposition states that if conditional mutual information between input modalities given output is below a certain threshold (cid:15) > 0 , then the utilty function f u ( · ) = I ( · ; Y ) admits a diminishing gain pattern controlled by (cid:15) .
This diminishing gain pattern is the deﬁnition of submodularity (Deﬁnition 2.1).
When conditional mutual information is zero, input modalities are strictly conditional independent, and I ( · ; Y ) is strictly submodular.
3.2 MODALITY SELECTION VIA APPROXIMATE SUBMODULARITY
With Proposition 3.3, we can formulate the problem of modality selection as a submodular function maximization problem with cardinality constraint, i.e., max S ⊆
V I ( S ; Y ) subject to | S | ≤ q .
Usually, q is considerably smaller than | V | .
However, Theorem 2.1 from Nemhauser et al.
[1978] is applicable to I ( · ; Y ) only if it is strictly submodular.
There the approximation guarantee differs in our case because the strength of submodularity of I ( · ; Y ) is controlled by the upper bound of conditional mutual information under
Assumption 2.1.
Under the approximate conditional independence assumption, we show the following result.
Theorem 3.1.
Under Assumption 2.1, let q ∈ Z
+ , and S p be the solution from Algorithm 1 at iteration p , we have:
[Equation]
To summarize, Theorem 3.1 states that any subset of selected modalities produced by Algorithm 1 has an approximation guarantee, in the setting of classiﬁcation with cross-entropy loss.
Since I ( · ; Y ) is monotonically non-decreasing, we can run Algorithm 1 for p = q iterations to get the best possible greedily-obtained value that is at least 1 − 1 e of the optimal value minus q(cid:15) .
The q(cid:15) term characterizes the fact that, if the to-be-optimized function is not always submodular, the upper bound of conditional mutual information (cid:15) could cause a larger approximation error as the algorithm runs longer.
Nonetheless, when (cid:15) = 0 , our result in Theorem 3. reduces to Theorem 2.1.
Using Theorem 3.1, we can further obtain a bound on the minimum of expected cross-entropy loss and expected zeroone loss achieved by the greedily-obtained set.
Let us ﬁrst denote optimal set arg max S : | S |≤ q I ( S ; Y ) as S ∗ , then:
Corollary 3.1.
Assume conditions in Theorem 3.1 hold, there exists optimal predictor h ∗ ( S p ) =
Pr( Y | S p ) such that
[Equation]
Corollary 3.1 shows that the minimum of both losses achieved by Pr( Y | S p ) are no more than the uncertainty of the target output minus the lower bound of our greedilyobtained value from Theorem 3.1.
We can also upper bound the difference in minimum cross-entropy losses achieved by the greedily-obtained set and the optimal set.
Corollary 3.2.
Assume conditions in Theorem 3.1 hold.
There exists optimal predictors h ∗ 1 =
Pr( Y
[Equation]
S
[Equation]
=
Pr( Y | S ∗ ) such that
[Equation]
This result expresses a guarantee on the maximum loss difference from the greedily-obtained set versus the optimal set using optimal predictors.
Both bounds from Corollary 3.
and Corollary 3.2 are paramterized by the duration and constraint ( p , q ) of Algorithm 1, as well as the approximation error induced by (cid:15) .
As the algorithm attempts to select a larger set of modalities, both bounds become looser.
Overall, under the setting described in Section 2, the (approximate) submodularity of the utility function allows us to have a solution in polynomial time with approximation guarantee for modality selection under cardinality constraint.
Under this theoretical formulation, we can directly extend results of other submodular optimization problems to solve modality selection problems with different constraints and objectives [Wolsey, 1982, Krause and Golovin, 2014].
3.3 MODALITY IMPORTANCE
We also examine the possibility of adapting feature importance scores to the context of modality selection, by using them to rank individual modalities.
Speciﬁcally, we consider
Shapley value and MCI.
We will show that both the computations of the exact Shapley value and MCI of a modality set is efﬁcient, if our utility function is used as the underlying evaluation function.
As previously shown, the utility of a modality f u ( { X i } )
=
I ( X i ; Y ) in the classiﬁcation with cross-entropy loss setting.
To proceed, we ﬁrst show the following propositions for I ( X i ; Y ) .
Proposition 3.4.
Under Assumption 2.1, I ( S ; Y ) is (cid:15) - approximately sub-additive for any S ⊆ V , i.e., I ( S ∪
S (cid:48) ; Y ) ≤ I ( S ; Y )
+ I ( S (cid:48) ;
Y ) + (cid:15) .
Shapley value.
In the classic deﬁnition (Deﬁnition 2.2), the complexity of computing the exact Shapley value of a player is exponential.
However, because Deﬁnition 2.2 involves a summation of the marginal contribution
I ( S ∪ { X i } ; Y ) −
I ( S ; Y ) , we can leverage the sub-additivity to provide an upper bound of the Shapley value φ I,X i via a summation of I ( X i ; Y ) s for all possible subsets.
Analogously, the super-additivity should provide a lower bound of φ I,X i again expressed by I ( X i ; Y ) .
Putting two bounds together gives us an efﬁcient approximation of φ I,X i .
Nonetheless, for I ( S ; Y ) to be super-additive, variables in S must be marginally independent.
Thus, we further introduced Assumption 3.1 for this setting.
Although Assumption 3.1 is seemingly stronger than Assumption 2.1, it will provide great convenience in approximating the Shapley value of a modality efﬁciently with a better guarantee parameterized by (cid:15) , as the following shows.
Assumption 3.1 ( (cid:15) -Approximate
Marginal Independence) .
There exists a positive constant (cid:15) > 0 such that, ∀ S, S (cid:48) ⊆
V, S ∩ S (cid:48)
= ∅ , we have I ( S ; S (cid:48) ) ≤ (cid:15) .
Proposition 3.5.
Under Assumption 3.1, I ( S ; Y ) is (cid:15) - approximately super-additive for any S ⊆ V , i.e., I ( S ∪
S (cid:48) ; Y ) ≥ I ( S ; Y )
+ I ( S (cid:48) ; Y ) − (cid:15) .
Proposition 3.6.
If conditions in Proposition 3.4 and Proposition 3.5 hold, we have I ( X i ; Y ) − (cid:15) ≤ φ I,X i ≤
I ( X i ; Y )+ (cid:15) for any X i ∈ V .
If Proposition 3.4 holds, the Shapley value of any modality
X i ∈ V will be upper bounded by its own prediction utility plus (cid:15) , i.e., φ I,X i ≤
I ( X i ; Y ) + (cid:15) .
On the other hand, we can further lower bound the Shapley value if Proposition 3.5 also holds, I ( X i ; Y ) + (cid:15) ≤ φ I,X i .
In both bounds,
I ( S ∪{ X i } ; Y ) − I ( S ; Y ) becomes I ( X i ; Y ) , and the summation of all fraction factors in fact equals to 1.
If both
Proposition 3.4 and Proposition 3.5 hold with (cid:15) = 0 , I ( · ; Y ) is additive, in which case, the Shapley value of a modality is exactly its prediction utility, i.e., φ I,X i
=
I
( X i ; Y ) .
Furthermore, by the efﬁciency property of the Shapley value, we must have I ( V ; Y ) =
(cid:80) X i ∈
V φ
I,X i .
MCI.
As claimed by Catav et al.
[2021], MCI has an extra beneﬁt over Shapley value (Section 2.3).
By its deﬁnition, solving MCI of a feature requires O (2 | F | ) , where | F | is the total number of features.
But if the evaluation function of
MCI is submodular, we can efﬁciently compute the exact
MCI.
Using Proposition 3.3, we have the following result.
Proposition 3.7.
Under Assumption 2.1, ∀ X i ∈ V , we have
[Equation]
If (cid:15) = 0 , I ( S ; Y ) will be strictly submodular for any
S ⊆ V , and the MCI of a modality is exactly its prediction utility, i.e., φ mciI,X i
=
I
( X i ; Y ) .
If Proposition 3. holds with (cid:15) = 0 , I ( · ; Y ) is sub-additive, then I ( V ; Y ) ≤ (cid:80)
X i ∈ V
I ( X i ; Y ) =
(cid:80) X i ∈ V φ mciI,X i .
If Proposition 3.5 further holds with (cid:15) = 0 , then I ( · ; Y ) is additive, we can obtain an efﬁciency property of the MCI in this problem setting, i.e., I ( V ; Y )
=
(cid:80) X i ∈ V φ mciI,X i .
Modality selection via MCI ranking.
In light of these properties, we can consider ranking individual modalities by Shapley value or MCI as an alternative for modality selection besides greedy maximization.
The ranking algorithm computes the Shapley value or MCI for all modalities, and returns the top- q modalities with maximum scores w.r.t.
a subset size limit q .
One advantage of this approach is its complexity of O ( | V | ) , while greedy maximization requires
O ( q | V | ) .
As shown above, solving Shapley value efﬁciently requires additional assumptions to hold (Assumption 3.1), thus MCI ranking would be more preferable.
EXPERIMENTS
We present empirical evaluation of greedy maximization (Algorithm 1) and MCI ranking on three classiﬁcation datasets.
Patch-MNIST.
Patch-MNIST is a semi-synthetic static dataset built upon MNIST [LeCun and Cortes, 1998].
Specifically, we divide each image in the original MNIST into non-overlapping square patches.
Each patch location represents a single modality.
We construct and experiment on two
Patch-MNIST variants, where one variant has 49 patches and each patch is of size 4 × 4 square pixel, and another has 9 patches and each patch has the side length of 9 orpixels.
Patch-MNIST has ten output classes, 50,000 training images, and 10,000 testing images.
PEMS-SF.
PEMS-SF is a real-world time-series dataset from UCI ( Dua and Graff [2017]).
This dataset represents the trafﬁc occupancy rate of different freeways of the San
Francisco bay area.
The classiﬁcation task is to predict the day of the week.
Data is obtained from 963 sensors placed across the bay area, where each sensor represents a single modality.
Each sensor has a time series with 144 time steps, which we down-sample to 36 via taking the regional means of size-4 windows.
Running Algorithm 1 requires O ( q | V | ) with | V | = 963 , and each step requires training a new model.
To mitigate extensive run-time, we experiment onout of 963 sensors by ﬁltering sensors in line for the same freeway.
There are a total of 440 instances (days), with the train-val-test split being 200, 67, 173 samples.
CMU-MOSI.
CMU-MOSI is a popular real-world benchmark dataset in affective computing and multimodal learning [Zadeh et al., 2016].
The task is 3-classes sentiment classiﬁcation (positive, neutral, negative) from 20 visual and 5 acoustic modalities with temporal features.
Speciﬁcally, CMU-MOSI collects time-series facial action units and phonetic units from short video clips (10-seconds clip sampled at 5Hz rate).
Each unit is a modality, and consists of a 50-dimensional feature vector.
Training and testing sample size are 1284 and 686 respectively.
Independence Assumption Validation We validate the independence conditions (e.g., Assumption 2.1) on all datasets by comparing the mean conditional Mutual Information (MI) and the mean marginal MI of disjoint modalities [Gao et al., 2017].
As shown in Table 1, the conditional MI is smaller than the marginal MI for MNIST and PEMS-SF.
Both conditional and marginal MI are small for CMU-MOSI.
This implies that modalities should be approximately conditionally independent in these datasets.
[Table]
4.1 IMPLEMENTATION
We implement greedy maximization based on the pseudocode in Algorithm 1.
We implement MCI ranking by com-
U t ili t y      0.
T e s t
C E L o ss
T e s t
A cc u r a c y ( % )   8 Modality subset size 1.
U t ili t y   8 Modality subset size 0.
T e s t
C E L o ss   8 Modality subset size
T e s t
A cc
[Figure]
r
[Figure]
c
[Figure]
MCI Greedy
Random Optimal
Average puting the MCI for each modality in the full set, and then select the top-ranked modalities with the largest MCIs.
Utility estimation.
From Proposition 3.1, utility f u ( S ) equals I ( S ; Y ) , and I ( S ; Y )
=
H ( Y ) − H
( Y | S ) .
Based on the variational formulation of the conditional entropy as the minimum cross-entropy, we approximate H ( Y | S ) by using the converged training loss on S to predict Y [Farnia and Tse, 2016].
Accordingly, to estimate the marginal gain
I ( X j ; Y | S i ) from Algorithm 1 over high dimensional data, we compute the difference H ( Y | S )
− H ( Y | S ∪ { X j } )
[McAllester and Stratos, 2020].
To compute MCI of each modality X j , we just need to compute I ( X j ; Y ) , according to Proposition 3.7.
Modeling.
We now describe models for prediction and utility estimation.
For Patch-MNIST, we use a convolutional neural network with one convolutional layer, one max pooling layer and two fully-connected layers with ReLU for both estimation and prediction.
The network is trained with
Adam optimizer on a learning rate of 1 e − 3 .
For PEMSSF, we use a 3-layer neural network with ReLU activation and batch normalization for estimation.
This is trained with
Adam optimizer on a learning rate of 5 e − 4 .
For prediction, we use a recent a time-series classiﬁcation pipeline
[Dempster et al., 2020] for time-series data processing, followed by a linear Ridge Classiﬁer [Löning et al., 2019].
For CMUMOSI, we experiment with two prediction model types: a linear classiﬁer with Rocket Transformation for time-series (same as the one for PEMS-SF); and a plain 3-layer fullyconnected neural network with ReLU activation.
On each dataset, the number of training epochs are the same for all evaluated approaches across different modality subset sizes.
4.2 EXPERIMENTAL PROCEDURES
In each iteration i of the Algorithm 1 we execute the following: (1) for each candidate modality X j : (a) train two models on S and S ∪{ X j } respectively until training losses converge, (b) take the loss difference to be I ( X j ; Y | S i ) ; (2) record test loss and accuracy from the model trained on S
i ∪ { X i } before the model over-ﬁts; (3) add selected modality
X i to S
i and go to next iteration.
We use model parameters before over-ﬁtting for prediction, and parameters after over-ﬁtting for utility estimation.
Step (2) for PEMS-SF and CMU-MOSI are slightly different, in which we record and show the training loss before over-ﬁtting instead of the test loss.
This is because PEMSSF and CMU-MOSI have a much smaller sample size than
Patch-MNIST with potentially noisier features, the model likely will not generalize stably.
Thus we ﬁrst examine Theorem 3.1 and MCI ranking on a larger sample set which better represents population and not inﬂuenced by the generalization gap.
Then we analyze with the test accuracy to accounting the generalization.
For Patch-MNIST with 49 modalities, PEMS-SF and CMUMOSI, we evaluate Algorithm 1 and MCI ranking against a randomized baseline at each set size.
The randomized baseline randomly selects a modality iteratively.
For PatchMNIST with 9 modalities, we further include optimal and average baselines.
At each set size q , the optimal baseline is the optimal value from all possible subsets of size q , and the average baseline is the average.
We only implement the optimal baseline for the 9 modalities case because evaluating on all possible subsets for a larger set is expensive.
Training cost.
At each iteration of Algorithm 1, the marginal utility gain for each candidate modality is evaluated.
Since we estimate the conditional mutual information by training a neural network, and we need to evaluate each modality subset at different set sizes, each iteration involves model training.
These experiments can be costly for large datasets and models.
The training cost at each iteration of
Algorithm 1 depends on different utility variants, or mutual information estimation methods in this setting.
[Figure]
4.3 RESULTS AND EMPIRICAL ANALYSIS 4.3.1 Patch-MNIST
Fig.
1 shows the Patch-MNIST experiment results.
In this ﬁgure, “Modality subset size” refers to the size of the selected modality set.
“Utility” refers to the utility of the selected set.
The “Test CE Loss” and “Test Accuracy” refers to the cross-entropy loss and prediction accuracy on test data from the model that is trained on the selected set.
Utility.
An immediate observation is the high correlation among the utility, test cross-entropy loss and accuracy in both rows.
The trend of test accuracy seems identical to the utility, although they mildly differ when the set size exceeds 30.
In addition, the utility and test loss is negatively correlated, matching to Deﬁnition 3.1.
Utility has a larger upper bound than test loss, potentially because the utility is estimated by converging training loss, which is often reduced in greater magnitude than test loss.
The utility has a trend of non-decreasing and diminishing gain, which matches the monotonicity and (approximate) submodularity shown in this setting.
Adding more modalities is unnecessary if the subset is already large: in the 49-modalities case, accuracy barely improves after 20 modalities selected; but in 9-modalities case, this pattern is less obvious.
Greedy maximization.
Algorithm 1 beats random selection in both cases.
In Fig.
1 (second row)
, it beats the average by selecting the modality with maximum utility from the start, and overlaps its trajectory with the optimal.
In Fig.(ﬁrst row), Algorithm 1 achieves near-maximum utility with only 7 modalities.
These results validate the approximate guarantee from Theorem 3.1.
In fact, the guarantee on utility is empirically much better than theoretically proven.
MCI ranking.
In the 9-modalities case, MCI ranking is as good as greedy maximization and the optimal baseline when the full set has fewer modalities.
When more modalities are available for selection (e.g., 49 modalities), Algorithmselect a subset that minimizes the loss slightly further than the highest ranked modalities when set size below 15.
Modality selection path.
We plot the modality selection paths from Algorithm 1 and MCI ranking in Fig.
We can see that MCI selects the modalities that each contain the most information to output – the center regions.
Whereas the modalities selected by Algorithm 1 are more diverse, covering different spatial locations of the original image, leading to an advantage in gaining more information collectively.
4.3.2 PEMS-SF
Fig.
3 shows our experiment results on PEMS-SF.
In Fig.
3, the two leftmost plots show the utility and cross-entropy loss on the training data.
The rightmost plot of Fig.
3 shows the moving average of test accuracy instead, because model was not generalized stably under small sample size.
Utility.
The difference in utility and loss among Algorithm 1, MCI ranking and random baseline are small, and all of them quickly converge to the minimum possible value after selecting only a few modalities.
This is potentially because almost each of the modality is sufﬁcient to make training loss small.
However greedily selected subsets still has slightly more utility than subsets from MCI ranking and random baseline at every set size.
Overall, we still observe the utility is monotone and (approximate) submodular; and
Algorithm 1’s achieved utility matches Theorem 3.1.
Generalization.
From the test accuracy plot, we can see a clear advantage from the greedily-obtained set over others when the subset size is small.
Meanwhile, MCI ranking is worse than random baseline, which could imply that MCI ranking does not have a robust performance guarantee as Algorithm 1.
Other than that, the test accuracy of Algorithmgradually decreases as more modalities are added.
This is inline with the over-ﬁtting artifact of greedy feature selection from Blanchet et al.
However, in the regime of good generalization, greedy maximization should preserve the performance guarantee during testing.
4.3.3 CMU-MOSI
The results are alike for both prediction model types mentioned in Section 4.1 for CMU-MOSI.
Thus we only use
Fig.
4 to show the CMU-MOSI evaluation results from the 3-layer fully-connected neural network.
In Fig.
4, the two leftmost plots show the utility and cross-entropy loss on the training data.
The rightmost plot of Fig.
4 shows the moving average of test accuracy since the model lacks the capacity to generalize well for this dataset under small sample size.
Overall, many previous observations from other datasets still hold for CMU-MOSI.
For example, the utility curve is approximately submodular and monotone as number of selected modalities increases.
Modalities selected by Algorithm 1 and MCI ranking outperform randomly selected modalities by having more utility, lower training loss, higher testing accuracy, especially when the number of modalities is still small.
On the other hand, potentially due to the sim-
[Figure] [Figure]
plicity of the model and noisy features, we are unable to observe an increase of testing accuracy as more modalities are included in Algorithm 1 and MCI ranking.
RELATED WORK
Multimodal Learning Multimodal learning is a vital research area with many applications [Liu et al., 2017, Pittermann et al., 2010, Frantzidis et al., 2010].
Theoretically,
Huang et al.
[2021] showed that learning with more modalities achieves a smaller population risk, and this marginal beneﬁt towards prediction could be upper bounded.
However, the existing measure of marginal beneﬁt [Huang et al., 2021] is hard to understand and cannot be easily estimated, and it does not provide further insight on the emerging modality selection problem.
Submodular Optimization Thanks to the benign property of submodularity, many subset selection problems, which are otherwise intractable, now admit efﬁcient approximate solutions [Fujishige, 2005, Iwata, 2008, Krause and Golovin, 2014].
The ﬁrst study of greedy algorithm over submodular set function dates back to Nemhauser et al.
Since then, submodular optimization has been widely applied to diverse domains such as machine learning [Wei et al., 2015] , distributed computing, and social network analysis [Zhuang et al., 2013].
A typical type of problem is submodular maximization, which can be subject to a variety of constraints such as cardinality, matroid, or knapsack constraints (Lee et al.
[2010], Iyer and Bilmes [2013]).
In our case, we extended results from Nemhauser et al.
[1978] to the case of approximate submodularity of mutual information in a multimodal learning setting.
Feature Selection Feature selection asks to ﬁnd a feature subset that can speed up learning, improve prediction and provide better interpretability to the data/model [Li et al., 2017, Chandrashekar and Sahin, 2014].
Here we brieﬂy touch related work on feature selection more relevant to our context.
Information-theoretic measures such as mutual information have been as a metric for feature selection [Brown et al., 2012, Fleuret, 2004, Chen et al., 2018].
For example,
Brown et al.
[2012] presents a uniﬁed information-theoretic feature selection framework via conditional likelihood maximisation.
There are also work on feature selection in regression problems through submodular optimization [Das and Kempe, 2011].
In our context, a distinction between the problems of modality selection and feature selection are the assumptions of the underlying data (Section 2).
CONCLUSION
In this paper, we formulate a theoretical framework for optimizing modality selection in multimodal learning.
In this framework, we propose a general utility function that quantify the impact of a modality towards prediction, and identify proper assumption(s) suitable for multimodal learning.
In the case of binary classiﬁcation under cross-entropy loss, we show the utility function conveniently manifests as Shannon mutual information, and preserves approximate submodularity that allows simple yet efﬁcient modality selection algorithms with approximation guarantee.
We also connects modality selection to feature importance scores by showing the computation advantages of using Shaply value and MCI to rank modality importance.
Lastly, we evaluated our results on a semi-synthetic dataset Patch-MNIST, and two real-world datasets PEMS-SF and CMU-MOSI.