From: auai.org/UAI/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Meta Review: The reviewers reach a consensus on the acceptance. The authors are encouraged to take all the comments into consideration and further improve the paper in the camera ready.

From: auai.org/UAI/2022/Conference/Paper695/Reviewer_PQSj
Q1 Summary and contributions: The motivation of the paper is to study modality selection, intending to efficiently select the most informative modalities under certain computational constraints. The contrbutions includes:
(1) The authors formulate a theoretical framework for optimizing modality selection in multimodal learning.
(2) They introduce a general utility function that measures the benefit of selecting a modality.
(3) They establish theoretical connections between modality selection and feature importance scores.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: On the whole, the paper has fluent description and clear topic. The paper formulates a good theoretical framework for optimizing modality selection in multimodal learning and the experiments performed in the article validate the proposed method.
Q4 Main weakness: The topic of the paper is novel but the algorithm used is not so novel. It is simple and not so novel that the paper uses the greedy maximization algorithm to solve modality selection problem.
Q5 Detailed comments to the authors: The experimental part of the article mainly uses two data sets for the binary classification task. If it is in other more complex tasks, is the proposed method effective?
Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: See the above comments
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper695/Reviewer_6YnH
Q1 Summary and contributions: The presented work proposes a selection method in constrained multi-modality settings. To this end, a utility function for quantification of the usefulness of modalities employed in a greedy search strategy is proposed. This modality utility is further connected to traditional feature selection measurements, such as the Shapley value, allowing for efficiency gains in the modality selection over the greedy approach.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: - Solid theoretical analysis of the method proposal incl. consideration of most recent related work
- Compelling link of the utility function to Shapley values / MCI
- Empirical experiments on different types of data (images and time series), overall convincing results
Q4 Main weakness: - Empirical evaluation involves rather homogeneous, interdependent modalities (contradictory to epsilon-approximate conditional independence, at least for small epsilons)
Q5 Detailed comments to the authors: - As e.g. for Patch-MNIST, the patches are embedded in a global spatial structure, which makes them interdependent (at least to some extent). Thus, the modalities cannot be considered as conditionally independent. Albeit your analysis weakens the strict conditional independence by an epsilon approximation, it is still internally relying on a somewhat small epsilon for meaningful / tight bounds. An empirical study with more independent modalities could be insightful.
- Limitations: As described in Section 4.2, each iteration in the greedy algorithm (Algorithm 1) involves a model training. When thinking of larger datasets and models, this is typically not feasible (or at least extremely costly). This should be addressed more explicitly. 
- How would your approach generalize to the multi-class setting? It would be nice to have a sort of “outlook”

Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: This work can certainly be considered as a valuable contribution, where solid theoretical and empirical results support the method proposal. Also, this work connects to previous measurements in the domain of feature selection, which I appreciate.
Q8 Confidence in your score: 2: Not very confident. I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details, or can't be sure about the novelty of the work.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper695/Reviewer_xYBK
Q1 Summary and contributions: The inputs space of a machine learning task exhibits multimodality by nature. Ideally, to train a good model, at every step when evaluating the loss function, we should consider all the inputs modalities simultaneously. However, it is not efficient or even infeasible in practice. The authors of this paper developed a greedy heuristic based on mutual information that guarantees the optimal modality selection via submodular maximization. 
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q2(5) Reproducibility: 3: Good: Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
Q2(6) Clarity of writing: 4: Excellent: The paper is well-organized and clearly written.
Q3 Main strengths: The motivation of this paper is clear and strong. Modality selection is an important topic and benefits the downstream machine learning tasks in practice. Though submodular optimization is not a novel method in general, people have used it for active learning also, the authors gave theoretical justifications for modality selection, which is counted as the novelty of this paper. 
Q4 Main weakness: If I didn't misread the figures, the improvement regards to other baselines are marginal. Seems to me, that the tasks the authors used are not challenging enough. I will suggest the authors consider more challenging tasks so that the audience could have a clear insight into when the proposed method is better. 
Q5 Detailed comments to the authors: It is totally fine to assume binary cross-entropy loss to derive the theories. However, could the authors show other loss functions used in the experiment? 
Q6 Overall score: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
Q7 Justification for your score: The study presented in this paper is comprehensive enough and easy to follow. This paper is well motivated and the authors gave theoretical justifications. A submodular optimization is a well-studied tool in active learning. The only concern seems the improvement is marginal. However, I am not an expert in feature learning. If authors could append more cases with variants of loss functions in the experiment. That would be more interesting.
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper695/Reviewer_Z7aX
Q1 Summary and contributions: The authors propose a new greedy algorithm for selecting features,
apply it to real data. They also extend known theoretical results
that are currently known for the case when features are independent
to a more general case when the features are almost independent in
some reasonable sense.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 2: Fair: The paper has minor, easily fixable, technical flaws that do not impact the validity of the main results.
Q2(4) Quality of experiments (Optional): 1: Poor: The experimental evaluation is flawed or the results fail to adequately support the main claims.
Q2(5) Reproducibility: 2: Fair: Key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: They extend known theoretical results
that are currently known for the case when features are independent
to a more general case when the features are almost independent in
some reasonable sense.
Q4 Main weakness: The empirical part is not convincing at all. The authors never
compare their result with previously known greedy algorithms, so it
is not clear whether the new method is better.

Also, the empirical part does not seem to be related to the
theoretical part at all: the authors never check whether the almost
independence condition is satisfied, and for what epsilon.
Q5 Detailed comments to the authors: The authors propose a new greedy algorithm for selecting features,
apply it to real data. They also extend known theoretical results
that are currently known for the case when features are independent
to a more general case when the features are almost independent in
some reasonable sense.

The empirical part is not convincing at all. The authors never
compare their result with previously known greedy algorithms, so it
is not clear whether the new method is better.

Also, the empirical part does not seem to be related to the
theoretical part at all: the authors never check whether the almost
independence condition is satisfied, and for what epsilon.

Even if the authors did not compare -- and thus, cannot claim
anything about the quality of their algorithm, this theoretical
part may form a nice short article.

There are, however, many other problems. For example, after
Definition 2.3, the author claim that Shapley value requires
exponential time to compute. This is nonsense, it is like claiming
that since a + b = a + 1 + ... + 1 (b times), the sum a + b
requires exponential time to compute for large b. Actually, there
exist efficient Monte-Carlo algorithms for computing the Shapley
value.

Definition 2.2 should say "defined" instead of "evaluated", it is a
definition after all.

There are also minor English typos: e.g., attempting instead of
tempting on p. 1; line 3 on right-hand side of p. 2 should be "and"
not "or".

Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: Empirical part is not ready, since the authors never compare the newly proposed method with what was known before. The theoretical part, whole not related to the empirical part, looks promising -- while having some flaws. 
Q8 Confidence in your score: 4: Quite confident. I tried to check the important points carefully. It is unlikely, though conceivable, that I missed some aspects that could otherwise have impacted my evaluation.
Q9 Complying with reviewing instructions: 1: Yes.