Paper: Optimal and Generalizable Multimodal Representation Learning Framework through Adaptive Graph Construction (ICLR-2024)

Reviewer comment (Experiments):
Although the paper claims to bind multimodalities, in practice, it deals with only three modalities (image, text, tabular) and their corresponding features (various columns in the data). Such a number of modalities is relatively common in multimodal learning, unlike current works that learn from up to five modalities that contain image, video, text, audio, and IMU.

Author response:
While the paper only deals with image, text and tabular data, the tabular data is categorised into different independent values and these are encoded independently, which leads to many different sources of data. 

Current works that use image, video, text, audio, and IMU data learns from these modalities separately (e.g. {image, video}, {image, text}, {text, audio}), whereas our approach considers the scenario where lots of modalities are available in a single dataset.

Reviewer comment (Novelty):
The modalities claimed in the current setting are more like different features in the same modalities (bedroom, bathroom, etc ...). Hence, the missing modality setting is more like missing features, which is a relatively explored problem.

Author response:
While the paper only deals with image, text and tabular data, the tabular data is categorised into different independent values and these are encoded independently, which leads to many different sources of data. 

Current works that use image, video, text, audio, and IMU data learns from these modalities separately (e.g. {image, video}, {image, text}, {text, audio}), whereas our approach considers the scenario where lots of modalities are available in a single dataset.

Reviewer comment (Evaluation):
The change between epochs 62, 63, and 64 is also not clear in Figure 4. In epochs 62 and 64, the label is dependent on various features. However, it is only correlated to two of them in epoch 63. It would be better for the author to elaborate more on these findings.

Author response:
We include the graphs for epoch 1, 2, 3 vs 62, 63, 64 to show the convergence of the MST graph. As shown in Figure 3: Graph Visualization of AutoBIND w/ 2D MRI, the (Image) - (Label) - (Biomarker) component of the graph is consistent for all 3 of the final epochs. We note that some other modality connections may fluctuate (e.g. demographics) since they are not correlated with the presence of AD.

Reviewer comment (Writing):
What is the specific definitions of correlated and uncorrelated modalities?

Author response:
We define correlated modalities as modalities that are capable of producing similar embeddings due to similarities in their data.

We define uncorrelated modalities as modalities that are not capable of this.

E.g. patient demographics is uncorrelated with the presence of Alzheimer's disease, while the MRI is highly correlated.

Reviewer comment (Theory):
The paper has the word "optimal" in the title, but there is no discussion or proof on the optimality.

Author response:
### The paper has the word "optimal" in the title, but there is no discussion or proof on the optimality.


The fundamental idea of contrastive learning leverages the inherent conceptual commonalities between different correlated modalities of data. These commonalities are mirrored by learnable correlations in the data itself. Therefore, by the transitive property, modalities with low contrastive loss (or high similarity) can be mapped to each other, and these mappings can be propagated only if the modalities are connected. This motivates the use of MST.

The MST is optimal due to the following:

Given two correlated modalities A and B:

- Consider that A and B are not directly connected due to particular biased batch or underfitting in the encoders.

- If A and B are never connected, then the correspondance between A and B will never be found. We cannot determine that modalities A and B are similar without comparing them.

- However, if A and B are indirectly connected, then their similarity will increase, as they are bound to one embedding space. Let A connect to C connect to B, where C is some component of the graph. In the embedding space of the vector of C, modalities A and B will be closer than if they were in disjoint graphs. Therefore, in the next iteration, the similarity between A and B increases.

- Hence, all data modalities should be connected whilst similar modalities are being grouped together.

Now further consider two uncorrelated modalities A and D:

- A and D will always be connected. However, they will be distant in the embedding space, since they have little/no similarity
- Hence, their edge will be pruned and they will be far apart on the graph
- Since their modalities have low correlation AND they are far apart on the graph, there is very little interference to the overall contrastive training process

The MST maintains connectivity of all modalities, whilst increasing the correspondance between similar modalities. After multiple iterations, the graph converges and similar modalities are grouped together.

As shown in Figure 3: Graph Visualization of AutoBIND w/ 2D MRI, the (Image) - (Label) - (Biomarker) component of the graph is consistent for all 3 of the final epochs. 

The converged MST highlights inherent similarities between two modalities across the entire dataset. 

After convergence, we can do node pruning. Node pruning eliminates the noise in the loss caused by the unrelated modalities. Furthermore, it reduces training time as less encoders require back-propagation.

Reviewer comment (Theory):
Eq (3) is not explained clearly. Is it a theorem?

Author response:
### Eq (3) is not explained clearly. Is it a theorem?

This is a claim that is part of the hypothesis. We say that the total loss, if we contrast uncorrelated modalities (Zn and Z'n) and correlated modalities (Zm, Z'm) separately (i.e. bring similar representations closer and move dissimilar representations apart), is less than if we contrast all modalities (Zp, Z'p) together, regardless of similarity.