Paper: Learning Multi-modal Representations Under Incomplete Data Via Dual Level Alignments (ICLR-2024)

Reviewer comment (Writing):
The entities, IDs, DIDs are not clearly defined.

Author response:
**1) The entities, IDs, DIDs are not clearly defined.**\
We have defined these many times in the text as follows, but do you need more specific examples?\
**Entities:** *``entity types (e.g., text, image, object)* in abstract, because this is public defintion (e.g., each image), so we are not defining it here''*.\
**IDs:** *``DID, and ID are the identifier of document, and entity, respectively''* in Figure 1 caption.\
**DIDs:** *Document ID (DID) is the identification of each entity (text or document) and is automatically assigned a random number if missing *in 3.2\
Specific examples of each are also presented in 4.1 and Supplementary Material.\
*``We conducted evaluations on Flickr30k 1, and MSCOCO 2, as both are publicly available, and are widely used in multi-modal search studies Wang et al. (2019); Li et al. (2020a) for modalities of texts and images (entity).''* in 4.1\
*``each ID can be converted into a product name by using a product name look-up table; the name is used only for interpretation of the results.''* in Supplementary Material

Reviewer comment (Writing):
The writing can be improved. A lot of terminology is left undefined when presented. The NTF was mentioned multiple times but never explained since it can refer to Non-negative Tensor Factorization and Neural Tensor Factorization. Also, the I2T and T2I weren't explained. I suggest it be text-to-image retrieval.

Author response:
We define NTF in *3.3 NEURAL TENSOR FACTORIZATION (NTF)*.
I2T and T2I are as you indicated, and we use the, without any other explanation learned because these terms are common in V-L tasks and multi-modal domain.

Reviewer comment (Writing):
The difference between contextual information and semantic information is not clear. Why is the proposed model learning semantics? Does semantic refer to global information from the TF-IDF document information? From my understanding of Figure 1, semantics was learned by TF-IDF document-level information, which is a form of text-only information.

Author response:
The contextual representation is obtained by the existing model from both text and features, 
while the semantic representation is obtained from only text for the first time by Mul2vec.
As shown in Figure 1 and stated in 3, Mul2vec learns semantics from the global information via NTF, like latent factor models (e.g., VAEs and Tensor Factorization) describe the statistical relationships between word occurrences, combining global with local information enables us to represent words in one semantic space
As stated in 3.1,
*``we try some weighting methods as preliminary experiment, compare their results, and confirm that this weighting (TF-IDF) offers greater effectiveness than using only frequency and others.''*
TF-IDF is form of text-only information as you point out, and we think it is an excellent initial value for learning semantics from the global infromation as used in other models.

Reviewer comment (Novelty):
The main component of Neural Tensor Factorization was proposed in previous work [Wu et al. (2017)]. Also, the training objective of ILM and DLM was from previous works, which weakens the contribution of this work.

Author response:
We have the same name as **Neural Tensor Factorization**, but it differs from  [Wu et al. (2019)] in 1) learning the dual level representations, 2) employing different layers (ie., TF and MLP) as its architecture, and 3) learning itself with Transformer based models.
The form within objective function ILM and DLM is indeed an existing one, 
but we are not proposing these forms.
From alternative candidates,
we are selecting, modifying, validating and proposing them as the form that meets our objectives function.

Reviewer comment (Experiments):
Although it claims to deal with missing modalities, it wasn't further explored by additional experiments (showing the effect of dealing with different levels of data missing, and the model is robust to it.)

Author response:
To test the robustness to missing data in the local information (e.g., text and image)
we created missing data and perform experiments, *``To evaluate the effect of semantic representations, we also prepared the original data and the data with 20% of each of these images and texts randomly masked, incomplete data, and compared the improvement obatained over the models without Mul2vec.''*, as shown in 4.2. 
Can you tell us what other validations or missing data we should have considered?

Reviewer comment (Evaluation):
In the experiment, as in tables 3 and 6, only the difference was presented in the table, which makes it hard for the reader to have a sense of whether the performance is good or bad.

Author response:
Because we believe that relative improvement value is more reliable and easier for readers to evaluate than absolute values, we show these values.
We fear that absolute values would be unreliable and unrepeatable because these values would change with the environment and other factors.

Reviewer comment (Writing):
The writing of the paper is below the bar, there are many typos and grammar errors, i.e., prameter -> parameter in Figure 1; many terms are written in short in the first place which is hard to understand, such as NTF, DLM and ILM in abstract.

Author response:
Sorry about both typos and errors. we will correct the areas pointed out or noticed.
Due to space limitations we put the abbreviation in the abstract, but not in it.

Reviewer comment (Writing):
Many terms and equations are not well-explained: how to use global information as topic? What are the example of topic? Others please refer to Questions section.

Author response:
Mul2vec learns semantic representations through NTF, where NTF discovers topics as the global information.
As stated in 1, *``Unlike other models, Mul2vec uses latent variables to quantify global information as topics, where these topics are discovered from context-independent information, word-occurrence, and sharing between entities.''*
More specifically, we create *``the document-word-entity's ID tensor $\bf{\mathcal{D}} \in \mathbb{R}^{N \times V \times L}$ for $N$ documents, $V$ words, and $L$ IDs, and feed it to Mul2vec.
Then NTF decomposes this 3rd order tensor into three latent factor matrices $\Theta=\{\theta_{n,k}\} \in \mathbb{R}^{N \times K}$, $\bf{T}=\{t_{v,k}\} \in \mathbb{R}^{V \times K}$ and $\bf{G}=\{g_{l,k}\} \in \mathbb{R}^{L \times K}$, where the $k$-th row vector of each matrix corresponds to the $k$-th factor of the tensor''*.
\
As stated in the paper and in the previous repley, topic is a vector representation (i.e.,$\theta_{n},g_{l},t_{v}$) in our approach, like contextual representations (i.e., embeddings), and is not readable.
\
As we answered in the question section, please see this section.

Reviewer comment (Experiments):
The experiments are limited in dataset and different type of downstream tasks, since the authors claim that Mul2vec learns semantic representations. Why only demonstrated the results on image-text retrieval tasks on two datasets? Also authors mentioned that their goal is to applied to multimodal tasks under incomplete data, is there any ablation studies on incomplete dataset? Or how did the authors set up experiments as noisy data or missing modality information?

Author response:
For fair comparison and to ensure reproducibility and model comprehensiveness, 
this experiment dataset and settings were limited to a representative test data set and basic tasks.
\
Yes. As shown in 4.2., we prepare the missing modality information as; *``To evaluate the effect of semantic representations, 
we also prepared the original data and the data with 20\% of each of these images and texts randomly masked, \textbf{incomplete data}, and compared the improvement obatained over the models without Mul2vec.''*

Reviewer comment (Presentation):
The discussion and limitations sections showed some models, datasets and results which were not discussed in the paper.

Author response:
Since we omitted these models, datasts, and results because they was not directly relevant.
We will add them in the next supplementary material.

Reviewer comment (Writing):
In section 3.3, what are the gaps between texts?

Author response:
The gaps mean the differences between texts with the same content but different wording.

Reviewer comment (Writing):
In section 3.4, how to use negative vector e_n, and is e_p is v_p in equation 6? I think authors need to clarify all the terms clearly in equations.

Author response:
Because it is common in contrastive learning and we have simplified,
$v_{p}$ is the semantic representation of DID (traget $l$-th ID's document) and $v_{p}$ is the semantic representation of the DID in the same batch (not traget $l$-th ID's document).

Reviewer comment (Presentation):
In Table 1, what are the differences between D and L, what are documents are ids? are they the captions in COCO and Flickr30 dataset?

Author response:
Yes. $D$, and $L$ denotes the number of captions, and ID, respectively.
As ID is the identification of images, it is the same number as images.

Reviewer comment (Experiments):
Are there any other algebraic operations in experiments? How these algebraic operations verify the representation learning ability of Mul2vec?

Author response:
Yes. Such operations can also evaluate the model's ability, but this was not the main objective and we do not show these examples in the main results because there was no dataset available to quantitatively evaluate this abliity.