Paper: Representation Matching Information Bottleneck for Text Matching in Asymmetrical Domains (ICLR-2024)

Reviewer comment (Theory):
The proof for Proposition 4 in the paper is somewhat perplexing, and the meaning and proof process for Theorem 4 are not well understood. Moreover, there is an error in Equation (67) within the proof of Proposition 4.

Author response:
There is a typo in Eq(67). The $I(Y ;Z_1, Z_2|Y )$ should be $I(Y ;Z_1, Z_2|Z )$, but the proof process is correct. In the previous line of Eq (68), we used  $I(Y ;Z_1, Z_2|Z )$ instead of $I(Y ;Z_1, Z_2|Y )$. We sincerely thank you for conducting such a meticulous review. The typos you pointed out are of great help in improving the quality of the paper.

Reviewer comment (Evaluation):
According to "3.1 DATASET AND METRIC," why do SICK and SciTail have F1 metrics, and why do the tables of experimental results not align with the F1 metric as described in the experiments?

Author response:
This is a typo. We have modified 'F1 values on SICK' to 'accuracy on SICK'

Reviewer comment (Theory):
Regarding the proof of Theorem 4: Can we directly derive equation (68) based on a Markov chain, and why are equations (66) and (67) necessary? My knowledge in this field is limited, so I seek your understanding if I have misunderstood.

Author response:
According to the definition of mutual information: $I(Y;Z_1,Z_2,Z)=I(Y;Z_1, Z_2)+I(Y;Z|Z_1,Z_2)=I(Y;Z)+I(Y;Z_1,Z_2|Z)$. Since $Y\to (X_1,X_2)\to(Z_1,Z_2)\to Z$ forms a Markov chain, $Y$ and $Z$ are independent of each other given $(Z_1,Z_2)$, that is, $I(Y;Z|Z_1,Z_2)=0$, then:
$$
\begin{equation} 
 I(Y;Z_1, Z_2)+0 =I(Y;Z)+I(Y;Z_1,Z_2|Z)
 \end{equation}
$$

Based on the non-negativity of mutual information, then: $ I(Y;Z_1,Z_2)\ge I(Y;Z) $. We can see that $I(Y;Z_1,Z_2)$ is the upper bound of $I(Y;Z)$.
Therefore, if the cost of optimizing $I(Y; Z_1, Z_2)$ and $I(Y; Z)$ to the same value is the same, this means that optimizing $I(Y; Z)$ will result in a higher $I(Y; Z_1, Z_2)$. Since one of the optimization goals of RMIB is to maximize $I(Y; Z_1, Z_2)$, the higher the value of $I(Y; Z_1, Z_2)$, the better $Z_1^\*,Z_2^\*$ learned.

Reviewer comment (Theory):
How does equation (68) directly lead to the conclusion, and can this step be explained in more detail?

Author response:
According to the definition of mutual information: $I(Y;Z_1,Z_2,Z)=I(Y;Z_1, Z_2)+I(Y;Z|Z_1,Z_2)=I(Y;Z)+I(Y;Z_1,Z_2|Z)$. Since $Y\to (X_1,X_2)\to(Z_1,Z_2)\to Z$ forms a Markov chain, $Y$ and $Z$ are independent of each other given $(Z_1,Z_2)$, that is, $I(Y;Z|Z_1,Z_2)=0$, then:
$$
\begin{equation} 
 I(Y;Z_1, Z_2)+0 =I(Y;Z)+I(Y;Z_1,Z_2|Z)
 \end{equation}
$$

Based on the non-negativity of mutual information, then: $ I(Y;Z_1,Z_2)\ge I(Y;Z) $. We can see that $I(Y;Z_1,Z_2)$ is the upper bound of $I(Y;Z)$.
Therefore, if the cost of optimizing $I(Y; Z_1, Z_2)$ and $I(Y; Z)$ to the same value is the same, this means that optimizing $I(Y; Z)$ will result in a higher $I(Y; Z_1, Z_2)$. Since one of the optimization goals of RMIB is to maximize $I(Y; Z_1, Z_2)$, the higher the value of $I(Y; Z_1, Z_2)$, the better $Z_1^\*,Z_2^\*$ learned.

Reviewer comment (Reproducibility):
I'm not quite sure how to implement the objective function using code. How do I calculate the KL divergence between a Gaussian distribution and the distribution of text representations?

Author response:
As we mentioned in Section 3.2, we use the reparameter trick to optimize KL terms in RMIB. Taking BERT as an example, the pseudocode of its feedforward calculation is:
```
# let x1 and x2 are the input texts
x1_token_embedding, x2_token_embedding=BERT(x1,x2)

# maxpooling to get the text embedding
x1_embedding=max_pooling(x1_token_embedding)
x2_embedding=max_pooling(x2_token_embedding)

# using reparameter trick
x1_mean=mlp_mean(x1_embedding)
x1_std=mlp_std(x1_embedding)
x1_repara_embedding=x1_mean + normal(mean=0, std=1)*x1_std

x2_mean=mlp_mean(x2_embedding)
x2_std=mlp_std(x2_embedding)
x2_repara_embedding=x2_mean + normal(mean=0, std=1)*x2_std

#concat text embedding
x=concat([x1_repara_embedding, x2_repara_embedding])

# get the output
logit=mlp(x)

# get class prob
class_prob=softmax(logit)
```

After reprocessing the text representation with reparameter trick, `x1_repara_embedding` can be represented as sampled from the distribution `normal(mean=x1_mean, std=x1_std)`. Therefore, we can optimize the KL item in RMIB:
$$ KL[p_\theta(x1\\_repara\\_embedding|x_1,x_2)\parallel p(\mathcal{Z})] \propto 0.5 * [ square(x_1\\_mean)+ square(x_1\\_std)-\log( square(x_1\\_std))-1 ]    $$ where $p(\mathcal{Z})$ is the standard normal distribution.

Reviewer comment (Experiments):
While most of the method descriptions are in the form of information theory, the actual model architecture deployed in the experiment should be also clarified. Some comparisons to baseline methods should be also conducted, otherwise we might not know if the gain is from the proposed idea or simply because of other factors like more model parameters.

Author response:
In fact, RMIB did not propose a new model structure. RMIB constrains the learned representations on existing models to better handle asymmetric domains text matching task, so we have not provided too detailed descriptions of these models. We have uploaded the main implementation code for this paper in Supplementary Material on how to implement the RMIB. To ensure the effectiveness of RMIB, we also replicated the baseline model in our experimental environment to ensure fairness in comparison. From the experimental results, it can be seen that in the scenario of asymmetric domains text matching, these baselines do not perform as well as RMIB. To the best of my knowledge, DDR-Match is currently the best model in this scenario. We have also added it to Table 2 for comparison. In addition to cross entropy loss, RMIB also includes three other items. Therefore, we also conducted ablation experiments to verify the overall effectiveness of RMIB. The results in Table 3 validate the effectiveness of RMIB.

Reviewer comment (Experiments):
With the same encoder, the representations are to some degree still in the same domain. The real asymmetric setup (like [d]) with different encoders or even different data types should be considered in the experiments.

Author response:
In asymmetric domain text matching tasks, asymmetric means that the texts to be matched are sampled from two different domains. For example, in the information retrieval tasks, the query and the retrieved document clearly belong to different domains. Therefore, for asymmetric text matching task, when the texts to be matched are obtained from different domains, it constitutes a real asymmetric setup, regardless of whether you use the same decoder or not. There are still certain differences between multimodal data matching tasks and asymmetric domain based matching tasks. Our current paper mainly focuses on asymmetric domain based matching. In future work, we will attempt to expand RMIB in multimodal data matching scenarios, as mentioned in section "CONFUSION AND LIMITED LIMITED".

Reviewer comment (Evaluation):
Many gains are huge in terms of values. I wonder if the authors conduct significance tests to have verification.

Author response:
In order to ensure that the improvement in experimental results is due to RMIB and not other factors, such as random initialization, learning rate schedule, model parameters, etc., we use the same environment and experimental settings for both baseline and RMIB. At the same time, in order to avoid the randomization error caused by parameter initialization, we used the same seed in all experiments to ensure the reproducibility of results. This is mentioned in section "IMPLEMENTATION DETAILS".

Reviewer comment (Experiments):
Following W3, the proposed method actually does not use any property about text, so theoretically it can be applied in representations of arbitrary data formats. I wonder if there could be some experiments on multi-modal settings.

Author response:
For asymmetric text matching tasks, there are currently two types of models: text encoding model and text interaction model. When using these two models, we are actually using some properties of the text. For example, for text interaction model, we can concatenate the two asymmetric domain texts to be matched as a whole for processing. However, for multimodal tasks, we cannot directly concatenate images and text as a whole representation. Our current work mainly focuses on the asymmetric domain text matching. We will attempt to expand RMIB in multimodal data matching scenarios, as mentioned in section "CONFUSION AND LIMITED LIMITED".

Reviewer comment (Evaluation):
In Table 1, it is interesting that the proposed method improves a lot in `SICK`, but also significantly underperform DDR-Match in `SNLI`. I wonder if the authors have conducted analysis to research this phenomenon.

Author response:
Compared to SNLI, we have noticed a more significant improvement in RMIB in SICK. We think this is because SNLI have significantly more data than SICK. For example, SNLI has about 57 times the data than SICK. [1] indicates that when the number of training data sets increases, the generalization performance of the model will also increase. When there are enough training sets, the model itself has good generalization performance, so RMIB can bring less improvement. We have analyzed and explained this phenomenon in the section "PERFORMANCE OF RMIB".

[1] Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. IEEE Transactions on Information Theory, 47(5):1902–1914, 200