Paper: Detecting Shortcuts using Mutual Information (ICLR-2024)

Reviewer comment (Theory):
In algorithm 1, a dataset is said to contain shortcut if $I(X_{test};Z)<I(X_{tr};Z)$. However, since $X_{test}$ have different distribution than $X_{tr}$, it seems natural to have $I(X_{test};Z)<I(X_{tr};Z)$. Therefore an important question need to be answered: are there any datasets satisfying $I(X_{test};Z)\geq I(X_{tr};Z)$? There lacks empirical evidences in this paper to answer the question and I do not think the algorithm 1 could effectively detect shortcuts.

Author response:
While $I(X_{test}; Z)$ cannot be greater than $I(X_{tr}; Z)$, however, for $X_{test}$ sampled from same distribution as $X_{tr}$, $I(X_{test}; Z) \approx I(X_{tr}; Z)$, as shown in the different experiments.

Reviewer comment (Experiments):
Unlike the proposed algorithm 1, the experiments in this paper, on the other hand, mainly compare networks trained on two datasets "with" and "without" shortcut. This approach also have problems since it requires comparing with a network trained on dataset that is "without" shortcut. By defining "without shortcut", it also involves domain knowledge and human expertise to detect shortcuts.

Author response:
We use `with' and 'without shortcut' to denote shortcuts in the controlled experiments to demonstrate that mutual information can be used to distinguish between the two datasets. We demonstrated the practical use case in Section 5 where we sample $X_{tr}$  and $X_{test}$ from two different medical imaging datasets. Our method can detect the presence of shortcuts, which is in fact corroborated when the model train on $X_{tr}$ is evaluated on $X_{test}$.

Reviewer comment (Experiments):
For those datasets without shortcut, are they satisfy $I(X_{test};Z)=I(X_{tr};Z)$? Please provide empirical evidences to show that algorithm 1 is feasible.

Author response:
$I(X_{test}; Z) and I(X_{tr}; Z)$ will similar when sampled from the same distribution. In the experiments reported, MI $ I(X_{tr}; Z) $ is calculated on different samples from the training distribution.

Reviewer comment (Evaluation):
At the beginning of the training, before the mutual information $I(X;Z)$ starts to be different between "with shortcut" and "without shortcut", the test loss has become different (e.g. 100 epoch in Fig.6 and 1000 epoch in Fig.7). Why is it?

Author response:
Our intuition in the initial phase network explores the more optimal region of the solution space in the initial training epochs before discovering shortcuts. This is consistent with the findings of Shwartz-Ziv & Tishby (2017) on the behaviour of SGD.

Reviewer comment (Novelty):
The core concept underpinning the method lacks novelty. In essence, during the training phase, the objective is to minimize $I(X; Z)$ on the training data $D_{tr}$. However, it becomes evident during the testing phase that $I(X_{te}; Z) < I(X_{tr}; Z)$, provided a domain gap exists between the training data $D_{tr}$ and the test data $D_{te}$. Subsequently, equations (4) - (10) are all employed to compute $I(X; Z)$ using well-established formulations from prior literature. Hence, the methods devised are not restricted to the detection of shortcuts and can be applied to data exhibiting substantial domain gaps. I recommend that the authors conduct additional experiments on out-of-distribution (OOD) detection.

Author response:
Thanks for the detailed feedback! We will run additional experiments on OOD for future submission.

Reviewer comment (Experiments):
The proposed method is limited in the real-world application scenario. The proposed method requires a 'without shortcuts dataset' to detect whether there are shortcuts in the training dataset. I am not sure how many cases we can prepare the 'without shortcuts dataset' before we know whether the training dataset has a shortcut.

Author response:
We use `with' and 'without shortcut' to denote shortcuts in the controlled experiments to demonstrate that mutual information can be used to distinguish between the two datasets. We demonstrated the practical use case in Section 5 where we sample $X_{tr}$
 and $X_{test}$  from two different medical imaging datasets. Our method can detect the presence of shortcuts, which is in fact corroborated when the model trained on $X_{tr}$ is evaluated on $X_{test}$.

Reviewer comment (Evaluation):
The experiment graphs show that the mutual information is less discriminative than the losses, which diminishes the necessity of using mutual information to detect the existence of shortcuts in the training dataset.

Author response:
While using a trained model on new data $X_{test}$, we usually don't have the corresponding label $Y_{test}$. In contrast to the test accuracy, calculating MI $I(X_{test}; Z)$ does not require a labelled dataset and hence our method can be used to detect shortcuts without requiring test labels. We plotted the test accuracy to empirically demonstrate the connection between $I(X_{test}; Z)$ and the test accuracy.

Reviewer comment (Experiments):
The proposed algorithm and the experiment setting are different. If I denote an original training and test dataset as $D_{tr}$ and $D_{te}$, respectively, and the shortcut added training dataset as $D_{tr}^{sc}$, then the Figure 3(c) plots '$I(X_{test};Z)$ trained on $D_{tr}$' and '$I(X_{test};Z)$ trained on $D_{tr}^{sc}$'. However, the algorithm seems to be denoted to compare '$I(X_{test};Z)$ trained on $D_{tr}$' and '$I(X_{tr};Z)$ trained on $D_{tr}$'.

Author response:
In our experiments, we calculated both $I(X_{tes}t; Z)$ and $I(X_{tr}; Z)$ using model trained $D_{tr}$

Reviewer comment (Theory):
Algorithm 1 step 1) Why $\mathcal{F}$ is initially required?

Author response:
We need a trained model $\mathcal{F}$ to calculate the mutual information.