Paper: SSL Framework for Causal Inconsistency between Structures and Representations (ICLR-2024)

Reviewer comment (Theory):
The hypothesis 2 is confusing. In images or text, people project the original data into latent space, where each dimension does not have to be entangled. Even in the feature space, they do not have to be entangled. The pixel one may not relate to pixel two.

Author response:
We fully agree with you, and in fact, we just wanted to illustrate a potential occurrence: the possibility of interconnections among pixels. Therefore, we believe it might be more suitable to revise our paper's statement to "meaning that the representation is probably causally entangled over dimensions". Additionally, Hypothesis 2 seeks to convey that we can grasp the information of Indefinite Data without sampling. For instance, if we want to understand the climate in Singapore, we need an adequate temperature sampling to learn about the weather characteristics, such as day-night temperature difference, or whether it's a tropical or subtropical climate. However, if we wanted to comprehend the meaning of a sentence, we would not need to collect many samples of this sentence. We could understand its semantics with just one "sample", as different dimensions in the multi-value representation could be causally entangled, thus containing sufficient information.

Reviewer comment (Experiments):
How do you adapt your method to other datasets where the intervention cannot be done, for example, celebA.

Author response:
We are delighted to discuss these issues with you. Intervention is determined based on how causal variables are identified. For example, if we believe that a particular part (like background) causes bias in the representation, then the 'background' should be a parent node of the representation. It should be removed in our framework (of course, the causal relationships cannot be observed with only these two variables). This is somewhat similar to finding shortcuts and causal patterns in XAI areas. Furthermore, if we think a person's recognition result is influenced by some people's face photos, we should sequentially intervene on different parent nodes (representations of other influential people) for different people.

Reviewer comment (Experiments):
Experimental section can be improved.

Author response:
We have included our new experimental results and analysis in the updated PDF, which can be downloaded from the Supplementary Material (they are not added to the manuscript). We will present solutions to each of the experiments you raised individually.

1, Why evaluate "graph" and "structure": 

We sincerely apologize for this serious error on our part. You probably saw "graph" and "structure" mentioned in Tables 3 and 4. However, the correct terms should be "structure" and "representation". We realized that it was this mistake that caused your major confusion about our evaluation. Since this error is critical, we have made corrections immediately in the manuscript. With the correct terms "structure" and "representation", it becomes easy to understand- our motivation is to resolve inconsistencies in structure and representation, hence the experiment must evaluate both sides ("structure" and "representation" instead of "graph" and "structure"). As for the detailed evaluation methods, the causal structure used AUROC and Hamming distance, while the causal representation used AUROC and F1 score - these are well-established metrics. We have already explained how to evaluate inconsistencies in our previous rebuttal. You can refer to A1 in the previous round of the rebuttal and Section 2 in the revision PDF.

2, Variance of Results: 

We have added boxplots in Section 4 of the revised PDF to demonstrate the variance comparison intuitively. Please download the PDF from the Supplementary Material for review. 

3, Scalability: 

We have added results of different training set scales in Section 5 of the revision PDF (Table 100). Please download the PDF from the Supplementary Material for review. Moreover, we remind you that in Section 6.4 of the manuscript, we conducted evaluations for three downstream tasks, which not only demonstrated scalability but also generality.

Finally, we once again apologize for our serious error. Given the overlength of this paper, such grave mistakes can lead to unexpected confusion. We also now understand why you think that our experiment was not perfect. We deeply regret the additional time and energy impacts this has caused. We hope that this response will fully resolve any confusion and dissatisfaction with regard to the experiments.

Reviewer comment (Writing):
The definition of "non-statistical data" forms like images, videos, and text is vague. These data still contains statistical information, and what do you mean by "non-statistical"?

Author response:
We can see why you might have misunderstood, so we think it would be better to drop this adjective. In fact, what we wanted to express is that data like text, images, etc., are harder to exploit statistical characteristics (for example, we would not calculate high-order statistics between two sentences) compared to traditional causal domain data, but this is not important. In addition, if you want to understand more about the definition of Semi-Definite Data and Indefinite Data, you can refer to the review [1]. To make this definition strong enough, a lot of definitions and analyses must be introduced, such as "Causal Variables and Causal Representation", "Single-value and Multi-value Variables," etc. Therefore, in this paper, we only hoped to explain Causal Consistency clearly. We would be very grateful if you could accept the definition of data paradigm as reasonable.

Reviewer comment (Theory):
Definition 1 (Causal Data). I still think the definition of "Indefinite Data" "Semi-definite" based on D and M only is weak. How to quantify causal consistency is still an open question. This also applies to section 2.3.

Author response:
We believe your question 2 and question 4 are pointing to the same issue: How to measure or define causal consistency? Indeed, in line with your thoughts and those of the other two reviewers, a few weeks ago, we also realized that the introduction to causal consistency in our paper was as unappetizing as garbage. Consequently, we uploaded a file that re-interprets what causal representation is, how to measure causal consistency, and why Indefinite Data would lead to such intense causal inconsistency. We hope these changes will make it more..."delicious" to you! Please refer to our Official Comment "We add a PDF file into Supplementary Material for some Major Revisions." We trust that after understanding these supplements, you will see why our experiments need to revolve around causal graphs (causal structure) and causal representation. If there are still any doubts, please tell us.

Reviewer comment (Writing):
Def 3. The symbols looks slightly wired.

Author response:
You are likely referring to the index of the causal partial order $\natural$ (music symbol). We will change it to $\Xi$ (greek symbol).

Reviewer comment (Evaluation):
Section 6. I am still puzzled that how you evaluate the "causal graph" and "structure" both. It seems that these two things basically align with the same aspect of the algorithm. Why not evaluate "variance of learning results" or ”scalability of the algorithm“?

Author response:
1, Why evaluate "graph" and "structure": 

We sincerely apologize for this serious error on our part. You probably saw "graph" and "structure" mentioned in Tables 3 and 4. However, the correct terms should be "structure" and "representation". We realized that it was this mistake that caused your major confusion about our evaluation. Since this error is critical, we have made corrections immediately in the manuscript. With the correct terms "structure" and "representation", it becomes easy to understand- our motivation is to resolve inconsistencies in structure and representation, hence the experiment must evaluate both sides ("structure" and "representation" instead of "graph" and "structure"). As for the detailed evaluation methods, the causal structure used AUROC and Hamming distance, while the causal representation used AUROC and F1 score - these are well-established metrics. We have already explained how to evaluate inconsistencies in our previous rebuttal. You can refer to A1 in the previous round of the rebuttal and Section 2 in the revision PDF.

2, Variance of Results: 

We have added boxplots in Section 4 of the revised PDF to demonstrate the variance comparison intuitively. Please download the PDF from the Supplementary Material for review. 

3, Scalability: 

We have added results of different training set scales in Section 5 of the revision PDF (Table 100). Please download the PDF from the Supplementary Material for review. Moreover, we remind you that in Section 6.4 of the manuscript, we conducted evaluations for three downstream tasks, which not only demonstrated scalability but also generality.

Reviewer comment (Writing):
Hypothesis 2 is not clear. What does the notation E(\hat{x}_{s,m,n}) = x_{s,m,n} mean? The E operator has not been defined, is it the expectation? If so, what is the expectation over?

Author response:
Yes, $E$ stands for expectation. $E(\hat{x_{s,m,n}}) \doteq x_{s,m,n}$ means that if there are many identically distributed samples of the causal representation $\hat{x_{s,m,n}}$, their expectation could equally reflect the corresponding causal variable $x_{s,m,n}$. This is the same as what we said in the lines 110-111: "For instance, any sentence is enough to express its semantics, a single image can be read for its content." For example, for the sentence "good morning", if we collect different causal representations of this sentence from different models, given a large enough number, its expected representation represents the semantics of the sentence itself (therefore, we didn't use $=$ but $\doteq$).

Reviewer comment (Writing):
Section 2.2 is incomprehensible. What is causal consistency has not been defined so far. What is been plotted in Figure 1? The caption says it is the MSE between similarity matrices of representation and structure. What are these similarity matrices? Appendix A.3 does not give these details. What is the reconstruction loss the authors are referring to here? For example, lines 130-133, the authors say in the M > 1 or D > 1 case, the optimization of causal strength f changes to a weighted linear combination of f_m for the different M structures. However, this optimization problem has never been defined. Numerous such issues plague the readability of the paper.

Author response:
We have revised Section 2.2, please refer to our Official Comment for details. Moreover, because everyone regards the generation process of causal representation as a general generation model, reconstruction loss, and optimization would emerge. For instance, section VI a) in the authoritative paper "Towards Causal Representation Learning".

Reviewer comment (Reproducibility):
Eq 3 is unclear. What is L_k, what is being optimized? I understand at a high level we have 2 causal models, we intervene on both of them, and then have a way of checking the consistency of the two models. But beyond this high-level detail, exact specifics of how the authors carry out the interventions and check for consistency is not present in the paper.

Author response:
There was a typo here, it should not be $\mathcal{L}_{k}$ but $\mathcal{L}$ because the correct optimization function should not contain parameters unique to the intervention $k$. The optimization aim is the entire self-supervision process, so it involves output and parameters. Additionally, you perhaps overlooked that we proposed two implementations in Section 4.2 to assist readers in understanding our SSL framework, the details of these two implementations are in Appendix C.