Paper: A Unified Causal View of Instruction Tuning (ICLR-2024)

Reviewer comment (Experiments):
This paper should compare the performance of their methods with different baselines and backbones to comprehensively validate the effectiveness of proposed method.

Author response:
We thank the reviewer for the constructive feedback. As suggested, we will compare our method to additional baselines using different backbones to further validate its effectiveness. We will also analyze the representations learned by our model and provide supporting evidence that causal mechanisms are captured.

We also sincerely appreciate the reviewer highlighting our key contributions - the novel meta-SCM, Uniform Identifiability Condition proof, and detailed experimental comparisons. Your feedback has been invaluable for improving our work. Please let us know if you have any other suggestions for strengthening the paper. Thank you again for your time in thoroughly reviewing our submission and providing these thoughtful suggestions.

Reviewer comment (Experiments):
In the experiment, this paper should give some evidence that the model learn causal representations and task-oriented causal generative mechanisms.

Author response:
We thank the reviewer for the constructive feedback. As suggested, we will compare our method to additional baselines using different backbones to further validate its effectiveness. We will also analyze the representations learned by our model and provide supporting evidence that causal mechanisms are captured.

We also sincerely appreciate the reviewer highlighting our key contributions - the novel meta-SCM, Uniform Identifiability Condition proof, and detailed experimental comparisons. Your feedback has been invaluable for improving our work. Please let us know if you have any other suggestions for strengthening the paper. Thank you again for your time in thoroughly reviewing our submission and providing these thoughtful suggestions.

Reviewer comment (Theory):
The authors claim that "Causally speaking, there exist backdoor paths between the target labels and non-causal latent factors through the task variable T.". But when we are doing a task, the task variable is given. And in Fig.1, we have $L_i \perp L_j|T$. So, there must be a problem arising from either the claim itself or the causal graph. Considering question Q1, I think it is more likely the latter. I would appreciate it if the authors could provide a specific example to illustrate how the non-causal factors can correlate with $Y_t$ (e.g., the simplest text classification). This is one of the most important questions for me, and I'm looking forward to the authors' response.

Author response:
This is an excellent question and we appreciate you raising this important issue. $T$ should be modified to $D$ to represent inherent dataset properties, which are unobservable. This would make $L_i$ not independent of $L_j$, leading to a backdoor path and inducing spurious correlation. We will correct the imprecision in our writing, which improperly used $T$ to denote both dataset properties and tasks. Thank you again for catching this critical misunderstanding.

Reviewer comment (Theory):
Following the above question, even if $T$ can act as a confounder and spurious correlations exist in the data, how does the traditional training process make the model learn the spurious correlations? This problem is also untouched, which makes the motivation informal and weak.

Author response:
We appreciate you raising this excellent point. The traditional training process optimizes the likelihood $P(Y|X)$, where $X$ contains a mixture of information - some factors causally influence $Y$ while others are spuriously correlated. Without explicitly disentangling the latent factors within $X$, the shortcuts from non-causal variables to $Y$ may be exploited, leading the model to learn these unintended spurious correlations along with the true causal relationships.

Reviewer comment (Writing):
Section 3.2 is really hard to follow. How to alleviate the influence of the spurious correlations? By backdoor adjustment or introducing intervened data?

Author response:
Since both the inherent dataset properties $D$ and latent factors are unobservable, we cannot directly perform backdoor adjustment or intervention. Therefore, to mitigate spurious correlation, an alternative approach is to only utilize the parent latent factors of each task's $Y_t$ for prediction.  This branch of research can be categorized as causal representation learning [1]. However, being able to select the parent factors relies on latent factor identifiability. Thus, guaranteeing identifiability of latent factors is a core challenge in causal representation learning.

[1] Towards Causal Representation Learning. Proceedings of the IEEE, 2021.

Reviewer comment (Theory):
The definition of identifiability is confusing. What's the identifiability of a SCM? When we say the identifiability, it usually refers to the causal quantity on two specific variables rather than a SCM. Also, the target of the UIC loss in Eq.6 is vague. What causal quantity do you want to measure?

Author response:
That is an excellent question. In causal inference, identifiability refers to " A causal quantity can be computed from a purely statistical quantity." [2]. However, in causal representation learning, identifiability has a different meaning - it indicates that "Representations for each latent factor can be learned without mixing information with others, while ensuring that the difference between the learned representations and the true representations remains within acceptable bounds of uncertainty" [3, 4, 5]. The goal of the UIC is to guarantee latent variable identifiability under this definition from causal representation learning.

[2] Introduction to causal inference. 2020.

[3] Self-supervised learning with data augmentations provably isolates content from style. NeurIPS 2021.

[4] Partial Identifiability for Domain Adaptation. ICML 2022.

[5] Identifiability results for multimodal contrastive learning. ICLR 2023.

Reviewer comment (Theory):
I guess the authors want to say that, if $L_i$ is identifiable, then it is the causal factor of $Y_t$. So, the UIC loss is used to promise the identifiability. There are two issues need to be addressed. One is why identifiable factors are causal ones, and another is why UIC can select identifiable factors. I'm not clear whether the second problem has been addressed by Section 3.2, since I think it's not well orangized. Nevertheless, I think the first problem is untouched.

Author response:
Thank you for your insightful question. We apologize that Section 3.2 did not effectively communicate our approach regarding identifiability and causal factor selection.

To clarify, we do not posit that "if $L_i$ is identifiable, then it is a causal factor of $Y_t$." For each task, whether $L_i$ is a causal parent of $Y_t$ is an intrinsic property. In our work, causal factors for different tasks are selected by a binary matrix with 0,1 values. This matrix is learned by maximizing likelihood, similar to score-based causal discovery [6]. Notably, selecting causal factors relies on the premise that the factors $L_i$ are identifiable. Thus, we introduce a UIC Loss as a regularization term to ensure $L_i$ identifiability. While UIC constrains the structure between $L_i$ and $Y_t$, this constraint is flexible. Causal factor selection is still primarily driven by the binary selection matrix.

[6] Generalized score functions for causal discovery. SIGKDD 2018.

Reviewer comment (Experiments):
The tasks are carefully chosen and the model tested is not a SOTA seq2seq model that is widely adopted for Instruction Tuning.

Author response:
We sincerely thank the reviewer for recognizing the intuitive merit of considering task-related factors and discarding spurious correlations. Due to computational constraints, the scale of our current experiments is limited. Per the reviewer's suggestions, we will explore scaling up our approach using larger models, such as Llama. We will also expand our method to additional tasks and datasets to further demonstrate its effectiveness. We deeply appreciate the reviewer providing this invaluable feedback to strengthen our work.