Paper: A Dynamical View of the Question of Why (ICLR-2024)

Reviewer comment (Novelty):
The general problem setup reminds me of what some in the field call "reverse causality". I.e., the aim is not to obtain the effects of causes (interventions), but rather obtain the causes of effects. I am not an expert in this area, but I think there are some existing works on this that are not mentioned in the paper, e.g., Gelman and Imbens (2013). I do not think that these approaches build upon a dynamic setting, but I still think that the paper would benefit from a section that relates these approaches.

Author response:
Thanks for the pointer, we would add the citation. This seems to be a short paper that suggests forward and reverse questions. While not providing any actual formulation (and computational framework), the view is interesting and must be acknowledged.

Reviewer comment (Theory):
Definition of cause/grit. I am wondering whether an event $A$ could be a cause of an event $B$ even though the expected grit of $B$ does not increase from before to after $A$. Imagine a medical setting, where a doctor prescribes treatment (event $A$), that only works in combination with another treatment (event $C$) to cure the patient (event ($B$). ($A$ -> $C$ -> $B$). Then $C$ would be a cause but $A$ would not be a cause because the grit would not increase from before to after $A$ (since the grit is the minimum probability of $B$ over all possible policies, also the one that does not administer $C$). However, intuitively both $A$ and $C$ should be causes.

Author response:
In your example, if $C$ happens (or can happen) with some non-zero probability, even unknown or non-stationary, then $A$ is a cause of $B$ and is also correctly captured by our theory [the grit indeed increases by $A$ as shown below]. On the other hand, if $C$ is chosen not to happen at all, then $B$ will never happen. In this case, $A$ will not be considered a cause because we violate condition C1 (in our definition of causation), where $A$ must happen before $B$ - here $B$ never happens! We note that in the lack of event $B$, there is no causal question. Put it a bit differently, $A$ is a cause of $B$ conditioned on $C$ being selected with non-zero probability (and this is exactly what our framework deduces). That is, grit is meaningful only if $B$ is not unlikely to happen. 

As a side note, we have also observed this computationally in the Diabetes example. W.r.t. insulin intakes, we observed that **any** insulin intake shows an increase in expected grit -- even when it is insufficient to cause event $B$ by itself and at least another sufficiently large intake is required. This is precisely what is expected from the theoretical analysis. 

**Analysis:** First, please recall that the definition of causation asserts the **expected** change of grit must be positive from before to after $A$. Let $C$ happen with probability $p > 0$ and if it happens, the grit of $B$ increases by $\alpha > 0$ (since $A$ has already happened). Then $\mathbb{E}\Delta\Gamma_{B}(A) = p\times\alpha + (1-p) \times 0 > 0 $. Hence, grit of $B$ cannot be the same from before to after $A$, and indeed **it increases**. Thus, $A$ is also a cause unless either $p$ or $\alpha$ is zero.

Reviewer comment (Theory):
Why is the notion of grit the correct way of defining causes (see weaknesses)? I am open to increasing my score if the authors make a convincing case.

Author response:
As we discussed above, the notion of grit works fully inline with what is expected intuitively as a causal relationship (even when it looks counterintuitive at the first glance). The minimum probability provides the natural way of dealing with possible future actions, as it ensures that $A$ is named a cause of $B$ only if it is so under any (even pessimistic) future trajectory. Importantly, if there is no action, then it automatically retrieves the standard definition (that $A$ should increase *probability* of $B$, rather than *grit* of $B$). Moreover, if a fixed policy is chosen to follow, either in general or only after event $A$, our framework still works readily, as $V^*$ will automatically be replaced by $V^{\pi}$.

Reviewer comment (Theory):
The notation of the state $X(t)$ seems inconsistent. Should it not depend on the actions $u(t)$? E.g., in Eq.(1) and (2) $u$ only appears on the right-hand side of the equations. Furthermore, $V^\pi$ on page 3 does not depend on $\pi$.

Author response:
This is a common way, which basically only sets time as the base argument (mostly to parallel the discrete-time format). It’s being used in literature, see for example RL/DP papers in continuous time, such as this one: 

Kim and Yang, ‘Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time’, in Conference on Learning for Dynamics and Control, 2020, vol. 120, pp. 739–748.

> notation $V^{\pi}$

We fixed it; thanks for the pointer. $\pi$ needs to be in the condition of expectation.

Reviewer comment (Writing):
The reviewer find the paper very hard to read and understand.

Author response:
We would like to highlight that this is a deeply interdisciplinary subject by nature and a certain level of complexity is unavoidable. This paper covers materials at the intersection of different communities (causality, systems control, reinforcement learning, and even philosophy). We tried to keep the presentation more formal and less descriptive with the hope that it helps the reader to follow the ideas in a precise manner, regardless of differences in community-related aspects. We indeed wrote several drafts of this work, and based on various internal feedback; the current version seems to be the most reliable as read by people with different backgrounds.

We, however, will do modifications before final submission to improve the presentation as much as possible.

Reviewer comment (Novelty):
Some of the assumptions are already well known in causality (“Causality necessitates time”. “Cause happens before effect”).

Author response:
We are respectfully not sure why this point is set as a weakness?
 
In any case, we would like to emphasize that these are NOT our assumptions, rather these are basic axioms which are well-known and are true in any causal arguments. We simply place them at the forefront. We will modify the text to clarify this point.

Reviewer comment (Theory):
The reliance of the proposed method on optimal value function corresponding to reward function seems a strong one.

Author response:
It is a fair point but it is also the case for **any** method that uses neural nets or approximation techniques (and we add a statement to reflect on this in the paper). Nonetheless, what is important here is that, using optimal value functions, our framework provides a strong basis (with formal grounds) to design a learning problem and gives all the computational tools to establish causal links in dynamical systems -- something which did not exist before, at least not at this level of formal rigour and explicitness, while still practically applicable and useful. Then, in practice, the optimal values will be replaced with the *learned* ones from data. As with ANY learning problem, the results will be reliable to the extent of approximation errors; this is not a specific problem of our framework, rather is a generic issue.
 
Of note, additional theoretical arguments can also be made here. For example, if the approximated value function is off by some maximum amount, then under certain assumptions, the degrade in validity of causal arguments will also be bounded (proof sketch: the bound over values will map to the same bound over grit, which will then be used in the decomposition lemma to bound the error in each component). Or, if the value of a certain policy is used in place of $V^{*}$, the causal arguments will be completely valid if that policy is followed after event $A$ (this result is immediate from the definition of value functions).
 
We would like to remark that presenting such results can go much beyond the scope of a single conference-paper.

Reviewer comment (Experiments):
Will be interesting to investigate how the proposed method works as the function of number of variables.

Author response:
Our framework scales well to even huge state-spaces simply because it maps the problem into RL settings (and current deep RL methods can easily handle extremely large state-spaces). Even in the provided Atari problem, the state comprises 28224 variables (corresponding to the pixels of four consecutive game screens)! The T1 diabetes simulator, on the other hand, deals with 13 state variables and 2 action variables. This reflects the range of variables our proposed method can handle. 

In a related context, as the base problem here is an RL problem (due to our Value Lemma), any value-based RL technique can be used, and the scalability will be inherited from the properties of the RL algorithm.

Reviewer comment (Presentation):
When the MDP is introduced, it is not clear how it relates to the system of interest. It only becomes clear later that the MDP is used for estimation of grit and reachability, and is not part of the definition of causality.

Author response:
We will add a clarification why MDP is defined (when it is introduced) to help address your point.

Reviewer comment (Writing):
Minor issue: At the top of page 3, "straight" should be changed to "straightforward."

Author response:
It stands for Halpern and Pearl. The explanatory sentence should unfortunately have been deleted in one of the last revisions. Will fix it.

Reviewer comment (Theory):
It is unclear whether this framework requires the ability to take actions (i.e., whether an MDP is necessary) or if that is just an estimation technique. If it is the former, that implies the existence of a simulator that plays the role of the causal model, allowing counterfactual queries by trying different actions from the same state. If this is the case, this limitation should be made explicit — for example, if one merely observes a dynamical system without the ability to take actions, causality cannot be inferred in this framework, correct?

Author response:
Our framework does not *need* the ability to take actions, and you certainly can use this theory for a system with only states (no action).

The reason to use MDP is basically the other way round: a system/environment may *include* actions and the framework needs to be able to cover such cases as well; hence, MDP is needed in general. Moreover, as we discussed in the paper, an event may also be defined as taking an action (a change in an action variable) rather than a change in a state variable. Our theory covers such cases too, which is again why we need MDP in general.

 > I just observe a dynamical system but don't have the ability to take actions, I can't infer causality in this framework, right?
 
You definitely can. This case can happen if either there is no action in the system by nature, or a fixed policy is taking care of actions with no freedom (hence, there is effectively no action). In all state-only problems, you simply use $\Gamma_{B}(X)$ and $V_{\Gamma}(X)$ and the rest of the theory remains valid and intact.