Paper: Amortized Network Intervention to Steer the Excitatory Point Processes (ICLR-2024)

Reviewer comment (Experiments):
I appreciate the authors applying their proposed method to two important pratical problems. However, I think the paper can benefit from comparison to alternative methods.

Author response:
As the suggestion to add more comparisons, we implemented two common RL baselines: SAC and PPO on the COVID data. The empirical results have been incorporated into Appendix I.2 (Figure 13). By using NJODE as our simulator, we observe that the proposed method has successfully reduced the number of infested people (with an average reduced intensity: 0.49) in six counties among the chosen nine counties while SAC and PPO struggle to have a positive intervention effect to control the pandemic.

One interesting alternative method would be to combine causal inference with RL. In fact, in our experiment, we have introduced a pre-learned Boolean mask to determine the support of the action space, which can be interpreted as leveraging the Granger causal structure to guide policy optimization. Conducting online causal inference given the data collected by RL provides an interesting direction, yet is still open in our context. Particularly, ensuring the identifiability of causal effect estimation requires careful investigation, especially considering our dynamics are modeled as neural ODEs with jumps.  

A most related reference we found is [1], which asks the counterfactual question given the observed temporal point processes by behavior policy and is related to off-policy evaluation for target policy.

References: 
[1] Noorbakhsh, K., & Rodriguez, M. (2022). Counterfactual temporal point processes. Advances in Neural Information Processing Systems, 35, 24810-24823.

Reviewer comment (Theory):
Is it possible for the authors to describe the potential limitation due to the use of mean field approximation for reward modeling?

Author response:
For the limitation, we have a detailed error-bound analysis for the presented mean-field approximation which is covered in the updated manuscript Appendix E.2. This analysis is built upon a Lipschitz assumption of the given dynamic. The theoretical analysis further provides the insight that, as long as the spectral norm of the ODE neural layer is smaller than one, the mean-field approximation error can be well bounded.

Reviewer comment (Experiments):
Are there experiments to demonstrate the incorporation of fairness constraints and more?

Author response:
We added experiments by considering two soft fairness-related constraints and provided an empirical study in Appendix Section I.2. Specifically, two fairness augmentation terms were added to the objective function: 1) intervention budget cost; and 2)  policy smoothing cost. The first cost penalizes implementing the intervention policy that exceeds the predefined budget and the second one trades off the policy-changing frequency. In addition to the two terms under consideration, more realistic constraints relevant to specific scenarios can also be included in practice.

Reviewer comment (Evaluation):
"For instance, when regulating the coronavirus, government interventions must balance health concerns with economic implications and public sentiment." It is not clear whether the experiments described in the paper achieve such a balance.

Author response:
During the intervention, we have three constraints related to the balance of health concerns and economic implications:
1. We have set up a hard constraint and implemented a dynamic mask to explicitly exclude actions that fall outside the feasible space, ruling out the action that consecutively locks down one county. This maximum length of consecutive lockdown for one city can give us a trade-off between enforcing long-time continuous lockdowns to control pandemics and allowing temporary ‘freedom’ to recover the economy.
2. We introduce a policy smoothing cost to penalize the abrupt difference between consecutive policies. This means a larger weight will heavily discourage the agent from frequently changing the policy (which will incur more financial burden) while a smaller weight will less discourage a frequent policy change.
3. We also introduce an intervention budget cost and penalize the intervention policy that exceeds the predefined budget. Intuitively, a large weight of the intervention cost will discourage the agent from intervening more edges and thus have limited control over the pandemic while a small weight of the intervention cost allows more connections to be controlled but will generate a higher expense.

To understand this trade-off, we have performed experiments of choosing different weights of intervention budget cost and policy smoothing cost in Appendix I.2.

Reviewer comment (Evaluation):
Can the author explain why the NHPI baseline almost have a constant intensity cost in Figure 1?

Author response:
As for the constant intensity cost of the NHPI baseline, it was originally proposed on less dense point processes (i.e., tweet data) and used an event-driven RL to steer the point process. We conjecture the less satisfied performance is mainly caused by two aspects: 
1. NHPI used a transformer-based sequential model on historical events to learn the dynamic model. This indeed is not easy to capture the inherent connections between different types of events and also not the best model representation for event sequences generated from a dense graph. On the contrary, the proposed NJODE is built on explicit graph embeddings and has a strong potential to model the mutual interaction between different types of events.
2. NHPI learns a stationary value function through the Bellman function on an infinite horizon semi-MDP. We think the single-step TD learning for semi-MDP presented in NHPI could cause bias to the learned value function, especially in the non-stationary environment. Instead, our policy is learned from a receding horizon window without learning additional value functions and can be adapted to a new model by iteratively updating the parameters of neural ODE.

Reviewer comment (Reproducibility):
More details are needed in some components of the proposed model, including the mean field approximation for the rewarding model and the construction of the amortized policy.

Author response:
> **[Q5]**: Mean-field approximation justification

Here, we provide a brief justification for the mean-field approximation. For supplementary numerical and theoretical justification, please refer to our modified draft (Appendix E).

To evaluate the policy gradient at each iteration, one needs to get sufficient roll-out trajectories. Numerically, obtaining one roll-out trajectory requires solving ODEs repeatedly within each time interval, and sampling random counts at the end of each interval, which will be fed back to the system to update the state. The mean-field approximation, however, only needs to compute the roll-out once – it uses a deterministic trajectory to approximate the mean of the stochastic trajectories, by replacing the random counts sampled at the end of each interval by their means. 

Mean-field approximation dramatically reduces the computational burden in policy optimization. We added numerical results (shown in Fig. 11, Appendix E.2) and theoretical analysis (in Appendix E.1) to demonstrate the approximation accuracy. 

The theoretical analysis further provides the insight that, as long as the spectral norm of the ODE neural layer is smaller than one, the mean-field approximation error can be well bounded. 

> **[Q6]**: Amortized policy construction

Mathematically, the proposed amortized policy $\pi_{ani}$ is learned over a distribution of random tasks from $\mathcal{M}$, i.e., $\pi_{ani} = E_{M_{i}}[\pi_{i}]$ where $M_{i} \sim \mathcal{M}$. The meta-policy gradient over different tasks is then updated in a similar fashion with Reptile [1]. We have clarified the point in the updated version.

[1] Nichol, A., Achiam, J., & Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.

Reviewer comment (Novelty):
The review of the work on networked excitatory point processes seems to only focus on work that used ODEs. However, I would like to point out that there are also other streams of research on networked excitatory point processes, for example, [1], [2], [3]. It is better to conduct a more comprehensive review of the topic.

Author response:
> **[Q1]**: Related work of networked excitatory point processes.

We have carefully reviewed and incorporated the listed publications in Appendix Section A (related work) in our updated version.

Reviewer comment (Writing):
In the definition of the temporal graph network, what are the definitions of edges in the two examples given in the paper? I do not find formal definitions. Could you please clarify?

Author response:
> **[Q2-Q4]**: Definition clarifications of the model details.

Here we make the following clarifications.
1. **Edge**: For pandemic control cases, the node is defined as county/city, and the edges are defined as the accessibility from one city to another city (see Figure 1). For traffic control cases, the node is defined as a road lane, and edges are defined as the connections between different lanes.

Reviewer comment (Theory):
On page 3, it is claimed that the "high-dimensional event sequences $\{X_t\}_{t\ge 0}$ has a stationary dynamics". What do you mean by "stationary"? If $X_t$ is the accumulated counts up to time $t$, it is unlikely to be a stationary time series. Please clarify.

Author response:
> **[Q2-Q4]**: Definition clarifications of the model details.

Here we make the following clarifications.
2. **Stationary Dynamic**: Here $X_{t}$ refers to the number of a sequence of spike counts within a time step, so $X_{t}$ can be stationary in a long-term period. Moreover, under a careful choice of the initial parameters for the neural network (as the contraction condition derived in Appendix E), the dynamic system is also stationary.

Reviewer comment (Reproducibility):
Is the influence matrix $W$ in equation~(4) a given matrix or learned from the data?

Author response:
> **[Q2-Q4]**: Definition clarifications of the model details.

Here we make the following clarifications.
3. **Influence matrix $W$**: In our cases, only the support of $W$ matters, and it can be obtained from prior knowledge. The reason is if we jointly learn the influence matrix $W$ and the discrete kernel function $\phi$ in the proposed system (Eq. (4) in the manuscript), W will become not identifiable and lose its original physical meaning. Therefore, we choose to fix the support of $W$ (i.e., $w_{ij} \in [0,1] $) and the specific value can be inferred from the data-driven method. For instance, in the COVID case, the influence matrix is restricted to $[0,1]$ and is proportional to the inverse distance between different counties.

Reviewer comment (Theory):
The use of mean field approximation on page 4 seems to be rather ad-hoc, without much theoretical or even heuristic justifications. Can you give an example to show the difference between the actual reward and the one calculated based on the mean-field approximation? Are they actually close? Perhaps some simulation studies can be carried out to show the difference in actual reduction in intensity using these two reward objectives in some simple settings.

Author response:
> **[Q5]**: Mean-field approximation justification

Here, we provide a brief justification for the mean-field approximation. For supplementary numerical and theoretical justification, please refer to our modified draft (Appendix E).

To evaluate the policy gradient at each iteration, one needs to get sufficient roll-out trajectories. Numerically, obtaining one roll-out trajectory requires solving ODEs repeatedly within each time interval, and sampling random counts at the end of each interval, which will be fed back to the system to update the state. The mean-field approximation, however, only needs to compute the roll-out once – it uses a deterministic trajectory to approximate the mean of the stochastic trajectories, by replacing the random counts sampled at the end of each interval by their means. 

Mean-field approximation dramatically reduces the computational burden in policy optimization. We added numerical results (shown in Fig. 11, Appendix E.2) and theoretical analysis (in Appendix E.1) to demonstrate the approximation accuracy. 

The theoretical analysis further provides the insight that, as long as the spectral norm of the ODE neural layer is smaller than one, the mean-field approximation error can be well bounded.

Reviewer comment (Reproducibility):
On page 5, it is stated that "Given a sequence of local policies $\{\pi_i\}$, $1\le i\le M$,  addressing $M$ distinct sub-problems, our goal is to create an amortized policy $\pi_{amo}$". However, I did not find any detail on how this goal can be achieved. Can you clarify?

Author response:
> **[Q6]**: Amortized policy construction

Mathematically, the proposed amortized policy $\pi_{ani}$ is learned over a distribution of random tasks from $\mathcal{M}$, i.e., $\pi_{ani} = E_{M_{i}}[\pi_{i}]$ where $M_{i} \sim \mathcal{M}$. The meta-policy gradient over different tasks is then updated in a similar fashion with Reptile [1]. We have clarified the point in the updated version.

[1] Nichol, A., Achiam, J., & Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.

Reviewer comment (Writing):
The label of the y-axis in Figure 3 reads "Intensity Cost". What is the definition of the "Intensity Cost"?

Author response:
> **[Q7]**: Figure label clarification

For brevity, here intensity cost means the average intensity throughout a fixed period and over all the nodes within a local region. We use this indicator to reflect how fast the pandemic/traffic will grow within a multivariate system. We have clarified this point in the updated version.

Reviewer comment (Novelty):
In my point of view, the authors did not clarify their contributions.

Author response:
We highlight the contributions part in the Introduction of the revised manuscript. Our contributions include three aspects: 1) a technically sound method to tackle the challenge of large-scale network intervention, 2) the proposal of a permutation-equivalent intervention paradigm, and 3) comprehensive experiments on traffic congestion and COVID data. Most importantly, our key contribution lies in introducing the permutation-equivalent intervention paradigm within a dynamically evolving large network topology, facilitated by the utilization of the proposed bi-contrastive loss function.

Reviewer comment (Novelty):
Policy learning within the permutation equivalence class through learning embedding $p^t, m^t$ via contrastive method is an interesting idea. However, the authors do not differentiate their work from (Chen et al., 2020b).

Author response:
As for the difference between the proposed Bi-contrastive loss and the previous SimCLR (Chen et al., 2020b):
 - Firstly, unlike SimCLR which directly performs augmentations on visual representations, we augment our network on its latent embedding space by utilizing graph augmentation techniques. 
- Secondly, our model does not require sampling additional data as negative examples and projecting the augmented sample to one embedding. Instead, we only use one anchor sample (here is the embedded network) project the sample to two different embeddings (p and m), and augment the anchor sample by permuting its node orders or adding noise to the node embeddings such that it can form two negative pairs naturally (as depicted in Figure 1). More details will be provided in Appendix Section F in the updated version.

Reviewer comment (Theory):
Each part in a large networks may have both global and local patterns. How to deal with the local patterns in model learning and policy optimization is not clear. Or the authors are just trying to obtain a policy that only has "overall" good performance.

Author response:
In our model, we have the permutation equivalent property to extract the global invariant patterns. Regarding the local pattern, we use the node embeddings $H_{t}$ learned by the Networked Jump ODE model to store the information. For instance, the elements of node embedding can have the meaning of a local region’s population and economy.  When performing policy optimization, the node embeddings will be used to learn the policy and global invariant patterns. Therefore, our policy can capture both local patterns and global patterns elegantly.

Reviewer comment (Experiments):
Another question is how to deal with the soft/hard constraints. I did not find evidence supporting the effectiveness of the constraint ensuring methods.

Author response:
For hard constraints like strictly controlling the maximum number of consecutive interventions, we use a ‘’dynamic mask” in our policy learning. For soft constraints, we can treat the constraints as ‘’regularization/augmentation terms” and add these terms to the objective function. 

We added experiments by considering two soft fairness-related constraints and provided an empirical study in Appendix Section I.2. Specifically, two fairness augmentation terms were added to the objective function: 1) intervention budget cost; and 2) policy smoothing cost. The first cost penalizes implementing the intervention policy that exceeds the predefined budget and the second one trades off the policy-changing frequency. In addition to the two terms under consideration, more realistic constraints relevant to specific scenarios can also be included in practice.

Reviewer comment (Experiments):
As I have mentioned before, is it possible for the current framework to capture the local patterns in a large network and perhaps steer the learned policy towards local patterns while maintaining the generalization power (such as regularized policy optimization).

Author response:
In our model, we have the permutation equivalent property to extract the global invariant patterns. Regarding the local pattern, we use the node embeddings $H_{t}$ learned by the Networked Jump ODE model to store the information. For instance, the elements of node embedding can have the meaning of a local region’s population and economy.  When performing policy optimization, the node embeddings will be used to learn the policy and global invariant patterns. Therefore, our policy can capture both local patterns and global patterns elegantly.