Original: L EARNING TO R EASON AND A CT IN C ASCADING P ROCESSES 
Revised: L EARNING TO R EASON AND A CT IN C ASCADING P ROCESSES D RIVEN BY S EMANTIC E VENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training agents to control a dynamic environment is a fundamental task in AI.
Revised: Training agents to control a dynamic environment is a fundamental task in AI.

Original: In many environments, the dynamics can be summarized by a small set of events that capture the semantic behavior of the system.
Revised: In many environments, the dynamics can be summarized by a small set of events that capture the semantic behavior of the system.

Original: Typically, these events form chains or cascades.
Revised: Typically, these events form chains or cascades.

Original: We often wish to change the system behavior using a single intervention that propagates through the cascade.
Revised: We often wish to change the system behavior using a single intervention that propagates through the cascade.

Original: For instance, one may trigger a biochemical cascade to switch the state of a cell or reroute a truck in logistic chains to meet an unexpected, urgent delivery.
Revised: For instance, one may trigger a biochemical cascade to switch the state of a cell or reroute a truck in logistic chains to meet an unexpected, urgent delivery.

Original: We introduce a new supervised learning setup called Cascade .
Revised: We introduce a new supervised learning setup called Cascade .

Original: An agent observes a system with known dynamics evolving from some initial state.
Revised: An agent observes a system with known dynamics evolving from some initial state.

Original: It is given a structured semantic instruction and needs to make an intervention that triggers a cascade of events, such that the system reaches an alternative (counterfactual) behavior.
Revised: It is given a structured semantic instruction and needs to make an intervention that triggers a cascade of events, such that the system reaches an alternative (counterfactual) behavior.

Original: We provide a test-bed for this problem, consisting of physical objects.
Revised: We provide a test-bed for this problem, consisting of physical objects.

Original: We combine semantic tree search with an event-driven forward model and devise an algorithm that learns to efficiently search in exponentially large semantic trees of continuous spaces.
Revised: We combine semantic tree search with an event-driven forward model and devise an algorithm that learns to efficiently search in exponentially large semantic trees of continuous spaces.

Original: We demonstrate that our approach learns to effectively follow instructions to intervene in new complex scenes.
Revised: We demonstrate that our approach learns to effectively follow instructions to intervene in new complex scenes.

Original: When provided with an observed cascade of events, it can also reason about alternative outcomes.
Revised: When provided with an observed cascade of events, it can also reason about alternative outcomes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Teaching agents to understand and control their dynamic environments is a fundamental problem in
Revised: Teaching agents to understand and control their dynamic environments is a fundamental problem in

Original: AI.
Revised: AI.

Original: It becomes extremely challenging when events trigger other events.
Revised: It becomes extremely challenging when events trigger other events.

Original: We denote such processes as cascading processes .
Revised: We denote such processes as cascading processes .

Original: As an example, consider a set of chemical reactions in a cellular pathway.
Revised: As an example, consider a set of chemical reactions in a cellular pathway.

Original: The synthesis of a new molecule is a discrete event that later enables other chemical reactions.
Revised: The synthesis of a new molecule is a discrete event that later enables other chemical reactions.

Original: Cascading processes are also prevalent in man-made systems.
Revised: Cascading processes are also prevalent in man-made systems:

Original: For instance, in assembly lines, when one task is completed, e.g., construction of gears, it may trigger another task, e.g. building the transmission system.
Revised: In assembly lines, when one task is completed, e.g., construction of gears, it may trigger another task, e.g. building the transmission system.

Original: Cascading processes are abundant in many environments, from natural processes like chemical reactions, through managing crisis situations for natural disasters (Zuccaro et al., 2018; Nakano et al., 2022) to logistic chains or water treatment plants (Cong et al., 2010).
Revised: Cascading processes are abundant in many environments, from natural processes like chemical reactions, through managing crisis situations for natural disasters (Zuccaro et al., 2018; Nakano et al., 2022) to logistic chains or water treatment plants (Cong et al., 2010).

Original: A major goal with cascading processes is to intervene and steer them towards a desired goal.
Revised: A major goal with cascading processes is to intervene and steer them towards a desired goal.

Original: For example, in biochemical cascades, one hopes to control chemical cascades in a cell by providing chemical signals; in logistics, a cargo dispatch plan may be completely modified by assigning a cargo plane to a different location.
Revised: For example, in biochemical cascades, one hopes to control chemical cascades in a cell by providing chemical signals; in logistics, a cargo dispatch plan may be completely modified by assigning a cargo plane to a different location.

Original: This paper addresses the problem of reasoning about a cascading process and controlling its qualitative behavior.
Revised: This paper addresses the problem of reasoning about a cascading process and controlling its qualitative behavior.

Original: We describe a new counterfactual reasoning setup called “ Cascade ”, which is trained via supervised learning.
Revised: We describe a new counterfactual reasoning setup called “ Cascade , which is trained via supervised learning.

Original: At inference time, an agent observes a dynamical system, evolving through a cascading process that was triggered from some initial state.
Revised: At inference time, an agent observes a dynamical system, evolving through a cascading process that was triggered from some initial state.

Original: We refer to it as the “unsatisfied” or “observed” cascade.
Revised: We refer to it as the “unsatisfied” or “observed” cascade

Original: The goal of the agent is to steer the system toward a different, counterfactual, configuration.
Revised: The goal of the agent is to steer the system toward a different counterfactual , configuration.

Original: That target configuration is given as a set of qualitative constraints about the end results and the intermediate properties of the cascade.
Revised: That target configuration is given as a set of qualitative constraints about the end results and the intermediate properties of the cascade

Original: We call these constraints the “instruction”.
Revised: We call these constraints the “instruction”.

Original: To satisfy that instruction, the agent intervenes with the system at a specific point in time by changing the state of one element which we call the “pivot”.
Revised: To satisfy that instruction the agent intervenes with the system at a specific point in time by changing the state of one element which we call the “pivot”.

Original: To solve the Cascade learning problem, we train an agent to select an intervention given a state of a system and an instruction.
Revised: To solve the Cascade learning problem, we train an agent to select an intervention given a state of a system and an instruction.

Original: Importantly, we operate in a counterfactual mode (See Pearl, 2000).
Revised: Importantly, we operate in a counterfactual mode (See Pearl, 2000).

Original: Dur- describes desired “coun

Original: Output (the sat isfied cascade): The agent intervenes sets the (continuous, 2D) initial velocity of the purple (the “pivot”) to achieve the goal Only keyframes shown.
Revised: During training, the agent only sees scenarios that are “satisfied”, in the sense that the system dynamics

Original: be/u1Io-ZWC1Sw (Anonymous)

Original: [Figure]
Revised: [Figure]

Original: ing training, the agent only sees scenarios that are “satisfied”, in the sense that the system dynamics obey the constraints given in the instruction.
Revised: obey the constraints given in the instruction.

Original: The reason is that in the real world it is not possible to rewind time and simultaneously obtain both a satisfied and an unsatisfied sequence of events.
Revised: The reason is that in the real world it is not possible to rewind time and simultaneously obtain both a satisfied and an unsatisfied sequence of events.

Original: Steering a cascade process is hard.
Revised: Steering a cascade process is hard.

Original: To see why, consider a natural but naive approach to the
Revised: To see why, consider a natural but naive approach to the

Original: Cascade problem: train an end-to-end regression model that takes the system and instruction as input and predicts the necessary intervention.
Revised: Cascade problem: train an end-to-end regression model that takes the system and instruction as input and predicts the necessary intervention.

Original: It is challenging because in many cases, a slight change in one part of the system can make a qualitative effect on the outcome.
Revised: It is challenging because in many cases, a slight change in one part of the system can make a qualitative effect on the outcome.

Original: This may lead to an exponential number of potential cascades.
Revised: This may lead to an exponential number of potential cascades.

Original: This “butterfly effect” (Lorenz, 1993) is typical in cascading systems, like a billiard ball missing another ball by a thread or a truck reaching a warehouse right after another truck has already left.
Revised: This “butterfly effect” (Lorenz, 1993) is typical in cascading systems, like a billiard ball missing another ball by a thread or a truck reaching a warehouse right after another truck has already left.

Original: Back to the regression approach, we empirically find that it fails, presumably because the set of possible chains of events is exponentially large, and the model fails to learn how to find an appropriate chain that satisfies the instruction.
Revised: Back to the regression approach, we empirically find that it fails, presumably because the set of possible chains of events is exponentially large, and the model fails to learn how to find an appropriate chain that satisfies the instruction.

Original: We discuss other challenges in Section 4.
Revised: We discuss other challenges in Section 4.

Original: Technical insights.
Revised: Technical insights.

Original: In designing our approach, we follow two key ideas.
Revised: In designing our approach, we follow two key ideas.

Original: First, instead of modeling the continuous dynamics of the system, we reduce the search space by focusing on a small number of discrete, semantic events.
Revised: First, instead of modeling the continuous dynamics of the system, we reduce the search space by focusing on a small number of discrete, semantic events.

Original: To do this, we design a representation called an “ Event Tree ” (Figure 2).
Revised: To do this, we design a representation called an “ Event Tree ” (Figure 2).

Original: In a billiard game, these events would be collisions of balls.
Revised: In a billiard game, these events would be collisions of balls.

Original: In logistic chains, these events would be deliveries of items to their target location or assembly of parts.
Revised: In logistic chains, these events would be deliveries of items to their target location or assembly of parts.

Original: To reduce the search space, we build a tree of possible future events, where the root holds the initial world-state.
Revised: To reduce the search space, we build a tree of possible future events, where the root holds the initial world-state.

Original: Each child node corresponds to a possible future subsequent event from its parent.
Revised: Each child node corresponds to a possible future subsequent event from its parent.

Original: Thus, a path in the tree from a root to a descendant captures a realizable sequence of events.
Revised: Thus, a path in the tree from a root to a descendant captures a realizable sequence of events.

Original: Our second idea is to learn how to efficiently search over the event tree.
Revised: Our second idea is to learn how to efficiently search over the event tree.

Original: This is critical because the tree grows exponentially with its depth.
Revised: This is critical because the tree grows exponentially with its depth.

Original: We learn a function that assigns scores to tree nodes conditioned on the instruction and use these scores to prioritize the search.
Revised: We learn a function that assigns scores to tree nodes conditioned on the instruction and use these scores to prioritize the search.

Original: We also derived a Bayesian correction termand use it to guide the search with the observed cascade: we first find the path in the event tree that corresponds to the observed cascade, and thenuse it to correct the scores of nodes along that path.
Revised: We also derived a Bayesian correction term to guide the search with the observed cascade: we first find the path in the event tree that corresponds to the observed cascade, and then correct the scores of nodes along that path.

Original: Modelling system dynamics with forward models.
Revised: Modelling system dynamics with forward models.

Original: A forward model describes the evolution of the dynamic systems in small time steps.
Revised: A forward model describes the evolution of the dynamic systems in small time steps.

Original: There is extensive literature on learning forward models from observations in physical systems (Fragkiadaki et al., 2016; Battaglia et al., 2016; Lerer et al., 2016;
Revised: There is extensive literature on learning forward models from observations in physical systems (Fragkiadaki et al., 2016; Battaglia et al., 2016; Lerer et al., 2016;

Original: Watters et al., 2017; Janner et al., 2019).
Revised: Watters et al., 2017; Janner et al., 2019).

Original: Recent work also studied learning forward models for cascades (Qi et al., 2021; Girdhar et al., 2021).
Revised: Recent work also studied learning forward models for cascades (Qi et al., 2021; Girdhar et al., 2021).

Original: However, once the forward model has been learned, the desired initial condition of the system is found by an exhaustive search.
Revised: However, once the forward model has been learned, the desired initial condition of the system is found by an exhaustive search.

Original: Here, we show that exhaustive search fails for complex cascades and with semantic constraints (Section 5).
Revised: Here, we show that exhaustive search fails for complex cascades and with semantic constraints (Section 5).

Original: Therefore, our paper focuses on learning to search not on learning the forward model .
Revised: Therefore, our paper focuses on learning to search not on learning the forward model .

Original: We assume that we are given a special kind of a “forward” model operating at the level of semantic events.
Revised: We assume that we are given a special kind of a “forward” model operating at the level of semantic events.

Original: Namely, given a state of the cascading system, our forward model allows to query for the next event (”which objects collide next?”), and predict the outcome of that event (velocities of objects after collision).
Revised: Namely, given a state of the cascading system, our forward model allows to query for the next event (”which objects collide next?”), and predict the outcome of that event (velocities of objects after collision).

Original: Test bed.
Revised: Test bed.

Original: Weabstract away from specific applications and designed a well-controlled environment that shares key ingredients with real-world cascading processes.
Revised: We designed a well-controlled environment that shares key ingredients with real-world cascading processes.

Original: In our test-bed several spheres move freely on a table. The spheres collide with each other and with static pins within a confinedfour-walled space (Figure 1).
Revised: In our test-bed several spheres move freely on a table, colliding with each other and with static pins within a confined space (Figure 1).

Original: The chain of collisions forms a complex cascading process.
Revised: The chain of collisions forms a complex cascading process.

Original: The appendix describes more use cases of our general cascade framework.
Revised: Naturally, a simulated test-bed cannot cover the full complexity of real-world scenarios and additional research may be required. We discuss these topics in Section 7 .

Original: Contributions.
Revised: Contributions.

Original: This paper proposes a novel approach for learning to efficiently search for a complex cascade in a dynamical system.
Revised: This paper proposes a novel approach for learning to efficiently search for a complex cascade in a dynamical system.

Original: Our contributions are: (1) A new learning setup, Cascade , where an agent observes a dynamical system and then changes its initial conditions to meet a given semantic goal.
Revised: Our contributions are: (1) A new learning setup, Cascade , where an agent observes a dynamical system and then changes its initial conditions to meet a given semantic goal.

Original: (2) Learning a principled probabilistic scoring function over an – Event Tree – data structure, for searching efficiently over the space of interventions.
Revised: (2) Learning a principled probabilistic scoring function over an – Event Tree – data structure, for searching efficiently over the space of interventions.

Original: A Bayesian formulation leveraging the observed cascade to guide the search in the event tree toward a counterfactual outcome.
Revised: A Bayesian formulation leveraging the observed cascade to guide the search in the event tree toward a counterfactual outcome.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Learning and reasoning in physical systems.
Revised: Learning and reasoning in physical systems.

Original: Several papers studied cascading events in the context of physical systems.
Revised: Several papers studied cascading events in the context of physical systems.

Original: There, the main focus was to use object interactions to learn a forward model from observations.
Revised: There, the main focus was to use object interactions to learn a forward model from observations.

Original: PHYRE, Virtual Tools, and CREATE (Bakhtin et al., 2019; Allen et al., 2020; Jain et al., 2020) are benchmarks for physical reasoning for computer vision.
Revised: PHYRE, Virtual Tools, and CREATE (Bakhtin et al., 2019; Allen et al., 2020; Jain et al., 2020) are benchmarks for physical reasoning for computer vision.

Original: They differ from our learning setup in three key aspects.
Revised: They differ from our learning setup in three key aspects.

Original: First, the current paper focuses on the search problem, looking to satisfy a set of semantic constraints on the event sequence.
Revised: First, the current paper focuses on the search problem, looking to satisfy a set of semantic constraints on the event sequence.

Original: Second, in the prior benchmarks, all tasks have to satisfy the same final goal, rather than being conditioned on a semantic goal.
Revised: Second, in the prior benchmarks, all tasks have to satisfy the same final goal, rather than being conditioned on a semantic goal.

Original: Last, their setup is a sequential decision reinforcement learning setup, allowing exploration, collecting rewards from the environment, and multiple retries, which are not allowed in our setup.
Revised: Last, their setup is a sequential decision reinforcement learning setup, allowing exploration, collecting rewards from the environment, and multiple retries, which are not allowed in our setup.

Revised: In addition, no event-driven forward model (EDFM) is currently available for these benchmarks, and training an EDFM requires additional annotations and is beyond the scope of this work.

Revised: There are several approaches to learn such models from temporal data, like dynamic Bayes nets (Bhattacharjya et al., 2020; Ghahramani, 1998; Gunawardana & Meek, 2016), which can also handle latent variables.

Original: Allen et al.
Revised: Allen et al.

Original: (2020) takes a Bayesian approach for updating the distribution of initial conditions given a reward.
Revised: (2020) takes a Bayesian approach for updating the distribution of initial conditions given a reward.

Original: We consider the underlying chain of events and update the value of the node scores in the event tree according to the observed cascade.
Revised: We consider the underlying chain of events and update the value of the node scores in the event tree according to the observed cascade.

Original: CLEVRER, CoPhy, CRAFT, CATER, and IntPhys (Yi et al., 2020; Baradel et al., 2020; Ates et al., 2021; Girdhar & Ramanan, 2020; Riochet et al., 2018) are benchmarks for reasoning over observed temporal and causal structures in video.
Revised: CLEVRER, CoPhy, CRAFT, CATER, and IntPhys (Yi et al., 2020; Baradel et al., 2020; Ates et al., 2021; Girdhar & Ramanan, 2020; Riochet et al., 2018) are benchmarks for reasoning over observed temporal and causal structures in video.

Original: They differ from our setup in a few key aspects: (1) They focus on video-tracking and question answering rather than taking an action.
Revised: They differ from our setup in a few key aspects: (1) They focus on video-tracking and question answering rather than acting .

Original: (2) In CLEVRER and CoPhy, the observed cascade is available during training time, which may not be a reasonable assumption for real-world problems (see Section 3).
Revised: (2) In CLEVRER and CoPhy, the observed cascade is available during training, which may not be a reasonable assumption for real-world problems (see Section 3).

Original: (3) CoPhy estimates the value of a staticproperty in the observed cascade, like gravity, while we focus on the cascade evolution.
Revised: (3) CoPhy estimates the value of a static observed property, like gravity, while we focus on the cascade evolution.

Original: Roussel et al.
Revised: Roussel et al.

Original: (2019) studied the chain reaction problem, with a different focus than ours.
Revised: (2019) studied the chain reaction problem, with a different focus than ours.

Original: Their cascading configuration is fully given and they study how to tune that configuration using a simulator.
Revised: Their cascading configuration is fully given and they study how to tune that configuration using a simulator.

Original: Our work focuses on finding a cascading configuration given a partial description of it.
Revised: Our work focuses on finding a cascading configuration given a partial description of it.

Original: Graph Neural Networks have been used in physical environments (Kim & Shimanuki, 2019;
Revised: Graph Neural Networks have been used in physical environments (Kim & Shimanuki, 2019;

Original: Shen et al., 2020; Bapst et al., 2019), representing the underlying state as a graph, without considering temporal consequences.
Revised: Shen et al., 2020; Bapst et al., 2019), representing the underlying state as a graph, without considering temporal consequences.

Original: Temporal consequences are vital for our decision system.
Revised: Temporal consequences are vital for our decision system.

Original: We propose how to transform a temporal sequence of events to a DAG.
Revised: We propose how to transform a temporal sequence of events to a DAG.

Original: Reinforcement learning: The “Cascade” learning setup is fundamentally different from a reinforcement learning framework (Sutton & Barto, 2005).
Revised: Reinforcement learning: The “Cascade” learning setup is fundamentally different from a reinforcement learning framework (Sutton & Barto, 2005).

Original: The problem we try to solve is not a standard planning problem, where a series of actions are taken sequentially.
Revised: The problem we try to solve is not a standard planning problem (Hafner et al., 2019), where a series of actions are taken sequentially.

Original: Here, an action is taken once and sets the cascade of events (“Fire and forget”).
Revised: Here, an action is taken once and sets the cascade of events (“Fire and forget”).

Original: Planning in robotics:
Revised: Planning in robotics:

Original: Pertsch et al.
Revised: Pertsch et al.

Original: ; Jayaraman et al.
Revised: ; Jayaraman et al.

Original: (2019) learned from video data to predict key-frames, conditioned on a start frame and an end frame (goal).
Revised: (2019) learned from video data to predict key-frames, conditioned on a start frame and an end frame (goal).

Original: These works rely on a visual end goal.
Revised: These works rely on a visual end goal.

Original: It is unclear how to use them with a semantic goal that includes constraints.
Revised: It is unclear how to use them with a semantic goal that includes constraints.

Original: They also rely on taking multiple actions, which is not applicable in our ”Fire and forget” setup.
Revised: They also rely on taking multiple actions, which is not applicable in our ”Fire and forget” setup.

Original: Causal inference: Counterfactual reasoning wasextensively studied in causal inference (Pearl, 2000).
Revised: Causal inference: Counterfactual reasoning was studied in causal inference (Pearl, 2000).

Original: Most relevant is (Buesing et al., 2019) that generated counterfactually augmented data for training a RL policy.
Revised: Most relevant is (Buesing et al., 2019) that used counterfactually augmented data for training a RL policy.

Original:  T HE “ Cascade ” LEARNING SETUP
Revised:  T HE “ Cascade ” LEARNING SETUP

Original: Cascade is a supervised learning problem.
Revised: Cascade is a supervised learning problem.

Original: At the inference phase, The agent is provided with a dynamical system and two inputs: (1) A sequence of events called the “observed cascade” together with the respective initial condition of the system.
Revised: At the inference phase, The agent is provided with a dynamical system and two inputs: (1) A sequence of events called the “observed cascade” together with the respective initial condition of the system.

Original: An instruction that describes desired semantic properties (”constraints”) of the solution.
Revised: An instruction that describes desired semantic properties (”constraints”) of the solution.

Original: The observed cascade does not satisfy the instruction.
Revised: The observed cascasde does not satisfy the instruction.

Original: The agent is asked to intervene by controlling the state of one “pivot element” in the system at a specific point in time.
Revised: The agent is asked to intervene by controlling the state of one “pivot element” in the system at a specific point in time.

Original: The goal is to find an intervention that makes the rollout of the dynamical system satisfy the instruction.
Revised: The goal is to find an intervention that makes the roll out of the dynamical system satisfy the instruction.

Original: At the training phase, we are only given “successful” labeled examples.
Revised: At the training phase, we are only given “successful” labeled examples.

Original: The “features” (x) of each label consist of (1) an instruction; and (2) the initial state of the system except the controllable pivot.
Revised: The “features” (x) of each label consists of (1) an instruction; and (2) the initial state of the system except the controllable pivot.

Original: The ”label” (y) of each sample is the initial state of the pivot, which yields the desired behavior of the system.
Revised: The ”label” (y) of each sample is the initial state of the pivot, which yields the desired behavior of the system.

Original: During training, we do not provide examples of failing sequences together with a successful sequence.
Revised: During training we do not provide examples of failing sequences together with a successful sequence.

Original: The reason is that in reality, one cannot ”roll back” time and obtain both a failed sequence and a successful sequence.
Revised: The reason is that in reality, one cannot ”roll-back” time and obtain both a failed sequence and a successful sequence.

Original: More formally, our training set
Revised: More formally, our training set

Original: D consists of
Revised: D consists of

Original: N labelled samples
Revised: N labelled samples

Original: D = { features = ( x n , g n ) , label =
Revised: D = { features = ( x n , g n ) , label =

Original: ( y ∗ n , Q ( y ∗ n ) , n = 1 . . .
Revised: ( y ∗ n , Q ( y ∗ n ) , n = 1 . . .

Original: N } , where:
Revised: N } , where:

Original: • x n is the initial state of a dynamical system, excluding those of its pivot element.
Revised: • x n is the initial state of a dynamical system, excluding those of its pivot element.

Original: • g n ∈ R G is a structured representation of an instruction.
Revised: • g n ∈ R G is a structured representation of an instruction.

Original: • y ∗ n ∈ Y ⊂ R
Revised: • y ∗ n ∈ Y ⊂ R

Original: d is the pivot’s initial state of the solution.
Revised: d is the pivot’s initial state of the solution.

Original: • Q ( y ∗ n ) =
Revised: • Q ( y ∗ n ) =

Original: { s k } Kk =1 is a sequence of events that occur when the system is played out with the pivot initial value y n .
Revised: { s k } Kk =1 is a sequence of events that occur when the system is played out with the pivot initial value y n .

Original: At test time , a novel sample is drawn, describing an unseen dynamical system and instruction ( x, g ) ; and an observed cascade roll-out ( y obs , Q ( y obs )) which fails to fulfill the instruction.
Revised: At test time , a novel sample is drawn, describing an unseen dynamical system and instruction ( x, g ) ; and an observed cascade roll-out ( y obs , Q ( y obs )) which fails to fulfill the instruction.

Original: Our goal is to provide an alternative (counterfactual) initial state for the pivot element ˆ y , such that the instruction is fulfilled when the system is rolled-out.
Revised: Our goal is to provide an alternative (counterfactual) initial state for the pivot element ˆ y , such that the instruction is fulfilled when the system is rolled-out.

Original: Our test bed: We introduce a new simulated test bed that abstracts away from specific applications.
Revised: Our test bed: We introduce a new simulated test bed that abstract away from specific applications.

Original: An agent observes a physical world with several moving and static objects going through a cascade of events (Figure 1 top), and it is given a complex instruction “ Push: purple ball ... ”.
Revised: An agent observes a physical world with several moving and static objects going through a cascade of events (Figure 1 top), and it is given a complex instruction “ Push: purple ball ... ”.

Original: It then manipulates the direction and speed of the purple ball (Figure 1 bottom) manifesting a new cascading process that satisfies the complex set of constraints given by the instruction.
Revised: It then manipulates the direction and speed of the purple ball (Figure 1 bottom) manifesting a new cascading process that satisfy the complex set of constraints given by the instruction.

Original: In the Cascade setup, the agent is trained on a set of scenes and their goals and is tested on new scenes and their goals.
Revised: In the Cascade setup, the agent is trained on a set of scenes and their goals, and is tested on new scenes and their goals.

Original:  M ETHODS
Revised:  M ETHODS

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Solving the Cascade setup poses three major challenges.
Revised: Solving the Cascade setup poses three major challenges.

Original: First, our model needs to match semantic events, but simulations of dynamical systems typically follow fixed and small timesteps, which are indifferent to events.
Revised: First, our model needs to match semantic events, but simulations of dynamical systems typically follow fixed and small timesteps, which are indifferent to events.

Original: Second, the set of desired constraints and dynamical systems is compositional and large.
Revised: Second, the set of desired constraints and dynamical systems is compositional and large.

Original: The agent should learn to generalize to different systems and configurations that were not observed during training.
Revised: The agent should learn to generalize to different systems and configurations that were not observed during training.

Original: Third, we wish to benefit from examples of failed cascades that are available at inference time (the counterfactual setup).
Revised: Third, we wish to benefit from examples of failed cascades that are available at inference time (the counterfactual setup).

Original: We develop an approach that addresses these three challenges.
Revised: We develop an approach that addresses these three challenges.

Original: To address the first, we develop a representation that focuses on key “semantic” events of the dynamics (e.g., collisions).
Revised: To address the first, we develop a representation that focuses on key “semantic” events of the dynamics (e.g., collisions).

Original: We build a tree of possible outcomes such that a path in the tree captures a realizable cascade of events.
Revised: We build a tree of possible outcomes such that a path in the tree captures a realizable cascade of events.

Original: To address the second challenge, we learn a scoring function that assigns values to tree nodes conditioned on the instruction.
Revised: To address the second challenge, we learn a scoring function that assigns values to tree nodes conditioned on the instruction.

Original: This allows us to generalize to unseen setups, and at inference time, we use the predicted node scores to search efficiently over the space of interventions.
Revised: This allows us to generalize to unseen setups, and at inference time, we use the predicted node scores to search efficiently over the space of interventions.

Original: To address the third challenge, We develop a
Revised: To address the third challenge, We develop a

Original: Bayesian formulation that allows to integrate the “counterfactual” information with the score predictions.
Revised: Bayesian formulation that allows to integrate the “counterfactual” information with the score predictions.

Original: Next, we describe each component in more detail.
Revised: Next, we describe each component in more detail.

Original: 4. T HE Event Tree : A TREE OF POSSIBLE FUTURES
Revised: 4. T HE Event Tree : A TREE OF POSSIBLE FUTURES

Original: We now describe the main data structure that we use to represent the search problem - the Event
Revised: We now describe the main data structure that we use to represent the search problem - the Event

Original: Tree .
Revised: Tree .

Original: The event tree is designed to provide a searchable data structure for realizable sequences of events.
Revised: The event tree is designed to provide a searchable data structure for realizable sequences of events.

Original: To make these searches efficient, we represent the system behavior at the semantic level.
Revised: To make these searches efficient, we represent the system behavior at the semantic level.

Original: These could be any key interactions between system components, like manufacturing an item in supply chains, or a protein binding the DNA to regulate the expression of a gene.
Revised: These could be any key interactions between system components, like manufacturing an item in supply chains, or a protein binding the DNA to regulate the expression of a gene.

Original: Importantly, in our approach, we require that it is possible to compute the state of the system right after an event.
Revised: Importantly, in our approach, we require that it is possible to compute the state of the system right after an event.

Original: Each node corresponds to a sequence of semantic events.
Revised: Each node corresponds to a sequence of semantic events.

Original: The node’s children correspond to realizable continuations of the event sequence.
Revised: The node’s children correspond to realizable continuations of the event sequence.

Original: Namely, all possible events that could happen after the sequence S u .
Revised: Namely, all possible events that could happen after the sequence S u .

Original: We now formally describe the node properties and the expansion of the event tree.
Revised: We now formally describe the node properties and the expansion of the event tree.

Original: Tree Nodes.
Revised: Tree Nodes.

Original: A node u of the event tree corresponds to the subset of interventions Y u ⊂ Y that share the same prefix of semantic events.
Revised: A node u of the event tree corresponds to the subset of interventions Y u ⊂ Y that share the same prefix of semantic events.

Original: Each node has a unique prefix S u ≜ ( s 1 , s 2 , . . .
Revised: Each node has a unique prefix S u ≜ ( s 1 , s 2 , . . .

Original: s u ) .
Revised: s u ) .

Original: Specifically, after the shared prefix, different sequences of events may follow for different y ∈ Y u .
Revised: Specifically, after the shared prefix, different sequences of events may follow for different y ∈ Y u .

Original: The root node describes the set of possible interventions at t = 0 and its sequence of events S root is empty.
Revised: The root node describes the set of possible interventions at t = 0 and its sequence of events S root is empty.

Original: Its intervention subset is Y root =
Revised: Its intervention subset is Y root =

Original: Y .
Revised: Y .

Original: See Figure 2, top.
Revised: See Figure 2, top.

Original: We define w uy as the state of the system after it evolved from y ∈ Y and yielded the sequence S u .
Revised: We define w uy as the state of the system after it evolved from y ∈ Y and yielded the sequence S u .

Original: Then, a “world-state” of a node is defined as the set W u = (cid:8) w uy |
Revised: Then, a “world-state” of a node is defined as the set W u = (cid:8) w uy |

Original: y ∈
Revised: y ∈

Original: Y u (cid:9) .
Revised: Y u (cid:9) .

Original: Given the world state of a current node, we propose an event-driven forward model f ( · ) .
Revised: Given the world state of a current node, we propose an event-driven forward model f ( · ) .

Original: It takes as input a state w uy and outputs the next immediate semantic event.
Revised: It takes as input a state w uy and outputs the next immediate semantic event.

Original: We parallelized it on a GPU to detect possible futures for the world state W u .
Revised: We parallelized it on a GPU to detect possible futures for the world state W u .

Original: Appendix H describes the forward model in detail.
Revised: Appendix I describes the forward model in detail.

Original: Node Expansion.
Revised: Node Expansion.

Original: Suppose we decide to expand node u .
Revised: Suppose we decide to expand node u .

Original: We apply the forward model f ( · ) to each w uy ∈ W u .
Revised: We apply the forward model f ( · ) to each w uy ∈ W u .

Original: A new node u ′ is composed of all f ( w uy ) that share a same next semantic event s ′ .
Revised: A new node u ′ is composed of all f ( w uy ) that share a same next semantic event s ′ .

Original: Then, the event sequence of the child node u ′ is S u ′ =
Revised: Then, the event sequence of the child node u ′ is S u ′ =

Original: concat ( S u , s ′ )
Revised: concat ( S u , s ′ )

Original: ; the intervention set is
Revised: ; the intervention set is

Original: Y u ′ =
Revised: Y u ′ =

Original: { y | Q ( y ) has prefix S u ′ } .
Revised: { y | Q ( y ) has prefix S u ′ } .

Original: Expanding the tree can be viewed as a tessellation refinement of the intervention space Y .
Revised: Expanding the tree can be viewed as a tessellation refinement of the intervention space Y .

Original: At each step, we pick one cell and split it into multiple cells, where each child cell represents a different event that occurs after a shared sequence of events, represented by the parent cell.
Revised: At each step, we pick one cell and split it into multiple cells, where each child cell represents a different event that occurs after a shared sequence of events, represented by the parent cell.

Original: If the tree is fully expanded, it covers all possible futures.
Revised: If the tree is fully expanded, it covers all possible futures.

Original: However, expanding the whole tree is expansive, as it grows exponentially with its depth.
Revised: However, expanding the whole tree is expansive, as it grows exponentially with its depth.

Original: In the next subsection, we discuss how one can learn a scoring function and use it to guide an efficient tree search.
Revised: In the next subsection, we discuss how one can learn a scoring function and use it to guide an efficient tree search.

Original: 4. A SSIGNING AND LEARNING A SCORING FUNCTION FOR NODES
Revised: 4. A SSIGNING AND LEARNING A SCORING FUNCTION FOR NODES

Original: To search the tree for a node that satisfies the goal, we prioritize which node to expand by learning a scoring function that assigns scores for nodes, conditioned on the instruction g .
Revised: To search the tree for a node that satisfies the goal, we prioritize which node to expand by learning a scoring function that assigns scores for nodes, conditioned on the instruction g .

Original: There are three key challenges in learning a score function.
Revised: There are three key challenges in learning a score function.

Original: First, we do not have ground-truth (target) scores for tree nodes, and it is unclear what would be an effective assignment of scores.
Revised: First, we do not have ground-truth (target) scores for tree nodes, and it is unclear what would be an effective assignment of scores.

Original: Second, the training data contains only positive examples of correctly designed plans.
Revised: Second, the training data contains only positive examples of correctly designed plans.

Original: Finally, we wish to leverage the information about the faulty observed cascade, which is only available during inference time.
Revised: Finally, we wish to leverage the information about the faulty observed cascade, which is only available during inference time.

Original: To motivate our approach, consider the following naive approach to set target scores.
Revised: To motivate our approach, consider the following naive approach to set target scores.

Original: For a given tree, let the “target” u ∗ be the node that represents the ground-truth sequence S u ∗ .
Revised: For a given tree, let the “target” u ∗ be the node that represents the ground-truth sequence S u ∗ .

Original: A natural choice for setting scores would be to set V ( u ∗ )
Revised: A natural choice for setting scores would be to set V ( u ∗ )

Original: = 1 , and set all other scores to zero.
Revised: = 1 , and set all other scores to zero.

Original: However, this provides little guidance for searching the tree, as no signal is provided until the search hits the target node.
Revised: However, this provides little guidance for searching the tree, as no signal is provided until the search hits the target node.

Original: Instead, a desired property of the learning algorithm would be to guide the search by assigning monotonically increasing scores along the path from the root to u ∗ .
Revised: Instead, a desired property of the learning algorithm would be to guide the search by assigning monotonically increasing scores along the path from the root to u ∗ .

Original: To address the three challenges we design a principled probabilistic approach for setting the score function.
Revised: To address the three challenges we design a principled probabilistic approach for setting the score function.

Original: We train our model to predict the likelihood that a sample from Y u , when rolled out, will satisfy the instruction g .
Revised: We train our model to predict the likelihood that a sample from Y u , when rolled out, will satisfy the instruction g .

Original: [Equation]
Revised: [Equation]

Original:  Here, nodes on the path from the root to u ∗ are assigned monotonically increasing scores, as the tessellation gets finer and concentrates on Y u ∗ .
Revised: u , g ) . [Equation] Here, nodes on the path from the root to u ∗ are assigned monotonically increasing scores, as the tessellation gets finer and concentrates on Y u ∗ .

Original: Additionally, this probabilistic perspective allows us to take a maximum-likelihood approach at inference time to prioritize nodes.
Revised: Additionally, this probabilistic perspective allows us to take a maximum-likelihood approach at inference time to prioritize nodes.

Original: We use the maximum likelihood estimate of V ( u ) to calculate the ground-truth scores for training.
Revised: We use the maximum likelihood estimate of V ( u ) to calculate the ground-truth scores for training.

Original: For that, we take a finite sample of ˆ Y root ⊂
Revised: For that, we take a finite sample of ˆ Y root ⊂

Original: Y , collecting say 10 6 points, and use it to expand the tree.
Revised: Y , collecting say 10 6 points, and use it to expand the tree.

Original: The node’s score is then the fraction of the samples from ˆ Y u that reach the target node u ∗ .
Revised: The node’s score is then the fraction of the samples from ˆ Y u that reach the target node u ∗ .

Original: [Equation]
Revised: [Equation]

Original: Nodes outside that sequence get a score of 0 .
Revised: Nodes outside that sequence get a score of 0 .

Original: In Section 5, we empirically explore alternative approaches for assigning ground-truth scores to nodes.
Revised: In Section 5, we empirically explore alternative approaches for assigning ground-truth scores to nodes.

Original: Counterfactual update for the score function.
Revised: Counterfactual update for the score function.

Original: During inference, we observe a cascade that does not satisfy the instruction, and are asked to retrospectively suggest a better solution.
Revised: During inference, we observe a cascade that does not satisfy the instruction, and are asked to retrospectively suggest a better solution.

Original: How can the observed cascade be used to find a solution?
Revised: How can the observed cascade be used to find a solution?

Original: The probabilistic score function we defined allows us to formalize this problem in a Bayesian setting.
Revised: The probabilistic score function we defined allows us to formalize this problem in a Bayesian setting.

Original: We treat the model predictions as a prior for the true score, and the information about the observed cascade as evidence .
Revised: We treat the model predictions as a prior for the true score, and the information about the observed cascade as evidence .

Original: We then ask how to update the score function given the observed evidence.
Revised: We then ask how to update the score function given the observed evidence.

Original: Formally, our goal is to solve Eq.
Revised: Formally, our goal is to solve Eq.

Original: (1) when it is conditioned by the evidence, V ( u | S u obs doesn’t satisfy g ) .
Revised: (1) when it is conditioned by the evidence, V ( u | S u obs doesn’t satisfy g ) .

Original: During training, our model learns to estimate the unconditioned score function V ( · ) .
Revised: During training, our model learns to estimate the unconditioned score function V ( · ) .

Original: In the appendix, we show that we can express the Bayesian update of the scores in terms of V ( u obs ) , V ( u ) ,
Revised: In the appendix, we show that we can express the Bayesian update of the scores in terms of V ( u obs ) , V ( u ) ,

Original: V ( u | S u obs doesn’t satisfy g )
Revised: V ( u | S u obs doesn’t satisfy g )

Original: =
Revised: =

Original: V ( u ) − V
Revised: V ( u ) − V

Original: ( u obs ) · fr ( y obs , y u ) .
Revised: ( u obs ) · fr ( y obs , y u ) .

Original: [Equation]

Original: 3) where fr ( u obs , u ) =
Revised: (3) where fr ( u obs , u ) =

Original: Pr( y ∈ Y
Revised: Pr( y ∈ Y

Original: u obs | y ∈ Y u ) is the probability that an intervention y ∈ Y u will result in sequence with prefix S u obs .
Revised: u obs | y ∈ Y u ) is the probability that an intervention y ∈ Y u will result in sequence with prefix S u obs .

Original: It is estimated in a fashion similar to Eq.
Revised: It is estimated in a fashion similar to Eq.

Original: A model for the score function.
Revised: A model for the score function.

Original: Next, we describe the representation and architecture for modelling the score function.
Revised: Next, we describe the representation and architecture for modelling the score function.

Original: The model takes as inputs the instruction g and sequence of events S u that define the node u , and predicts a scalar score with ground-truth labels according to Eq.
Revised: The model takes as inputs the instruction g and sequence of events S u that define the node u , and predicts a scalar score with ground-truth labels according to Eq.

Original: A naive approach is to represent S u as a sequence.
Revised: A naive approach is to represent S u as a sequence.

Original: However, such representation may not convey well the relations describing the cascade of events.
Revised: However, such representation may not convey well the relations describing the cascade of events.

Original: For illustration, in the following sequence of collision events [(A, B), (C, D), (A, E)] , the collision (A,E) is driven by (A,B), because A is common for both, while (C,D) is less relevant for describing the events that lead to (A,E).
Revised: For illustration, in the following sequence of collision events [(A, B), (C, D), (A, E)] , the collision (A,E) is driven by (A,B), because A is common for both, while (C,D) is less relevant for describing the events that lead to (A,E).

Original: Instead, we transform each sequence to a Directed Acyclic Graph (DAG) that captures relations in the cascade of events.
Revised: Instead, we transform each sequence to a Directed Acyclic Graph (DAG) that captures relations in the cascade of events.

Original: A node in this DAG is an event that involves some elements.
Revised: A node in this DAG is an event that involves some elements.

Original: Each edge represents an element shared by two subsequent events.
Revised: Each edge represents an element shared by two subsequent events.

Original: See Figure 3 for a concrete illustration.
Revised: See Figure 3 for a concrete illustration.

Original: Architecture We use a Graph Neural Network (GNN) to parameterize our score function.
Revised: Architecture We use a Graph Neural Network (GNN) to parameterize our score function.

Original: We represent the graph as a tuple ( A, X, E, z ) where A ∈ { 0 , 1 } n × n is the graph adjacency matrix,
Revised: We represent the graph as a tuple ( A, X, E, z ) where A ∈ { 0 , 1 } n × n is the graph adjacency matrix,

Original: Y ∈ R n × d is a node feature matrix, E ∈ R m ×
Revised: Y ∈ R n × d is a node feature matrix, E ∈ R m ×

Original: d
Revised: d

Original: ′ is an edge feature matrix, and z ∈ R d ′′ is a global graph feature.
Revised: ′ is an edge feature matrix, and z ∈ R d ′′ is a global graph feature.

Original: We chose to use a popular message passing GNN model (Battaglia et al., 2018).
Revised: We chose to use a popular message passing GNN model (Battaglia et al., 2018).

Original: We describe its architecture in detail in the appendix.
Revised: We describe its architecture in detail in the appendix.

Original: 4. I NFERENCE
Revised: I NFERENCE

Original: Our agent searches the tree for the maximum scored node u MAX .
Revised: Our agent searches the tree for the maximum scored node u MAX .

Original: Then, it randomly selects an intervention from its intervention subset y ∈ Y u MAX .
Revised: Then, it randomly selects an intervention from its intervention subset y ∈ Y u MAX .

Original: We consider two variants.
Revised: We consider two variants.

Original: Maximum likelihood search: The agent performs a tree search that expands the most likely nodes.
Revised: Maximum likelihood search: The agent performs a tree search that expands the most likely nodes.

Original: At any given step, the agent stores a sorted list of nodes together with their likelihood scores, it then picks the highest scoring node from this list and expands it.
Revised: At any given step, the agent stores a sorted list of nodes together with their likelihood scores, it then picks the highest scoring node from this list and expands it.

Original: The node children are then added to the list with their predicted scores, and the agent resorts the list.
Revised: The node children are then added to the list with their predicted scores, and the agent resorts the list.

Original: We limit the tree search to expand only 80 nodes, whereas in our test bed a full event tree, which contains all possible realizations, have billions of nodes, ∼ × 2 .
Revised: We limit the tree search to expand only 80 nodes, whereas in our test bed a full event tree, which contains all possible realizations, have billions of nodes, ∼ × 2 .

Original: 8 per unit of depth (empirically).
Revised: 8 per unit of depth (empirically).

Original: Counterfactual search: Here we explain how we leverage the information in the observed cascade for inference.
Revised: Counterfactual search: Here we explain how we leverage the information in the observed cascade for inference.

Original: Consider the case where the sequence of the solution is complex and the observed sequence diverges from the solution at a late point.
Revised: Consider the case where the sequence of the solution is complex and the observed sequence diverges from the solution at a late point.

Original: In this case, it is likely that a part of the observed chain will be informative about the solution, and will diverge at some point.
Revised: In this case, it is likely that a part of the observed chain will be informative about the solution, and will diverge at some point.

Original: To use that information, we apply the Bayesian correction (Eq. 3) term to the predicted score of every node along the observed sequence.
Revised: To use that information, we apply the Bayesian correction (Eq. 3) term to the predicted score of every node along the observed sequence.

Original: We pick the highest scoring node, and initialize the search up to that node.
Revised: We pick the highest scoring node, and initialize the search up to that node.

Original: Then we continue the search as described by the “Maximum likelihood search”.
Revised: Then we continue the search as described by the “Maximum likelihood search”.

Original: In practice, we trim the observed sequence, at N observed nodes.
Revised: In practice, we trim the observed sequence, at N observed nodes.

Original: N observed is a hyper-parameter.
Revised: N observed is a hyper-parameter.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compared our approach to state-of-the-art baselines, including human performance.
Revised: We compared our approach to state-of-the-art baselines, including human performance.

Original: Then, we follow with an ablation study to examine the contributions of different components in our approach.
Revised: Then, we follow with an ablation study to examine the contributions of different components in our approach.

Original: Next, we describe our experimental protocol, compared methods, and evaluation metrics.
Revised: Next, we describe our experimental protocol, compared methods, and evaluation metrics.

Original: 5. A SIMULATION BENCHMARK
Revised: 5. A SIMULATION BENCHMARK

Original: [Figure]
Revised: [Figure]

Original:  of events (top) to a DAG (bottom)
Revised: Figure 3: Illustrates transforming a sequence of events (top) to a DAG (bottom).

Original: It  to the video in Figure 1 bottom
Revised: It corresponds to the video in Figure 1 bottom.

Original: We designed a well-controlled environment that share key ingredients with real-world systems of cascading events.
Revised: We designed a well-controlled environment that share key ingredients with real-world systems of cascading events.

Original: 1 Making it (1) sensitive to initial conditions, to allow a diverse set of future cascades, (2) containing diverse scenes, each describing a unique dynamical system.
Revised: 1 Making it (1) sensitive to initial conditions, to allow a diverse set of future cascades, (2) containing diverse scenes, each describing a unique dynamical system.

Original: (3) including semantic goals that depend on intermediate outcomes; and (4) can benchmark counterfactual scenarios.
Revised: (3) including semantic goals that depend on intermediate outcomes; and (4) can benchmark counterfactual scenarios.

Original: Scenes.
Revised: Scenes.

Original: In our test-bed, several spheres move freely on a frictionless table, colliding with each other and with static pins within a confined four-walled space (Figure 1).
Revised: In our test-bed, several spheres move freely on a frictionless table, colliding with each other and with static pins within a confined four-walled space (Figure 1).

Original: Each episode describes a different scene, which includes tens of collisions.
Revised: Each episode describes a different scene, which includes tens of collisions.

Original: Instructions.
Revised: Instructions.

Original: A structured instruction describes (i)
Revised: A structured instruction describes (i)

Original: A pivot element to manipulate “ Push: green ball ”; (ii) A target semantic event (collision) to fulfill “ Target: red hits black pin ”; and (iii) constraints, of two possible types.
Revised: A pivot element to manipulate “ Push: green ball ”; (ii) A target semantic event (collision) to fulfill “ Target: red hits black pin ”; and (iii) constraints, of two possible types.

Original: First, is a “count” constraint.
Revised: First, is a “count” constraint.

Original: It resembles constraining the total amount of resources available on a logistic chain.
Revised: It resembles constraining the total amount of resources available on a logistic chain.

Original: It specifies an accumulated number of collisions, on all the paths from the pivot to the target collision, e.g. “ Chain Count: 3 ”.
Revised: It specifies an accumulated number of collisions, on all the paths from the pivot to the target, e.g. “ Chain Count: 3 ”.

Original: Second, is a “bottleneck” constraint, which resembles a bottleneck along a logistic chain.
Revised: Second, is a “bottleneck” constraint, which resembles a bottleneck along a logistic chain.

Original: It enforces a specific collision that lies on any path that starts from the pivot and reaches the target collision, e.g. “ Bottleneck: red hits top wall ”.
Revised: It enforces a specific collision that lies on any path that starts from the pivot and reaches the target collision, e.g. “ Bottleneck: red hits top wall ”.

Original: The appendix describes the instruction generation process with more examples.
Revised: The appendix describes the instruction generation process with more examples.

Original: The task.
Revised: The task.

Original: The objective of the agent is to intervene with the initial state of the scene, by setting the velocity vector of the pivot object to reach a set of collisions specified by the instruction.
Revised: The objective of the agent is to intervene with the initial state of the scene, by setting the velocity vector of the pivot object to reach a set of collisions specified by the instruction.

Original: This often requires ahighly precise “trick shot”, that takes into account detailed reasoning on how downstream events will roll out.
Revised: This often requires a precise “trick shot”, that requires detailed reasoning on how downstream events will roll out.

Original: Dataset: We generated a dataset with ∼ 46 K scenes, each includes 4-6 moving balls, 0-2 pins, andwalls and up to 5 semantic instructions ( ∼ 4 . 25 on average).
Revised: Dataset: We generated a dataset with ∼ 46 K scenes (we limited generation time to 80 hours), each includes 4-6 moving balls, 0-2 pins, and 4 walls and up to 5 semantic instructions ( ∼ 4 . 25 on average).

Original: The data is split by unique scenes , into 470 unseen scenes for test, 69 scenes for selecting hyper-parameters (val. set), and the rest are used for training.
Revised: The data is split by unique scenes , into 470 unseen scenes for test, 69 scenes for selecting hyper-parameters (val. set), and the rest are used for training.

Original: See the appendix for more details.
Revised: See Appendix H for more details.

Original: 5. E XPERIMENT DETAILS
Revised: 5. E XPERIMENT DETAILS

Original: Compared Methods: We compared the following methods.
Revised: Compared Methods: We compared the following methods.

Original: (1) ROSETTE (
Revised: (1) ROSETTE (

Original: Reasoning On SEmanTic TreEs) : Our approach described in Section 4.
Revised: Reasoning On SEmanTic TreEs) : Our approach described in Section 4.

Original: Search uses the “counterfactual” variant of the tree search (Section 4.3), by first expanding the nodes along the “observed” sequence.
Revised: Search uses the “counterfactual” variant of the tree search (Section 4.3), by first expanding the nodes along the “observed” sequence.

Original: ROSETTE-max-l. : Like #1, but using “Maximum likelihood search” (Section 4.3) - not using the “observed” sequence.
Revised: ROSETTE-max-l. : Like #1, but using “Maximum likelihood search” (Section 4.3) - not using the “observed” sequence.

Original: For a fair comparison, we make sure that ROSETTE expands the same number of nodes in total as ROSETTE-max-l. (3) (Qi et al., 2021) , The SOTA on PHYRE, using a learned forward model, goal-satisfaction classifier and exhaustive search.
Revised: For a fair comparison, we make sure that ROSETTE expands the same number of nodes in total as ROSETTE-max-l. (3) (Qi et al., 2021) , The SOTA on PHYRE, using a learned forward model, goal-satisfaction classifier and exhaustive search.

Original: For a fair comparison we replace their learned forward model by the full simulator of Makoviychuk et al.
Revised: For a fair comparison we replace their learned forward model by the full simulator of Makoviychuk et al.

Original: (4) Sequential :
Revised: (4) Cross Entropy :

Revised: A standard planner (de Boer et al., 2005; Greenberg et al., 2022) that optimizes the objective function learned by compared method (3).

Original:  Using a sequential representation for a tree chain, instead of a DAG.
Revised: Sequential : Using a sequential representation for a tree chain, instead of a DAG.

Original: Specifically, we represent the sequence as a graph with edges along the sequence.
Revised: Specifically, we represent the sequence as a graph with edges along the sequence.

Original: (Litany et al., 2022) compared a recurrent versus standard synchronous propagation in GNN models and found them empirically equivalent.
Revised: (Litany et al., 2022) compared a recurrent versus standard synchronous propagation in GNN models and found them empirically equivalent.

Original:  Deep Sets regression :
Revised: (6) Deep Sets regression :

Original: Embedding the instruction and the initial world state to predict a continuous intervention.
Revised: Embedding the instruction and the initial world state to predict a continuous intervention.

Original: We embed the objects’ initial positions and velocities using the permutation-invariant “Deep Sets” architecture (Zaheer et al., 2017), and use an L 2 loss with respect to ground-truth interventions in the “counterfactual” training samples.
Revised: We embed the objects’ initial positions and velocities using the permutation-invariant “Deep Sets” architecture (Zaheer et al., 2017), and use an L 2 loss with respect to ground-truth interventions in the “counterfactual” training samples.

Original: (6) Random : Sample interventions at random from an estimated distribution of ground-truth interventions.
Revised: (7) Random : Sample interventions at random from an estimated distribution of ground-truth interventions.

Original: More details appear in the appendix.
Revised: More details appear in the appendix.

Original: Ablation: We also carry a thorough ablation study:
Revised: Ablation: We also carry a thorough ablation study:

Original: First, we explore alternative approaches to label node scores along the ground-truth sequence.
Revised: First, we explore alternative approaches to label node scores along the ground-truth sequence.

Original: Linear :
Revised: Linear :

Original: Linearly increases the score by V ( u ) = depth ( u ) /depth
Revised: Linearly increases the score by V ( u ) = depth ( u ) /depth

Original: ( u ∗ ) .
Revised: ( u ∗ ) .

Original: Step : Give a fixed medium score to nodes along the sequence, and a maximal score to the target node: V ( u ) = 0 .
Revised: Step : Give a fixed medium score to nodes along the sequence, and a maximal score to the target node: V ( u ) = 0 .

Original: 5+0 .
Revised: 5+0 .

Original: 5 1 u ∗ ( u ) .
Revised: 5 1 u ∗ ( u ) .

Original: Dirac-Delta : Sets V ( u ) = 1 u ∗ ( u ) , this baseline is equivalent to naive approach discussed in Section 4.2.
Revised: Dirac-Delta : Sets V ( u ) = 1 u ∗ ( u ) , this baseline is equivalent to naive approach discussed in Section 4.2.

Original: Second, we compare the “Counterfactual” to the “Maximum Likelihood” search by emphasizing the former effect with a dataset that employs more complex instructions using a third constraint, and compare the two types of search across “Easy” and “Hard” instructions.
Revised: Second, we compare the “Counterfactual” to the “Maximum Likelihood” search by emphasizing the former effect with a dataset that employs more complex instructions using a third constraint, and compare the two types of search across “Easy” and “Hard” instructions.

Original: We describe this dataset in the appendix.
Revised: We describe this dataset in the appendix.

Original: Last, we test how ablating parts of the instruction affects the ROSETTE model performance.
Revised: Last, we test how ablating parts of the instruction affects the ROSETTE model performance.

Original: Implementation details of the baselines and ablations are described in Section C.
Revised: Implementation details of the baselines and ablations are described in Section D.

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: Evaluation metrics: For each episode and goal, we predict an intervention and evaluate their success rate using the following metrics.
Revised: Evaluation metrics: For each episode and goal, we predict an intervention and evaluate their success rate using the following metrics.

Original: Simulator success rate : The success rate when rolling out the predicted intervention using a physical simulator (Makoviychuk et al., 2021).
Revised: Simulator success rate : The success rate when rolling out the predicted intervention using a physical simulator (Makoviychuk et al., 2021).

Original: This metric mimics experimenting in the real world.
Revised: This metric mimics experimenting in the real world.

Original: Tree success rate (where applicable): Each node in the tree represents a sequence of events.
Revised: Tree success rate (where applicable): Each node in the tree represents a sequence of events.

Original: A tree based algorithm selects a node.
Revised: A tree based algorithm selects a node.

Original: A “tree success” is when the selected node’s sequence satisfy the instruction.
Revised: A “tree success” is when the selected node’s sequence satisfy the instruction.

Original: This metric evaluates the performance of the score function and tree search, independently from errors that may be introduced due to the event-driven forward model.
Revised: This metric evaluates the performance of the score function and tree search, independently from errors that may be introduced due to the event-driven forward model.

Original: We further measured refinements of these metrics by conditioning on various properties of the instruction and scene.
Revised: We further measured refinements of these metrics by conditioning on various properties of the instruction and scene.

Original: (1) Condition tree success rate on instruction type: Unconstrained: The instruction only specifies target collisions.
Revised: (1) Condition tree success rate on instruction type: Unconstrained: The instruction only specifies target collisions.

Original: Bottleneck: also contains an “bottleneck” constraint.
Revised: Bottleneck: also contains an “bottleneck” constraint.

Original: Count: contains a “count” constraint.
Revised: Count: contains a “count” constraint.

Original: B&C: contains both “bottleneck” and “count” constraints.
Revised: B&C: contains both “bottleneck” and “count” constraints.

Original: (2) Condition tree success rate on complex scenarios: (2.1) Instructions with 2 or more constraints are marked as “Hard”, and the rest as “Easy”; (2.2) Instructions with a “count” constraint value ≥ 5 are considered hard.
Revised: (2) Condition tree success rate on complex scenarios: (2.1) Instructions with 2 or more constraints are marked as “Hard”, and the rest as “Easy”; (2.2) Instructions with a “count” constraint value ≥ 5 are considered hard.

Original: Complex scenario conditioning was evaluated on the complex in- struction dataset.
Revised: Complex scenario conditioning was evaluated on the complex instruction dataset.

Original: Conditioning on the main dataset demonstrate a similar trend (See appendix J)
Revised: Using the main dataset demonstrate a similar trend (See appendix K).

Original: We report mean value and standard error of the mean across 5different model seeds.
Revised: We report mean value and standard error of the mean across 5 model seeds.

Original: 5. H UMAN EVALUATION
Revised: 5. H UMAN EVALUATION

Original: [Figure]

Original: To assess a human baseline, we conducted a user study with Amazon Mechanical Turk.
Revised: To assess a human baseline, we conducted a user study with Amazon Mechanical Turk.

Original: We designed a game, where a player is given a video of the observed cascade and is asked to select one of 44 combinations of orientations (11) and speeds (4).
Revised: We designed a game, where a player is given a video of the observed cascade and is asked to select one ofcombinations of orientations (11) and speeds (4).

Original: The game is based on 30 test episodes.
Revised: The game is based on 30 test episodes.

Original: For comparing with ROSETTE, we select the one (of 44) which is nearest (in L 2 ) to ROSETTE’s predicted velocity.
Revised: For comparing with ROSETTE, we select the one (of 44) which is nearest (in L 2 ) to ROSETTE’s predicted velocity.

Original: Appendix Section B, describes the experiment design and further analysis of the results
Revised: Appendix Section C, describes the experiment design and further analysis of the results.

Original:  R ESULTS
Revised:  R ESULTS

Original: We first compare the performance of ROSETTE with baseline methods and human performance.
Revised: We first compare the performance of ROSETTE with baseline methods and human performance.

Original: We then study it properties in greater depth, through a series of ablation experiments.
Revised: We then study it properties in greater depth, through a series of ablation experiments.

Original: Finally, we  provide qualitative examples at the appendix.
Revised: Finally, we discuss the results of the baselines in Appendix A.1, and we provide qualitative examples in Appendix B.

Original: Table 1 (Left) describes the Tree and the Simulator success rates of ROSETTE and compared methods.
Revised: Table 1 (Left) describes the Tree and the Simulator success rates of ROSETTE and compared methods.

Original: ROSETTE achieves the highest success rate for both the “Tree” success rate ( 60 . 8% ) and the “Simulated” success rate ( 48 . 8% ).
Revised: ROSETTE achieves the highest success rate for both the “Tree” success rate ( 60 . 8% ) and the “Simulated” success rate ( 48 . 8% ).

Original: Achieving ∼ 80% conversion rate from Tree to Simulated .
Revised: Achieving ∼ 80% conversion rate from Tree to Simulated .

Original: The random baseline success rate is ( 17 . 6% ), which is close to the performance of the regression model.
Revised: The random baseline success rate is ( 17 . 6% ), which is close to the performance of the regression model.

Original: We conjecture that the regression model fails, because it can’t “imagine” the outcomes of the input states as ROSETTE can.
Revised: We conjecture that the regression model fails, because it can’t “imagine” the outcomes of the input states as ROSETTE can.

Original: The Sequential approach is the strongest baseline, reaching Tree = 52 .
Revised: The Sequential approach is the strongest baseline, reaching Tree = 52 .

Original: 4% and Simulated = 43 .
Revised: 4% and Simulated = 43 .

Original: 1% success rates.
Revised: 1% success rates.

Original: Next, we describe the success rate in the human study.
Revised: Next, we describe the success rate in the human study.

Original: ROSETTE achieves the highest average success rate ( 43 .
Revised: ROSETTE achieves the highest average success rate ( 43 .

Original: 3% ± 1 .
Revised: 3% ± 1 .

Original: 3% vs 23 .
Revised: 3% vs 23 .

Original: 9% ± 2 . 6% ).
Revised: 9% ± 2 . 6% ).

Original: Humans displayed a large range of success rates, with the best human achieving 41 .
Revised: Humans displayed a large range of success rates, with the best human achieving 41 .

Original: 4% , while the median and worst humans were 25% and 10% .
Revised: 4% , while the median and worst humans were 25% and 10% .

Original: ROSETTE performed more persistent, with 46 .
Revised: ROSETTE performed more persistent, with 46 .

Original: 3% and 40% for the best, median, and worst.
Revised: 3% and 40% for the best, median, and worst.

Original:  (1) Table 1 (Right) shows the advantage of the probabilistic formulation of the score function (ROSETTE-max-l), compared to the several heuristics described in Section 5.
Revised: Ablation experiments: (1) Table 1 (Right) shows the advantage of the probabilistic formulation of the score function (ROSETTE-max-l), compared to the several heuristics described in Section 5.

Original: The strongest baseline (“Linear”) only reaches 48 .
Revised: The strongest baseline (“Linear”) only reaches 48 .

Original: 7% vs. 59 .
Revised: 7% vs. 59 .

Original: 7% for ROSETTE-max-l. (2) Figure 4 quantifies the benefit gained by using “Counterfactual” search (ROSETTE) over Maximum-Likelihood search (Section 4.3).
Revised: 7% for ROSETTE-max-l. (2) Figure 4 quantifies the benefit gained by using “Counterfactual” search (ROSETTE) over MaximumLikelihood search (Section 4.3).

Original: ROSETTE shows a relative improvement of 7 .
Revised: ROSETTE shows a relative improvement of 7 .

Original: 1% vs 41 .
Revised: 1% vs 41 .

Original: 9% ) for complex instructions.
Revised: 9% ) for complex instructions.

Original: (3) Table 2  allows an in-depth examination of the strengths and weaknesses of ROSETTE,dissecting the success rate across 4 types of instructions, as described in Section 5.
Revised: (3) Table 2 (Appendix A) allows an in-depth examination of the strengths and weaknesses of ROSETTE, across 4 types of ablations, as described in Section 5.

Original: First, we observe that the sequential baseline can find target collisions that depend on a bottleneck collision, as well as ROSETTE.
Revised: First, we observe that the sequential baseline can find target collisions that depend on a bottleneck collision, as well as ROSETTE.

Original: However, it fails with “count” instructions ( 46 .
Revised: However, it fails with “count” instructions ( 46 .

Original: 3% vs 60 . 8% ), since it has no capacity for that reasoning task.
Revised: 3% vs 60 . 8% ), since it has no capacity for that reasoning task.

Original: Second, we observe that ROSETTE effectively uses the instruction, since any ablated part of the instruction hurt the respective success rate.
Revised: Second, we observe that ROSETTE effectively uses the instruction, since any ablated part of the instruction hurt the respective success rate.

Original:  C ONCLUSION
Revised:  D ISCUSSION

Original: In this paper, we took a first step towards understanding how to affect a complex system of cascading events.
Revised: In this paper, we took a first step towards understanding how to affect a complex system of cascading events.

Original: We presented a new learning setup, called Cascade , where an agent observes a cascade of events in a dynamical system and is asked to intervene and change its initial state to make the system meet a given goal.
Revised: We presented a new learning setup, called Cascade , where an agent observes a cascade of events in a dynamical system and is asked to intervene and change its initial state to make the system meet a given goal.

Original: We use an event-tree representation and a principled probabilistic score function for searching efficiently over the space of interventions.
Revised: We use an event-tree representation and a principled probabilistic score function for searching efficiently over the space of interventions.

Original: We also describe an approach to counterfactually reason about an observed cascade during the tree search.
Revised: We also describe an approach to counterfactually reason about an observed cascade during the tree search.

Revised: Our approach is best applied in problems that are naturally described by event-driven dynamics.

Revised: As an example, consider cascading failures in power grids (Sch ¨ afer et al., 2018).

Revised: Here, semantic events are failures of nodes (transformers, power generators, ...) or edges (power lines).

Revised: The power flow obeys a known set of ODE for a given grid.

Original: The reverse question of how to avoid a cascade of events by modifying by modifying some dynamics may require different tools andhas a practical significance of its own.
Revised: When flow exceeds a powerline capacity, that line fails (an event), resulting in an effectively different grid and a different set of ODEs that govern the dynamics. The transmission system operator may wish to define goals like “no more than three failures”, “no more than n people affected”, “that important node must not fail”. We elaborate on this use-case and other use-cases, such as logistics and evolution of natural disasters in Appendix J. One important question remains, how do studies that use our toy testbed can generalize to real-world scenarios.

Revised: We believe it can follow a similar paths as in other areas of AI where approaches mature from toy datasets to realistic problems: First, by creating a benchmark dataset for a real-world domain, annotated with semantic events.

Revised: Some fields have datasets that can be very natural for the problem we discussed.

Revised: These may include logistics (Appendix J), evolution of natural disasters and their consequences (Zuccaro et al., 2018), and cascading failures in power grids Sch¨afer et al.

Revised: Second, an event-driven forward model (EDFM) needs to be trained using this dataset.

Revised: Domain specific properties can be used to improve the accuracy and robustness of an EDFM learned.

Original:  Reproducibility Statement We provide full experimental detail about our approach and baselines in Appendix C. Appendix D describes the model architecture, feature represntation, and the data for training the scoring model for the event tree.
Revised: Finally, given the EDFM, our approach can be applied. Reproducibility Statement We provide full experimental detail about our approach and baselines in Appendix D. Appendix E describes the model architecture, feature represntation, and the data for training the scoring model for the event tree.

Original: Appendix G describes the details about the data generation.
Revised: Appendix H describes the details about the data generation.

Original: Appendix H describes the derivation and implementation of the event driven forward model.
Revised: Appendix I describes the derivation and implementation of the event driven forward model.

Original: Finally, Appendix B describes the details about the user study
Revised: Finally, Appendix C describes the details about the user study

Original: A Q UALITATIVE EXAMPLES
Revised: A A DDITIONAL RESULTS

Revised: Here we describe additional results and provide further discussion.

Revised: [Table]

Revised: A. B ASELINE RESULTS DISCUSSION

Revised: (Qi et al., 2021) baseline: We believe that the (Qi et al., 2021) baseline fails for three main reasons: (1) The baseline does not use an event-based representation.

Revised: (2) It employs a classifier that is trained to provide an all-or-none signal, rather than guiding the search.

Revised: The ablation study (Table 1, right) demonstrates the importance of guiding the search (compare “Ours” 59.7% vs “Dirac-Delta” 33.5% ) (3) The baseline architecture cannot reason over the temporal DAG structure of a cascade, as our

Revised: GNN can.

Revised: The importance of capturing the DAG structure is demonstrated when comparing “Our” (60.8%) to the SEQUENTIAL baseline (52.4%)

Revised: (Table 1, left)

Revised: Existing Planners: We wish to provide further insight into why it is challenging to apply existing planners to this setup.

Revised: The main challenge is that the optimization objective is given in semantic terms about the end goal.

Revised: To apply a Cross-Entropy-Method (CEM) planner, we derive a corresponding objective function by training a classifier that checks if the goal was achieved for a given scene and plan.

Revised: Specifically, we used the existing SoTA PHYRE classifier Qi et al.

Revised: A main drawback is that classifiers provide an “all or none” signal, hence fail in guiding the planner through optimization.

Revised: Conversely, our approach provides a score (Eq. 1) that monotonically increases through the tree, and it is constructed to assist the search.

Revised: B Q UALITATIVE EXAMPLES

Original: Here we provide links to qualitative examples that we uploaded to YouTube.
Revised: Here we provide links to qualitative examples that we uploaded to YouTube.

Original: They are best viewed in × 0 .
Revised: They are best viewed in × 0 .

Original: 25 slow motion.
Revised: 25 slow motion.

Original: The YouTube account we use is anonymous.
Revised: The YouTube account we use is anonymous.

Original: For each episode, we show a side-to-side video of the observed cascade, the ROSETTE successful case, and ROSETTE-max-l failure case.
Revised: For each episode, we show a side-to-side video of the observed cascade, the ROSETTE successful case, and ROSETTE-max-l failure case.

Original: The instruction is displayed on top of each video.
Revised: The instruction is displayed on top of each video.

Original: • Push: cyan ball, Target: blue hits red, Bottleneck: cyan hits bottom wall, Chain Count:
Revised: • Push: cyan ball, Target: blue hits red, Bottleneck: cyan hits bottom wall, Chain Count:

Original: In this example, ROSETTE semantically follows the first 10 collisions (in chonological order) as in the observed cascade.
Revised: In this example, ROSETTE semantically follows the first 10 collisions (in chonological order ) as in the observed cascade.

Original: It then diverges from the observed cascade, making the blue hit the red.
Revised: It then diverges from the observed cascade, making the blue hit the red.

Original: The cyan pivot comes into play already on the 1st collision, and the agent adjusts its velocity such that it shall yield the goal.
Revised: The cyan pivot comes into play already on the 1st collision, and the agent adjusts its velocity such that it shall yield the goal.

Original: The ROSETTE-max-l baseline, hits the target, however it fails with the count constraint.
Revised: The ROSETTE-max-l baseline, hits the target, however it fails with the count constraint.

Original: The bottleneck collision occurs, but not on the chain fromthe pivot to the target.
Revised: The bottleneck collision occurs, but not on the chain from

Revised: the pivot to the target.

Original: See the complete video here: 
Revised: See the complete video here: https:

Original: https: //youtu.be/RCKFBRrCRw
Revised: //youtu. be

Original:  • Push: cyan ball, Target: green hits red, Bottleneck: purple hits red, Chain Count:
Revised: /RCKFBRrCRw • Push: cyan ball, Target: green hits red, Bottleneck: purple hits red, Chain Count:

Original: In this example, ROSETTE semantically follows the first 5 collisions (in chonological order) as in the observed cascade.
Revised: In this example, ROSETTE semantically follows the first 5 collisions (in chonological order) as in the observed cascade.

Original: The cyan pivot comes into play on the 3rd collision.
Revised: The cyan pivot comes into play on the 3rd collision.

Original: It then diverges from the observed cascade, and follows another chain of events, making the purple hit the red, and concluding with the target hit within 4 collisions in the chain that started at the cyan ball.
Revised: It then diverges from the observed cascade, and follows another chain of events, making the purple hit the red, and concluding with the target hit within 4 collisions in the chain that started at the cyan ball.

Original: This task is too hard for the ROSETTE-max-l baseline, as it completely fails to satisfy the instruction.
Revised: This task is too hard for the ROSETTE-max-l baseline, as it completely fails to satisfy the instruction.

Original: See the complete video here: https://youtu.be/ 4s9MmY2J I
Revised: See the complete video here: https://youtu.be/4s9MmY2J__I

Original: • Push: green ball, Target: green hits cyan, Bottleneck: green hits purple, Chain Count:
Revised: • Push: green ball, Target: green hits cyan, Bottleneck: green hits purple, Chain Count:

Original: In this example, ROSETTE semantically follows the first 6 collisions (in chonological order) as in the observed cascade.
Revised: In this example, ROSETTE semantically follows the first 6 collisions (in chonological order) as in the observed cascade.

Original: It then diverges from the observed cascade, making the green hit the purple, fulfiling the bottleneck 
Revised: It then diverges from the observed cascade, making the green hit the purple, fulfiling the bottleneck constraint

Original: constraint.
Revised: The

Original: The cyan pivot comes into play only on the 6th collision, and the agent adjusts its velocity such that both the target and the count constraint will by satisfied.
Revised: cyan pivot comes into play only on the 6th collision, and the agent adjusts its velocity such that both the target and the count constraint will by satisfied

Original: The ROSETTE-max-l baseline, hits the bottleneck, however it fails to hit the target.
Revised: The ROSETTE-max-l baseline, hits the bottleneck, however it fails to hit the target.

Revised: See the complete video here:

Revised: https: //youtu.be/iMedd_

Original: See the complete video here: https: //youtu.be/iMedd_7YndQ • Push: yellow ball, Target: cyan hits red, Bottleneck: purple hits red, Chain Count: - In this example, ROSETTE semantically follows the first 5 collisions (in chonological order) as in the observed cascade.
Revised: 7YndQ • Push: yellow ball, Target: cyan hits red, Bottleneck: purple hits red, Chain Count: - In this example ROSETTE semantically follows the first 5 collisions (

Revised: in

Revised: chonological or der

Revised: ) as in

Original:  The yellow pivot comes into play only on the 5th collision, and the agent adjusts its velocity to satisfy the bottleneck constraint andthe target.
Revised: the observed cascade. The yellow pivot comes into play only on the 5th collision, and the agent adjusts its velocity to satisfy the bottleneck constraint and

Original:  The ROSETTE-max-l baseline, completely fails in this task.
Revised: the target The ROSETTE max-l baseline, completely fails in this task.

Original: See the complete video here: https://youtu.be/QLMTD6R2Z • Push: red ball, Target: blue hits red, Bottleneck: red hits bottom wall, Chain Count: - In this example, ROSETTE semantically follows the first 4 collisions (in chonological order) as in the observed cascade.
Revised: See the complete video here: https://youtu.be/QLMTD6R2Z • Push: red ball, Target: blue hits red, Bottleneck: red hits bottom wall, Chain Count: - In this example, ROSETTE semantically follows the first 4 collisions (in chonological order) as in the observed cascade.

Original: The red pivot comes into play already on the 2nd collision, and the agent adjusts its velocity to satisfy the bottleneck constraint and the target.
Revised: The red pivot comes into play already on the 2nd collision, and the agent adjusts its velocity to satisfy the bottleneck constraint and the target.

Original: The ROSETTE-max-l baseline, satisfy the bottleneck but does not satisfy the target.
Revised: The ROSETTE-max-l baseline, satisfy the bottleneck but does not satisfy the target.

Revised: See the complete video

Revised: here: https:

Original: See the complete video here: https://youtu.be/vT1ivd1ECJs • Push: yellow ball, Target: blue hits purple, Bottleneck: blue hits right wall, Chain Count: - In this example, ROSETTE semantically follows only the first 2 collisions (in chonological order) as in the observed cascade.
Revised: youtu. be/vT1ivd1ECJs • Push: yellow ball, Target: blue hits purple, Bottleneck: blue hits right wall, Chain Count: - In this example, ROSETTE semantically follows only the first 2 collisions (in chonological order as in the observed cascade.

Original: The red pivot comes into play on the 3rd collision, and the agent adjusts its velocity to satisfy the bottleneck constraint and the target.
Revised: The red pivot comes into play on the 3rd collision, and the agent adjusts its velocity to satisfy the bottleneck constraint and the target.

Original: The ROSETTE-max-l baseline, completely fails in this task.
Revised: The ROSETTE-max-l baseline, completely fails in this task.

Original: See the complete video here: https://youtu.be/rgzWBfx-LqY
Revised: See the complete video here: https://youtu .be/rgzWBfx-LqY

Original: Importantly, these examples demonstrate the usefulness of the observed cascade for tree search.
Revised: Importantly, these examples demonstrate the usefulness of the observed cascade for tree search.

Original: ROSETTE followed the observed cascade along the part of the path that was useful to satisfy the instruction.
Revised: ROSETTE followed the observed cascade along the part of the path that was useful to satisfy the instruction.

Original: It diverged from the path when necessary, and found a solution when long cascades were essential, while ROSETTE-max-l struggled.
Revised: It diverged from the path when necessary, and found a solution when long cascades were essential, while ROSETTE-max-l struggled.

Original: Finally, we note that this observation is also quantitatively supported: As we show in Figure 4 and
Revised: Finally, we note that this observation is also quantitatively supported: As we show in Figure 4 and

Original: Figure A.3.
Revised: Figure A.3.

Original: When conditioning the Tree Success rate on producing long cascades, with Chain count constraint values greater or equal to 5, ROSETTE performs at 34 .
Revised: When conditioning the Tree Success rate on producing long cascades, with Chain count constraint values greater or equal to 5, ROSETTE performs at 34 .

Original: 8 ± 0 .
Revised: 8 ± 0 .

Original: 8% , while ROSETTE-max-l performs at 31 .
Revised: 8% , while ROSETTE-max-l performs at 31 .

Original: 3 ± 1 .
Revised: 3 ± 1 .

Original: 1% , showing ∼ 11 .
Revised: 1% , showing ∼ 11 .

Original: 1% improvement.
Revised: 1% improvement.

Original: For Chain count values smaller than 5, they are statistically equivalent 75 .
Revised: For Chain count values smaller than 5, they are statistically equivalent 75 .

Original: 1 ± 0 .
Revised: 1 ± 0 .

Original: 4% and 75 .
Revised: 4% and 75 .

Original: 4 ± 0 .
Revised: 4 ± 0 .

Original: B U SER STUDY
Revised: C U SER STUDY

Original: We conducted a user study with Amazon Mechanical Turk (AMT) using 30 test episodes.
Revised: We conducted a user study with Amazon Mechanical Turk (AMT) using 30 test episodes.

Original: We designed a game where a player (rater) is given a video of the observed cascade and is asked to select one of 44 combinations of orientations (11) and relative speeds (4) (magnitude of velocity).
Revised: We designed a game where a player (rater) is given a video of the observed cascade and is asked to select one of 44 combinations of orientations (11) and relative speeds (4) (magnitude of velocity).

Original: One combination of orientation and speed was aligned with the ground-truth solution, and the rest were spaced in relation to that solution.
Revised: One combination of orientation and speed was aligned with the ground-truth solution, and the rest were spaced in relation to that solution.

Original: In an offline stage, we tested which of the other orientations and speeds satisfy the goal and included those as valid solutions.
Revised: In an offline stage, we tested which of the other orientations and speeds satisfy the goal and included those as valid solutions.

Original: We allowed the players to freely replay the observed video.
Revised: We allowed the players to freely replay the observed video.

Original: We paid 1$ per game.
Revised: We paid 1$ per game.

Original: Figure A.1 shows one test episode.
Revised: Figure A.1 shows one test episode.

Original: The upper panel provides an instruction that states the goal of that specific episode.
Revised: The upper panel provides an instruction that states the goal of that specific episode.

Original: On the left, we provide a set of simple guidelines.
Revised: On the left, we provide a set of simple guidelines.

Original: The center panel provides the observed (failed) video.
Revised: The center panel provides the observed (failed) video.

Original: The right panel shows the initial frame, overlaid with the set of possible orientations and a set of HTML radio buttons to select the orientation and speed.
Revised: The right panel shows the initial frame, overlaid with the set of possible orientations and a set of HTML radio buttons to select the orientation and speed.

Original: The upper tab provides a set of four examples with solutions and explanations.
Revised: The upper tab provides a set of four examples with solutions and explanations.

Original: Those examples are given in
Revised: Those examples are given in

Original: Figure A.2.
Revised: Figure A.2.

Original: To maintain the quality of the queries, we only picked users with AMT “masters” qualification, demonstrating a high degree of approval rate over a wide range of tasks.
Revised: To maintain the quality of the queries, we only picked users with AMT “masters” qualification, demonstrating a high degree of approval rate over a wide range of tasks.

Original: Furthermore, we also executed a qualification test with a few curated episodes that are very simple.
Revised: Furthermore, we also executed a qualification test with a few curated episodes that are very simple.

Original: To qualify users, we made sure they do not randomly pick an answer by only qualifying users who completedepisodes and had a single error at most.
Revised: To qualify users, we made sure they do not randomly pick an answer by only qualifying users who completedepisodes and had a single error at most.

Original: Additionally, we deleted queries from one qualified user, who submitted answers at a rate of 3-4 episodes per minute, as we qualitatively observed that it should take 1-3 minutes to complete an episode.
Revised: Additionally, we deleted queries from one qualified user, who submitted answers at a rate of 3-4 episodes per minute, as we qualitatively observed that it should take 1-3 minutes to complete an episode.

Original: Qualified users received a bonus of 0.5$, accompanied with the following message:
Revised: Qualified users received a bonus of 0.5$, accompanied with the following message:

Original: Thank you for doing the qualification batch for our colliding balls game.
Revised: Thank you for doing the qualification batch for our colliding balls game.

Original: Our full study is now online.
Revised: Our full study is now online.

Original: You can start doing it.
Revised: You can start doing it.

Original: Please remember to PLAY
Revised: Please remember to PLAY

Original: THE VIDEO and use it to decide about your answer.
Revised: THE VIDEO and use it to decide about your answer.

Original: And also, take another look at the examples, as they can provide more intuition about the task.
Revised: And also, take another look at the examples, as they can provide more intuition about the task.

Original: 11 players have passed our qualification tests, playing 25 episodes on average.
Revised: 11 players have passed our qualification tests, playing 25 episodes on average.

Original: Table 3 compares the human success rate with ROSETTE and a Random baseline.
Revised: Table 3 compares the human success rate with ROSETTE and a Random baseline.

Original: Showing Average, Median and Best statistics.
Revised: Showing Average, Median and Best statistics.

Original: For the Median and Best statistics, we only included users who played a minimum number of 20 episodes (8 of 11 users).
Revised: For the Median and Best statistics, we only included users who played a minimum number of 20 episodes (8 of 11 users).

Original: [Table]
Revised: [Table]

Original: C A DDITIONAL EXPERIMENTAL DETAILS C. H YPERPARAMETER TUNING
Revised: D A DDITIONAL EXPERIMENTAL DETAILS D. H YPERPARAMETER TUNING

Original: We train the model and baselines for 15 epochs.
Revised: We train the model and baselines for 15 epochs.

Original: Batch size was set to 8192 to maximize the GPU memory usage.
Revised: Batch size was set to 8192 to maximize the GPU memory usage.

Original: We use the PyTorch’ default learning rate for Adam (Kingma & Ba, 2015) (0.001).
Revised: We use the PyTorch’ default learning rate for Adam (Kingma & Ba, 2015) (0.001).

Original: For inference, we set N observed to 9, the maximal tree depth to 30, we sample 10 6 initial states and expand 80 nodes per episode which takes ∼ 13 seconds.
Revised: For inference, we set N observed to 9, the maximal tree depth to 30, we sample 10 6 initial states and expand 80 nodes per episode which takes ∼ 13 seconds.

Original: The GNN uses 5 layers, with a hidden state dimension of 128.
Revised: The GNN uses 5 layers, with a hidden state dimension of 128.

Original: Hyper parameters were tuned one at a time, during an early experiment on a validation set.
Revised: Hyper parameters were tuned one at a time, during an early experiment on a validation set.

Original: C. R ANDOM B ASELINE
Revised: D. R ANDOM B ASELINE

Original: We sample an intervention at random from an estimated distribution of ground-truth interventions.
Revised: We sample an intervention at random from an estimated distribution of ground-truth interventions.

Original: The distribution is estimated by calculating a 2D-histogram with 30 × 30 , and approximating the distribution within each bin to be uniform.
Revised: The distribution is estimated by calculating a 2D-histogram with 30 × 30 , and approximating the distribution within each bin to be uniform.

Original: C. D EEPSET REGRESSION B ASELINE
Revised: D. D EEPSET REGRESSION B ASELINE

Original: Overview : For the Deepset regression baseline, we embed the instruction and the initial world state to predict a continuous intervention.
Revised: Overview : For the Deepset regression baseline, we embed the instruction and the initial world state to predict a continuous intervention.

Original: We use the permutation-invariant “Deep Sets” architecture (Zaheer et al., 2017), and use an L 2 loss with respect to ground-truth interventions in the “counterfactual” training samples.
Revised: We use the permutation-invariant “Deep Sets” architecture (Zaheer et al., 2017), and use an L 2 loss with respect to ground-truth interventions in the “counterfactual” training samples.

Original: The input to the Deepset architecture (Zaheer et al., 2017) is a set of feature vectors.
Revised: The input to the Deepset architecture (Zaheer et al., 2017) is a set of feature vectors.

Original: Each feature vector corresponds to a dynamic or static object in the scene.
Revised: Each feature vector corresponds to a dynamic or static object in the scene.

Original: The output is a vector ∈ R 2 , for predicting the controlled velocity of the pivot object.
Revised: The output is a vector ∈ R 2 , for predicting the controlled velocity of the pivot object.

Original: Feature representation : Each feature vector in the set is represented by a concatenation of the following fields [ obj feat ( o ) , instruction emb, position, velocity ] , where obj feat ( o ) is defined by Eq.
Revised: Feature representation : Each feature vector in the set is represented by a concatenation of the following fields [ obj feat ( o ) , instruction emb, position, velocity ] , where obj feat ( o ) is defined by Eq.

Original: (4), instruction emb , is defined by Eq.
Revised: (4), instruction emb , is defined by Eq.

Original: (7), position , velocity are the initial position and velocity of the object, as given by the observed cascade.
Revised: (7), position , velocity are the initial position and velocity of the object, as given by the observed cascade.

Original: Labels and loss : For ground-truth labels, we use the ground-truth velocity of the solution.
Revised: Labels and loss : For ground-truth labels, we use the ground-truth velocity of the solution.

Original: We use a L 2 loss comparing the ground-truth labels with the output of the Deepset architecture.
Revised: We use a L 2 loss comparing the ground-truth labels with the output of the Deepset architecture.

Original: C.
Revised: D. (Q I ET AL ., 2021) B ASELINE

Original: [Equation] B ASELINE

Original: Overview : Qi2021 is the state-of-the-art approach for solving PHYRE.
Revised: Overview : Qi2021 is the state-of-the-art approach for solving PHYRE.

Original: It uses a learned forward model, a learned goal-satisfaction classifier, and exhaustive search.
Revised: It uses a learned forward model, a learned goal-satisfaction classifier, and exhaustive search.

Original: For a fair comparison with our analytic event-driven forward model, we replace their learned forward model by a full simulator (Makoviychuk et al., 2021).
Revised: For a fair comparison with our analytic event-driven forward model, we replace their learned forward model by a full simulator (Makoviychuk et al., 2021).

Original: Therefore, for the goal-satisfaction classifier, in each frame, we replace the set of input feature vectors coming from the region-proposal-interaction-network (RPIN) of (Qi et al., 2021) by a set of feature vectors corresponding to each object in the scene, and its kinematic state as given by the simulator.
Revised: Therefore, for the goal-satisfaction classifier, in each frame, we replace the set of input feature vectors coming from the region-proposal-interaction-network (RPIN) of (Qi et al., 2021) by a set of feature vectors corresponding to each object in the scene, and its kinematic state as given by the simulator.

Original: To condition the classifier on the goal, we concatenate the instruction representation to each feature vector.
Revised: To condition the classifier on the goal, we concatenate the instruction representation to each feature vector.

Original: Feature representation : Each feature vector of an object in a frame, is represented by a concatenation of the following fields [ obj feat ( o ) , instruction emb, position, velocity, time ] , where obj feat ( o ) is defined by Eq.
Revised: Feature representation : Each feature vector of an object in a frame, is represented by a concatenation of the following fields [ obj feat ( o ) , instruction emb, position, velocity, time ] , where obj feat ( o ) is defined by Eq.

Original: (4), instruction emb , is defined by Eq.
Revised: (4), instruction emb , is defined by Eq.

Original: (7), position , velocity , time are the respective readings from the simulator in the frame.
Revised: (7), position , velocity , time are the respective readings from the simulator in the frame.

Original: Positive and Negative examples : For training the goal-satisfaction classifier with positive examples, we use the simulation of the solution cascade.
Revised: Positive and Negative examples : For training the goal-satisfaction classifier with positive examples, we use the simulation of the solution cascade.

Original: For negative examples, we use the simulation of the observed cascade.
Revised: For negative examples, we use the simulation of the observed cascade.

Original: Classifier Architecture : We use the classifier architecture of (Qi et al., 2021), as provided in their public implementation, with the following adaptations: (1) We replace the RPIN representation by the simulator-driven representation described above.
Revised: Classifier Architecture : We use the classifier architecture of (Qi et al., 2021), as provided in their public implementation, with the following adaptations: (1) We replace the RPIN representation by the simulator-driven representation described above.

Original: (2) We allow replacing the last fully connected layer by a multi-layer-perceptron (MLP) (3) We allowed more than four equally spaced input frames.
Revised: (2) We allow replacing the last fully connected layer by a multi-layer-perceptron (MLP) (3) We allowed more than four equally spaced input frames.

Original: Simulator configuration: The RPIN forward model is a fixed timestamp model, working atframe-per-second.
Revised: Simulator configuration: The RPIN forward model is a fixed timestamp model, working atframe-per-second.

Original: In the full simulator we used (Makoviychuk et al., 2021), we observed that it does not perform well in such a coarse-grained resolution, making objects to sometimes go through the walls.
Revised: In the full simulator we used (Makoviychuk et al., 2021), we observed that it does not perform well in such a coarse-grained resolution, making objects to sometimes go through the walls.

Original: Therefore, we increased the simulator resolution to 10 frames-per-second.
Revised: Therefore, we increased the simulator resolution to 10 frames-per-second.

Original: Hyperparam search: We searched for the best hyper-parameters configuration that minimizes the validation loss over the following ranges: Number of MLP hidden-layers ∈ [0 , 1 , . . .
Revised: Hyperparam search: We searched for the best hyper-parameters configuration that minimizes the validation loss over the following ranges: Number of MLP hidden-layers ∈ [0 , 1 , . . .

Original: 6] , Number of input frames ∈ [4 , 6 , 10 , 20] , batch-size ∈ [128 , 256] .
Revised: 6] , Number of input frames ∈ [4 , 6 , 10 , 20] , batch-size ∈ [128 , 256] .

Original: Number of training epoch was set according to early stopping on the validation set.
Revised: Number of training epoch was set according to early stopping on the validation set.

Original: Finally, we used the following hyper-parameters to evaluate the model performance on the test set:
Revised: Finally, we used the following hyper-parameters to evaluate the model performance on the test set:

Original: Number of MLP hidden-layers = 2 , Number of input frames = 4 (as in (Qi et al., 2021) paper), batch-size=128 , Number of training epoch = 17 .
Revised: Number of MLP hidden-layers = 2 , Number of input frames = 4 (as in (Qi et al., 2021) paper), batch-size=128 , Number of training epoch = 17 .

Original: Evaluation: For evaluation, we randomly selected a subset of 208 episodes ( 0 . 1% of the dataset), because inference for a single episodes took ∼ 5 . 
Revised: Evaluation: For evaluation, we randomly selected a subset of 208 episodes (10% of the test subset ), because inference for a single episodes took ∼ 5 . 5 minutes.

Revised: D. C ROSS ENTROPY B ASELINE

Revised: Overview : The cross entropy method is a black box optimizer for solving optimization problems.

Revised: We used (Qi et al., 2021) baseline’s classifier as our objective function.

Revised: At each step, we sampled 100 points and updated the sampling distribution based on their score.

Original: 5 minutes.
Revised: We have repeated this process for 100 iterations, and chosen the highest scored intervention for evaluation. Our code is based on a standard implementation Greenberg et al. (2022) of the cross entropy method. For evaluation, we randomly selected a subset of 208 episodes (10% of the test subset), because inference for a single episodes took ∼ 4 minutes.

Original: C. S EQUENTIAL B ASELINE
Revised: D. S EQUENTIAL B ASELINE

Original: We used the validation set to select the number of layers for this baseline, ∈ [5 , 10 , 20 , 30] .
Revised: We used the validation set to select the number of layers for this baseline, ∈ [5 , 10 , 20 , 30] .

Original: There wasn’t any significant difference when using 5 or 10 layers, and the success rate degraded for 20 or 30 layers.
Revised: There wasn’t any significant difference when using 5 or 10 layers, and the success rate degraded for 20 or 30 layers.

Original: Therefore, we used 5 layers for evaluating performance on the test set.
Revised: Therefore, we used 5 layers for evaluating performance on the test set.

Original: C. I NSTRUCTION ABLATION B ASELINES
Revised: D. I NSTRUCTION ABLATION B ASELINES

Original: We report the “count” and “bottleneck” ablations by zeroing their respective features in the instruction and using the same model weights that were used to report the performance of the ROSETTE model.
Revised: We report the “count” and “bottleneck” ablations by zeroing their respective features in the instruction and using the same model weights that were used to report the performance of the ROSETTE model.

Original: We did not retrain the model for these cases because the ROSETTE model was trained to handle these cases, as is evident by the “Unconstained” metric.
Revised: We did not retrain the model for these cases because the ROSETTE model was trained to handle these cases, as is evident by the “Unconstained” metric.

Original: For ablating the “full” instruction, we retrained the model, while completely zeroing the representation vector of the input instruction.
Revised: For ablating the “full” instruction, we retrained the model, while completely zeroing the representation vector of the input instruction.

Original: D I MPLEMENTATION DETAILS OF THE MODEL OF THE SCORE FUNCTION
Revised: E I MPLEMENTATION DETAILS OF THE MODEL OF THE SCORE FUNCTION

Original: We use a Graph Neural Network (GNN) to parameterize our score function.
Revised: We use a Graph Neural Network (GNN) to parameterize our score function.

Original: We represent the graph as a tuple ( A, X, E, z ) where A ∈ { 0 , 1 } n × n is the graph adjacency matrix, Y ∈ R n × d is a node feature matrix,
Revised: We represent the graph as a tuple ( A, X, E, z ) where A ∈ { 0 , 1 } n × n is the graph adjacency matrix, Y ∈ R n × d is a node feature matrix,

Original: E ∈ R m × d ′ is an edge feature matrix, and z ∈ R d ′′ is a global graph feature.
Revised: E ∈ R m × d ′ is an edge feature matrix, and z ∈ R d ′′ is a global graph feature.

Original: we chose to use a popular message passing GNN model (Battaglia et al., 2018) that maintains learnable node, edge and global graph representations.
Revised: we chose to use a popular message passing GNN model (Battaglia et al., 2018) that maintains learnable node, edge and global graph representations.

Original: Architecture The model is composed of several message passing layers, L k ◦ · · · ◦ L 1 where each
Revised: Architecture The model is composed of several message passing layers, L k ◦ · · · ◦ L 1 where each

Original: L i updates all representations, i.e.:
Revised: L i updates all representations, i.e.:

Original: [Equation]
Revised: [Equation]

Original: Each layer L i updates the features sequentially: the node and edge features are updated by aggregating local information, while the global feature is updated by aggregating over the whole graph.
Revised: Each layer L i updates the features sequentially: the node and edge features are updated by aggregating local information, while the global feature is updated by aggregating over the whole graph.

Original: We denote the parameters of the MLPs that are used in a layer L i as θ i , and note that these are the only learnable parameters in the model.
Revised: We denote the parameters of the MLPs that are used in a layer L i as θ i , and note that these are the only learnable parameters in the model.

Original: At the last layer i
Revised: At the last layer i

Original: =
Revised: =

Original: k we use a single dimension for the global feature, i.e., d ′ = 1 , which is then used as the score of the event node.
Revised: k we use a single dimension for the global feature, i.e., d ′ = 1 , which is then used as the score of the event node.

Original: Feature representation We describe next the feature representation of the inputs to the node feature matrix Y , the edge feature matrix E , and the global graph feature z .
Revised: Feature representation We describe next the feature representation of the inputs to the node feature matrix Y , the edge feature matrix E , and the global graph feature z .

Original: We start by describing a feature representation of any of the dynamic and static objects in the scene:
Revised: We start by describing a feature representation of any of the dynamic and static objects in the scene:

Original: An object o feature representation, noted by obj feat ( o ) , is a concatenation of the following fields
Revised: An object o feature representation, noted by obj feat ( o ) , is a concatenation of the following fields

Original: [Equation]
Revised: [Equation]

Original:  where one hot ( o ) is a one-hot vector ∈ R 12 , as represented by the instruction; is stationary indicates whether the object is stationary; is active means that in the context of a current collision, the object dynamics were coming from a collision chain that included the pivot; instruct inner prod is the results of an inner product of one hot ( o ) with each of the 5 object representations at the instruction embedding (Section G.3).
Revised: instruct [Equation] where one hot ( o ) is a one-hot vector ∈ R 12 , as represented by the instruction; is stationary indicates whether the object is stationary; is active means that in the context of a current collision, the object dynamics were coming from a collision chain that included the pivot; instruct inner prod is the results of an inner product of one hot ( o ) with each of the 5 object representations at the instruction embedding (Section H.3).

Original: Finally, bottleneck ind, count, count ind are copied from the instruction embedding.
Revised: Finally, bottleneck ind, count, count ind are copied from the instruction embedding.

Original: The graph node and edge features are derived from the DAG representation (Figure 3).
Revised: The graph node and edge features are derived from the DAG representation (Figure 3).

Original: Each row of the node feature matrix Y concatenates the two objects that participate at a collision [ obj feat ( obj a ) , obj feat ( obj b )] .
Revised: Each row of the node feature matrix Y concatenates the two objects that participate at a collision [ obj feat ( obj a ) , obj feat ( obj b )] .

Original: Each row at the edge feature matrix E represents obj feat ( o ) of the object on that edge.
Revised: Each row at the edge feature matrix E represents obj feat ( o ) of the object on that edge.

Original: Last, the global feature z is a copy of the instruction embedding Eq.
Revised: Last, the global feature z is a copy of the instruction embedding Eq.

Original: Training data For calculating the training labels of the score function, we traverse the semantic tree along the ground-truth sequence of the solution cascade and collect the positive labels using Eq.
Revised: Training data For calculating the training labels of the score function, we traverse the semantic tree along the ground-truth sequence of the solution cascade and collect the positive labels using Eq.

Original: If the event tree cannot reproduce the solution sequence of a sample (due to errors accumulated by the event-driven forward model), then Eq.
Revised: If the event tree cannot reproduce the solution sequence of a sample (due to errors accumulated by the event-driven forward model), then Eq.

Original: (2) cannot be calculated, and we drop that sample from the training set.
Revised: (2) cannot be calculated, and we drop that sample from the training set.

Original: We collect negative examples (with V = 0 ) by (1) taking the child nodes that diverge from the path to the ground-truth solution.
Revised: We collect negative examples (with V = 0 ) by (1) taking the child nodes that diverge from the path to the ground-truth solution.

Original: (2) Traverse a random path along the tree with the same length as the ground truth sequence, and set the score of all the nodes along that path to 0.
Revised: (2) Traverse a random path along the tree with the same length as the ground truth sequence, and set the score of all the nodes along that path to 0.

Revised: Note that setting the scores of every node along these paths to V = 0 is a heuristic and may introduce some label noise with respect to negative examples.

Revised: Additional research may be required to analyze the label-noise consequences and address it.

Original: E C OUNTERFACTUAL UPDATE FOR THE SCORE FUNCTION
Revised: F C OUNTERFACTUAL UPDATE FOR THE SCORE FUNCTION

Original: In this section, we derive the expression of the score function update according to the observed cascade (Eq.
Revised: In this section, we derive the expression of the score function update according to the observed cascade (Eq.

Original: We start the derivation by repeating the preliminary derivation steps introduced in the main text in more detail.
Revised: We start the derivation by repeating the preliminary derivation steps introduced in the main text in more detail.

Original: During inference, we observe a cascade that does not satisfy the instruction, and are asked to retrospectively suggest a better solution.
Revised: During inference, we observe a cascade that does not satisfy the instruction, and are asked to retrospectively suggest a better solution.

Original: How can the information can be used to find a better solution?
Revised: How can the information can be used to find a better solution?

Original: The probabilistic score function allows us to formalize this problem in a Bayesian setting.
Revised: The probabilistic score function allows us to formalize this problem in a Bayesian setting.

Original: We treat the model predictions as a prior for the true score, and the information about the observed cascade as evidence .
Revised: We treat the model predictions as a prior for the true score, and the information about the observed cascade as evidence .

Original: We then ask how to update the score function given the observed evidence.
Revised: We then ask how to update the score function given the observed evidence.

Original: Formally, we condition Eq.
Revised: Formally, we condition Eq.

Original: (1) by the evidence, V ( u | S u obs doesn’t satisfy g ) .
Revised: (1) by the evidence, V ( u | S u obs doesn’t satisfy g ) .

Original: We denote the set of interventions that satisfy the instruction g as G g ⊂ Y , and the evidence by E .
Revised: We denote the set of interventions that satisfy the instruction g as G g ⊂ Y , and the evidence by E .

Original: Note that an equivalent definition for the unconditioned score function V ( · ) is
Revised: Note that an equivalent definition for the unconditioned score function V ( · ) is

Original: [Equation]
Revised: [Equation]

Original: Our evidence is that for a particular ˜ y ∈ Y obs , we have ˜ y / ∈
Revised: Our evidence is that for a particular ˜ y ∈ Y obs , we have ˜ y / ∈

Original: G g .
Revised: G g .

Original: Now, by definition, every y, y ′ ∈ Y obs share the same observed cascade S u obs .
Revised: Now, by definition, every y, y ′ ∈ Y obs share the same observed cascade S u obs .

Original: Therefore, the evidence E can be equally formulated 
Revised: Therefore, the evidence E can be equally formulated as y ′ / ∈ G g for any y ′ sampled uniformly from Y u obs , y ′ ∼ U ( Y u obs ) . For brevity, we set Y obs = Y u obs .

Original: The conditioned score function is then,
Revised: The conditioned score function is then,

Original: [Equation]
Revised: [Equation]

Original: We use the law of total probability and write,
Revised: We use the law of total probability and write,

Revised: Pr ( y ∈ G g | y ∼ U

Original: [Equation]
Revised: ( Y u ) , E ) = Pr ( y ∈ G g y ∼ U ( Y u ) , E, y ∈ Y obs )

Revised: Pr ( y ∈ Y obs | y ∼ U

Revised: ( Y u ) , E ) + Pr ( y ∈ G g |

Revised: y ∼ U

Revised: ( Y u ) , E, y ∈ Y cobs )

Revised: Pr ( y ∈ Y cobs | y ∼ U

Revised: ( Y u ) , E ) =

Revised: Pr ( y ∈ G g | y ∼ U

Revised: ( Y u ∩ Y obs )

Revised: , E ) Pr ( y ∈ Y obs | y ∼ U

Revised: ( Y u ) , E ) + Pr ( y ∈ G g |

Revised: y ∼ U ( Y u ∩ Y cobs ) , E ) Pr ( y ∈ Y cobs

Revised: | y ∼ U

Revised: ( Y u ) , E )

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: As the conditioned event y ∈ Y obs is independent of E .
Revised: As the conditioned event y ∈ Y obs is independent of E .

Original: The relations between node U obs and u can be one of the three: a) the observed node is a descendant of u (and
Revised: The relations between node U obs and u can be one of the three: a) the observed node is a descendant of u (and

Original: therefore Y u ∩ Y obs =
Revised: therefore Y u ∩ Y obs =

Original: Y u ) b) u and the observed node belong to different branches, and therefore Y u ∩ Y obs = ∅ , or c) u is a descendant of the observed node (and therefore Y u ∩ Y obs =
Revised: Y u ) b) u and the observed node belong to different branches, and therefore Y u ∩ Y obs = ∅ , or c) u is a descendant of the observed node (and therefore Y u ∩ Y obs =

Original: Y obs ).
Revised: Y obs ).

Original: However, u obs represents a complete cascade rather than a partial sequence, and therefore the observed node does have any children, and we can ignore c).
Revised: However, u obs represents a complete cascade rather than a partial sequence, and therefore the observed node does have any children, and we can ignore c).

Original: Let us consider each case separately.
Revised: Let us consider each case separately.

Original: u and the observed node are along different paths .
Revised: u and the observed node are along different paths .

Original: In this case,
Revised: In this case,

Original: [Equation]
Revised: [Equation]

Original: and we’re left to evaluate Pr ( y ∈ G g |
Revised: and we’re left to evaluate Pr ( y ∈ G g |

Original: y ∼ U
Revised: y ∼ U

Original: ( Y u ) , E ) .
Revised: ( Y u ) , E ) .

Original: Since the evidence in this case provides information about a set that y is not conditioned on, it is independent of y , and therefore we conclude with,
Revised: Since the evidence in this case provides information about a set that y is not conditioned on, it is independent of y , and therefore we conclude with,

Original: [Equation]
Revised: [Equation]

Original: u is a descendant of the observed node.
Revised: u is a descendant of the observed node.

Original: Here,
Revised: Here,

Original: [Equation]
Revised: [Equation]

Original: In this case,
Revised: In this case,

Original: [Equation]
Revised: [Equation]

Original: Now,
Revised: Now,

Original: [Equation]
Revised: [Equation]

Original: Since the evidence indicates that for every y ′ ∈ Y obs the resulting sequence S u obs does not satisfy the goal.
Revised: Since the evidence indicates that for every y ′ ∈ Y obs the resulting sequence S u obs does not satisfy the goal.

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: As E does not add information when we sample from ( Y u ∩ Y cobs ) .
Revised: As E does not add information when we sample from ( Y u ∩ Y cobs ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: Namely,
Revised: Namely,

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: Plugging this back to Eq.
Revised: Plugging this back to Eq.

Original: 6 we obtain:
Revised: 6 we obtain:

Original: [Equation]
Revised: [Equation]

Original: which is our final result.
Revised: which is our final result.

Original: F R ELATION TO C AUSAL -I NFERENCE
Revised: G R ELATION TO C AUSAL -I NFERENCE

Original: The DAG representation (Section 4.2) is useful for graphically representing one instance of a cascade, but we intentionally avoid naming it a Causal DAG, because it can’t represent dependencies between events that are not explicitly observed in the video.
Revised: The DAG representation (Section 4.2) is useful for graphically representing one instance of a cascade, but we intentionally avoid naming it a Causal DAG, because it can’t represent dependencies between events that are not explicitly observed in the video.

Original: E.g., in the example [(A, B), (C, D), (A, E)] in Section 4.2, it may be that (A,E) depends on (C,D) because C blocks D from reaching to E before A do.
Revised: E.g., in the example [(A, B), (C, D), (A, E)] in Section 4.2, it may be that (A,E) depends on (C,D) because C blocks D from reaching to E before A do.

Original: The event tree can simulate this behaviour, while the DAG (C,D) ; (A,B)-¿(A,E) is unaware of it.
Revised: The event tree can simulate this behaviour, while the DAG (C,D) ; (A,B)-¿(A,E) is unaware of it.

Original: From a formal causal inference perspective (Pearl, 2000), our event tree is the part of our approach that can be related to the formal ”Structured” Causal Model (SCM).
Revised: From a formal causal inference perspective (Pearl, 2000), our event tree is the part of our approach that can be related to the formal ”Structured” Causal Model (SCM).

Original: As it is a generative model that reflects the data generation process; it can account for complex dependencies between events; and every edge corresponds to a function, namely, the event-driven forward model.
Revised: As it is a generative model that reflects the data generation process; it can account for complex dependencies between events; and every edge corresponds to a function, namely, the event-driven forward model.

Original: G D ATA GENERATION DETAILS G. V IDEO GENERATION DETAILS
Revised: H D ATA GENERATION DETAILS H. V IDEO GENERATION DETAILS

Original: In this section, we describe the generation process of the dynamical scene.
Revised: In this section, we describe the generation process of the dynamical scene.

Original: We first create an “unperturbed” video.
Revised: We first create an “unperturbed” video.

Original: Then, we perturb the video by modifying the velocity of a specific element, which will be later designated as the pivot.
Revised: Then, we perturb the video by modifying the velocity of a specific element, which will be later designated as the pivot.

Original: We let the perturbed video roll out, validate that it is indeed semantically different than the unperturbed video, and label it as the “observed” video.
Revised: We let the perturbed video roll out, validate that it is indeed semantically different than the unperturbed video, and label it as the “observed” video.

Original: The unperturbed video can now be used as reference for our instruction generation process.
Revised: The unperturbed video can now be used as reference for our instruction generation process.

Original: It is a realization of a specific, complex, semantic chain of events that is both semantically different than the perturbed (“observed”) video and is also feasible, e.g, by setting the intervention value as to revert the perturbation.
Revised: It is a realization of a specific, complex, semantic chain of events that is both semantically different than the perturbed (“observed”) video and is also feasible, e.g, by setting the intervention value as to revert the perturbation.

Original: This flow guarantees that we can ask meaningful instructions on the “observed” that are guaranteed to be realizable.
Revised: This flow guarantees that we can ask meaningful instructions on the “observed” that are guaranteed to be realizable.

Original: The unperturbed video.
Revised: The unperturbed video.

Original: We construct the unperturbed video by iteratively adding spheres and collisions in a physical simulator (IsaacGym (Makoviychuk et al., 2021)) increasing the video complexity.
Revised: We construct the unperturbed video by iteratively adding spheres and collisions in a physical simulator (IsaacGym (Makoviychuk et al., 2021)) increasing the video complexity.

Original: We start by placing a sphere in the confined four-walled space and assign it a random velocity.
Revised: We start by placing a sphere in the confined four-walled space and assign it a random velocity.

Original: The dynamics of a sphere moving freely in a confined square area can be expressed analytically.
Revised: The dynamics of a sphere moving freely in a confined square area can be expressed analytically.

Original: We pick a random time t 1 , hitting velocity, and hitting angle for the first collision.
Revised: We pick a random time t 1 , hitting velocity, and hitting angle for the first collision.

Original: We analytically solve for the initial position and velocity at t 0 = 0 that will result in the a collision at t 1 with the specific hitting velocity and angle.
Revised: We analytically solve for the initial position and velocity at t 0 = 0 that will result in the a collision at t 1 with the specific hitting velocity and angle.

Original: We assign these value to a randomly colored sphere.
Revised: We assign these value to a randomly colored sphere.

Original: Due to discrepancies between the simulator dynamics and the kinematic analytic model, we roll out the dynamical system in the simulator, and record the system state immediately after a collision.
Revised: Due to discrepancies between the simulator dynamics and the kinematic analytic model, we roll out the dynamical system in the simulator, and record the system state immediately after a collision.

Original: We continue adding spheres iteratively.
Revised: We continue adding spheres iteratively.

Original: Given a state at t i , we randomly select a sphere O i from the existing spheres, collision time t i +1 , hitting angle and velocity.
Revised: Given a state at t i , we randomly select a sphere O i from the existing spheres, collision time t i +1 , hitting angle and velocity.

Original: We solve analytically and find the initial position and velocity at t 0 = 

Revised: We solve analytically and find the initial position and velocity at t 0 =

Original: that will result in a collision with O i corresponding parameters.
Revised: that will result in a collision with O i corresponding parameters.

Original: We roll out the dynamical system, and update the velocities and positions records after each collision with the empirical values from the simulator.
Revised: We roll out the dynamical system, and update the velocities and positions records after each collision with the empirical values from the simulator.

Original: Our simple kinematic model assumes the target sphere and the newly added move freely.
Revised: Our simple kinematic model assumes the target sphere and the newly added move freely.

Original: However, other spheres may cross their trajectories, resulting in an a collision that will distract the spheres from their designated path.
Revised: However, other spheres may cross their trajectories, resulting in an a collision that will distract the spheres from their designated path.

Original: However, this simply means that the planned random collision was replaced by a different collision.
Revised: However, this simply means that the planned random collision was replaced by a different collision.

Original: Since we update our records of the resulting collisions and corresponding output velocities and positions using the simulator, this does not pose any serious limitations.
Revised: Since we update our records of the resulting collisions and corresponding output velocities and positions using the simulator, this does not pose any serious limitations.

Original: The observed video.
Revised: The observed video.

Original: We pick a random sphere from the set of spheres and assign it a different velocity at t = 0 .
Revised: We pick a random sphere from the set of spheres and assign it a different velocity at t = 0 .

Original: We roll out the system in the simulator and log all resulting collisions.
Revised: We roll out the system in the simulator and log all resulting collisions.

Original: We validate that the resulting collision sequence is different than the unperturbed video collision sequence.
Revised: We validate that the resulting collision sequence is different than the unperturbed video collision sequence.

Original: We now have two videos that differ only in the initial velocity of a specific sphere, but result in a substantially different semantic chain of events.
Revised: We now have two videos that differ only in the initial velocity of a specific sphere, but result in a substantially different semantic chain of events.

Original: G. I NSTRUCTION GENERATION DETAILS
Revised: H. I NSTRUCTION GENERATION DETAILS

Original: We describe the instruction generation process when given an “observed” video, and a “counterfactual” video that displays an alternative cascade of events.
Revised: We describe the instruction generation process when given an “observed” video, and a “counterfactual” video that displays an alternative cascade of events.

Original: Given a ground-truth video, its sequence of collisions, and a pivot, we randomly sample an instruction: Starting by randomly sampling a target collision from the sequence.
Revised: Given a ground-truth video, its sequence of collisions, and a pivot, we randomly sample an instruction: Starting by randomly sampling a target collision from the sequence.

Original: And then, we randomly sample up to two constraints that accompany the goal.
Revised: And then, we randomly sample up to two constraints that accompany the goal.

Original: For constructing the constraints, we first represent the sequence of collisions using a DAG, in a similar fashion as described in Figure 3, then we use standard NetworkX functionality (Hagberg et al., 2008) for graph traversal: (1) We use “dag.ancestors()” to get a list of nodes for the “bottleneck” constraint.
Revised: For constructing the constraints, we first represent the sequence of collisions using a DAG, in a similar fashion as described in Figure 3, then we use standard NetworkX functionality (Hagberg et al., 2008) for graph traversal: (1) We use “dag.ancestors()” to get a list of nodes for the “bottleneck” constraint.

Original: (2) We use ”all simple paths()” to count the nodes in a chain reaction between the pivot and the target collision.
Revised: (2) We use ”all simple paths()” to count the nodes in a chain reaction between the pivot and the target collision.

Original: To avoid trivial goals, we drop an instruction if it is fulfilled by the observed video (rather than the “counterfactual” video).
Revised: To avoid trivial goals, we drop an instruction if it is fulfilled by the observed video (rather than the “counterfactual” video).

Original: We sample up to 5 unique instructions for each scene ( ∼ 4 on average).
Revised: We sample up to 5 unique instructions for each scene ( ∼ 4 on average).

Original: G. I NSTRUCTION FEATURE REPRESENTATION
Revised: H. I NSTRUCTION FEATURE REPRESENTATION

Original: We assume perfect lexical perception, and provide the agent with the a structured vector representation of each instruction, by concatenating the following fields:
Revised: We assume perfect lexical perception, and provide the agent with the a structured vector representation of each instruction, by concatenating the following fields:

Original: [Equation]
Revised: [Equation]

Original: (7) where target obj a, target obj b are the object representations of the target collision.
Revised: where target obj a, target obj b are the object representations of the target collision.

Original: pivot obj represents the pivot.
Revised: pivot obj represents the pivot.

Original: bottleneck obj a, bottleneck obj b, bottleneck ind represent the two “bottleneck” objects and a binary indicator scalar.
Revised: bottleneck obj a, bottleneck obj b, bottleneck ind represent the two “bottleneck” objects and a binary indicator scalar.

Original: If an “bottleneck” constraint is not applicable for an instruction, we them all to 0.
Revised: If an “bottleneck” constraint is not applicable for an instruction, we them all to 0.

Original: count, count ind are 2 scalar values: One for the number of collisions of the chain “count” constraint, and another used as a binary indicator for the “count” constraint.
Revised: count, count ind are 2 scalar values: One for the number of collisions of the chain “count” constraint, and another used as a binary indicator for the “count” constraint.

Original: Similarly if a “count” constraint is not applicable for an instruction, we set both count and count ind to 0.
Revised: Similarly if a “count” constraint is not applicable for an instruction, we set both count and count ind to 0.

Original: Finally, note that each object is represented by a one-hot vector ∈ R 12 , because the environment has 12 types of unique objects: 6 colored balls, 2 static pins, and 4 walls.
Revised: Finally, note that each object is represented by a one-hot vector ∈ R 12 , because the environment has 12 types of unique objects: 6 colored balls, 2 static pins, and 4 walls.

Original: G. C OMPLEX INSTRUCTIONS DATASET
Revised: H. C OMPLEX INSTRUCTIONS DATASET

Original: For the complex instructions dataset, we add a third object centric constraint that counts the number of interactions a specific object makes on the paths from the pivot to the target collision.
Revised: For the complex instructions dataset, we add a third object centric constraint that counts the number of interactions a specific object makes on the paths from the pivot to the target collision.

Original: It resembles constraining the amount of resources available per instance on a logistic chain.
Revised: It resembles constraining the amount of resources available per instance on a logistic chain.

Original: With an additional constraint we can test our approach on a more challenging task that has a large variety of instructions that have 2 or more constraints.
Revised: With an additional constraint we can test our approach on a more challenging task that has a large variety of instructions that have 2 or more constraints.

Original: We split the evaluation set to “Hard” instructions that have 2 or more constraints, and “Easy” instruction with 0-1 constraints.
Revised: We split the evaluation set to “Hard” instructions that have 2 or more constraints, and “Easy” instruction with 0-1 constraints.

Original: We generated instructions for the same scenes as in the main dataset, which yields ∼ 4 .
Revised: We generated instructions for the same scenes as in the main dataset, which yields ∼ 4 .

Original: 5 instructions per scene.
Revised: 5 instructions per scene.

Original: The test set consists of 2190 episodes, where 54% are “Hard” instructions.
Revised: The test set consists of 2190 episodes, where 54% are “Hard” instructions.

Original: H T HE FORWARD MODEL
Revised: I T HE FORWARD MODEL

Original: In our physical setup, the dynamics are prescribed by the position and velocity c ji =
Revised: In our physical setup, the dynamics are prescribed by the position and velocity c j i =

Original: ( pos ji , vel ji ) , j = 1 . . .
Revised: ( pos ji , vel ji ) , j = 1 . . .

Original: n of each of n objects in the environment.
Revised: n of each of n objects in the environment.

Original: The world state w ui of a node u is then a tuplew ui
Revised: The world state w ui of a node u is then a tuple

Original: = ( c 1 i , c 2 i , ...
Revised: [Equation]

Original: c ni , t i ) , where for the root node t i = 0 for all x i .
Revised: where for the root node t i = 0 for all x i .

Original: The forward module takes as input a world state w i it outputs the next semantic event ( s ′ ), and a state f ( w i ) =
Revised: The forward module takes as input a world state w i it outputs the next semantic event ( s ′ ), and a state f ( w i ) =

Original: w ′ i
Revised: w ′ i

Original: =
Revised: =

Original: ( c ′ 1 i , c ′ 2 i , ...
Revised: ( c ′ 1 i , c ′ 2 i , ...

Original: c ′ ni , t ′ i ) immediately after the predicted semantic event at t ′ i .
Revised: c ′ ni , t ′ i ) immediately after the predicted semantic event at t ′ i .

Original: The section is divided into three parts.
Revised: The section is divided into three parts.

Original: First, we describe the analytical equations that control if two objects will collide.
Revised: First, we describe the analytical equations that control if two objects will collide.

Original: Then we show how can leverage the analytic model to efficiently branch out from a node in the event tree.
Revised: Then we show how can leverage the analytic model to efficiently branch out from a node in the event tree.

Original: Finally, we fill in the missing details and present the full forward model.
Revised: Finally, we fill in the missing details and present the full forward model.

Original: The collision detector .
Revised: The collision detector .

Original: Assume two spheres i
Revised: Assume two spheres i

Original: = α, β moving freely on a plane with anexample initial velocity of v i and position p i at t = 0 .
Revised: = α, β moving freely on a plane with an initial velocity of v i and position r i at t = 0 .

Original: Each sphere has a radius of r i .
Revised: Each sphere has a radius of l i .

Original: If the two sphere collide, then, at the moment of collision, the spheres intersect at a single point.
Revised: If the two sphere collide, then, at the moment of collision, the spheres intersect at a single point.

Original: A simple geometric calculation shows their planar distance at collision is
Revised: We can use a simple geometric calculation to find their planar distance.

Revised: The distance between the center of spheres is l α + l β , while the vertical distance between the two centers is | l α − l β | .

Revised: The resulting planar distance is then:

Original: [Equation]
Revised: [Equation]

Original: Therefore, in order to check if the spheres collide, we can check if the  distance between the two spheres is ever equal to d ,
Revised: Therefore, in order to check if the spheres collide, we can check if the planar distance between the two spheres is ever equal to d ,

Original: [Equation]
Revised: [Equation]

Original: This is a quadratic equation in t , which we can solve for analytically.
Revised: This is a quadratic equation in t , which we can solve for analytically.

Original: If the discriminant is nonnegative, the collision time corresponds to the smaller root.
Revised: If the discriminant is nonnegative, the collision time corresponds to the smaller root.

Original: The spheres’ velocities immediately after the collision are given by:
Revised: The spheres’ velocities immediately after the collision are given by:

Original: [Equation]
Revised: [Equation]

Original: Likewise, it is trivial to obtain an analytical expression for the collision time and output velocity of a collision between a freely moving sphere and each of the static walls bounding the spheres (should the collision occur).
Revised: Likewise, it is trivial to obtain an analytical expression for the collision time and output velocity of a collision between a freely moving sphere and each of the static walls bounding the spheres (should the collision occur).

Revised: The sphere’s velocity in the direction orthogonal to the walls flips, while the parallel velocity remains the same.

Original: Parallelizing collision detection.
Revised: Parallelizing collision detection.

Original: The collision detector provides an analytic condition that validates whether a specific collision occurs.
Revised: The collision detector provides an analytic condition that validates whether a specific collision occurs.

Original: Therefore, we can solve it in parallel for multiple tuples of ( r 1 , v 1 , r 2 , v 2 ) on a GPU using packages such as PyTorch.
Revised: [Equation] Eqs. 9-13 can be solved in parallel for multiple tuples of ( r 1 , v 1 , r 2 , v 2 ) on a GPU using packages such as PyTorch.

Original: Therefore, given an intervention set of Y u , and a corresponding world-state set W u , we iterate over all possible collisions S ij =
Revised: Given an intervention set of Y u , and a corresponding world-state set W u , we iterate over all possible collisions S ij =

Original: ( O i , O j )  and  apply our collision detector on all  w ∈ W u simultaneously.
Revised: ( O i , O j ) . For each collision between object i and j we can apply our collision detector by extracting the corresponding coordinates c iw , c jw , t w from w ∈ W u (Eq 8). We can do in parallel for all world states w ∈ W u .

Revised: If a collision is predicted, we construct a new node child u ′ of u .

Revised: We associate with it the interventions for which the collision detector returned a non-null time for the collision, Y ′ u , the corresponding set of post-collision world state W ′ u , and the event sequence S ′ u

Revised: = concat ( S u , ( O i , O j ))

Original: The complexity is quadratic in the number of object rather than linear in the number of interventions.
Revised: The complexity is quadratic in the number of object rather than linear in the number of interventions.

Original: This allows us to apply our algorithm with a high number of interventions, and therefore enable us to consider delicate sequences of collision that would require refined ”trick shots”.
Revised: This allows us to apply our algorithm with a high number of interventions, and therefore enable us to consider delicate sequences of collision that would require refined ”trick shots”.

Original: This approach considers every two objects O i , O j as moving freely.
Revised: This approach considers every two objects O i , O j as moving freely.

Original: However, another object in the environment, e.g, O k , may interact with O i (without loss of generality) before the collision.
Revised: However, another object in the environment, e.g, O k , may interact with O i (without loss of generality) before the collision.

Original: This necessarily means that the collision time t ik precedes t ij .
Revised: This necessarily means that the collision time t ik precedes t ij .

Original: In order to account for this, we hold an additional structure that maintains the minimal collision time for every w ∈ W u .
Revised: In order to account for this, we hold an additional structure that maintains the minimal collision time for every w ∈ W u .

Original: We update it as we iterate over all possible collisions.
Revised: We update it as we iterate over all possible collisions.

Original: Then, we associate each w ∈ W u and its corresponding y ∈ Y u to the event node corresponding to the collision with the earliest collision time.
Revised: Then, we associate each w ∈ W u and its corresponding y ∈ Y u to the event node corresponding to the collision with the earliest collision time.

Original: I A DDITIONAL SETUP - LOGISTICS
Revised: J A DDITIONAL SETUPS

Original: Here, we present an example foran additional setup for which our formalism can be applied. 
Revised: Here, we present examples for additional setups for which our formalism can be applied. J . L OGISITICS

Original: While logistics is a complex field, we describe a simple model that captures the essential components of a logistics problem.
Revised: While logistics is a complex field, we describe a simple model that captures the essential components of a logistics problem.

Original: Consider a large logistics enterprise that needs to coordinate shipping from multiple locations.
Revised: Consider a large logistics enterprise that needs to coordinate shipping from multiple locations.

Original: The enterprise has multiple carriers (e.g, trucks or airplanes) v i , i = 1 ..m and routes them between logistic centers at r j , j = 1 ..n .
Revised: The enterprise has multiple carriers (e.g, trucks or airplanes) v i , i = 1 ..m and routes them between logistic centers at r j , j = 1 ..n .

Original: A plan is a schedule for each carrier τ j , where a schedule τ j is a sequence of arrivals and departures between various logistic centers,
Revised: A plan is a schedule for each carrier τ j , where a schedule τ j is a sequence of arrivals and departures between various logistic centers,

Original: [Equation]
Revised: [Equation]

Original: Not all plans are feasible .
Revised: Not all plans are feasible .

Original: Each carrier can travel at a range of velocities, resulting in a range of arrival times to the possible destinations.
Revised: Each carrier can travel at a range of velocities, resulting in a range of arrival times to the possible destinations.

Original: Carriers can exchange cargo is they are present at the same logistic center.
Revised: Carriers can exchange cargo is they are present at the same logistic center.

Original: Now, assume a logistic center is suddenly shut down.
Revised: Now, assume a logistic center is suddenly shut down.

Original: Rescheduling all carriers is unfeasible, as some may be already committed to a route, or may not be easily diverted (e.g., are airborne).
Revised: Rescheduling all carriers is unfeasible, as some may be already committed to a route, or may not be easily diverted (e.g., are airborne).

Original: Furthermore, recomputing a new plan for the complete enterprise might be computationally heavy.
Revised: Furthermore, recomputing a new plan for the complete enterprise might be computationally heavy.

Original: Finally, it seems reasonable that re-planning of only the routes of carriers that were suppose to arrive to the closed logistic center may be enough.
Revised: Finally, it seems reasonable that re-planning of only the routes of carriers that were suppose to arrive to the closed logistic center may be enough.

Original: We denote those k carriers as the rescheduled carriers .
Revised: We denote those k carriers as the rescheduled carriers .

Original: Note that while only some of routes may be re-planned, other carriers might be affected as well due to a cascade of delays or even cargo exchange cancellations.
Revised: Note that while only some of routes may be re-planned, other carriers might be affected as well due to a cascade of delays or even cargo exchange cancellations.

Original: Such re-planning may be constrained by semantic instruction.
Revised: Such re-planning may be constrained by semantic instruction.

Original: For example: ”Carrier X should only make two deliveries”, ”Carrier Y should meet carrier Z before meeting Carrier W”, etc. .
Revised: For example: ”Carrier X should only make two deliveries”, ”Carrier Y should meet carrier Z before meeting Carrier W”, etc. .

Original: We now cast this problem into our general framework, described in Section 4.
Revised: We now cast this problem into our general framework, described in Section 4.

Original: An event is the arrival or departure of a carrier to a logistic center.
Revised: An event is the arrival or departure of a carrier to a logistic center.

Original: Each intervention y ∈ Y is a set of plans the rescheduled carriers,
Revised: Each intervention y ∈ Y is a set of plans the rescheduled carriers,

Original: [Equation]
Revised: [Equation]

Original: A world state w j is the position of the k carriers at different time (( p 0 j , t 0 j ) , ..., ( p kj , t kj ))
Revised: A world state w j is the position of the k carriers at different time (( p 0 j , t 0 j ) , ..., ( p kj , t kj ))

Original: The forward model takes as input a world state and outputs all world state that obey the following two rules: 1)
Revised: The forward model takes as input a world state and outputs all world state that obey the following two rules: 1)

Original: At least one carrier moved to a different logistic center.
Revised: At least one carrier moved to a different logistic center.

Original: The transition of carriers follow physical constraints.
Revised: The transition of carriers follow physical constraints.

Original: If carrier i can move at velocity range [ v imin , v imax ] and it moves between two logistics centers r a and r b , then the transition time must be in
Revised: If carrier i can move at velocity range [ v imin , v imax ] and it moves between two logistics centers r a and r b , then the transition time must be in

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we assume that there is no cargo limit.
Revised: For simplicity, we assume that there is no cargo limit.

Original: The expressions for the induced probability, Eqs.
Revised: The expressions for the induced probability, Eqs.

Original: 2 -3, remains the same.
Revised: 2 -3, remains the same.

Revised: J. F AILURE CASCADES IN POWER GRIDS

Revised: Cascading failures in power grid may cause large blackout with substantial economical damage

Original: [Figure]
Revised: Sch ¨ afer et al. Cascading Power failures may be induced due to random fluctuations and can develop on orders of seconds. Human operators or complex control mechanism may not be able react in time. The transmission system operator may use an event-driven forward model to find fast automated reactions for unseen dynamical configurations to avoid cascading failures.

Revised: Here, semantic events are failures of nodes (nodes; e.g., transformers, power generators, etc.) or power lines (edges).

Revised: Power flow follows a known set of ODE for a given grid (eqs 14-15 in Sch ¨ afer et al.

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: where, θ i , ω i are the dynamical variable at node i , P

Revised: i is the power input (or output) at node i , and

Revised: K ij is a weighted adjacency matrix representing the grid connectivity.

Revised: If at some point in time the flow F i j exceeds the powerline capacity αk ij , α ∈ [0 , 1] (eqs 1-2), the line fails.

Revised: This condition can be formally written as

Revised: [Equation]

Revised: If the line fails, the dynamics are governed by a new effective coupling matrix K i j , and the dynamics in Eqs.

Revised: 15-16 changes accordingly.

Revised: A failure of a node may induce outage to some region.

Revised: The transmission system operator (TSO) might define goals such as “no more than three failures”, “the maximal number of affected people should be less than n ”, “these highly important nodes should not fail” etc.

Revised: J. E VOLUTION OF NATURAL DISASTERS

Revised: Finally, another use case is the evolution of natural disasters.

Revised: Zuccaro et al.

Revised: (2018) provides a full description of an event tree.

Revised: It models transitions between events like a “seismic shock” which can lead to “landslide” and result with “traffic accident”, and how taking preventive measures like “evacuate population” can influence the total damage caused by the crisis.

Original: J C OMPLEX SCENARIO CONDITIONING FOR THE MAIN DATASET
Revised: K C OMPLEX SCENARIO CONDITIONING FOR THE MAIN DATASET

Original: In Figure A.3 we provide the results for complex scenario conditioning for the main dataset (with two type of constraints).
Revised: In Figure A.3 we provide the results for complex scenario conditioning for the main dataset (with two type of constraints).

Original: The results demonstrate a similar trend as in the complex instruction dataset in Figure 
Revised: The results demonstrate a similar trend as in the complex instruction dataset in Figure 4. [Figure]