Original: Learning Casual Overhypotheses through Exploration in Children and Computational Models
Revised: Learning Causal Overhypotheses through Exploration in Children and Computational Models

Original: Author names withheld Editor: Under Review for CLeaR
Revised: Editors: Bernhard Sch ¨ olkopf, Caroline Uhler and Kun Zhang

Original: Abstract
Revised: Abstract

Original: Despite recent progress in reinforcement learning (RL), RL algorithms for exploration still remain an active area of research.
Revised: Despite recent progress in reinforcement learning (RL), RL algorithms for exploration still remain an active area of research.

Original: Existing methods often focus on state-based metrics, which do not consider the underlying causal structures of the environment, and while recent research has begun to explore RL environments for causal learning, these environments primarily leverage causal information through causal inference or induction rather than exploration.
Revised: Existing methods often focus on state-based metrics, which do not consider the underlying causal structures of the environment, and while recent research has begun to explore RL environments for causal learning, these environments primarily leverage causal information through causal inference or induction rather than exploration.

Original: In contrast, human children— some of the most proﬁcient explorers—have been shown to use causal information to great beneﬁt.
Revised: In contrast, human children— some of the most proficient explorers—have been shown to use causal information to great benefit.

Original: In this work, we introduce a novel RL environment designed with a controllable causal structure, which allows us to evaluate exploration strategies used by both agents and children in a uniﬁed environment.
Revised: In this work, we introduce a novel RL environment designed with a controllable causal structure, which allows us to evaluate exploration strategies used by both agents and children in a unified environment.

Original: In addition, through experimentation on both computation models and children, we demonstrate that there are signiﬁcant differences between information-gain optimal RL exploration in causal environments and the exploration of children in the same environments.
Revised: In addition, through experimentation on both computation models and children, we demonstrate that there are significant differences between information-gain optimal RL exploration in causal environments and the exploration of children in the same environments.

Original: We leverage this new insight to lay the groundwork for future research into efﬁcient exploration and disambiguation of causal structures for RL algorithms.
Revised: We conclude with a discussion of how these findings may inspire new directions of research into efficient exploration and disambiguation of causal structures for RL algorithms.

Original: 1. Introduction
Revised: 1. Introduction

Original: Exploration is a fundamental problem in reinforcement learning (RL).
Revised: Exploration is a fundamental problem in reinforcement learning (RL).

Original: In order to act on the world effectively, an agent needs to be able to efﬁciently and actively gather information about how the environment works.
Revised: In order to act on the world effectively, an agent needs to be able to efficiently and actively gather information about how the environment works.

Original: Gathering causal information is particularly helpful for action planning and generalization (de Haan et al., 2019; Rezende et al., 2020; Ke et al., 2021).
Revised: Gathering causal information is particularly helpful for action planning and generalization (de Haan et al., 2019; Rezende et al., 2020; Ke et al., 2021).

Original: For example, if the agent’s task is to turn on a lamp, the corresponding causal relationship is that the lamp will turn on only if all of the following are true: 1) the switch is ﬂipped to the on position, 2) the lamp is plugged into a source of electricity and 3) the light bulb is working.
Revised: For example, if the agent’s task is to turn on a lamp, the corresponding causal relationship is that the lamp will turn on only if all of the following are true: 1) the switch is flipped to the on position, 2) the lamp is plugged into a source of electricity, and 3) the light bulb is working.

Original: Understanding this causal relationship enables the agent to systematically diagnose a problem—if, for instance, it is in a new room and ﬂips the switch to on but the lamp does not turn on—and systematically explore to ﬁnd solutions.
Revised: Understanding this causal relationship enables the agent to systematically diagnose a problem—if, for instance, it is in a new room and flips the switch to on but the lamp does not turn on—and systematically explore to find solutions.

Original: Existing RL exploration methods typically do not focus on such causal exploration: they do not form and test causal hypotheses, or plan active interventions to obtain causal data (Amin et al., 2021).
Revised: Existing RL exploration methods typically do not focus on such causal exploration: they do not form and test causal hypotheses, or plan active interventions to obtain causal data (Amin et al., 2021).

Original: Instead, existing RL exploration methods primarily focus on expanding the set of experiences of the agent, for instance by visiting novel or surprising areas of the state space.
Revised: Instead, existing RL exploration methods primarily focus on expanding the set of experiences of the agent, for instance by visiting novel or surprising areas of the state space.

Original: This may be sufﬁcient for the agent to solve the particular task it is trained for, but limits its ability to generalize to new tasks and environments (Packer et al., 2018; Cobbe et al., 2019).
Revised: This may be sufficient for the agent to solve the particular task it is trained for, but limits its ability to generalize to new tasks and environments (Packer et al., 2018; Cobbe et al., 2019).

Original: Although there is growing interest in causal learning in RL, this work has focused on extracting a particular underlying causal graph from given data in a particular environment (Nair et al., 2019; Ke et al., 2021;
Revised: Although there is growing interest in causal learning in RL, this work has focused on extracting a particular underlying causal graph from given data in a particular environment (Nair et al., 2019; Ke et al., 2021;

Original: Wang et al., 2021).
Revised: Wang et al., 2021).

Original: There is very limited amount work that attempts to utilize causal information for exploration in RL, or to learn abstract causal structure through exploration.
Revised: There is very limited amount work that attempts to utilize causal information for exploration in RL, or to learn abstract causal structure through exploration.

Original: How might we integrate causal learning and reasoning into exploration in RL agents?
Revised: How might we integrate causal learning and reasoning into exploration in RL agents?

Original: We propose to draw inspiration from cognitive science.
Revised: We propose to draw inspiration from cognitive science.

Original: In contrast to RL agents, even young children learn and reason about causal relations and actively explore to collect causal data.
Revised: In contrast to RL agents, even young children learn and reason about causal relations and actively explore to collect causal data.

Original: Moreover, they can learn and use causal overhypotheses —hypotheses about which classes of causal relationships are more or less likely (Kemp et al., 2007; Lucas et al., 2014).
Revised: Moreover, they can learn and use causal overhypotheses —hypotheses about which classes of causal relationships are more or less likely (Kemp et al., 2007; Lucas et al., 2014).

Original: Causal overhypotheses are a key component that allows humans to learn causal models from a sparse amount of data (Grifﬁths and
Revised: Causal overhypotheses are a key component that allows humans to learn causal models from a sparse amount of data (Griffiths and

Original: Tenenbaum, 2009), because they can help narrow down the possible causal relationships that we consider and test.
Revised: Tenenbaum, 2009), because they can help narrow down the possible causal relationships that we consider and test.

Original: In the previous light example, suppose that we enter a new apartment, and want to turn on the lights in the living room.
Revised: In the previous light example, suppose that we enter a new apartment, and want to turn on the lights in the living room.

Original: If we lacked any causal overhypothesis about what causes a light to turn on, then this would be an essentially hopeless task.
Revised: If we lacked any causal overhypothesis about what causes a light to turn on, then this would be an essentially hopeless task.

Original: We might try knocking on the wall, shufﬂing on the carpet, rotating the lamp, and so forth—the way that an RL agent starts out acting in an environment in which it has no prior experience.
Revised: We might try knocking on the wall, shuffling on the carpet, rotating the lamp, and so forth—the way that an RL agent starts out acting in an environment in which it has no prior experience.

Original: Instead, if our causal overhypothesis is that ﬂipping a light switch on the wall will turn on one or more lights, then that helps our hypothesis testing.
Revised: Instead, if our causal overhypothesis is that flipping a light switch on the wall will turn on one or more lights, then that helps our hypothesis testing.

Original: We might try ﬂipping various combinations of light switches in the apartment, and quickly identify the correct causal relationships.
Revised: We might try flipping various combinations of light switches in the apartment, and quickly identify the correct causal relationships.

Original: In our experiments, we draw inspiration from the blicket detector experiment (Gopnik and Sobel, 2000a), which is a classic setting for evaluating causal learning and reasoning in children.
Revised: In our experiments, we draw inspiration from the blicket detector experiment (Gopnik and Sobel, 2000a), which is a classic setting for evaluating causal learning and reasoning in children.

Original: Blocks are placed on a “blicket machine”.
Revised: Blocks are placed on a “blicket machine”.

Original: Some blocks are “blickets” and the blicket machine lights up when blickets are placed on it, according to some rule.
Revised: Some blocks are “blickets” and the blicket machine lights up when blickets are placed on it, according to some rule.

Original: The participants must learn which blocks are blickets, and use those blocks to activate the machine.
Revised: The participants must learn which blocks are blickets, and use those blocks to activate the machine.

Original: The blicket machine requires children to learn the structure of a novel causal system, and allows researchers to present children with relatively complex patterns of statistical correlation and intervention.
Revised: The blicket machine requires children to learn the structure of a novel causal system, and allows researchers to present children with relatively complex patterns of statistical correlation and intervention.

Original: It does so in a concrete, simple and intuitive way.
Revised: It does so in a concrete, simple and intuitive way.

Original: As a result, a large body of studies using this method have demonstrated a remarkable range of causal inference capacities in children as young as 18 months (Gopnik et al., 2001; Cook et al., 2011; Gopnik, 2012; Gopnik et al., 2004;
Revised: As a result, a large body of studies using this method have demonstrated a remarkable range of causal inference capacities in children as young as 18 months (Gopnik et al., 2001; Cook et al., 2011; Gopnik, 2012; Gopnik et al., 2004;

Original: Gopnik and Sobel, 2000b; Gopnik and Wellman, 2012; Kushnir and Gopnik, 2007; Lucas et al., 2014; Meltzoff et al., 2012).
Revised: Gopnik and Sobel, 2000b; Gopnik and Wellman, 2012; Kushnir and Gopnik, 2007; Lucas et al., 2014; Meltzoff et al., 2012).

Original: In particular, studies have tested how well children learn more abstract overhypotheses about the rules by which the machine works.
Revised: In particular, studies have tested how well children learn more abstract overhypotheses about the rules by which the machine works.

Original: For example, the causal relationship can be either disjunctive or conjunctive (Lucas et al., 2014).
Revised: For example, the causal relationship can be either disjunctive or conjunctive (Lucas et al., 2014).

Original: In the disjunctive case, if at least one blicket is placed on the machine, then it lights up.
Revised: In the disjunctive case, if at least one blicket is placed on the machine, then it lights up.

Original: In the conjunctive case, at least two blickets must be placed on the machine in order for it to light up.
Revised: In the conjunctive case, at least two blickets must be placed on the machine in order for it to light up.

Original: Participants must infer whether the machine is conjunctive or disjunctive.
Revised: Participants must infer whether the machine is conjunctive or disjunctive.

Original: Interestingly, prior work has found not only that that children can learn these overhypotheses from data, but also that they are more ﬂexible than adults in these tasks.
Revised: Interestingly, prior work has found not only that that children can learn these overhypotheses from data, but also that they are more flexible than adults in these tasks.

Original: They are better at learning unusual overhypotheses, like those involving conjunctive causal relationships (Lucas et al., 2014; Gopnik et al., 2017)
Revised: They are better at learning unusual overhypotheses, like those involving conjunctive causal relationships (Lucas et al., 2014; Gopnik et al., 2017)

Original: However, in prior experiments investigating causal overhypotheses, children are given the relevant data by the experimenter rather than generating the data themselves through causal exploration.
Revised: However, in prior experiments investigating causal overhypotheses, children are given the relevant data by the experimenter rather than generating the data themselves through causal exploration.

Original: Could children also actively generate data that would allow them to learn the right causal overhypotheses?
Revised: Could children also actively generate data that would allow them to learn the right causal overhypotheses?

Original: And would causal overhypotheses shape their exploration?
Revised: And would causal overhypotheses shape their exploration?

Original: Developmental psychology has found that children are active and curious learners, with strong intrinsic motivation to systematically explore their environment (Schulz and Bonawitz, 2007; Schulz, 2012).
Revised: Developmental psychology has found that children are active and curious learners, with strong intrinsic motivation to systematically explore their environment (Schulz and Bonawitz, 2007; Schulz, 2012).

Original: Even young children engage in hypothesis-testing behavior in settings with ambiguous (Cook et al., 2011) or inconsistent (Legare, 2012) evidence.
Revised: Even young children engage in hypothesis-testing behavior in settings with ambiguous (Cook et al., 2011) or inconsistent (Legare, 2012) evidence.

Original: However, there is limited prior work on spontaneous causal exploration in children and none on how causal overhypotheses affect children’s exploration.
Revised: However, there is limited prior work on spontaneous causal exploration in children and none on how causal overhypotheses affect children’s exploration.

Original: In the current research, we seek to understand how young children, between the ages of four and six, explore and learn causal structure and how their exploration compares to that exhibited by typical ideal observer or RL models.
Revised: In this work, we seek to understand how young children, between the ages of four and six, explore and learn causal structure and how their exploration compares to that exhibited by typical ideal observer or RL models.

Original: In particular, we investigate how children use exploration to learn causal overhypotheses and how causal overhypotheses inﬂuence their exploration.
Revised: In particular, we investigate how children use exploration to learn causal overhypotheses and how causal overhypotheses influence their exploration.

Original: We ﬁrst show them speciﬁc sequences of interactions with the blicket machine, which are consistent with either a disjunctive, conjunctive, or ambiguous causal relationship.
Revised: We first show them specific sequences of interactions with the blicket machine, which are consistent with either a disjunctive, conjunctive, or ambiguous causal relationship.

Original: We then give them as much time as they want to experiment with a new blicket machine, to “ﬁgure out how to make it go.”
Revised: We then give them as much time as they want to experiment with a new blicket machine, to “figure out how to make it go.”

Original: We ﬁnd that children exhibit rich and diverse exploratory behavior, and that their overhypotheses are indeed inﬂuenced by their prior exposure to the environment.
Revised: We find that children exhibit rich and diverse exploratory behavior, and that their overhypotheses are indeed influenced by their prior exposure to the environment.

Original: In contrast, we consider an ideal learner with a particular causal overhypothesis and ﬁnd that its exploratory behavior in this task is quite different from that of children.
Revised: In contrast, we consider an ideal learner with a particular causal overhypothesis and find that its exploratory behavior in this task is quite different from that of children.

Original: We conclude by discussing how our ﬁndings can inform the development of
Revised: We conclude by discussing how our findings can inform the development of

Original: RL agents that are capable of causal exploration, learning and reasoning.
Revised: RL agents that are capable of causal exploration, learning and reasoning.

Revised: Our findings cannot be directly applied to existing RL algorithms, but rather may inspire entirely new directions of research.

Original: To summarize, the main contributions of this paper are as follows.
Revised: To summarize, the main contributions of this paper are as follows.

Original: 1) We develop an online environment based on the classical blicket machine (Gopnik and Sobel, 2000a) which can be used with both children and agents.
Revised: 1) We develop an online environment based on the classical blicket machine (Gopnik and Sobel, 2000a) which can be used with both children and agents.

Original: We collect experimental data from children in this environment, and demonstrate that they exhibit diverse structured exploration strategies, and that they learn from this exploration.
Revised: We collect experimental data from children in this environment, and demonstrate that they exhibit diverse structured exploration strategies, and that they learn from this exploration.

Original: We compare children’s exploration strategies with a set of simple ideal observer models, and show that children’s actions do not directly reﬂect either simple overhypothesis information gain or reward maximization.
Revised: We compare children’s exploration strategies with a set of simple ideal observer models, and show that children’s actions do not directly reflect either simple overhypothesis information gain or reward maximization.

Original: The results suggest that children rely on a broad set of causal assumptions and exploration behaviors that may generalize to many environments.
Revised: The results suggest that children rely on a broad set of causal assumptions and exploration behaviors that may generalize to many environments.

Original: This paves the way for future research which will enable artiﬁcial agents to exhibit richer, causally motivated, exploration strategies.
Revised: This paves the way for future research which will enable artificial agents to exhibit richer, causally motivated, exploration strategies.

Original: 2. Related Work
Revised: 2. Related Work

Original: The relevant previous work includes studies of exploration in reinforcement learning (RL), multitask RL, causal learning in RL, and various versions of the blicket environment used in cognitive science experiments.
Revised: The relevant previous work includes studies of exploration in reinforcement learning (RL), multitask RL, causal learning in RL, and various versions of the blicket environment used in cognitive science experiments.

Original: Exploration in reinforcement learning
Revised: Exploration in reinforcement learning

Original: Causal exploration is a relatively understudied area in reinforcement learning.
Revised: Causal exploration is a relatively understudied area in reinforcement learning.

Original: Most techniques do not consider explicit causal hypotheses, but instead rely on adding an exploration bonus to the task reward.
Revised: Most techniques do not consider explicit causal hypotheses, but instead rely on adding an exploration bonus to the task reward.

Original: This exploration bonus may be given for visiting novel states (Bellemare et al., 2016;
Revised: This exploration bonus may be given for visiting novel states (Bellemare et al., 2016;

Original: Ostrovski et al., 2017; Martin et al., 2017; Tang et al., 2017; Machado et al., 2018a), surprising dynamics (Schmidhuber, 1991; Pathak et al., 2017), uncertainty (Osband et al., 2016; Burda et al., 2018) or disagreement (Pathak et al., 2019).
Revised: Ostrovski et al., 2017; Martin et al., 2017; Tang et al., 2017; Machado et al., 2018a), surprising dynamics (Schmidhuber, 1991; Pathak et al., 2017), uncertainty (Osband et al., 2016; Burda et al., 2018) or disagreement (Pathak et al., 2019).

Original: Please refer to Amin et al.
Revised: Please refer to Amin et al.

Original: (2021) for a comprehensive survey of exploration in deep reinforcement learning.
Revised: (2021) for a comprehensive survey of exploration in deep reinforcement learning.

Original: Our proposed work using the blicket environment lays the groundwork for potential exploration algorithms based on the empirical exploration patterns of children in causal environments.
Revised: Our proposed work using the blicket environment lays the groundwork for potential exploration algorithms based on the empirical exploration patterns of children in causal environments.

Original: Multi-task learning in reinforcement learning
Revised: Multi-task learning in reinforcement learning

Original: There are several benchmarks for multi-task learning for robotics (Yu et al., 2019; James et al., 2020), for physical reasoning (Allen et al., 2020; Bakhtin et al., 2019) and video games (Cobbe et al., 2018;
Revised: There are several benchmarks for multi-task learning for robotics (Yu et al., 2019; James et al., 2020), for physical reasoning (Allen et al., 2020; Bakhtin et al., 2019) and video games (Cobbe et al., 2018;

Original: Machado et al., 2018b; Nichol et al., 2018; Chevalier-Boisvert et al., 2018).
Revised: Machado et al., 2018b; Nichol et al., 2018; Chevalier-Boisvert et al., 2018).

Original: The relevant causal overhypotheses for these environments are not clear, however, making it difﬁcult to evaluate the inﬂuence of causal information on agents’ exploration.
Revised: The relevant causal overhypotheses for these environments are not clear, however, making it difficult to evaluate the influence of causal information on agents’ exploration.

Original: In our work, we introduce a novel blicket environment, where the causal overhypotheses are clearly deﬁned.
Revised: In our work, we introduce a novel blicket environment, where the causal overhypotheses are clearly defined.

Original: Causal learning in reinforcement learning
Revised: Causal learning in reinforcement learning

Original: There are several standard reinforcement learning benchmarks and environments for causal discovery, including Causal World (Ahmed et al., 2020),
Revised: There are several standard reinforcement learning benchmarks and environments for causal discovery, including Causal World (Ahmed et al., 2020),

Original: Causal City (McDuff et al., 2021), Alchemy (Wang et al., 2021), ACRE (Zhang et al., 2021), and the work of Ke et al. (2021).
Revised: Causal City (McDuff et al., 2021), Alchemy (Wang et al., 2021), ACRE (Zhang et al., 2021), and the work of Ke et al. (2021).

Original: The causal hypotheses for many of these environments are not clear, or do not allow control over overhypotheses.
Revised: The causal hypotheses for many of these environments are not clear, or do not allow control over overhypotheses.

Original: Furthermore, most environments are concerned with causal induction or generalization, rather than focusing on exploration (though see Sontakke et al., 2021).
Revised: Furthermore, most environments are concerned with causal induction or generalization, rather than focusing on exploration (though see Sontakke et al., 2021).

Original: We instead focus on developing an environment with a controllable causal structure designed to allow us to measure the agents’ ability to explore using causal overhypotheses.
Revised: We instead focus on developing an environment with a controllable causal structure designed to allow us to measure the agents’ ability to explore using causal overhypotheses.

Original: Moreover, none of these environments have been used with children.
Revised: Moreover, none of these environments have been used with children.

Original: We aim to put agents and children in exactly the same environments, allowing us to use information about the real life causal exploration of these very effective child causal learners to inform agents.
Revised: We aim to put agents and children in exactly the same environments, allowing us to use information about the real life causal exploration of these very effective child causal learners to inform agents.

Original: 3. The Virtual “Blicket Detector” Environment
Revised: 3. The Virtual “Blicket Detector” Environment

Original: Although there is a large body of work on causal learning in children, and some experimental studies of active learning in the lab, as noted above, there are no studies analyzing children’s spontaneous actions as they freely explore the blicket machine, and none looking at how such exploration might lead to the learning of overhypotheses or how overhypotheses might inﬂuence learning.
Revised: Although there is a large body of work on causal learning in children, and some experimental studies of active learning in the lab, as noted above, there are no studies analyzing children’s spontaneous actions as they freely explore the blicket machine, and none looking at how such exploration might lead to the learning of overhypotheses or how overhypotheses might influence learning.

Original: We introduce a virtual, internet-hosted, representation of the standard blicket detector (Gopnik and Sobel, 2000a; Lucas et al., 2014) which is suitable not only for interaction with children, but also enables an exact comparison between children and reinforcement learning agents through an inferface with OpenAI Gym (Brockman et al., 2016).
Revised: We introduce a virtual, internet-hosted, representation of the standard blicket detector (Gopnik and Sobel, 2000a; Lucas et al., 2014) which is suitable not only for interaction with children, but also enables an exact comparison between children and reinforcement learning agents through an inferface with OpenAI Gym (Brockman et al., 2016).

Original: In particular, this environment allows us to precisely record and analyze both children’s and agent’s actions.
Revised: In particular, this environment allows us to precisely record and analyze both children’s and agent’s actions.

Original: Unlike previous experiments (Gopnik and Sobel, 2000a; Lucas et al., 2014) which were mainly conducted in person, the internetbased environment also allows for a more diverse set of participants.
Revised: Unlike previous experiments (Gopnik and Sobel, 2000a; Lucas et al., 2014) which were mainly conducted in person, the internet-based environment also allows for a more diverse set of participants.

Original: A visualization of the online blicket detector is shown in Figure 1.
Revised: A visualization of the online blicket detector is shown in Figure 1.

Original: We plan to open-source this environment upon publication.
Revised: We plan to open-source this environment upon publication.

Original: virtual blicket detector.

Original: Children can interact with the blicket detector through a touch interface on an iPad, or through a point and click interface on any web-enabled machine.

Original: Environment details
Revised: Environment details

Original: Observations are a 3D rendering ( 256 × 256 RGB pixels) of the detector and a set of objects (Figure 1).
Revised: Observations are a 3D rendering ( 256 × 256 RGB pixels) of the detector and a set of objects (Figure 1).

Original: In each episode, there are three objects, (which we refer to as A , B , and C for the rest of the paper) and the blicket detector present in the environment.
Revised: In each episode, there are three objects, (which we refer to as A , B , and C for the rest of the paper) and the blicket detector present in the environment.

Original: Each object in the environment has a unique color and shape.
Revised: Each object in the environment has a unique color and shape.

Original: The environment is Markovian, as all objects and blicket detector states are visible at any given time.
Revised: The environment is Markovian, as all objects and blicket detector states are visible at any given time.

Original: The children can put any combination of objects on the detector in any order, and click the check mark to test to see if its works or not.
Revised: The children can put any combination of objects on the detector in any order, and click the check mark to test to see if its works or not.

Original: It lights up and makes a sound when the correct set of objects (i.e., blickets) are placed on top, and object combinations are permutation invariant in our setup (though children do not necessarily have this prior, see Section 4).
Revised: It lights up and makes a sound when the correct set of objects (i.e., blickets) are placed on top, and object combinations are permutation invariant in our setup (though children do not necessarily have this prior, see Section 4).

Original: The required combinations of blickets to trigger the detector vary per episode and condition, according to a set of causal models.
Revised: The required combinations of blickets to trigger the detector vary per episode and condition, according to a set of causal models.

Original: The action space consists of seven discrete actions.
Revised: The action space consists of seven discrete actions.

Original: There are six actions for moving the objects (on/off for each of the three objects) and an additional action for pressing the check mark on the blicket detector, which checks if the existing objects make the blicket detector light up.
Revised: There are six actions for moving the objects (on/off for each of the three objects) and an additional action for pressing the check mark on the blicket detector, which checks if the existing objects make the blicket detector light up.

Original: Note, that the blicket detector will light up as long as a subset of the objects on the detector are blickets (e.g. putting three objects on the detector will also light up the detector).
Revised: Note, that the blicket detector will light up as long as a subset of the objects on the detector are blickets (e.g. putting three objects on the detector will also light up the detector).

Original: Causal overhypotheses
Revised: Causal overhypotheses

Original: Our environment setup consists of a hierarchical causal structure, where the higher level structure is a causal hypotheses that determines the number of objects needed to light-up the blicket detector, and the lower level describes which particular objects are blickets.
Revised: Our environment setup consists of a hierarchical causal structure, where the higher level structure is a causal hypotheses that determines the number of objects needed to light-up the blicket detector, and the lower level describes which particular objects are blickets.

Original: Similar to the setup in (Lucas et al., 2014), we consider two causal hypotheses in our environment:
Revised: Similar to the setup in (Lucas et al., 2014), we consider two causal hypotheses in our environment:

Original: CONJUNCTIVE and DISJUNCTIVE .
Revised: CONJUNCTIVE and DISJUNCTIVE .

Original: In the CONJUNCTIVE condition, a pair of blickets must be on the machine (together) to activate it.
Revised: In the CONJUNCTIVE condition, a pair of blickets must be on the machine (together) to activate it.

Original: In this case, A and B turn on the machine, but only when placed on the detector together so the only possible combinations that turn on the detector in the
Revised: In this case, A and B turn on the machine, but only when placed on the detector together so the only possible combinations that turn on the detector in the

Original: CONJUNCTIVE condition are AB , or ABC .
Revised: CONJUNCTIVE condition are AB , or ABC .

Original: The combinations that wouldn’t work are: A, B, C, AC and BC .
Revised: The combinations that wouldn’t work are: A, B, C, AC and BC .

Original: In the other condition or DISJUNCTIVE , A and B are blickets which individually turn the blicket machine on.
Revised: In the other condition or DISJUNCTIVE , A and B are blickets which individually turn the blicket machine on.

Original: Therefore, the following combinations tun on the detector in the DISJUNCTIVE condition: A, AB, ABC, B, BC and AC .
Revised: Therefore, the following combinations tun on the detector in the DISJUNCTIVE condition: A, AB, ABC, B, BC and AC .

Original: 4. Measuring Childrens’ Causal Exploration
Revised: 4. Measuring Childrens’ Causal Exploration

Original: We designed an experiment modeled on the blicket detector tasks (Gopnik and Sobel, 2000a) which allowed us to measure and analyze children’s exploration behavior and use of overhypotheses in an online environment that could also be given to an agent.
Revised: We designed an experiment modeled on the blicket detector tasks (Gopnik and Sobel, 2000a) which allowed us to measure and analyze children’s exploration behavior and use of overhypotheses in an online environment that could also be given to an agent.

Original: We tested N = 85 children aged 4-5 years (20-23 children per condition) at a local science museum following IRB protocols.
Revised: We tested N = 85 children aged 4-5 years (20-23 children per condition) at a local science museum following IRB protocols.

Revised: the exact script we used can be found in the appendix.

Original: Conditions
Revised: Conditions

Original: Our experimental setup consisted of a 2 × 2 design.
Revised: Our experimental setup consisted of a 2 × 2 design.

Original: Our ﬁrst tier of conditions were
Revised: Our first tier of conditions were

Original: CONJUNCTIVE and DISJUNCTIVE , reﬂecting different ways the detector could work, as described in Section 3.
Revised: CONJUNCTIVE and DISJUNCTIVE , reflecting different ways the detector could work, as described in Section 3.

Original: Children also received one of two forms of evidence about the blicket detector, either suggesting ( GIVEN HYPOTHESIS ) or failing to suggest ( NOT GIVEN HYPOTHESIS ) a relevant hypothesis space. 
Revised: Children also received one of two forms of evidence about the blicket detector, either suggesting ( GIVEN HYPOTHESIS ) or failing to suggest ( NOT GIVEN HYPOTHESIS ) a relevant hypothesis space. [Figure]

Original: Demonstration phase
Revised: Demonstration phase

Original: In both the DISJUNCTIVE and CONJUNCTIVE conditions, children ﬁrst saw a video of a live demonstration of two different blicket machines.
Revised: In both the DISJUNCTIVE and CONJUNCTIVE conditions, children first saw a video of a live demonstration of two different blicket machines.

Original: The machines had either a polka dot pattern or a stripe pattern and the objects varied in color and shape (sphere, pyramid and cube) as well, counterbalanced across conditions.
Revised: The machines had either a polka dot pattern or a stripe pattern and the objects varied in color and shape (sphere, pyramid and cube) as well, counterbalanced across conditions.

Original: The demonstrations for each machine and for each of the GIVEN HYPOTHESIS and NOT GIVEN HYPOTHESIS conditions are illustrated in Figure 2.
Revised: The demonstrations for each machine and for each of the GIVEN HYPOTHESIS and NOT GIVEN HYPOTHESIS conditions are illustrated in Figure 2.

Revised: First, in the testing portion of the experiment, we specifically chose colors and shapes that were different from those used in the demonstration phase, so that children could not directly apply attribute-based overhypotheses from the demonstration phase to the exploration phase.

Original: In the GIVEN HYPOTHESIS condition the children received evidence that the blicket detectors could work in either a conjunctive or disjunctive way.
Revised: In the GIVEN HYPOTHESIS condition the children received evidence that the blicket detectors could work in either a conjunctive or disjunctive way.

Original: In the NOT GIVEN HYPOTHESIS condition children only received ambiguous evidence about how the blicket machine might work, consistent with many overhypotheses.
Revised: In the NOT GIVEN HYPOTHESIS condition children only received ambiguous evidence about how the blicket machine might work, consistent with many overhypotheses.

Revised: To avoid biasing the children, we did not give them any other evidence about how the machine works, beyond the demonstration video.

Original: Exploration and test phase
Revised: Exploration and test phase

Original: After they watched the demonstrations, the children were shown a new detector with a checkerboard pattern and ring, triangle and half dome objects (as in Figure 1).
Revised: After they watched the demonstrations, the children were shown a new detector with a checkerboard pattern and ring, triangle and half dome objects (as in Figure 1).

Original: They were told, “Look, I have a 3rd blicket detector.
Revised: They were told, “Look, I have a 3rd blicket detector.

Original: It could work like the polka dot one, or it could work like the striped one.
Revised: It could work like the polka dot one, or it could work like the striped one.

Original: Can you ﬁgure out how it works and which blocks make it go, and make the detector go yourself?”
Revised: Can you figure out how it works and which blocks make it go, and make the detector go yourself?”

Original: After making the machine light up the ﬁrst time, the children were then asked “Great, is there something else you want to try?”.Once the child responded no to that [Figure] [Figure]
Revised: After making the machine light up the first time, the children were then asked “Great, is there something else you want to try?”.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] question they were asked two ﬁnal test questions.
Revised: Once the child responded no to that question they were asked two final test questions.

Original: First, they were asked whether each object was or was not a blicket, and then they were asked how the machine worked.
Revised: First, they were asked whether each object was or was not a blicket, and then they were asked how the machine worked.

Original: 4.1. Results
Revised: 4.1. Results

Original: A major aim of this study was as a proof of concept that young children would actively and intelligently explore in this online environment and treat it as a causal system, as they do with the real-life blicket machines.
Revised: A major aim of this study was as a proof of concept that young children would actively and intelligently explore in this online environment and treat it as a causal system, as they do with the real-life blicket machines.

Original: We see this study as itself exploratory, but broadly, we hypothesized that children would exhibit systematic causal exploration rather than exploring randomly, and, as a result, make correct causal inferences.
Revised: We see this study as itself exploratory, but broadly, we hypothesized that children would exhibit systematic causal exploration rather than exploring randomly, and, as a result, make correct causal inferences.

Original: Theory-driven exploration
Revised: Theory-driven exploration

Original: Following previous work on exploration in children (Schulz and
Revised: Following previous work on exploration in children (Schulz and

Original: Bonawitz, 2007; Schulz, 2012), we expected that children would not simply try to make the light go on, but that they would explore more extensively.
Revised: Bonawitz, 2007; Schulz, 2012), we expected that children would not simply try to make the light go on, but that they would explore more extensively.

Original: As shown in Table 1, children in all conditions took less than 30 seconds to activate the detector for the ﬁrst time but continued to explore for at  least several minutes more.
Revised: As shown in Table 1, children in all conditions took less than 30 seconds to activate the detector for the first time but continued to explore for at [Table] least several minutes more.

Original: Moreover, children tried fewer unique combinations of objects than the total number of checks (i.e., the number of times they pressed the check mark to test the effect of the combination), indicating that children tested some combinations multiple times.
Revised: Moreover, children tried fewer unique combinations of objects than the total number of checks (i.e., the number of times they pressed the check mark to test the effect of the combination), indicating that children tested some combinations multiple times.

Original: Note that if a child clicked the check mark multiple times without any action in between, only the ﬁrst time was counted and all subsequent checks were discarded.
Revised: Note that if a child clicked the check mark multiple times without any action in between, only the first time was counted and all subsequent checks were discarded.

Original: However, if the child performed any action, including taking all objects off and placing them back on in the same order, the second check would be included in our data.
Revised: However, if the child performed any action, including taking all objects off and placing them back on in the same order, the second check would be included in our data.

Original: The number of combinations refers to how many different sets of objects the child tried, even if children placed the objects on the machine in different orders, which they frequently did.
Revised: The number of combinations refers to how many different sets of objects the child tried, even if children placed the objects on the machine in different orders, which they frequently did.

Original: As we will discuss in Section 5.1, the observation that children tried the same com- [Figure] binations multiple times and that they varied the order of the objects might indicate that children are considering additional hypotheses which were not part of our initial analysis.
Revised: As we will discuss in Section 5.1, the observation that children tried the same combinations multiple times and that they varied the order of the objects might indicate that children are considering additional hypotheses which were not part of our initial analysis.

Original: Inferential success
Revised: Inferential success

Original: Prior work has shown that children can make correct causal inferences about blicket machines given sufﬁciently informative data (Lucas et al., 2014).
Revised: Prior work has shown that children can make correct causal inferences about blicket machines given sufficiently informative data (Lucas et al., 2014).

Original: We similarly hypothesized that children would successfully be able to distinguish blickets from non-blickets given the evidence they generated during exploration.
Revised: We similarly hypothesized that children would successfully be able to distinguish blickets from non-blickets given the evidence they generated during exploration.

Original: To test this, we compared how likely children were to report that true blickets are blickets (true positive rate) to how likely they are to report that non-blickets are blickets (false positive rate).
Revised: To test this, we compared how likely children were to report that true blickets are blickets (true positive rate) to how likely they are to report that non-blickets are blickets (false positive rate).

Original: The results are shown in Figure 3 (left).
Revised: The results are shown in Figure 3 (left).

Original: Across all conditions, children were more likely to report that blickets were blickets than that non-blickets were blickets.
Revised: Across all conditions, children were more likely to report that blickets were blickets than that non-blickets were blickets.

Original: While children were not perfectly able to determine the correct causal structure from the evidence they generated themselves, they showed some ability to do so across conditions.
Revised: While children were not perfectly able to determine the correct causal structure from the evidence they generated themselves, they showed some ability to do so across conditions.

Revised: We did not ask the children to identify if the detector was conjunctive or disjunctive because it is challenging to get a meaningful answer on questions like this from children, especially in the NOT GIVEN HYPOTHESIS condition where they are not introduced to the concepts of conjunctive and disjunctive.

Original: Effect of conjunctive vs. disjunctive
Revised: Effect of conjunctive vs. disjunctive

Original: Lucas et al.
Revised: Lucas et al.

Original: (2014) showed that, given sufﬁcient evidence, children are equally good at making causal inferences about conjunctive and disjunctive structures.
Revised: (2014) showed that, given sufficient evidence, children are equally good at making causal inferences about conjunctive and disjunctive structures.

Original: Similarly, we expected that children would be equally good at making inferences in the DISJUNCTIVE condition as the CONJUNCTIVE condition, and that they would exhibit similar amounts of exploration.
Revised: Similarly, we expected that children would be equally good at making inferences in the DISJUNCTIVE condition as the CONJUNCTIVE condition, and that they would exhibit similar amounts of exploration.

Original: To test this, we measured how long children explored (both in terms of time and number of actions taken) in the different conditions, as well as their success at discriminating between blickets and non-blickets.
Revised: To test this, we measured how long children explored (both in terms of time and number of actions taken) in the different conditions, as well as their success at discriminating between blickets and non-blickets.

Original: As reported in Table 1, Children took approximately the same amount of time in both the CONJUNCTIVE ( 185 . 84 seconds) and DISJUNCTIVE ( 187 . 19 seconds) conditions.
Revised: As reported in Table 1, Children took approximately the same amount of time in both the CONJUNCTIVE ( 185 . 84 seconds) and DISJUNCTIVE ( 187 . 19 seconds) conditions.

Original: The same is true for the number of actions they performed ( 12 . 45 in CONJUNCTIVE and 12 . 39 in DISJUNCTIVE ). 
Revised: The same is true for the number of actions they performed ( 12 . 45 in CONJUNCTIVE and 12 . 39 in DISJUNCTIVE ). In-

Original: Interestingly, even though it is easier to illuminate the blicket detector in the DISJUNCTIVE condition and children were faster at turning the blicket machine on for the ﬁrst time in the DISJUNCTIVE condition, their exploration of both conditions was similar.
Revised: [Figure] terestingly, even though it is easier to illuminate the blicket detector in the DISJUNCTIVE condition and children were faster at turning the blicket machine on for the first time in the DISJUNCTIVE condition, their exploration of both conditions was similar.

Original: We also looked at how well children could discriminate between blickets and non-blickets across the different conditions.
Revised: We also looked at how well children could discriminate between blickets and non-blickets across the different conditions.

Original: As illustrated in Figure 3 (left), there is little difference between the conditions (with the exception of GIVEN HYPOTHESIS ( DISJUNCTIVE ), which we discuss further below).
Revised: As illustrated in Figure 3 (left), there is little difference between the conditions (with the exception of GIVEN HYPOTHESIS ( DISJUNCTIVE ), which we discuss further below).

Original: Taken together, these results indicate that the true causal structure of the blicket detector does not substantially inﬂuence how much children explore or how likely they are to come to a correct answer, consistent with earlier results (Lucas et al., 2014).
Revised: Taken together, these results indicate that the true causal structure of the blicket detector does not substantially influence how much children explore or how likely they are to come to a correct answer, consistent with earlier results (Lucas et al., 2014).

Original: Hypothesis space effects
Revised: Hypothesis space effects

Original: If an actor has evidence about the hypothesis space, the actor should be more efﬁcient in the exploration of that space, and causal inference should be an easier task.
Revised: If an actor has evidence about the hypothesis space, the actor should be more efficient in the exploration of that space, and causal inference should be an easier task.

Original: Thus, we hypothesized that children in the GIVEN HYPOTHESIS conditions would explore less and be more accurate in their inferences than in the NOT GIVEN HYPOTHESIS condition.
Revised: Thus, we hypothesized that children in the GIVEN HYPOTHESIS conditions would explore less and be more accurate in their inferences than in the NOT GIVEN HYPOTHESIS condition.

Original: First, to measure the amount of exploration that children performed, we looked at the amount of time they took and the number of actions they explored, as given in Table 1 and Figure S.1.
Revised: First, to measure the amount of exploration that children performed, we looked at the amount of time they took and the number of actions they explored, as given in Table 1 and Figure S.1.

Original: When collapsing across causal structures, the results did not entirely align with our hypotheses: on average children tried more actions in a shorter time in the GIVEN HYPOTHESIS condition ( 178 .
Revised: When collapsing across causal structures, the results did not entirely align with our hypotheses: on average children tried more actions in a shorter time in the GIVEN HYPOTHESIS condition ( 178 .

Original: 65 seconds, 14 . 07 actions) than in the NOT GIVEN HYPOTHESIS condition ( 195 . 38 seconds, 10 .
Revised: 65 seconds, 14 . 07 actions) than in the NOT GIVEN HYPOTHESIS condition ( 195 . 38 seconds, 10 .

Original: 58 actions).
Revised: 58 actions).

Original: Second, we examined if the data generated during exploration was sufﬁcient to disambiguate the hypothesis space.
Revised: Second, we examined if the data generated during exploration was sufficient to disambiguate the hypothesis space.

Original: Figure 3 (right) indicates that children were substantially more likely to do so in the GIVEN HYPOTHESIS conditions ( 71% of the time) than in the NOT GIVEN HYPOTHESIS conditions ( 32% of the time, p = 0 .
Revised: Figure 3 (right) indicates that children were substantially more likely to do so in the GIVEN HYPOTHESIS conditions ( 71% of the time) than in the NOT GIVEN HYPOTHESIS conditions ( 32% of the time, p = 0 .

Original: 002 using a standard 2-proportion Z test).
Revised: 002 using a standard 2-proportion Z test).

Original: Although children only tried a few more combinations more in the GIVEN HYPOTHESIS conditions, this exploration was more effective.
Revised: Although children only tried a few more combinations more in the GIVEN HYPOTHESIS conditions, this exploration was more effective.

Original: This result is consistent with the hypothesis that without guidance, children may explore unbounded sets of overhypotheses, which may not be useful for discriminating between conjunctive and disjunctive structures.
Revised: This result is consistent with the hypothesis that without guidance, children may explore unbounded sets of overhypotheses, which may not be useful for discriminating between conjunctive and disjunctive structures.

Original: Third, we looked at tests that might naturally occur to children but should be ruled out in the
Revised: Third, we looked at tests that might naturally occur to children but should be ruled out in the

Original: NOT GIVEN HYPOTHESIS condition.
Revised: NOT GIVEN HYPOTHESIS condition.

Original: The ﬁrst is to test whether the machine would turn on without any objects on it.
Revised: The first is to test whether the machine would turn on without any objects on it.

Original: 7% and 17 .
Revised: 7% and 17 .

Original: 5% of children performed this test in the GIVEN HYPOTHESIS and
Revised: 5% of children performed this test in the GIVEN HYPOTHESIS and

Original: NOT GIVEN HYPOTHESIS conditions, respectively.
Revised: NOT GIVEN HYPOTHESIS conditions, respectively.

Original: The second test is to try different orderings of the same set of objects (our environment preserves the order in which the the objects are put on top of the blicket detector).
Revised: The second test is to try different orderings of the same set of objects (our environment preserves the order in which the the objects are put on top of the blicket detector).

Original: 7%8 and 65% of children tried different orderings of the same set of objects in the GIVEN HYPOTHESIS and NOT GIVEN HYPOTHESIS conditions, respectively.
Revised: 7%8 and 65% of children tried different orderings of the same set of objects in the GIVEN HYPOTHESIS and NOT GIVEN HYPOTHESIS conditions, respectively.

Original: Both of the tests mentioned above are reasonable for children in the NOT GIVEN HYPOTHESIS condition; however, the children in the GIVEN HYPOTHESIS condition could have reasoned that the blicket detector would not turn on without any objects, and that the ordering of objects does not matter.
Revised: Both of the tests mentioned above are reasonable for children in the NOT GIVEN HYPOTHESIS condition; however, the children in the GIVEN HYPOTHESIS condition could have reasoned that the blicket detector would not turn on without any objects, and that the ordering of objects does not matter.

Original: Thus it is surprising that children in both conditions performed similarly.
Revised: Thus it is surprising that children in both conditions performed similarly.

Original: Finally, we looked at whether children’s inferences about blickets and non-blickets were affected by being given the hypothesis space or not.
Revised: Finally, we looked at whether children’s inferences about blickets and non-blickets were affected by being given the hypothesis space or not.

Original: As shown in Figure 3 (left), there were generally no clear differences between conditions, again with the exception of the GIVEN HYPOTHESIS ( DISJUNCTIVE ) condition in which children were substantially more likely to correctly identify blickets.
Revised: As shown in Figure 3 (left), there were generally no clear differences between conditions, again with the exception of the GIVEN HYPOTHESIS ( DISJUNCTIVE ) condition in which children were substantially more likely to correctly identify blickets.

Original: Although they were not more likely to generate sufﬁcient data in this condition (Figure 3, right), they did try substantially more actions (Table 1).
Revised: Although they were not more likely to generate sufficient data in this condition (Figure 3, right), they did try substantially more actions (Table 1).

Original: Summary
Revised: Summary

Original: Overall, our results suggest that children are able to effectively explore—particularly when given the relevant hypothesis space—and that they are often able to correctly identify blickets.
Revised: Overall, our results suggest that children are able to effectively explore—particularly when given the relevant hypothesis space—and that they are often able to correctly identify blickets.

Original: However, children did not always generate sufﬁcient evidence and could not perfectly discriminate between blickets and non-blickets (Figure 3).
Revised: However, children did not always generate sufficient evidence and could not perfectly discriminate between blickets and non-blickets (Figure 3).

Original: One explanation for this ﬁnding could be that the children were acting optimally (with noise); however, we favor another explanation: that children were optimizing for a wider range of alternative hypotheses.
Revised: One explanation for this finding could be that the children were acting optimally (with noise); however, we favor another explanation: that children were optimizing for a wider range of alternative hypotheses.

Original: To differentiate between these explanations, in the next section we compare the results of children with several optimal models.
Revised: To differentiate between these explanations, in the next section we compare the results of children with several optimal models.

Original: 5. Modeling Causal Learning in the Blicket Environment
Revised: 5. Modeling Causal Learning in the Blicket Environment

Original: Given our experimental results, we aimed to characterize the optimal behavior for the blicket detector task, and used this optimal behavior to interpret the children’s decisions.
Revised: Given our experimental results, we aimed to characterize the optimal behavior for the blicket detector task, and used this optimal behavior to interpret the children’s decisions.

Original: As a ﬁrst pass at understanding the motivations and priors of the children, we leveraged the relatively simple causal structure of the blicket environment to build several policies, which we used as a baseline for the children’s behavior.
Revised: As a first pass at understanding the motivations and priors of the children, we leveraged the relatively simple causal structure of the blicket environment to build several policies, which we used as a baseline for the children’s behavior.

Original: Our approach is similar to previous work (Oaksford and Chater, 1994; Nelson et al., 2010; Coenen et al., 2019) which has taken a Bayesian approach to optimal data selection and used such a model to describe adult behavior for various causal reasoning tasks.
Revised: Our approach is similar to previous work (Oaksford and Chater, 1994; Nelson et al., 2010; Coenen et al., 2019) which has taken a Bayesian approach to optimal data selection and used such a model to describe adult behavior for various causal reasoning tasks.

Original: We used the uniﬁed environment described in Section 3, and the same experimental setup.
Revised: We used the unified environment described in Section 3, and the same experimental setup.

Original: Here, one action corresponds to putting any number of blickets on the detector and pressing the check button.
Revised: Here, one action corresponds to putting any number of blickets on the detector and pressing the check button.

Original: For each action there are two possible observations (blicket machine turns on or remains off).
Revised: For each action there are two possible observations (blicket machine turns on or remains off).

Original: Policy Design
Revised: Policy Design

Original: In the blicket game, there are eleven possible causal structures, seven of which are disjunctive ( A -dis, B -dis, C -dis, AB -dis, AC -dis, BC -dis, and ABC -dis, where “ A -dis” refers to a disjunctive structure where A is a blicket and B and C are not), and four of which are conjunctive ( AB -con, AC -con, BC -con, and ABC -con).
Revised: In the blicket game, there are eleven possible causal structures, seven of which are disjunctive ( A -dis, B -dis, C -dis, AB -dis, AC -dis, BC -dis, and ABC -dis, where “ A -dis” refers to a disjunctive structure where A is a blicket and B and C are not), and four of which are conjunctive ( AB -con, AC -con, BC -con, and ABC -con).

Original: We excluded  causal structures such asA -con and
Revised: We excluded degenerate causal structures such as

Original:  B -con where no observations can distinguish between them.
Revised: A -con and B -con where no observations can distinguish between them, since conjunctive detectors require two blickets to light up.

Original: The goal of our models was to determine the causal model through intervention in the environment.
Revised: The goal of our models was to determine the causal model through intervention in the environment.

Original: In this work, we investigated two policies through which we explored optimal behavior in the state space: a policy based on per-step information gain maximization (the PER - STEP model), and a policy which minimized the expected time to full disambiguation of the hypothesis space (the MINIMUM - STEP model).
Revised: In this work, we investigated two policies through which we explored optimal behavior in the state space: a policy based on per-step information gain maximization (the PER - STEP model), and a policy which minimized the expected time to full disambiguation of the hypothesis space (the MINIMUM - STEP model).

Original: PER - STEP model
Revised: PER - STEP model

Original: The ﬁrst policy optimizes for the expected per-action information gain, and aims to optimally discriminate between a set of ﬁxed hypotheses through its actions by taking actions which minimize the uncertainty in the conditional posterior.
Revised: The first policy optimizes for the expected per-action information gain, and aims to optimally discriminate between a set of fixed hypotheses through its actions by taking actions which minimize the uncertainty in the conditional posterior.

Original: We measured uncertainty over all hypotheses using KL-divergence between the hypothesis distribution over causal hypotheses and some current prior distribution.
Revised: We measured uncertainty over all hypotheses using KL-divergence between the hypothesis distribution over causal hypotheses and some current prior distribution.

Original: More formally, let h ∈ H be the possible hypotheses considered by this policy.
Revised: More formally, let h ∈ H be the possible hypotheses considered by this policy.

Original: The “usefulness” of a particular observation, o , resulting from an action a , is given by the difference between the posterior and prior distribution: D KL ( p ( h | o ) (cid:107) p ( h )) .
Revised: The “usefulness” of a particular observation, o , resulting from an action a , is given by the difference between the posterior and prior distribution: D KL ( p ( h | o ) ∥ p ( h )) .

Original: The posterior probability for each individual hypothesis h is given by Bayes’ rule: p ( h | o )
Revised: The posterior probability for each individual hypothesis h is given by Bayes’ rule: p ( h | o )

Original: =
Revised: =

Original: p ( o | h ) p ( h ) /p
Revised: p ( o | h ) p ( h ) /p

Original: ( o ) .
Revised: ( o ) .

Original: Before the ﬁrst action, the prior distribution p ( o ) is initialized using one of two possible choices (see below).
Revised: Before the first action, the prior distribution p ( o ) is initialized using one of two possible choices (see below).

Original: For modeling multiple actions in sequence, the prior is deﬁned to be to the posterior of the previous timestep, and actions are selected by maximizing the sum of the per-timestep divergences over the sequence.
Revised: For modeling multiple actions in sequence, the prior is defined to be to the posterior of the previous timestep, and actions are selected by maximizing the sum of the per-timestep divergences over the sequence.

Original: The PER - STEP model returns action sequences that are optimal for all possible ground-truth states of the blicket detector (i.e. A and B are blickets and the detector is conjunctive,
Revised: The PER - STEP model returns action sequences that are optimal for all possible ground-truth states of the blicket detector (i.e. A and B are blickets and the detector is conjunctive,

Original: A and C are blickets and the detector is disjunctive, etc.).
Revised: A and C are blickets and the detector is disjunctive, etc.).

Original: MINIMUM - STEP model
Revised: MINIMUM - STEP model

Original: Instead of greedily optimizing for disambiguation as in the PER - STEP model, the second policy that we explored (the MINIMUM - STEP model) optimizes for the minimum number of steps required to fully disambiguate the hypothesis space.
Revised: Instead of greedily optimizing for disambiguation as in the PER - STEP model, the second policy that we explored (the MINIMUM - STEP model) optimizes for the minimum number of steps required to fully disambiguate the hypothesis space.

Original: To determine this policy, we modeled the possible hypothesis space as a tree search problem, where each vertex represents the possible causal hypotheses, and each edge represents an action.
Revised: To determine this policy, we modeled the possible hypothesis space as a tree search problem, where each vertex represents the possible causal hypotheses, and each edge represents an action.

Original: The policy selects sequences of actions which minimize the expected depth of the tree rooted at each vertex.
Revised: The policy selects sequences of actions which minimize the expected depth of the tree rooted at each vertex.

Original: Unlike the PER - STEP model, this policy actively attempts to minimize the number of steps that are required for resolution.
Revised: Unlike the PER - STEP model, this policy actively attempts to minimize the number of steps that are required for resolution.

Original: Choice of prior
Revised: Choice of prior

Original: We considered two possible prior beliefs for our models.
Revised: We considered two possible prior beliefs for our models.

Original: The ﬁrst prior, the
Revised: The first prior, the

Original: UNIFORM prior, posits that that all causal structures are equally likely.
Revised: UNIFORM prior, posits that that all causal structures are equally likely.

Original: However, in the GIVEN
Revised: However, in the GIVEN

Original: HYPOTHESIS condition, we implied equal likelihood between the CONJUNCTIVE and DISJUNCTIVE overhypotheses, and our experiments with children only contained causal structures with two blickets.
Revised: HYPOTHESIS condition, we implied equal likelihood between the CONJUNCTIVE and DISJUNCTIVE overhypotheses, and our experiments with children only contained causal structures with two blickets.

Original: Therefore, children could reasonably be expected to follow a different prior with three conjunctive structures and three disjunctive structures, are of which are equally likely.
Revised: Therefore, children could reasonably be expected to follow a different prior with three conjunctive structures and three disjunctive structures, are of which are equally likely.

Original: We encoded this possibility in a prior (the EXPERIMENTAL prior) which placed equal weight on conjunctive and disjunctive hypotheses.
Revised: We encoded this possibility in a prior (the EXPERIMENTAL prior) which placed equal weight on conjunctive and disjunctive hypotheses.

Original: 5.1. Results, Analysis and Comparison to Children’s Exploration
Revised: 5.1. Results, Analysis and Comparison to Children’s Exploration

Original: The PER - STEP model and the MINIMUM - STEP models generated subtly different strategies for playing the game.
Revised: The PER - STEP model and the MINIMUM - STEP models generated subtly different strategies for playing the game.

Original: In all experiments, the causal structure (or type of causal structure) was not known a priori and the policies had to determine both which causal structure (conjunctive vs. disjunctive) was present, and which objects in the scene were blickets.
Revised: In all experiments, the causal structure (or type of causal structure) was not known a priori and the policies had to determine both which causal structure (conjunctive vs. disjunctive) was present, and which objects in the scene were blickets.

Original: Because of the relatively small policy space, we could succinctly describe the policies, and Figure S.2, S.3, S.4, and S.5 in the supplementary materials demonstrate the optimal strategies learned by each model.
Revised: Because of the relatively small policy space, we could succinctly describe the policies, and Figure S.2, S.3, S.4, and S.5 in the supplementary materials demonstrate the optimal strategies learned by each model.

Original: As expected, the MINIMUM - STEP agent optimizing for the minimum number of steps obtained a policy which is slightly better than PER - STEP , with MINIMUM - STEP achieving an expected time to full information of 3 .
Revised: As expected, the MINIMUM - STEP agent optimizing for the minimum number of steps obtained a policy which is slightly better than PER - STEP , with MINIMUM - STEP achieving an expected time to full information of 3 .

Original: 55 actions under the UNIFORM prior, and 3 .
Revised: 55 actions under the UNIFORM prior, and 3 .

Original: 50 actions under the EXPERIMENTAL prior, while PER - STEP achieved 3 .
Revised: 50 actions under the EXPERIMENTAL prior, while PER - STEP achieved 3 .

Original: 72 expected actions to full information under the UNIFORM prior, and 4 . 0 expected actions under the EXPERIMENTAL prior.
Revised: 72 expected actions to full information under the UNIFORM prior, and 4 . 0 expected actions under the EXPERIMENTAL prior.

Original: It is interesting to note that the PER - STEP  model performed worse when given incorrect information (the EXPERIMENTAL prior), while the
Revised: It is interesting to note that the PER - STEP [Figure] model performed worse when given incorrect information (the EXPERIMENTAL prior), while the

Original: MINIMUM - STEP model is able to perform better in this scenario (the downside to the MINIMUMSTEP being the fact that it is usually intractable to compute in practice).
Revised: MINIMUM - STEP model is able to perform better in this scenario (the downside to the MINIMUMSTEP being the fact that it is usually intractable to compute in practice).

Original: Next, we contrasted the models’ behavior with that of the children.
Revised: Next, we contrasted the models’ behavior with that of the children.

Original: As shown in Figure 4 (left), children always took more steps than the agents before they ﬁnished exploring the environment.
Revised: As shown in Figure 4 (left), children always took more steps than the agents before they finished exploring the environment.

Original: Surprisingly, children did not show any notable difference between GIVEN HYPOTHESIS ( CONJUNCTIVE ) and GIVEN HYPOTHESIS ( DISJUNCTIVE ), whereas the agents differed signiﬁcantly, with both optimal methods taking longer to explore for conjunctive spaces (note that the agents have no variance in expected time to goal, as these numbers are analytic).
Revised: Surprisingly, children did not show any notable difference between GIVEN HYPOTHESIS ( CONJUNCTIVE ) and GIVEN HYPOTHESIS ( DISJUNCTIVE ), whereas the agents differed significantly, with both optimal methods taking longer to explore for conjunctive spaces (note that the agents have no variance in expected time to goal, as these numbers are analytic).

Original: This increase in required actions for conjunctive causal structures demonstrates the bias towards disjunctive structures in the environment, as there are more possible disjunctive than conjunctive structures (7 vs. 4).
Revised: This increase in required actions for conjunctive causal structures demonstrates the bias towards disjunctive structures in the environment, as there are more possible disjunctive than conjunctive structures (7 vs. 4).

Original: While the agents always took enough actions in the space to fully disambiguate the causal structure, the children often did not take enough actions, especially when they were not given any information about the hypothesis space.
Revised: Even though children are explicitly encouraged to determine how the machine functions, while the agents always took enough actions in the space to fully disambiguate the causal structure, the children often did not take enough actions, especially when they were not given any information about the hypothesis space.

Original: Figure 4 (right) shows the percentage of children who took enough actions to fully disambiguate the space.
Revised: Figure 4 (right) shows the percentage of children who took enough actions to fully disambiguate the space.

Original: Unsurprisingly, when in GIVEN HYPOTHESIS , the children were far more likely to fully disambiguate the environment, demonstrating the power of fully guided exploration.
Revised: Unsurprisingly, when in GIVEN HYPOTHESIS , the children were far more likely to fully disambiguate the environment, demonstrating the power of fully guided exploration.

Original: There is little difference between the CONJUNCTIVE and DISJUNCTIVE conditions when it comes to exploration, suggesting that the children explored equally in both conditions, and did not favor disambiguation of conjunctive vs. disjunctive environments.
Revised: There is little difference between the CONJUNCTIVE and DISJUNCTIVE conditions when it comes to exploration, suggesting that the children explored equally in both conditions, and did not favor disambiguation of conjunctive vs. disjunctive environments.

Revised: This effect raises an interesting question: unlike RL agents, we do not have any explicit understanding of the reward function that children are optimizing during training time, and even though an effort was made to encourage the children to optimize the reward function of maximum information gain, the children may not have been optimizing this reward alone.

Revised: A key contribution of this work is to raise this question: can we discover in future work what kinds of reward functions children optimize during overhypothesis discovery, and can we leverage these functions to build more powerful machine learning models?

Original: Further analyses of children’s exploration
Revised: Further analyses of children’s exploration

Original: As the previous results show, children’s exploration behavior was not well-captured by the proposed optimal models.
Revised: As the previous results show, children’s exploration behavior was not well-captured by the proposed optimal models.

Original: By qualitatively examining children’s behavior in more detail, we observed that they appeared to be testing many different kinds of overhypotheses beyond the ground-truth conjunctive and disjunctive structures.
Revised: By qualitatively examining children’s behavior in more detail, we observed that they appeared to be testing many different kinds of overhypotheses beyond the ground-truth conjunctive and disjunctive structures.

Original: Figure 5 shows some example sequences of the combinations the children tried, indicating the various hypotheses[Figure] they were testing.
Revised: Figure 5 shows some example sequences of the combinations the children tried, indicating the various hypotheses they were testing.

Original: These qualitative examples suggest that children have a richer and larger hypothesis space than the models.
Revised: These qualitative examples suggest that children have a richer and larger hypothesis space than the models.

Original: For example, Participant 1 seemed to only consider the possibility that all three objects must trigger the detector, but attempted to determine if the order in which blickets are placed on the detector is important, as well as if the detector might be stochastic.
Revised: For example, Participant 1 seemed to only consider the possibility that all three objects must trigger the detector, but attempted to determine if the order in which blickets are placed on the detector is important, as well as if the detector might be stochastic.

Original: Participant 2’s behavior was consistent with trying disjunctive conditions ﬁrst, and then conjunctive conditions after that: a good example of systematic testing.
Revised: Participant 2’s behavior was consistent with trying disjunctive conditions first, and then conjunctive conditions after that: a good example of systematic testing.

Original: Participant 3 only tested conjunctive behavior (with some testing of order).
Revised: Participant 3 only tested conjunctive behavior (with some testing of order).

Original: Participant 4 is another example where the child quickly discovered the causal structure, but then tested order and stochasticity
Revised: Participant 4 is another example where the child quickly discovered the causal structure, but then tested order and stochasticity.

Revised: In addition to causal structure, order, and stochasticity, childrens’ overhypotheses may also take into account the attributes of objects, for example that blue objects are blickets or circular objects are blickets.

Revised: Taking into account attributes would exponentially increase the size of the overhypothesis space, so we designed our experiment to try to minimize the likelihood that children would consider attribute-based overhypotheses.

Revised: First, we used distinctly different colors and shapes in the exploration phase, compared to in the demonstration phase, so that children would not be able to directly apply attribute-based overhypotheses from the demonstration phase to the exploration phase.

Revised: In addition, for each child we randomized which two of the three objects were blickets, in both the demonstration and exploration phases.

Original: Unlikethe behavior displayed by children, the optimal policies do not consider that the blicket detector may be stochastic or that there may be additional hypotheses outside of conjunctive and disjunctive.
Revised: Unlike children, the optimal policies do not consider that the blicket detector may be stochastic or that there may be additional hypotheses outside of conjunctive and disjunctive.

Original: Determining a set of hypotheses that more accurately represents what the children actually consider would lead to a model which is more predictive of the children’s behavior.
Revised: Determining a set of hypotheses that more accurately represents what the children actually consider would lead to a model which is more predictive of the children’s behavior.

Original: 6. Discussion & Conclusion
Revised: 6. Discussion & Conclusion

Original: Exploration—and in particular, causal exploration—remains a challenge for modern RL agents.
Revised: Exploration—and in particular, causal exploration—remains a challenge for modern RL agents.

Original: In this paper, we investigated how human children engage in this type of exploration in order to draw insights that can be applied to artiﬁcial agents as well.
Revised: In this paper, we investigated how human children engage in this type of exploration in order to draw insights that can be applied to artificial agents as well.

Original: To do so, we both introduced a novel “blicket” environment which enables testing exploration strategies of both humans and agents, and collected data on young children to observe how they explore in this domain.
Revised: To do so, we both introduced a novel “blicket” environment which enables testing exploration strategies of both humans and agents, and collected data on young children to observe how they explore in this domain.

Original: We found that children exhibited a diverse set of exploration strategies not well captured by na¨ıve models optimizing for a small set of known overhypotheses.
Revised: We found that children exhibited a diverse set of exploration strategies not well captured by na¨ıve models optimizing for a small set of known overhypotheses.

Original: Notably, from our results it seems that children bring to bear extensive prior knowledge regarding how objects and mechanisms behave, and use this knowledge to generate a wide range of potential hypotheses.
Revised: Notably, from our results it seems that children bring to bear extensive prior knowledge regarding how objects and mechanisms behave, and use this knowledge to generate a wide range of potential hypotheses.

Original: These results have important implications for research aimed at developing RL agents.
Revised: These results have important implications for research aimed at developing RL agents.

Original: Most importantly, they suggest that effective exploration requires rich prior knowledge structured via causal overhypotheses.
Revised: Most importantly, they suggest that effective exploration requires rich prior knowledge structured via causal overhypotheses.

Original: While in our experiments children’s behavior may technically be “suboptimal” in the context of simply disambiguating conjunctive vs. disjunctive hypotheses, in more realistic environments, children spend their time exploring an incredibly vast range of phenomena: block towers, bugs in the forest, digital devices, pets, social dynamics, and so on.
Revised: While in our experiments children’s behavior may technically be “suboptimal” in the context of simply disambiguating conjunctive vs. disjunctive hypotheses, in more realistic environments, children spend their time exploring an incredibly vast range of phenomena: block towers, bugs in the forest, digital devices, pets, social dynamics, and so on.

Original: To effectively explore in such a broad range of domains, children need to leverage rich prior knowledge; our experiments reﬂect that they do this by default even in simple settings.
Revised: To effectively explore in such a broad range of domains, children need to leverage rich prior knowledge; our experiments reflect that they do this by default even in simple settings.

Original: If we want agents that can perform a similar breadth of exploration, they similarly need to bring to bear rich causal knowledge about the world.
Revised: If we want agents that can perform a similar breadth of exploration, they similarly need to bring to bear rich causal knowledge about the world.

Original: Moreover, it may not be sufﬁcient to have a large “bag-of-hypotheses”: like children, agents may need to organize their knowledge into hierarchical representations like overhypotheses in order to swiftly narrow in on the relevant subset of causal structures.
Revised: Moreover, it may not be sufficient to have a large “bag-of-hypotheses”: like children, agents may need to organize their knowledge into hierarchical representations like overhypotheses in order to swiftly narrow in on the relevant subset of causal structures.

Original: Although we have only presented results here for simple optimal models, these results pose a clear set of questions regarding RL, in particular: what is required for RL agents to exhibit exploration strategies similar to those children use?
Revised: Although we have only presented results here for simple optimal models, these results pose a clear set of questions regarding RL, in particular: what is required for RL agents to exhibit exploration strategies similar to those children use?

Original: There are two challenges: what reward function should be used to train such agents, and in what environments?
Revised: There are two challenges: what reward function should be used to train such agents, and in what environments?

Original: A maximally “blank-slate” approach to our task might be: on each episode, sample whether the agent is in the CONJUNCTIVE or
Revised: A maximally “blank-slate” approach to our task might be: on each episode, sample whether the agent is in the CONJUNCTIVE or

Original: DISJUNCTIVE condition, and then reward it for making the blicket detector turn on.
Revised: DISJUNCTIVE condition, and then reward it for making the blicket detector turn on.

Original: Unfortunately, this training regime would not result in exploratory behavior at all: the optimal solution here would be to always just put all blocks on the detector.
Revised: Unfortunately, this training regime would not result in exploratory behavior at all: the optimal solution here would be to always just put all blocks on the detector.

Original: Alternatively, we could consider rewarding agents for correctly identifying blickets, however this strategy would just demonstrate identical behavior to our optimal models (Dasgupta et al., 2019; Mikulik et al., 2020).
Revised: Alternatively, we could consider rewarding agents for correctly identifying blickets, however this strategy would just demonstrate identical behavior to our optimal models (Dasgupta et al., 2019; Mikulik et al., 2020).

Original: We might consider training agents on a broader set of overhypotheses (for example, that order matters, that the detector is stochastic, etc.), but it is unclear what this set should be, or how to avoid it being overly task-speciﬁc.
Revised: We might consider training agents on a broader set of overhypotheses (for example, that order matters, that the detector is stochastic, etc.), but it is unclear what this set should be, or how to avoid it being overly task-specific.

Original: Such “blank-slate” and “constructed” approaches, do not appear consistent with how children learn about the world or how humans have developed over the course of evolution.
Revised: Such “blank-slate” and “constructed” approaches, do not appear consistent with how children learn about the world or how humans have developed over the course of evolution.

Original: We believe that developing environments and training regimes to tackle these questions is an essential aspect of future work in building agents that can efﬁciently and effectively explore. 
Revised: We believe that developing environments and training regimes to tackle these questions is an essential aspect of future work in building agents that can efficiently and effectively explore. Acknowledgments

Revised: We would like to thank the research assistants who conducted the research: Azzaya Munkhbat, Eli

Revised: Phipps, Zane Levine and Michael Larosa.

Revised: Supplementary Material - Learning Casual Overhypotheses through Exploration in Children and Computational Models

Revised: Below is the exact testing script we use with the children per condition:

Revised: Experimental Script:

Revised: During the training phases the children see a video demonstration for how two blicket detectors work, and they are pre-recorded for consistency.

Revised: We have attached the links here, but cropped out the portion where you can see the researcher giving instructions to keep the submission double-blind.

Revised: We will also include this in the supplementary material for the camera-ready.

Revised: Given hypothesis space condition: Part 1: Training: https://drive.google.com/file/ d/1Z8bLzXjHSdXQL-0ap3OcpSq4HDPuhU5D/view?usp

Revised: =sharing

Revised: And this one since the order is counterbalanced https://drive.google.com/file/d/1CGdxi_VBpVuiq-VNUtU_ ogynSqkOwFfm/view?usp

Revised: =sharing

Revised: Part 2: Testing: “Look, I have a third blicket detector.

Revised: It could work like the polka dot one, or it could work like the striped one.

Revised: Can you figure out how it works and make the detector go, and which blocks make it go?”

Revised: “Great!

Revised: Is there something else you want to try?”

Revised: Repeat this until they say “no” Stop them after 10 minutes.

Revised: Then ask which objects are blickets by pointing to each one, then clicking the star button.

Revised: “Is this object a blicket?”

Revised: (point to leftmost) “Is this object a blicket?”

Revised: (point to middle) “Is this object a blicket?”

Revised: (point to rightmost) “Now click the shapes that will make the machine go.”

Revised: “Now that you tried it, can you tell me how this machine works?”

Revised: Not given hypothesis condition: Part 1: Training: https://drive.google.com/file/ d/1DhuYXZ48BuuT92l2Iq29NAzxjLB6s5dV/view?usp

Revised: =sharing

Revised: Part

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Detectoron

Revised: [Figure]

Revised: Detectoroff

Revised: [Figure]

Revised: Detectoroff

Revised: [Figure]

Revised: [Figure]