Original: G ROWING A CTION S
Revised: G ROWING A CTION S

Original: PACES
Revised: PACES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefﬁcient to achieve meaningful learning progress.
Revised: In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefﬁcient to achieve meaningful learning progress.

Original: In this work, we use a curriculum of progressively growing action spaces to accelerate learning.
Revised: In this work, we use a curriculum of progressively growing action spaces to accelerate learning.

Original: We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space.
Revised: We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space.

Original: Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efﬁciently transfers data, value estimates, and state representations from restricted action spaces to the full task.
Revised: Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efﬁciently transfers data, value estimates, and state representations from restricted action spaces to the full task.

Original: We show the efﬁcacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large, multi-agent action spaces.
Revised: We show the efﬁcacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large, multi-agent action spaces.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The value of curricula has been well established in machine learning, reinforcement learning, and in biological systems.
Revised: The value of curricula has been well established in machine learning, reinforcement learning, and in biological systems.

Original: When a desired behaviour is sufﬁciently complex, or the environment too unforgiving, it can be intractable to learn the behaviour from scratch through random exploration.
Revised: When a desired behaviour is sufﬁciently complex, or the environment too unforgiving, it can be intractable to learn the behaviour from scratch through random exploration.

Original: Instead, by “starting small” (Elman, 1993), an agent can build skills, representations, and a dataset of meaningful experiences that allow it to accelerate its learning.
Revised: Instead, by “starting small” (Elman, 1993), an agent can build skills, representations, and a dataset of meaningful experiences that allow it to accelerate its learning.

Original: Such curricula can drastically improve sample efﬁciency (Bengio et al., 2009).
Revised: Such curricula can drastically improve sample efﬁciency (Bengio et al., 2009).

Original: Typically, curriculum learning uses a progression of tasks or environments.
Revised: Typically, curriculum learning uses a progression of tasks or environments.

Original: Simple tasks that provide meaningful feedback to random agents are used ﬁrst, and some schedule is used to introduce more challenging tasks later during training (Graves et al., 2017).
Revised: Simple tasks that provide meaningful feedback to random agents are used ﬁrst, and some schedule is used to introduce more challenging tasks later during training (Graves et al., 2017).

Original: However, in many contexts neither the agent nor experimenter has such unimpeded control over the environment.
Revised: However, in many contexts neither the agent nor experimenter has such unimpeded control over the environment.

Original: In this work, we instead make use of curricula that are internal to the agent, simplifying the exploration problem without changing the environment.
Revised: In this work, we instead make use of curricula that are internal to the agent, simplifying the exploration problem without changing the environment.

Original: In particular, we grow the size of the action space of reinforcement learning agents over the course of training.
Revised: In particular, we grow the size of the action space of reinforcement learning agents over the course of training.

Original: At the beginning of training, our agents use a severely restricted action space.
Revised: At the beginning of training, our agents use a severely restricted action space.

Original: This helps exploration by guiding the agent towards rewards and meaningful experiences, and provides low variance updates during learning.
Revised: This helps exploration by guiding the agent towards rewards and meaningful experiences, and provides low variance updates during learning.

Original: The action space is then grown progressively.
Revised: The action space is then grown progressively.

Original: Eventually, using the most unrestricted action space, the agents are able to ﬁnd superior policies.
Revised: Eventually, using the most unrestricted action space, the agents are able to ﬁnd superior policies.

Original: Each action space is a strict superset of the more restricted ones.
Revised: Each action space is a strict superset of the more restricted ones.

Original: This paradigm requires some domain knowledge to identify a suitable hierarchy of action spaces.
Revised: This paradigm requires some domain knowledge to identify a suitable hierarchy of action spaces.

Original: However, such a hierarchy is often easy to ﬁnd.
Revised: However, such a hierarchy is often easy to ﬁnd.

Original: Continuous action spaces can be discretised with increasing resolution.
Revised: Continuous action spaces can be discretised with increasing resolution.

Original: Similarly, curricula for coping with the large combinatorial action spaces induced by many agents can be obtained from the prior that nearby agents are more likely to need to coordinate.
Revised: Similarly, curricula for coping with the large combinatorial action spaces induced by many agents can be obtained from the prior that nearby agents are more likely to need to coordinate.

Original: For example, in routing or trafﬁc ﬂow problems nearby agents or nodes may wish to adopt similar local policies to alleviate global congestion.
Revised: For example, in routing or trafﬁc ﬂow problems nearby agents or nodes may wish to adopt similar local policies to alleviate global congestion.

Original: Our method will be valuable when it is possible to identify a restricted action space in which random exploration leads to signiﬁcantly more meaningful experiences than random exploration in the full action space.
Revised: Our method will be valuable when it is possible to identify a restricted action space in which random exploration leads to signiﬁcantly more meaningful experiences than random exploration in the full action space.

Original: We propose an approach that uses off-policy reinforcement learning to improve sample efﬁciency in this type of curriculum learning.
Revised: We propose an approach that uses off-policy reinforcement learning to improve sample efﬁciency in this type of curriculum learning.

Original: Since data from exploration using a restricted action space is still valid in the Markov Decision Processes (MDPs) corresponding to the less restricted action spaces, we can learn value functions in the less restricted action space with ‘off-action-space’ data collected by exploring in the restricted action space.
Revised: Since data from exploration using a restricted action space is still valid in the Markov Decision Processes (MDPs) corresponding to the less restricted action spaces, we can learn value functions in the less restricted action space with ‘off-action-space’ data collected by exploring in the restricted action space.

Original: In our approach, we learn value functions corresponding to each level of restriction simultaneously.
Revised: In our approach, we learn value functions corresponding to each level of restriction simultaneously.

Original: We can use the relationships of these value functions to each other to accelerate learning further, by using value estimates themselves as initialisations or as bootstrap targets for the less restricted action spaces, as well as sharing learned state representations.
Revised: We can use the relationships of these value functions to each other to accelerate learning further, by using value estimates themselves as initialisations or as bootstrap targets for the less restricted action spaces, as well as sharing learned state representations.

Original: Empirically, we ﬁrst demonstrate the efﬁcacy of our approach in two simple control tasks, in which the resolution of discretised actions is progressively increased.
Revised: Empirically, we ﬁrst demonstrate the efﬁcacy of our approach in two simple control tasks, in which the resolution of discretised actions is progressively increased.

Original: We then tackle a more challenging set of problems with combinatorial action spaces, in the context of StarCraft micromanagement with large numbers of agents (50-100).
Revised: We then tackle a more challenging set of problems with combinatorial action spaces, in the context of StarCraft micromanagement with large numbers of agents (50-100).

Original: Given the heuristic prior that nearby agents in a multiagent setting are likely to need to coordinate, we use hierarchical clustering to impose a restricted action space on the agents.
Revised: Given the heuristic prior that nearby agents in a multiagent setting are likely to need to coordinate, we use hierarchical clustering to impose a restricted action space on the agents.

Original: Agents in a cluster are restricted to take the same action, but we progressively increase the number of groups that can act independently of one another over the course of training.
Revised: Agents in a cluster are restricted to take the same action, but we progressively increase the number of groups that can act independently of one another over the course of training.

Original: Our method substantially improves sample efﬁciency on a number of tasks, outperforming learning any particular action space from scratch, a number of ablations, and an actor-critic baseline that learns a single value function for the behaviour policy, as in the work of Czarnecki et al.
Revised: Our method substantially improves sample efﬁciency on a number of tasks, outperforming learning any particular action space from scratch, a number of ablations, and an actor-critic baseline that learns a single value function for the behaviour policy, as in the work of Czarnecki et al.

Original: Code is available, but redacted here for anonymity.
Revised: Code is available, but redacted here for anonymity.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Curriculum learning has a long history, appearing at least as early as the work of Selfridge et al.
Revised: Curriculum learning has a long history, appearing at least as early as the work of Selfridge et al.

Original: (1985) in reinforcement learning, and for the training of neural networks since Elman (1993).
Revised: (1985) in reinforcement learning, and for the training of neural networks since Elman (1993).

Original: In supervised learning, one typically has control of the order in which data is presented to the learning algorithm.
Revised: In supervised learning, one typically has control of the order in which data is presented to the learning algorithm.

Original: For learning with deep neural networks, Bengio et al.
Revised: For learning with deep neural networks, Bengio et al.

Original: (2009) explored the use of curricula in computer vision and natural language processing.
Revised: (2009) explored the use of curricula in computer vision and natural language processing.

Original: Many approaches use handcrafted schedules for task curricula, but others (Zaremba & Sutskever, 2014; Pentina et al., 2015; Graves et al., 2017) study diagnostics that can be used to automate the choice of task mixtures throughout training.
Revised: Many approaches use handcrafted schedules for task curricula, but others (Zaremba & Sutskever, 2014; Pentina et al., 2015; Graves et al., 2017) study diagnostics that can be used to automate the choice of task mixtures throughout training.

Original: In a self-supervised control setting, Murali et al.
Revised: In a self-supervised control setting, Murali et al.

Original: (2018) use sensitivity analysis to automatically deﬁne a curriculum over action dimensions and prioritise their search space.
Revised: (2018) use sensitivity analysis to automatically deﬁne a curriculum over action dimensions and prioritise their search space.

Original: In some reinforcement learning settings, it may also be possible to control the environment so as to induce a curriculum.
Revised: In some reinforcement learning settings, it may also be possible to control the environment so as to induce a curriculum.

Original: With a resettable simulator, it is possible to use a sequence of progressively more challenging initial states (Asada et al., 1996; Florensa et al., 2017).
Revised: With a resettable simulator, it is possible to use a sequence of progressively more challenging initial states (Asada et al., 1996; Florensa et al., 2017).

Original: With a procedurally generated task, it is often possible to automatically tune the difﬁculty of the environments (Tamar et al., 2016).
Revised: With a procedurally generated task, it is often possible to automatically tune the difﬁculty of the environments (Tamar et al., 2016).

Original: Similar curricula also appear often in hierarchical reinforcement learning, where skills can be learned in comparatively easy settings and then composed in more complex ways later (Singh, 1992).
Revised: Similar curricula also appear often in hierarchical reinforcement learning, where skills can be learned in comparatively easy settings and then composed in more complex ways later (Singh, 1992).

Original: Taylor et al.
Revised: Taylor et al.

Original: (2007) use more general inter-task mappings to transfer Q -values between tasks that do not share state and action spaces.
Revised: (2007) use more general inter-task mappings to transfer Q -values between tasks that do not share state and action spaces.

Original: In adversarial settings, one may also induce a curriculum through self-play (Tesauro, 1995; Sukhbaatar et al., 2017; Silver et al., 2017).
Revised: In adversarial settings, one may also induce a curriculum through self-play (Tesauro, 1995; Sukhbaatar et al., 2017; Silver et al., 2017).

Original: In this case, the learning agents themselves deﬁne the changing part of the environment.
Revised: In this case, the learning agents themselves deﬁne the changing part of the environment.

Original: A less invasive manipulation of the environment involves altering the reward function.
Revised: A less invasive manipulation of the environment involves altering the reward function.

Original: Such reward shaping allows learning policies in an easier MDP, which can then be transferred to the more difﬁcult sparse-reward task (Colombetti & Dorigo, 1992; Ng et al., 1999).
Revised: Such reward shaping allows learning policies in an easier MDP, which can then be transferred to the more difﬁcult sparse-reward task (Colombetti & Dorigo, 1992; Ng et al., 1999).

Original: It is also possible to learn reward shaping on simple tasks and transfer it to harder tasks in a curriculum (Konidaris & Barto, 2006).
Revised: It is also possible to learn reward shaping on simple tasks and transfer it to harder tasks in a curriculum (Konidaris & Barto, 2006).

Original: In contrast, learning with increasingly complex function approximators does not require any control of the environment.
Revised: In contrast, learning with increasingly complex function approximators does not require any control of the environment.

Original: In reinforcement learning, this has often taken the form of adaptively growing the resolution of the state space considered by a piecewise constant discretised approximation (Moore, 1994; Munos & Moore, 2002; Whiteson et al., 2007).
Revised: In reinforcement learning, this has often taken the form of adaptively growing the resolution of the state space considered by a piecewise constant discretised approximation (Moore, 1994; Munos & Moore, 2002; Whiteson et al., 2007).

Original: Stanley & Miikkulainen (2004) study continual complexiﬁcation in the context of coevolution, growing the complexity of neural network architectures through the course of training.
Revised: Stanley & Miikkulainen (2004) study continual complexiﬁcation in the context of coevolution, growing the complexity of neural network architectures through the course of training.

Original: These works progressively increase the capabilities of the agent, but not with respect to its available actions.
Revised: These works progressively increase the capabilities of the agent, but not with respect to its available actions.

Original: In the context of planning on-line with a model, there are a number of approaches that use progressive widening to consider increasing large action spaces over the course of search (Chaslot et al., 2008), including in planning for continuous action spaces (Cou¨etoux et al., 2011).
Revised: In the context of planning on-line with a model, there are a number of approaches that use progressive widening to consider increasing large action spaces over the course of search (Chaslot et al., 2008), including in planning for continuous action spaces (Cou¨etoux et al., 2011).

Original: However, these methods cannot directly be applied to grow the action space in the model-free setting.
Revised: However, these methods cannot directly be applied to grow the action space in the model-free setting.

Original: A recent related work tackling our domain is that of Czarnecki et al.
Revised: A recent related work tackling our domain is that of Czarnecki et al.

Original: (2018), who train mixtures of two policies with an actor-critic approach, learning a single value function for the current mixture of policies.
Revised: (2018), who train mixtures of two policies with an actor-critic approach, learning a single value function for the current mixture of policies.

Original: The mixture contains a policy that may be harder to learn but has a higher performance ceiling, such as a policy with a larger action space as we consider in this work.
Revised: The mixture contains a policy that may be harder to learn but has a higher performance ceiling, such as a policy with a larger action space as we consider in this work.

Original: The mixing coefﬁcient is initialised to only support the simpler policy, and adapted via population based training
Revised: The mixing coefﬁcient is initialised to only support the simpler policy, and adapted via population based training

Original: (Jaderberg et al., 2017).
Revised: (Jaderberg et al., 2017).

Original: In contrast, we simultaneously learn a different value function for each policy, and exploit the properties of the optimal value functions to induce additional structure on our models.
Revised: In contrast, we simultaneously learn a different value function for each policy, and exploit the properties of the optimal value functions to induce additional structure on our models.

Original: We further use these properties to construct a scheme for off-action-space learning which means our approach may be used in an off-policy setting.
Revised: We further use these properties to construct a scheme for off-action-space learning which means our approach may be used in an off-policy setting.

Original: Empirically, in our settings, we ﬁnd our approach to perform better and more consistently than an actor-critic algorithm modeled after Czarnecki et al.
Revised: Empirically, in our settings, we ﬁnd our approach to perform better and more consistently than an actor-critic algorithm modeled after Czarnecki et al.

Original: (2018), although we do not take on the signiﬁcant additional computational requirements of population based training in any of our experiments.
Revised: (2018), although we do not take on the signiﬁcant additional computational requirements of population based training in any of our experiments.

Revised: A number of other works address the problem of generalisation and representation for value functions with large discrete action spaces, without explicitly addressing the resulting exploration problem (e.g. (Dulac-Arnold et al., 2015; Pan et al., 2018)).

Revised: These approaches typically rely on action representations from prior knowledge.

Revised: Such representations could be used in combination with our method to construct a hierarchy of action spaces with which to shape exploration.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We formalise our problem as a MDP, speciﬁed by a tuple < S , A , P, r, γ > .
Revised: We formalise our problem as a MDP, speciﬁed by a tuple < S , A , P, r, γ > .

Original: The set of possible states and actions are given by S and A , P is the transition function that speciﬁes the environment dynamics, and γ is a discount factor used to specify the discounted return R = (cid:80) Tt =

Revised: The set of possible states and actions are given by S and A , P is the transition function that speciﬁes the environment dynamics, and γ is a discount factor used to specify the discounted return R = (cid:80) Tt =

Original: γ t r t for an episode of length T .
Revised: γ t r t for an episode of length T .

Original: We wish our agent to maximise this return in expectation by learning a policy π that maps states to actions.
Revised: We wish our agent to maximise this return in expectation by learning a policy π that maps states to actions.

Original: The state-action value function ( Q -function) is given by
Revised: The state-action value function ( Q -function) is given by

Original: Q π = Eπ
Revised: Q [Equation] E

Revised: [Equation]

Original: [ R | s, a ] .
Revised: [ R | s, a ] .

Original: The optimal Q -function Q ∗ satisﬁes the Bellman optimality equation:
Revised: The optimal Q -function Q ∗ satisﬁes the Bellman optimality equation:

Original: [Equation]
Revised: [Equation]

Original: Q -learning (Watkins & Dayan, 1992) uses a sample-based approximation of the Bellman optimality operator T to iteratively improve an estimate of Q ∗ .
Revised: Q -learning (Watkins & Dayan, 1992) uses a sample-based approximation of the Bellman optimality operator T to iteratively improve an estimate of Q ∗ .

Original: Q -learning is an off-policy method, meaning that samples from any policy may be used to improve the value function estimate.
Revised: Q -learning is an off-policy method, meaning that samples from any policy may be used to improve the value function estimate.

Original: We use this property to engage Q -learning for off-action-space learning, as described in the next section.
Revised: We use this property to engage Q -learning for off-action-space learning, as described in the next section.

Original: We also introduce some notation for restricted action spaces.
Revised: We also introduce some notation for restricted action spaces.

Original: In particular, for an MDP with unrestricted action space
Revised: In particular, for an MDP with unrestricted action space

Original: A we deﬁne a set of N action spaces A (cid:96) , (cid:96) ∈ { 0 , . . .
Revised: A we deﬁne a set of N action spaces A (cid:96) , (cid:96) ∈ { 0 , . . .

Original: , N − 1 } .
Revised: , N − 1 } .

Original: Each action space is a subset of the next: A 0 ⊂ A 1 ⊂ . . .
Revised: Each action space is a subset of the next: A 0 ⊂ A 1 ⊂ . . .

Original: ⊂
Revised: ⊂

Original: A N − 1 ⊆ A .
Revised: A N − 1 ⊆ A .

Original: A policy restricted to actions A (cid:96) is denoted π (cid:96) ( a | s ) .
Revised: A policy restricted to actions A (cid:96) is denoted π (cid:96) ( a | s ) .

Original: The optimal policy in this restricted policy class is π ∗ (cid:96) ( a | s ) , and its corresponding action-value and value functions are Q ∗ (cid:96) ( s, a ) and V ∗ (cid:96) ( s )
Revised: The optimal policy in this restricted policy class is π ∗ (cid:96) ( a | s ) , and its corresponding action-value and value functions are Q ∗ (cid:96) ( s, a ) and V ∗ (cid:96) ( s )

Original: = max a Q ∗ (cid:96) ( s, a ) .
Revised: = max a Q ∗ (cid:96) ( s, a ) .

Original: Additionally, we deﬁne a hierarchy of actions by identifying for every action a ∈ A (cid:96) , (cid:96) > 0 a parent action parent (cid:96) ( a ) in the space of A (cid:96) − 1 .
Revised: Additionally, we deﬁne a hierarchy of actions by identifying for every action a ∈ A (cid:96) , (cid:96) > 0 a parent action parent (cid:96) ( a ) in the space of A (cid:96) − 1 .

Original: Since action spaces are subsets of larger action spaces, for all a ∈ A (cid:96) − 1 , parent (cid:96) ( a ) = a , i.e., one child of each action is itself.
Revised: Since action spaces are subsets of larger action spaces, for all a ∈ A (cid:96) − 1 , parent (cid:96) ( a ) = a , i.e., one child of each action is itself.

Original: Simple pieces of domain knowledge are often sufﬁcient to deﬁne these hierarchies.
Revised: Simple pieces of domain knowledge are often sufﬁcient to deﬁne these hierarchies.

Original: For example, a discretised continuous action can identify its nearest neighbour in A (cid:96) − 1 as a parent.
Revised: For example, a discretised continuous action can identify its nearest neighbour in A (cid:96) − 1 as a parent.

Original: In Section 5 we describe a possible hierarchy for multi-agent action spaces.
Revised: In Section 5 we describe a possible hierarchy for multi-agent action spaces.

Original: One could also imagine using action-embeddings (Tennenholtz & Mannor, 2019) to learn such a hierarchy from data.
Revised: One could also imagine using action-embeddings (Tennenholtz & Mannor, 2019) to learn such a hierarchy from data.

Original:  C URRICULUM LEARNING WITH GROWING ACTION SPACES
Revised:  C URRICULUM LEARNING WITH GROWING ACTION SPACES

Original: We build our approach to growing action spaces (GAS) on off-policy value-based reinforcement learning.
Revised: We build our approach to growing action spaces (GAS) on off-policy value-based reinforcement learning.

Original: Q -learning and its deep-learning adaptations have shown strong performance (Hessel et al., 2018), and admit a simple framework for off-policy learning.
Revised: Q -learning and its deep-learning adaptations have shown strong performance (Hessel et al., 2018), and admit a simple framework for off-policy learning.

Original: 4. O FF - ACTION - SPACE LEARNING
Revised: 4. O FF - ACTION - SPACE LEARNING

Original: A value function for an action space A (cid:96) may be updated with transitions using actions drawn from its own action space, or any more restricted action spaces, if we use an off-policy learning algorithm.
Revised: A value function for an action space A (cid:96) may be updated with transitions using actions drawn from its own action space, or any more restricted action spaces, if we use an off-policy learning algorithm.

Original: The restricted transitions simply form a subset of the data required to learn the value functions of the less restricted action spaces.
Revised: The restricted transitions simply form a subset of the data required to learn the value functions of the less restricted action spaces.

Original: To exploit this, we simultaneously learn an estimated optimal value function ˆ Q ∗ (cid:96) ( s, a ) for each action space A (cid:96) , and use samples drawn from a behaviour policy based on a value function for low (cid:96) to directly train the higher (cid:96) value functions.
Revised: To exploit this, we simultaneously learn an estimated optimal value function ˆ Q ∗ (cid:96) ( s, a ) for each action space A (cid:96) , and use samples drawn from a behaviour policy based on a value function for low (cid:96) to directly train the higher (cid:96) value functions.

Original: At the beginning of each episode, we sample (cid:96) according to some distribution.
Revised: At the beginning of each episode, we sample (cid:96) according to some distribution.

Original: The experiences generated in that episode are used to update all of the ˆ Q ∗≥ (cid:96) ( s, a ) .
Revised: The experiences generated in that episode are used to update all of the ˆ Q ∗≥ (cid:96) ( s, a ) .

Original: This off-action-space learning is a type of off-policy learning that enables efﬁcient exploration by restricting it to the low- (cid:96) regime.
Revised: This off-action-space learning is a type of off-policy learning that enables efﬁcient exploration by restricting it to the low- (cid:96) regime.

Original: We sample at the beginning of the episode rather than at each timestep because, if the agent uses a high- (cid:96) action, it may enter a state that is inaccessible for a lower- (cid:96) policy, and we do not wish to force a low- (cid:96) value function to generalise to states that are only accessible at higher (cid:96) .
Revised: We sample at the beginning of the episode rather than at each timestep because, if the agent uses a high- (cid:96) action, it may enter a state that is inaccessible for a lower- (cid:96) policy, and we do not wish to force a low- (cid:96) value function to generalise to states that are only accessible at higher (cid:96) .

Original: Since data from a restricted action space only supports a subset of the state-action space relevant for the value functions of less restricted action spaces, we hope that a suitable function approximator still allows some generalisation to the unexplored parts of the less restricted state-action space.
Revised: Since data from a restricted action space only supports a subset of the state-action space relevant for the value functions of less restricted action spaces, we hope that a suitable function approximator still allows some generalisation to the unexplored parts of the less restricted state-action space.

Original: 4. V ALUE ESTIMATES
Revised: 4. V ALUE ESTIMATES

Original: Note that:
Revised: Note that:

Original: [Equation]
Revised: [Equation]

Original: This is because each action space is a strict subset of the larger ones, so the agent can always in the worst case fall back to a policy using a more restricted action space.
Revised: This is because each action space is a strict subset of the larger ones, so the agent can always in the worst case fall back to a policy using a more restricted action space.

Original: This monotonicity intuitively recommends an iterative decomposition of the value estimates, in which ˆ Q ∗ (cid:96) +

Revised: This monotonicity intuitively recommends an iterative decomposition of the value estimates, in which ˆ Q ∗ (cid:96) +

Original: ( s, a ) is estimated as a sum of ˆ Q ∗ (cid:96) ( s, a ) and some positive ∆ (cid:96) ( s, a ) .
Revised: ( s, a ) is estimated as a sum of ˆ Q ∗ (cid:96) ( s, a ) and some positive ∆ (cid:96) ( s, a ) .

Original: This is not immediately possible due to the mismatch in the support of each function.
Revised: This is not immediately possible due to the mismatch in the support of each function.

Original: However, we can leverage a hierarchical structure in the action spaces when present, as described in Section 3.
Revised: However, we can leverage a hierarchical structure in the action spaces when present, as described in Section 3.

Original: In this case we can use:
Revised: In this case we can use:

Original: [Equation]
Revised: [Equation]

Original: This is a task-speciﬁc upsampling of the lower- (cid:96) value function to intialise the next value function.
Revised: This is a task-speciﬁc upsampling of the lower- (cid:96) value function to intialise the next value function.

Original: Both ˆ Q ∗ (cid:96) ( s, a ) and ∆ (cid:96) ( s, a ) are learned components.
Revised: Both ˆ Q ∗ (cid:96) ( s, a ) and ∆ (cid:96) ( s, a ) are learned components.

Original: We could further regularise or restrict the functional form of ∆ (cid:96) to ensure its positivity when parent (cid:96) ( a ) = a .
Revised: We could further regularise or restrict the functional form of ∆ (cid:96) to ensure its positivity when parent (cid:96) ( a ) = a .

Original: However, we did not ﬁnd this to be valuable in our experiments, and simply initialised ∆ (cid:96) to be small.
Revised: However, we did not ﬁnd this to be valuable in our experiments, and simply initialised ∆ (cid:96) to be small.

Original: The property (2) also implies a modiﬁed Bellman optimality equation:
Revised: The property (2) also implies a modiﬁed Bellman optimality equation:

Original: [Equation]
Revised: [Equation]

Original: The max i<(cid:96) are redundant in their role as conditions on the optimal value function Q ∗ (cid:96) .
Revised: The max i<(cid:96) are redundant in their role as conditions on the optimal value function Q ∗ (cid:96) .

Original: However, the Bellman optimality equation also gives us the form of a Q -learning update, where the term in the expectation on the RHS is used as an operator that iteratively improves an estimate of Q ∗ .
Revised: However, the Bellman optimality equation also gives us the form of a Q -learning update, where the term in the expectation on the RHS is used as an operator that iteratively improves an estimate of Q ∗ .

Original: When these estimates are inaccurate, the modiﬁed form of the Bellman equation may lead to different updates, allowing the solutions to higher (cid:96) to be bootstrapped from those at lower (cid:96) .
Revised: When these estimates are inaccurate, the modiﬁed form of the Bellman equation may lead to different updates, allowing the solutions to higher (cid:96) to be bootstrapped from those at lower (cid:96) .

Original: We expect that policies with low (cid:96) are easier to learn, and that therefore the corresponding ˆ Q ∗ (cid:96) is higher value and more accurate earlier in training than those at high (cid:96) .
Revised: We expect that policies with low (cid:96) are easier to learn, and that therefore the corresponding ˆ Q ∗ (cid:96) is higher value and more accurate earlier in training than those at high (cid:96) .

Original: These high values could be picked up by the extra maximisation in the modiﬁed bootstrap, and thereby rapidly learned by the higher- (cid:96) value functions.
Revised: These high values could be picked up by the extra maximisation in the modiﬁed bootstrap, and thereby rapidly learned by the higher- (cid:96) value functions.

Original: Empirically however, we ﬁnd that using this form for the target in our loss function performs no better than just maximising over ˆ Q ∗ (cid:96) ( s (cid:48) , a (cid:48) ) .
Revised: Empirically however, we ﬁnd that using this form for the target in our loss function performs no better than just maximising over ˆ Q ∗ (cid:96) ( s (cid:48) , a (cid:48) ) .

Original: We discuss the choice of target and these results in more detail in Section 6.2.
Revised: We discuss the choice of target and these results in more detail in Section 6.2.

Original: 4. R EPRESENTATION
Revised: 4. R EPRESENTATION

Original: By sharing parameters between the function approximators of each Q (cid:96) , we can learn a joint state representation, which can then be iteratively decoded into estimates of Q ∗ for each (cid:96) .
Revised: By sharing parameters between the function approximators of each Q (cid:96) , we can learn a joint state representation, which can then be iteratively decoded into estimates of Q ∗ for each (cid:96) .

Original: This shared embedding can be iteratively reﬁned by, e.g., additional network layers for each ˆ Q ∗ (cid:96) to maintain ﬂexibility along with transfer of useful representations.
Revised: This shared embedding can be iteratively reﬁned by, e.g., additional network layers for each ˆ Q ∗ (cid:96) to maintain ﬂexibility along with transfer of useful representations.

Original: This simple approach has had great success in improving the efﬁciency of many multi-task solutions using deep learning (Ruder, 2017).
Revised: This simple approach has had great success in improving the efﬁciency of many multi-task solutions using deep learning (Ruder, 2017).

Original: 4. C URRICULUM SCHEDULING
Revised: 4. C URRICULUM SCHEDULING

Original: It may be difﬁcult to choose agood distribution from which to sample (cid:96) for the behaviour policy, and an appropriate schedule with which to increase its mean during training.
Revised: We need to choose a schedule with which to increase the (cid:96) used by the behaviour policy over the course of training.

Original: Czarnecki et al.
Revised: Czarnecki et al.

Original: (2018) use population based training (Jaderberg et al., 2017) to choose a mixing parameter on the ﬂy.
Revised: (2018) use population based training (Jaderberg et al., 2017) to choose a mixing parameter on the ﬂy.

Original: However, this comes at signiﬁcant computational cost, and optimises greedily for immediate performance gains.
Revised: However, this comes at signiﬁcant computational cost, and optimises greedily for immediate performance gains.

Original: We use a simple linear schedule on a mixing parameter α ∈ [0 , N ] , picking{+. Initially α = 
 and we always choose (cid:96) = 0 . Later, we pick+} (cid:96) =
Revised: We use a simple linear schedule on a mixing parameter α ∈ [0 , N ] {+. Initially α = 
 and we always choose (cid:96) = 0 . Later, we pick+} (cid:96) =

Original: (cid:98) α (cid:99) with probability (cid:100) α
Revised: (cid:98) α (cid:99) with probability (cid:100) α

Original: (cid:101) − α and (cid:96) =
Revised: (cid:101) − α and (cid:96) =

Original: (cid:100)
Revised: (cid:100)

Original: α (cid:101) with probability α − (cid:98)
Revised: α (cid:101) with probability α − (cid:98)

Original: α (cid:99)  . 
Revised: α (cid:99) (e.g. if α = 1 . 1 , we choose (cid:96) = 1 with 90% chance and (cid:96) = with 10% chance).

Original: This worked well empirically with little effort to tune.
Revised: This worked well empirically with little effort to tune.

Original: Many other strategies exist for tuning a curriculum automatically (such as those explored by Graves et al.
Revised: Many other strategies exist for tuning a curriculum automatically (such as those explored by Graves et al.

Original: (2017)), and could be beneﬁcial, at the cost of additional overhead and algorithmic complexity.
Revised: (2017)), and could be beneﬁcial, at the cost of additional overhead and algorithmic complexity.

Original:  G ROWING ACTION SPACES FOR MULTI - AGENT CONTROL
Revised:  G ROWING ACTION SPACES FOR MULTI - AGENT CONTROL

Original: In cooperative multi-agent control, the full action space allows each of N agents to take actions from a set A agent , resulting in an exponentially large action space of size |A agent | N .
Revised: In cooperative multi-agent control, the full action space allows each of N agents to take actions from a set A agent , resulting in an exponentially large action space of size |A agent | N .

Original: Random exploration in this action space is highly unlikely to produce sensical behaviours, so growing the action space as we propose is particularly valuable in this setting.
Revised: Random exploration in this action space is highly unlikely to produce sensical behaviours, so growing the action space as we propose is particularly valuable in this setting.

Original: One approach would be to limit the actions available to each agent, as done in our discretised continuous control experiments (Section 6.1) and those of Czarnecki et al.
Revised: One approach would be to limit the actions available to each agent, as done in our discretised continuous control experiments (Section 6.1) and those of Czarnecki et al.

Original: However, the joint action space would still be exponential in N .
Revised: However, the joint action space would still be exponential in N .

Original: We propose instead to use hierarchical clustering, and to assign the same action to nearby agents.
Revised: We propose instead to use hierarchical clustering, and to assign the same action to nearby agents.

Original: At the ﬁrst level of the hierarchy, we treat the whole team as a single group, and all agents are constrained to take the same action.
Revised: At the ﬁrst level of the hierarchy, we treat the whole team as a single group, and all agents are constrained to take the same action.

Original: At the next level of the hierarchy, we split the agents into k groups using an unsupervised clustering algorithm, allowing each group to act independently.
Revised: At the next level of the hierarchy, we split the agents into k groups using an unsupervised clustering algorithm, allowing each group to act independently.

Original: At each further level, every group is split once again into k smaller groups.
Revised: At each further level, every group is split once again into k smaller groups.

Original: In practice, we simply use k -means clustering based on the agent’s spatial position, but this can be easily extended to more complex hierarchies using other clustering approaches.
Revised: In practice, we simply use k -means clustering based on the agent’s spatial position, but this can be easily extended to more complex hierarchies using other clustering approaches.

Original: To estimate the value function, we compute a state-value score ˆ V ( s ) , and a group-action delta ∆ (cid:96) ( s, a g , g ) for each group g at each level (cid:96) .
Revised: To estimate the value function, we compute a state-value score ˆ V ( s ) , and a group-action delta ∆ (cid:96) ( s, a g , g ) for each group g at each level (cid:96) .

Original: Then, we compute an estimated group-action value for each group, at each level, using a per-group form of (3): ˆ Q ∗ (cid:96) +

Revised: Then, we compute an estimated group-action value for each group, at each level, using a per-group form of (3): ˆ Q ∗ (cid:96) +

Original: ( s, a g )
Revised: ( s, a g )

Original: =
Revised: =

Original: ˆ Q ∗ (cid:96) ( s, parent k ( a g ))
Revised: ˆ Q ∗ (cid:96) ( s, parent k ( a g ))

Original: + ∆ (cid:96) ( s, a g , g ) .
Revised: + ∆ (cid:96) ( s, a g , g ) .

Original: We use ˆ Q ∗− 1 ( s, · ) =
Revised: We use ˆ Q ∗− 1 ( s, · ) =

Original: ˆ V ( s ) to initialise the iterative computation, similarly to the dueling architecture of Wang et al. (2015).
Revised: ˆ V ( s ) to initialise the iterative computation, similarly to the dueling architecture of Wang et al. (2015).

Original: The estimated value of the parent action is the estimated value of the entire parent group all taking the same action as the child group.
Revised: The estimated value of the parent action is the estimated value of the entire parent group all taking the same action as the child group.

Original: At each level (cid:96) we now have a set of group-action values.
Revised: At each level (cid:96) we now have a set of group-action values.

Original: In effect, a multi-agent value-learning problem still remains at each level (cid:96) , but with a greatly reduced number of agents at low (cid:96) .
Revised: In effect, a multi-agent value-learning problem still remains at each level (cid:96) , but with a greatly reduced number of agents at low (cid:96) .

Original: We could simply use independent Q -learning (Tan, 1993), but instead choose to estimate the joint-action value at each level as the mean of the group-action values for the groups at that (cid:96) , as in the work of Sunehag et al.
Revised: We could simply use independent Q -learning (Tan, 1993), but instead choose to estimate the joint-action value at each level as the mean of the group-action values for the groups at that (cid:96) , as in the work of Sunehag et al.

Original: A less restrictive representation, such as that proposed by Rashid et al.
Revised: A less restrictive representation, such as that proposed by Rashid et al.

Original: (2018), could help, but we leave this direction to future work.
Revised: (2018), could help, but we leave this direction to future work.

Original: A potential problem is that the clustering changes for every state, which may interfere with generalisation as group-actions will not have consistent semantics.
Revised: A potential problem is that the clustering changes for every state, which may interfere with generalisation as group-actions will not have consistent semantics.

Original: We address this in two ways.
Revised: We address this in two ways.

Original: First, we include the clustering as part of the state, and the cluster centroids are re-initialised from the previous timestep for t > 0 to keep the cluster semantics approximately consistent.
Revised: First, we include the clustering as part of the state, and the cluster centroids are re-initialised from the previous timestep for t > 0 to keep the cluster semantics approximately consistent.

Original: Second, we use a functional representation that produces group-action values that are broadly agnostic to the identiﬁer of the group.
Revised: Second, we use a functional representation that produces group-action values that are broadly agnostic to the identiﬁer of the group.

Original: In particular, we compute a spatially resolved embedding, and pool over the locations occupied by each group.
Revised: In particular, we compute a spatially resolved embedding, and pool over the locations occupied by each group.

Original: See Figure 2 and Section 6.2 for more details.
Revised: See Figure 2 and Section 6.2 for more details.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We investigate two classes of problems that have a natural hierarchy in the action space.
Revised: We investigate two classes of problems that have a natural hierarchy in the action space.

Original: First, simple control problems where a coarse action discretisation can help accelerate exploration, and ﬁne action discretisation allows for a more optimal policy.
Revised: First, simple control problems where a coarse action discretisation can help accelerate exploration, and ﬁne action discretisation allows for a more optimal policy.

Original: Second, the cooperative multi-agent setting, discussed in Section 5, using large-scale StarCraft micromanagement scenarios.
Revised: Second, the cooperative multi-agent setting, discussed in Section 5, using large-scale StarCraft micromanagement scenarios.

Original: 6. D ISCRETISED CONTINUOUS CONTROL
Revised: 6. D ISCRETISED CONTINUOUS CONTROL

Original: As a proof-of-concept, we look at two simple examples: versions of the classic Acrobot and Mountain Car environments with discretised action spaces.
Revised: As a proof-of-concept, we look at two simple examples: versions of the classic Acrobot and Mountain Car environments with discretised action spaces.

Original: Both tasks have a sparse reward of +1 when the goal is reached, and we make the exploration problem more challenging by terminating episodes  with a penalty of -1 if the goal is not reached within 500 timesteps.
Revised: Both tasks have a sparse reward of +1 when the goal is reached, and we make the exploration problem more challenging by terminating episodes [Figure] with a penalty of -1 if the goal is not reached within 500 timesteps.

Original: The normalised remaining time is concatenated to the state so it remains Markovian despite the time limit.
Revised: The normalised remaining time is concatenated to the state so it remains Markovian despite the time limit.

Original: There is a further actuation cost of 0 .
Revised: There is a further actuation cost of 0 .

Original: 05 (cid:107) a (cid:107) 2 .
Revised: 05 (cid:107) a (cid:107) 2 .

Original: At A 0 , the actions apply a force of +1 and − 1 .
Revised: At A 0 , the actions apply a force of +1 and − 1 .

Original: At each subsequent A (cid:96)> 0 , each action is split into two children, one that is the same as the parent action, and the other applying half the force.
Revised: At each subsequent A (cid:96)> 0 , each action is split into two children, one that is the same as the parent action, and the other applying half the force.

Original: Thus,there are 2 (cid:96) actions in A (cid:96) .
Revised: Thus,

Original: The results of our experiments are shown in Figure 1.
Revised: The results of our experiments are shown in Figure 1.

Original: Training with the lower resolutions A 0 and
Revised: Training with the lower resolutions A 0 and

Original: A 1 from scratch converges to ﬁnding the goal, but incurs signiﬁcant actuation costs.
Revised: A 1 from scratch converges to ﬁnding the goal, but incurs signiﬁcant actuation costs.

Original: Training with
Revised: Training with

Original: A 2 from scratch almost never ﬁnds the goal with (cid:15) -greedy exploration.
Revised: A 2 from scratch almost never ﬁnds the goal with (cid:15) -greedy exploration.

Original: We also tried decaying the (cid:15)[Figure] at a quarter of the  ( A  success.
Revised: We also tried decaying the (cid:15) at a quarter of the rate ( A 2 slow (cid:15) ) without success.

Original: In these cases, the policy converges to the one that minimises actuation costs, never ﬁnding the goal.
Revised: In these cases, the policy converges to the one that minimises actuation costs, never ﬁnding the goal.

Original: Training with a growing action space explores to ﬁnd the goal early, and then uses this experience to transition smoothly into a solution that ﬁnds the goal but takes a slower route that minimises actuation costs while achieving the objective.
Revised: Training with a growing action space explores to ﬁnd the goal early, and then uses this experience to transition smoothly into a solution that ﬁnds the goal but takes a slower route that minimises actuation costs while achieving the objective.

Original: 6. C OMBINATORIAL ACTION SPACES : S TAR C RAFT BATTLES 6.2. L ARGE - SCALE S TAR C RAFT M ICROMANAGEMENT
Revised: 6. C OMBINATORIAL ACTION SPACES : S TAR C RAFT BATTLES 6.2. L ARGE - SCALE S TAR C RAFT M ICROMANAGEMENT

Original: The real-time strategy game StarCraft and its sequel StarCraft II have emerged as popular platforms for benchmarking reinforcement learning algorithms (Synnaeve et al., 2016; Vinyals et al., 2017). 
Revised: The real-time strategy game StarCraft and its sequel StarCraft II have emerged as popular platforms for benchmarking reinforcement learning algorithms (Synnaeve et al., 2016; Vinyals et al., 2017). [Table]

Original: Full game-play has been tackled by e.g. (Lee et al., 2018; Vinyals et al., 2019), while other works focus on sub-problems such as micromanagement , the low-level control of units engaged in a battle between two armies (e.g. (Usunier et al., 2016)).
Revised: has been tackled by e.g. (Lee et al., 2018; Vinyals et al., 2019), while other works focus on sub-problems such as micromanagement , the low-level control of units engaged in a battle between two armies (e.g. (Usunier et al., 2016)).

Original: Efforts to approach the former problem have required some subset of human demonstrations, hierarchical methods, and massive compute scale, and so we focus on the latter as a more tractable benchmark to evaluate our methods.
Revised: Efforts to approach the former problem have required some subset of human demonstrations, hierarchical methods, and massive compute scale, and so we focus on the latter as a more tractable benchmark to evaluate our methods.

Original: Most previous work on RL benchmarking with StarCraft micromanagement is restricted to maximally 20-30 units (Samvelyan et al., 2019; Usunier et al., 2016).
Revised: Most previous work on RL benchmarking with StarCraft micromanagement is restricted to maximally 20-30 units (Samvelyan et al., 2019; Usunier et al., 2016).

Original: In our experiments we focus on much larger-scale micromanagement scenarios with 50-100 units on each side of the battle.
Revised: In our experiments we focus on much larger-scale micromanagement scenarios with 50-100 units on each side of the battle.

Original: To further increase the difﬁculty of these micromanagement scenarios, in our setting the starting locations of the armies are randomised, and the opponent is controlled by scripted logic that holds its position until any agent-controlled unit is in range, and then focus-ﬁres on the closest enemy.
Revised: To further increase the difﬁculty of these micromanagement scenarios, in our setting the starting locations of the armies are randomised, and the opponent is controlled by scripted logic that holds its position until any agent-controlled unit is in range, and then focus-ﬁres on the closest enemy.

Original: This increases the exploration challenge, as our agents need to learn to ﬁnd the enemy ﬁrst, while they hold a strong defensive position.
Revised: This increases the exploration challenge, as our agents need to learn to ﬁnd the enemy ﬁrst, while they hold a strong defensive position.

Original: The action space for each unit permits an attack-move or move action in eight cardinal directions, as well as a stop action that causes the unit to passively hold its position.
Revised: The action space for each unit permits an attack-move or move action in eight cardinal directions, as well as a stop action that causes the unit to passively hold its position.

Original: In our experiments, we use k = 2 for k -means clustering and split down to at most four or eight groups.
Revised: In our experiments, we use k = 2 for k -means clustering and split down to at most four or eight groups.

Original: The maximum number of groups in an experiment with A (cid:96) is 2 (cid:96) .
Revised: The maximum number of groups in an experiment with A (cid:96) is 2 (cid:96) .

Original: Although our approach is designed for off-policy learning, we follow the common practice of using n -step Q -learning to accelerate the propagation of values (Hessel et al., 2018).
Revised: Although our approach is designed for off-policy learning, we follow the common practice of using n -step Q -learning to accelerate the propagation of values (Hessel et al., 2018).

Original: Our base algorithm uses the objective of n -step Q -learning from the work of Mnih et al.
Revised: Our base algorithm uses the objective of n -step Q -learning from the work of Mnih et al.

Original: and collects data from multiple workers into a short queue similarly to Espeholt et al.
Revised: and collects data from multiple workers into a short queue similarly to Espeholt et al.

Original: Full details can be found in the Appendix.
Revised: Full details can be found in the Appendix.

Original: 6.2. M ODEL ARCHITECTURE
Revised: 6.2. M ODEL ARCHITECTURE

Original: We propose an architecture to efﬁciently represent the value functions of the action-space hierarchy.
Revised: We propose an architecture to efﬁciently represent the value functions of the action-space hierarchy.

Original: The overall structure is shown in Figure 2.
Revised: The overall structure is shown in Figure 2.

Original: We start with the state of the scenario (1).
Revised: We start with the state of the scenario (1).

Original: Ally units are blue and split into two groups.
Revised: Ally units are blue and split into two groups.

Original: From the state, features are extracted from the units and map (see Appendix for full details).
Revised: From the state, features are extracted from the units and map (see Appendix for full details).

Original: These features are concatenated with a one-hot representation of the unit’s group (for allied agents), and are embedded with a small MLP.
Revised: These features are concatenated with a one-hot representation of the unit’s group (for allied agents), and are embedded with a small MLP.

Original: A 2-D grid of embeddings is constructed by adding up the unit embeddings for all units in each cell of the grid (2).
Revised: A 2-D grid of embeddings is constructed by adding up the unit embeddings for all units in each cell of the grid (2).

Original: The embeddings are passed through a residual CNN to produce a ﬁnal embedding (3), which is copied several times and decoded as follows.
Revised: The embeddings are passed through a residual CNN to produce a ﬁnal embedding (3), which is copied several times and decoded as follows.

Original: First, a state-value branch computes a scalar value by taking a global mean pooling (4) and passing the result through a 2-layer MLP (6).
Revised: First, a state-value branch computes a scalar value by taking a global mean pooling (4) and passing the result through a 2-layer MLP (6).

Original: Then, for each (cid:96) , a masked mean-pooling is used to produce an embedding for each group at that A (cid:96) by masking out the positions in the spatial embedding where there are no units of that group (5a, 5b, 5c).
Revised: Then, for each (cid:96) , a masked mean-pooling is used to produce an embedding for each group at that A (cid:96) by masking out the positions in the spatial embedding where there are no units of that group (5a, 5b, 5c).

Original: A single evaluation MLP for each (cid:96) is used to decode this embedding into a group action-score (7a, 7b, 7c).
Revised: A single evaluation MLP for each (cid:96) is used to decode this embedding into a group action-score (7a, 7b, 7c).

Original: This architecture allows a shared state representation to be efﬁciently decoded into value-function contributions for groups of any size, at any level of restriction in the action space.
Revised: This architecture allows a shared state representation to be efﬁciently decoded into value-function contributions for groups of any size, at any level of restriction in the action space.

Original: We consider two approaches for combining these outputs.
Revised: We consider two approaches for combining these outputs.

Original: In our default approach, described in
Revised: In our default approach, described in

Original: Section 5, each group’s action-value is given by the sum of the state-value and group-action-scores for the group and its parents (8a, 8b).
Revised: Section 5, each group’s action-value is given by the sum of the state-value and group-action-scores for the group and its parents (8a, 8b).

Original: In ‘SEP-Q’, each group’s action-value is simply given by the state-value added to the group-action score, i.e., ˆ Q ∗ (cid:96) ( s, a g ) =
Revised: In ‘SEP-Q’, each group’s action-value is simply given by the state-value added to the group-action score, i.e., ˆ Q ∗ (cid:96) ( s, a g ) =

Original: ˆ V ( s ) + ∆ (cid:96) ( s, a g , g ) .
Revised: ˆ V ( s ) + ∆ (cid:96) ( s, a g , g ) .

Original: This is an ablation in which the action-value estimates for restricted action spaces do not initialise the actionvalue estimates of their child actions.
Revised: This is an ablation in which the action-value estimates for restricted action spaces do not initialise the actionvalue estimates of their child actions.

Original: 6.2. R ESULTS AND DISCUSSION
Revised: 6.2. R ESULTS AND DISCUSSION

Original: Figure 3 presents the results of our method, as well as a number of baselines and ablations, on a variety of micromanagement tasks.
Revised: Figure 3 presents the results of our method, as well as a number of baselines and ablations, on a variety of micromanagement tasks.

Original: Our method is labeled Growing Action Spaces GAS( (cid:96) ) , such that GAS(2) will grow from A 0 to A 2 .
Revised: Our method is labeled Growing Action Spaces GAS( (cid:96) ) , such that GAS(2) will grow from A 0 to A 2 .

Original: Our primary baselines are policies trained with action spaces
Revised: Our primary baselines are policies trained with action spaces

Original: A 0 or A 2 from scratch.
Revised: A 0 or A 2 from scratch.

Original: GAS(2) consistently outperforms both of these variants.
Revised: GAS(2) consistently outperforms both of these variants.

Original: Policies trained from scratch on A 2 struggle with exploration, in particular in the harder scenarios where the opponent has a numbers advantage.
Revised: Policies trained from scratch on A 2 struggle with exploration, in particular in the harder scenarios where the opponent has a numbers advantage.

Original: Policies trained from scratch on A 0 learn quickly, but plateau comparatively low, due to the limited ability of a single group to position effectively.
Revised: Policies trained from scratch on A 0 learn quickly, but plateau comparatively low, due to the limited ability of a single group to position effectively.

Original: GAS(2) beneﬁts from the efﬁcient exploration enabled by an intialisation at A 0 , and uses the data gathered under this policy to efﬁciently transfer to A 2 ; enabling a higher asymptotic performance.
Revised: GAS(2) beneﬁts from the efﬁcient exploration enabled by an intialisation at A 0 , and uses the data gathered under this policy to efﬁciently transfer to A 2 ; enabling a higher asymptotic performance.

Original: We also compare against a Mix&Match (MM) baseline following the actor-critic approach of Czarnecki et al.
Revised: We also compare against a Mix&Match (MM) baseline using the actor-critic approach of Czarnecki et al.

Original: (2018), but adapted for our new multi-agent setting and supporting a third level in the mixture of policies ( A 0 , A 1 , A 2 ).
Revised: (2018), but adapted for our new multi-agent setting and supporting a third level in the mixture of policies ( A 0 , A 1 , A 2 ).

Original: We tuned hyperparameters for all algorithms on the easiest, fastest-training scenario (80 marines vs. 80 marines).
Revised: We tuned hyperparameters for all algorithms on the easiest, fastesttraining scenario (80 marines vs. 80 marines).

Original: On this scenario, MM learns faster but plateaus at the same level as GAS(2) .
Revised: On this scenario, MM learns faster but plateaus at the same level as GAS(2) .

Original: MM underperforms on all other scenarios to varying degrees.
Revised: MM underperforms on all other scenarios to varying degrees.

Original: Learning separate value functions for each A (cid:96) , as in our approach, appears to accelerate the transfer learning in the majority of settings.
Revised: Learning separate value functions for each A (cid:96) , as in our approach, appears to accelerate the transfer learning in the majority of settings.

Original: Another possible explanation is that MM may be more sensitive to hyperparameters.
Revised: Another possible explanation is that MM may be more sensitive to hyperparameters.

Original: We do not use population based training to tune hyperparameters on the ﬂy, which could otherwise help MM adapt to each scenario.
Revised: We do not use population based training to tune hyperparameters on the ﬂy, which could otherwise help MM adapt to each scenario.

Original: However, GAS would presumably also beneﬁt from population based training, at the cost of further computation and sample efﬁciency.
Revised: However, GAS would presumably also beneﬁt from population based training, at the cost of further computation and sample efﬁciency.

Original: The policies learned by GAS exhibit good tactics.
Revised: The policies learned by GAS exhibit good tactics.

Original: Control of separate groups is used to position our army so as to maximise the number of attacking units by forming a wall or a concave that surrounds the enemy, and by coordinating a simultaneous assault.
Revised: Control of separate groups is used to position our army so as to maximise the number of attacking units by forming a wall or a concave that surrounds the enemy, and by coordinating a simultaneous assault.

Original: Figure 4 in the Appendix shows some example learned policies.
Revised: Figure 5 in the Appendix shows some example learned policies.

Original: In scenarios where MM fails to learn well, it typically falls into a local minimum of attacking head-on.
Revised: In scenarios where MM fails to learn well, it typically falls into a local minimum of attacking head-on.

Original: In each scenario, we test an ablation GAS (2): ON-AC that does not use our off-action-space update, instead training each level of the Q -function only with data sampled at that level.
Revised: In each scenario, we test an ablation GAS (2): ON-AC that does not use our off-action-space update, instead training each level of the Q -function only with data sampled at that level.

Original: This ablation performs slightly, or considerably, worse  in each scenario.
Revised: This ablation performs somewhat worse on average, although the impact various in different scenarios.

Original: As expected, it is beneﬁcial to accelerate learning for ﬁner action spaces using data drawn from the off-action-space policy.
Revised: As expected, it is beneﬁcial to accelerate learning for ﬁner action spaces using data drawn from the off-actionspace policy.

Revised: In appendix A.1.1, the same ablation shows signiﬁcantly worse performance on the

Revised: Mountain Car task and comparable performance on Acrobot.

Original: We present a number of further ablations on two scenarios.
Revised: We present a number of further ablations on two scenarios.

Original: The most striking failure is of the ‘SEPQ’ variant which does not compose the value function as a sum of scores in the hierarchy.
Revised: The most striking failure is of the ‘SEPQ’ variant which does not compose the value function as a sum of scores in the hierarchy.

Original: It is critical to ensure that values are well-initialised as we move to less restricted action spaces.
Revised: It is critical to ensure that values are well-initialised as we move to less restricted action spaces.

Revised: In the discretised continuous control tasks, ‘SEP-Q’ underperforms, although less dramatically.

Original: The choice of target is less important: performing a max over coarser action spaces to construct the target as described in Section 4.2 does not improve learning speed as intended.
Revised: The choice of target is less important: performing a max over coarser action spaces to construct the target as described in Section 4.2 does not improve learning speed as intended.

Original: One potential reason is that maximising over more potential targets increases the maximisation bias already present in
Revised: One potential reason is that maximising over more potential targets increases the maximisation bias already present in

Original: Q -learning (Hasselt, 2010).
Revised: Q -learning (Hasselt, 2010).

Original: Additionally, we use an n -step objective which combines a partial onpolicy return with the bootstrap target, which could reduce the relative impact of the choice of target.
Revised: Additionally, we use an n -step objective which combines a partial onpolicy return with the bootstrap target, which could reduce the relative impact of the choice of target.

Original: Finally, we experiment with a higher (cid:96) .
Revised: Finally, we experiment with a higher (cid:96) .

Original: Unfortunately, asymptotic performance is degraded slightly once we use A 3 or higher.
Revised: Unfortunately, asymptotic performance is degraded slightly once we use A 3 or higher.

Original: One potential reason is that it decreases the average group size, pushing against the limits of the spatial resolution that may be captured by our CNN architecture.
Revised: One potential reason is that it decreases the average group size, pushing against the limits of the spatial resolution that may be captured by our CNN architecture.

Original: Higher (cid:96) increases the amount of time that there are fewer units than groups, leaving certain groups empty and rendering our masked pooling operation degenerate.
Revised: Higher (cid:96) increases the amount of time that there are fewer units than groups, leaving certain groups empty and rendering our masked pooling operation degenerate.

Original: We do not see a fundamental limitation that should restrict the further growth of the action space, although we note that most hierarchical approaches in the literature avoid too many levels of depth.
Revised: We do not see a fundamental limitation that should restrict the further growth of the action space, although we note that most hierarchical approaches in the literature avoid too many levels of depth.

Original: For example, Czarnecki et al.
Revised: For example, Czarnecki et al.

Original: (2018) only mix between two sizes of action spaces rather than the three we progress through in the majority of our GAS experiments.
Revised: (2018) only mix between two sizes of action spaces rather than the three we progress through in the majority of our GAS experiments.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we presented an algorithm for growing action spaces with off-policy reinforcement learning to efﬁciently shape exploration.
Revised: In this work, we presented an algorithm for growing action spaces with off-policy reinforcement learning to efﬁciently shape exploration.

Original: We learn value functions for all levels of a hierarchy of restricted action spaces simultaneously and transfer data, value estimates and representations from more restricted to less restricted action spaces.
Revised: We learn value functions for all levels of a hierarchy of restricted action spaces simultaneously, and transfer data, value estimates, and representations from more restricted to less restricted action spaces.

Original: We also present a strategy for using this approach in cooperative multi-agent control.
Revised: We also present a strategy for using this approach in cooperative multi-agent control.

Original: In discretised continuous control tasks and challenging multiagent StarCraft micromanagement scenarios, we demonstrate empirically the effectiveness of our approach and the value of off-action-space learning.
Revised: In discretised continuous control tasks and challenging multiagent StarCraft micromanagement scenarios, we demonstrate empirically the effectiveness of our approach and the value of off-action-space learning.

Original: An interesting avenue for future work is to automatically identify how to restrict action spaces for efﬁcient exploration, potentially through meta-optimisation.
Revised: An interesting avenue for future work is to automatically identify how to restrict action spaces for efﬁcient exploration, potentially through meta-optimisation.

Original: We also look to explore more complex and deeper hierarchies of action spaces
Revised: We also look to explore more complex and deeper hierarchies of action spaces.

Original: A A PPENDIX A. D ISCRETISED CONTINUOUS CONTROL 
Revised: A A PPENDIX A. D ISCRETISED CONTINUOUS CONTROL A.1. A DDITIONAL A BLATIONS

Revised: Here, we present results on some additional ablations of GAS on the discretised continous control tasks.

Revised: SEP-Q performs slightly worse on both tasks, a less dramatic failure than in the StarCraft experiments.

Revised: These value functions are simpler, and it is easier to learn the new action space’s value without relying so much on the previous one.

Revised: ON-AC performs worse only on Mountain

Original: For our experiments in discretised continous control, we use  a standard DQN trainer (Mnih et al., 2015) with the following parameters.
Revised: Car, suggesting once again that the signiﬁcance of this component of the algorithm is somewhat problem-dependent. We also test a version that follows the intuition of the ‘Match’ objective of M&M more closely, adapted for the value-based setting: instead of using an adaptive initialisation of each level’s Q -function as described in the main text, we use an L2 penalty to ‘Match’ the new level’s value function to its parent action, which should have a similar effect. This variant performs similarly here, perhaps slightly worse than the algorithm we present.

Original: [Table]
Revised: [Figure]

Revised: A.1. H YPERPARAMETERS

Revised: For our experiments in discretised continous control, we use a standard DQN trainer (Mnih et al., 2015) with the following parameters.

Revised: Parameter

Original: For GAS experiments, we keep the mixing coefﬁcient α = 0 for 25000 environment steps, and then increase it linearly by 1 every 25000 steps until reaching the maximum value.
Revised: Value batch size replay buffer size target update interval (cid:15) initial 1. (cid:15) ﬁnal 0. (cid:15) decay 25000 env steps (cid:96) lead-in 25000 env steps (cid:96) growth 25000 env steps env steps per model udpate Adam learning rate 5eAdam (cid:15) 1eFor GAS experiments, we keep the mixing coefﬁcient α = 0 for 25000 environment steps, and then increase it linearly by 1 every 25000 steps until reaching the maximum value.

Original: We use γ = 0 .
Revised: We use γ = 0 .

Original: 998 for our Acrobot experiments, but reduce it to γ = 0 .
Revised: 998 for our Acrobot experiments, but reduce it to γ = 0 .

Original: 99 for Mountain Car to prevent diverging Q -values.
Revised: 99 for Mountain Car to prevent diverging Q -values.

Original: Our model consists of fully-connected ReLU layers, with 128 hidden units for the ﬁrst and 64 hidden units for all subsequent layers.
Revised: Our model consists of fully-connected ReLU layers, with 128 hidden units for the ﬁrst and 64 hidden units for all subsequent layers.

Original: Two layers are applied as an encoder.
Revised: Two layers are applied as an encoder.

Original: Then, for each (cid:96) one layer is applied on the current embedding to produce a new embedding, and an evaluation layer on that embedding produces the Q -values for that level.
Revised: Then, for each (cid:96) one layer is applied on the current embedding to produce a new embedding, and an evaluation layer on that embedding produces the Q -values for that level.

Original: A. S TAR C RAFT MICROMANAGEMENT SCENARIOS A.2. S CENARIOS AND LEARNED STRATEGIES
Revised: A. S TAR C RAFT MICROMANAGEMENT SCENARIOS A.2. S CENARIOS AND LEARNED STRATEGIES

Original: We explore ﬁve Starcraft micromanagement scenarios: 50 hydralisks vs 50 hydralisks, 80 marines vs 80 marines, 80 marines vs 85 marines, 60 marines vs 65 marines, 95 zerglings vs 50 marines.
Revised: We explore ﬁve Starcraft micromanagement scenarios: 50 hydralisks vs 50 hydralisks, 80 marines vs 80 marines, 80 marines vs 85 marines, 60 marines vs 65 marines, 95 zerglings vs 50 marines.

Original: In these scenarios, our model controls the ﬁrst set of units, and the opponent controls the second set.
Revised: In these scenarios, our model controls the ﬁrst set of units, and the opponent controls the second set.

Original: The opponent is a scripted opponent that holds its location until an opposing unit is within range to attack.
Revised: The opponent is a scripted opponent that holds its location until an opposing unit is within range to attack.

Original: Then, the opponent will engage in an ”attack-closest” behavior, as described in Usunier et al.
Revised: Then, the opponent will engage in an ”attack-closest” behavior, as described in Usunier et al.

Original: (2016), where each unit individually targets the closest unit to it.
Revised: (2016), where each unit individually targets the closest unit to it.

Original: Having the opponent remain stationary until engaged makes this a more difﬁcult problem – the agent must ﬁnd its opponent, and attack into a defensive position, which requires good positions prior to engagement.
Revised: Having the opponent remain stationary until engaged makes this a more difﬁcult problem – the agent must ﬁnd its opponent, and attack into a defensive position, which requires good positions prior to engagement.

Original: As mentioned in section 6.2, all of our scenarios require control of a much larger number of units than previous work.
Revised: As mentioned in section 6.2, all of our scenarios require control of a much larger number of units than previous work.

Original: The 50 hydralisks and 80v80 marines scenarios are both imbalanced as a result of attacking into a defensive position.
Revised: The 50 hydralisks and 80v80 marines scenarios are both imbalanced as a result of attacking into a defensive position.

Original: The optimal strategy for 80 marines vs 85 marines and 60 vs 65 marines requires slightly more sophisticated unit positioning, and the 95 zerglings vs 50 marines scenario requires the most precise positioning.
Revised: The optimal strategy for 80 marines vs 85 marines and 60 vs 65 marines requires slightly more sophisticated unit positioning, and the 95 zerglings vs 50 marines scenario requires the most precise positioning.

Original: The agent can use the enemy’s initial stationary positioning to its advantage by slightly surrounding the opponent in a concave, ensuring that the outermost units are in its attack range, but far enough away to be out of range of the center-most enemy units.
Revised: The agent can use the enemy’s initial stationary positioning to its advantage by slightly surrounding the opponent in a concave, ensuring that the outermost units are in its attack range, but far enough away to be out of range of the center-most enemy units.

Original: Ideally, the timing of the groups in all scenarios should be coordinated such that all units get in range of the opponent at roughly the same point in time.
Revised: Ideally, the timing of the groups in all scenarios should be coordinated such that all units get in range of the opponent at roughly the same point in time.

Original: Figure 4 shows how our model is able to exhibit this level of unit control.
Revised: Figure 5 shows how our model is able to exhibit this level of unit control.

Original: A.2. F EATURES
Revised: A.2. F EATURES

Original: We use a standard features for the units and map, given by TorchcraftAI
Revised: We use a standard features for the units and map, given by TorchcraftAI

Original: For each of the units, the following features are extracted:
Revised: For each of the units, the following features are extracted:

Original: • Current x, y positions.
Revised: • Current x, y positions.

Original: • Current x, y velocities.
Revised: • Current x, y velocities.

Original: • Current hitpoints • Armor and damage values • Armor and damage types • Range versus both ground and air units • Current weapon cooldown •
Revised: • Current hitpoints • Armor and damage values • Armor and damage types • Range versus both ground and air units • Current weapon cooldown •

Original: A few boolean ﬂags on some miscellaneous unit attributes
Revised: A few boolean ﬂags on some miscellaneous unit attributes

Original: Approximate normalization for each feature keep its value approximately between 0-1.
Revised: Approximate normalization for each feature keep its value approximately between 0-1.

Original: For the map, the following features are extracted for each tile in the map:
Revised: For the map, the following features are extracted for each tile in the map:

Original: • a one-hot encoding of tile’s the ground height (4 channels) • boolean representing or not the given tile is walkable • boolean representing or not the given tile is buildable • and boolean representing or not the given tile is covered by fog of war.
Revised: • a one-hot encoding of tile’s the ground height (4 channels) • boolean representing or not the given tile is walkable • boolean representing or not the given tile is buildable • and boolean representing or not the given tile is covered by fog of war.

Original: The features form a HxWx 7 tensor, where our map has height H and width W .
Revised: The features form a HxWx 7 tensor, where our map has height H and width W .

Original: A.2. E NVIRONMENT DETAILS
Revised: A.2. E NVIRONMENT DETAILS

Original: We use a frame-skip of 25, approximately 1 second of real time, allowing for reasonably ﬁne-grained control but without making the exploration and credit assignment problems too challenging.
Revised: We use a frame-skip of 25, approximately 1 second of real time, allowing for reasonably ﬁne-grained control but without making the exploration and credit assignment problems too challenging.

Original: We calculate at every timestep the difference in total health points (HP) and number of units for the enemy from the last step, normalised by the total starting HP and unit count.
Revised: We calculate at every timestep the difference in total health points (HP) and number of units for the enemy from the last step, normalised by the total starting HP and unit count.

Original: As a reward function, we use the normalised damage dealt, plus 4 times the normalised units killed, plus an additional reward of 8 for winning the scenario by killing all enemy units.
Revised: As a reward function, we use the normalised damage dealt, plus 4 times the normalised units killed, plus an additional reward of 8 for winning the scenario by killing all enemy units.

Original: This reward function is designed such that the agent gets some reward for doing damage and killing units, but the reward from doing damage will never be greater than from winning the scenario.
Revised: This reward function is designed such that the agent gets some reward for doing damage and killing units, but the reward from doing damage will never be greater than from winning the scenario.

Original: Ties and timeouts are considered losses.
Revised: Ties and timeouts are considered losses.

Original: A. E XPERIMENTAL DETAILS A.3. M ODEL
Revised: A. E XPERIMENTAL DETAILS A.3. M ODEL

Original: As described in Section 6.2.2 a custom model architecture is used for Starcraft micromanagement.
Revised: As described in Section 6.2.2 a custom model architecture is used for Starcraft micromanagement.

Original: Each unit’s feature vector is embedded to size 128 in step 2 of Figure 2.
Revised: Each unit’s feature vector is embedded to size 128 in step 2 of Figure 2.

Original: The grid where the unit features and map features are scattered onto is the size of the Starcraft map of the scenario in walktiles downsampled by a factor of 8.
Revised: The grid where the unit features and map features are scattered onto is the size of the Starcraft map of the scenario in walktiles downsampled by a factor of 8.

Original: After being embedded, the unit features for ally and enemy units are concatenated with the downsampled map features and sent into a ResNet encoder with four residual blocks (stride 7 padding 3).
Revised: After being embedded, the unit features for ally and enemy units are concatenated with the downsampled map features and sent into a ResNet encoder with four residual blocks (stride 7 padding 3).

Original: The output is an embedding of size 64.
Revised: The output is an embedding of size 64.

Original: The decoder uses a mean pooling over the embedding cells as described in Section 6.2.2.
Revised: The decoder uses a mean pooling over the embedding cells as described in Section 6.2.2.

Original: Each evaluator is a 2-layer MLP with 64 hidden units and 17 outputs, one for each action.
Revised: Each evaluator is a 2-layer MLP with 64 hidden units and 17 outputs, one for each action.

Original: All layers are separated with ReLU nonlinearities.
Revised: All layers are separated with ReLU nonlinearities.

Original: A.3. T RAINING HYPERPARAMETERS
Revised: A.3. T RAINING HYPERPARAMETERS

Original: We use 64 parallel actors to collect data in a short queue from which batches are removed when they are consumed by the learner.
Revised: We use 64 parallel actors to collect data in a short queue from which batches are removed when they are consumed by the learner.

Original: We use batches of 32 6-step segments for each update.
Revised: We use batches of 32 6-step segments for each update.

Original: For the Q -learning experiments, we used the Adam optimizer with a learning rate of 2 .
Revised: For the Q -learning experiments, we used the Adam optimizer with a learning rate of 2 .

Original: 5 × 10 −and (cid:15)
Revised: 5 × 10 −and (cid:15)

Original: = 1 × 10 − 4 .
Revised: = 1 × 10 − 4 .

Original: For the MM baseline experiments, we use a learning rate of 1 × 10 − 4 , entropy loss coefﬁcient of 8 × 10 − 3 and value loss coefﬁcient 0 .
Revised: For the MM baseline experiments, we use a learning rate of 1 × 10 − 4 , entropy loss coefﬁcient of 8 × 10 − 3 and value loss coefﬁcient 0 .

Original: The learning rates and entropy loss coefﬁcient were tuned by random search, training with A 0 from scratch on the 80 marines vsmarines scenario with 10 conﬁgurations sampled from log uniform ( − 5 , − 3) for the learning rate and log uniform ( − 3 , − 1) for the entropy loss coefﬁcient.
Revised: The learning rates and entropy loss coefﬁcient were tuned by random search, training with A 0 from scratch on the 80 marines vs marines scenario with 10 conﬁgurations sampled from log uniform ( − 5 , − 3) for the learning rate and log uniform ( − 3 , − 1) for the entropy loss coefﬁcient.

Original: For Q -learning, we use an (cid:15) -greedy exploration strategy , decaying (cid:15) linearly from 1.0 to 0.1 over the ﬁrst 10000 model updates.
Revised: For Q -learning, we use an (cid:15) -greedy exploration strategy , decaying (cid:15) linearly from 1.0 to 0.1 over the ﬁrst 10000 model updates.

Original: We also use a target network that copies the behaviour model’s parameters every 200 model updates.
Revised: We also use a target network that copies the behaviour model’s parameters every 200 model updates.

Original: We also use a linear schedule to grow the action-space.
Revised: We also use a linear schedule to grow the action-space.

Original: There is a lead in of 5000 model updates, during which the action-space is held constant at A 0 , to prevent the action space from growing when (cid:15) or the policy entropy is too high.
Revised: There is a lead in of 5000 model updates, during which the action-space is held constant at A 0 , to prevent the action space from growing when (cid:15) or the policy entropy is too high.

Original: The action-space is then grown linearly at a rate of 10000 model updates per level of restriction, so that after 10000 updates, we act entirely at A 1 and after 20000, entirely at A 2 .
Revised: The action-space is then grown linearly at a rate of 10000 model updates per level of restriction, so that after 10000 updates, we act entirely at A 1 and after 20000, entirely at A 2 .