L EARNING TO R EPRESENT A CTION V ALUES AS A H YPERGRAPH ON THE A CTION V ERTICES
Anonymous authors
Paper under double-blind review
A BSTRACT
Action values are ubiquitous in reinforcement learning (RL) methods, with the sample complexity of such methods relying heavily on how fast a good estimator for action value can be learned.
By viewing this problem through the lens of representation learning, good representations of both state and action can facilitate action-value estimation.
While advances in deep learning have seamlessly driven progress in learning state representations, given the speciﬁcity of the notion of agency to RL, little attention has been paid to learning action representations.
We conjecture that leveraging the combinatorial structure of multidimensional action spaces is a key ingredient for learning good representations of action.
To test this, we set forth the action hypergraph networks framework—a class of functions for learning action representations in multidimensional discrete action spaces with a structural inductive bias.
Using this framework we realise an agent class based on a combination with deep Q-networks, which we dub hypergraph Q-networks .
We show the effectiveness of our approach on a myriad of domains: illustrative prediction problems under minimal confounding effects, Atari 2600 games, and discretised physical control benchmarks.
I NTRODUCTION
Representation learning methods have helped shape recent progress in RL by enabling a capacity for learning good representations of state.
This is in spite of the fact that, traditionally, representation learning was less often explored in the RL context.
As such, the de facto representation learning techniques which are widely used in RL were developed under other machine learning paradigms (Bengio et al., 2013).
Nevertheless, RL brings some unique problems to the topic of representation learning, with exciting headway being made in identifying and exploring such topics.
Action-value estimation is a critical component of the RL paradigm (Sutton & Barto, 2018).
Hence, how to effectively learn estimators for action value from training samples is one of the major problems studied in RL.
In this work, we set out to study this problem through the lens of representation learning, focusing particularly on learning representations of action in multidimensional discrete action spaces.
While action values are conditioned on both state and action and, as such, good representations of both would be beneﬁcial, there has been comparatively little research on the latter.
We frame this problem as learning a decomposition of the action-value function that is structured in such a way to leverage the combinatorial structure of multidimensional discrete action spaces.
This structure is an inductive bias which we incorporate in the form of architectural assumptions.
We present this approach as a framework to ﬂexibly build architectures for learning representations of multidimensional discrete actions by leveraging various orders of their underlying sub-action combinations.
Our architectures can be combined in succession with any other architecture for learning state representations and trained end-to-end using backpropagation, without imposing any change to the RL algorithm.
We remark that designing representation learning methods by incorporating some form of structural inductive biases is highly common in deep learning, resulting in highly-publicised architectures such as convolutional, recurrent, and graph networks (Battaglia et al., 2018).
We ﬁrst show the effectiveness of our approach in illustrative, structured prediction problems.
Then, we argue for the ubiquity of similar structures and test our approach in standard RL problems.
Our results advocate for the general usefulness of leveraging the combinatorial structure of multidimensional discrete action spaces, especially in problems with larger action spaces.
B ACKGROUND
We consider the RL problem in which the interaction of an agent and the environment is modeled as a Markov decision process (MDP) ( S , A , P, R, S 0 ) , where S denotes the state space, A the action space, P the transition distribution, R the reward distribution, and S
the initial state distribution (Sutton & Barto, 2018).
At each time step t the agent observes a state s t ∈ S and produces an action a t ∈ A , drawn from its policy π ( .
| s t ) .
The agent then transitions to and observes the next state s t +1 ∈ S , drawn from P ( .
| s t , a t ) , and receives a reward r t , drawn from R ( .
| s t , a t , s t +1 ) .
The standard MDP formulation generally abstracts away the combination of sub-actions that are activated when an action a t is chosen.
That is to say, if a problem has a natural N v -dimensional action space, each action a t maps onto an N v -tuple ( a 1 t , a 2 t , . .
. , a N v t ) where each a it is a sub-action from the i th sub-action space.
Therefore, the action space could have an underlying combinatorial structure where the set of actions are formed as a Cartesian product of the sub-action spaces.
To make this explicit, we express the action space as A .
=
A 1 ×
A 2 × · · · × A N v where each A i is a ﬁnite set of sub-actions.
Furthermore, we amend our notation for the actions a t into a t (in bold) to reﬂect that actions are generally a combination of several sub-actions.
Within our framework, we refer to each sub-action space
A i as an action vertex .
Naturally, the cardinality of the set of action vertices is the number of action dimensions N v .
Given a policy π that maps states onto distributions over the actions, the discounted sum of future rewards under π is denoted by the random variable Z π ( s, a ) = (cid:80) ∞ t =0 γ t r t , where s
=
s , a 0 = a , s t +1 ∼ P ( .
| s t , a t ) , r t ∼ R ( .
| s t , a t , s t +1 ) , a t ∼ π ( .
| s t ) , and 0 ≤ γ ≤ 1 is a discount factor.
The action-value function is then deﬁned as Q π ( s, a ) =
E [ Z π ( s, a )] .
Evaluating the action-value function Q π of a policy π is referred to as a prediction problem.
In a control problem, the objective is to ﬁnd an optimal policy π ∗ which maximises the action-value function.
The main thesis of this paper applies to any method for prediction or control, provided that they involve estimating an action-value function.
The canonical example of such a method for control is Q-learning (Watkins, 1989; Watkins & Dayan, 1992) which iteratively improves an estimate Q of the optimal action-value function Q ∗ via
[Equation]
where 0 ≤ α ≤ 1 is a learning rate.
The action-value function is typically approximated using a parameterised function Q θ , where θ is a vector of parameters, and trained by minimising a sequence of squared temporal-difference errors
[Equation]
[Equation]
) over samples ( s t , a t , r t , s t +1 ) .
Deep Q-networks (DQN) (Mnih et al., 2015) combine Q-learning with deep neural networks as function approximators to achieve human-level performance in the
Atari 2600 games.
2. D EFINITION OF HYPERGRAPH
A hypergraph (Berge, 1989) is a generalisation of a graph in which an edge, known as a hyperedge , can join any number of vertices.
Let V =
{A 1 , A 2 , . . . ,
A N v } be a ﬁnite set representing the set of action vertices A i .
A hypergraph on V is a family of subsets (or hyperedges)
H =
{ E 1 , E 2 , . . .
, E N e } such that
[Equation]
Thus, each hyperedge E j is a member of P ( V ) \∅ where P ( V ) , called the powerset of V , is the set of all possible subsets on V .
We denote P ( V ) \∅ by E as the space of all possible hyperedges on V .
The rank r of a hypergraph is deﬁned as the maximum cardinality of any of its hyperedges.
We deﬁne a c -hyperedge , where c ∈ { 1 , 2 , . . .
, N v } , as one with cardinality or order c .
The number of all c -hyperedges on V is given by the binomial coefﬁcient (cid:0) N v c (cid:1) .
We deﬁne a c -uniform hypergraph as one with only c -hyperedges.
As a special case, a c -complete hypergraph, denoted K c , is one with all possible c -hyperedges.
A CTION HYPERGRAPH NETWORKS FRAMEWORK
We now describe our framework using the example in Fig.
Consider the sample physical system of
Fig.
1a with six action vertices (solid circles).
A sample hypergraph is depicted for this system with four hyperedges (dashed shapes), featuring a 1-hyperedge, two 2-hyperedges, and a 3-hyperedge.
We remark that this set of hyperedges constitutes a hypergraph as there are no empty hyperedges (see Eq.
and the union of hyperedges spans the set of action vertices (see Eq.
We wish to enable learning a representation of each hyperedge in an arbitrary hypergraph.
To achieve this, we create a parameterised function U E j (e.g. a neural network) for each hyperedge E j which receives a state representation ψ ( s ) as input and returns as many values as the possible combinations of the sub-actions for the action vertices enclosed by its respective hyperedge.
In other words, U E j has as many outputs as the cardinality of a Cartesian product of the action vertices in E j .
Each such hyperedge-speciﬁc function U E j is a building block of our action hypergraph networks framework.
Figure 1b depicts the block corresponding to the 3-hyperedge from the hypergraph of Fig.
1a.
We remark that for any action a = ( a 1 , a 2 , . .
. , a N v ) from the action space A
, each block has only one output that corresponds to a and, thus, contributes exactly one value as a representation of its respective hyperedge at the given action.
This output U E j (cid:0) ψ ( s ) , a E j (cid:1) is identiﬁed by a E j which denotes the combination of sub-actions in a that correspond to the action vertices enclosed by E j .
We can realise an architecture within our framework by composing several such building blocks, one for each hyperedge in a hypergraph of our choosing.
Figure 1c shows an instance architecture corresponding to the sample hypergraph of Fig.
1a.
The forward view through this architecture is as follows.
A shared representation of the input state s is fed into multiple blocks, each of which features a unique hyperedge.
Then, a representation vector of size N e is obtained for each action a , where N e is the number of hyperedges or blocks.
These action-speciﬁc representations are then mixed (on an action-by-action basis) using a function f
(e.g. a ﬁxed non-parametric function or a neural network).
The output of this mixing function is our estimator for action value at the stateaction pair s, a .
Concretely,
[Equation]
Such an architecture can be combined with any action-value algorithm for prediction or control and trained end-to-end using backpropagation, without imposing any change to the RL algorithm.
While only a single output from each block is relevant for an action, the reverse is not the case.
That is, an output from a block generally contributes to more than a single action’s value estimate.
In fact, the lower the cardinality of a hyperedge, the larger the number of actions’ value estimates to which a block output contributes.
This can be thought of as a form of combinatorial generalisation.
Particularly, action value can be estimated for an insufﬁciently-explored or unexplored action by mixing the action’s corresponding representations which have been trained as parts of other actions.
Moreover, this structure enables a capacity for learning faster on lower-order hyperedges by receiving more updates, and slower on higher-order ones by receiving less updates.
This is a desirable intrinsic property as, ideally, we would wish to learn representations that exhaust their capacity for representing action values using lower-order hyperedges before they resort to higher-order ones.
3. M IXING FUNCTION SPECIFICATION
The mixing function receives as input a state-conditioned representation vector for each action.
We view these action-speciﬁc representation vectors as action representations .
Explicitly, these action representations are learned as a decomposition of the action-value function under the mixing function.
Without a priori knowledge about its appropriate form, the mixing function should be learned by a universal function approximator.
However, joint learning of the mixing function together with a good decomposition under its dynamic form could be challenging.
Moreover, increasing the number of hyperedges expands the space of possible decompositions, thereby making it even harder to reach a good one.
The latter is due to lack of identiﬁability in value decomposition in that there is not a unique decomposition to reach.
Nevertheless, this issue is not unique to our setting as representations learned using neural networks are in general unidentiﬁable.
We demonstrate the potential beneﬁt of using a universal mixer (a neural network) in our illustrative bandit problems.
However, to allow ﬂexible experimentation without re-tuning standard agent implementations, in our RL benchmarks we choose to use summation as a non-parametric mixing function.
This boils down the task of learning action representations to reaching a good linear decomposition of the action-value function under the summation mixer.
We remark that the summation mixer is commonly-used with valuedecomposition methods in the context of cooperative multi-agent RL (Sunehag et al., 2017).
In an informal evaluation in our RL benchmarks, we did not ﬁnd any advantage for a universal mixer over the summation one.
Nonetheless, this could be a matter of tuning the learning hyperparameters.
3. H YPERGRAPH SPECIFICATION
We now consider the question of how to specify a good hypergraph.
Actually, there is not an all-encompassing answer to this question.
For example, the choice of hypergraph could be treated as a way to incorporate a priori knowledge about a speciﬁc problem.
Nonetheless, we can outline some general rules of thumb.
In principle, including as many hyperedges as possible enables a richer capacity for discovering useful structures.
Correspondingly, an ideal representation is one that returns neutral values (e.g. near-zero inputs to the summation mixer) for any hyperedge whose contribution is not necessary for accurate action-value estimation, provided that lower-order hyperedges are able to represent its contribution.
However, as described in Sec. 3.1, having many mixing terms could complicate reaching a good decomposition due to lack of identiﬁability.
Taking these into consideration, we frame hypergraph speciﬁcation as choosing a rank r , whereby we specify the hypergraph that comprises all possible hyperedges of orders up to and including r .
Therefore,
[Equation]
where hypergraph H is speciﬁed as the union of c -complete hypergraphs K c .
Figure 2b depicts a class of models in our framework where the space of possible hyperedges E is ordered by cardinality.
On the whole, not including the highest-order ( N v ) hyperedge limits the representational capacity of an action-value estimator.
This could introduce (statistical) bias due to estimating N actions’ values using a model with M < N unique outputs.
In a structured problem M < N could sufﬁce, otherwise such bias is inevitable.
Consequently, choosing any r < N v could affect the estimation accuracy in a prediction problem and cause sub-optimality in a control problem.
Thus, preferably, we wish to use hypergraphs of rank r = N v .
We remark that this bias is additional to—and should be distinguished from—the bias of function approximation which also affects methods such as DQN.
We can view this in terms of the bias-variance tradeoff with r acting as a knob: lower r means more bias but less variance, and vice versa.
Notably, when the N v -hyperedge is present even the simple summation mixer can be used without causing bias.
However, when this is not the case, the choice of mixing function could signiﬁcantly inﬂuence the extent of bias.
In this work, we generally ﬁx the choice of mixing function to summation and instead try to include high-order hyperedges as much as possible.
I LLUSTRATIVE PREDICTION PROBLEMS
We set out to illustrate the essence of problems in which we expect improvements by our approach.
To do so, we minimise confounding effects in both our problem setting and learning method.
Given that we are interested purely in studying the role of learning representations of action (and not of state), we consider a multi-armed bandit problem setting.
We specify our bandits such that they have a combinatorial action space of three dimensions.
Moreover, the reward functions are deterministic but differ per random seed.
Despite using a different reward function in each independent trial, they are all generated to feature a good degree of decomposability with respect to the combinatorial structure of the action space.
We train predictors for reward using minimalist parameterised models that resemble tabular ones as closely as possible (e.g. our baseline corresponds to a standard tabular model) and train them using supervised learning.
For our approach, we consider summation and universal mixers as well as increasingly more complete hypergraphs, speciﬁed using Eq.
(6) by varying rank r from 1 to 3.
See Appendix A for details about the reward functions and hyperparameters.
Figure 3a shows the normalised RMS prediction error curves (each averaged over 64 random seeds) for two variants of our approach that leverage all possible hyperedges versus the baseline.
As shown, the ordering of the curves remains consistent with increasing number of actions, where the baseline is outperformed by our approach regardless of the mixing function.
Nevertheless, our model with a universal mixer performs signiﬁcantly better in terms of sample efﬁciency.
Moreover, the performance gap becomes signiﬁcantly wider as the action-space size increases, attesting to the utility of leveraging the combinatorial structure of actions for scaling to more complex action spaces.
Figure 3b shows the average RMS prediction error at the 400th iteration (as a proxy for “ﬁnal” performance) for all variants in our study.
As expected, including higher-order hyperedges and/or using a more generic mixing function improves ﬁnal prediction accuracy in every case.
Going beyond these simple prediction tasks to control benchmarks in RL we expect a general advantage for our approach, following the same pattern of yielding more signiﬁcant improvements in larger action spaces.
Indeed, this is if similar structures are ubiquitous in practical RL tasks.
A TARI 2600 GAMES
We tested our approach in the Atari 2600 games using the Arcade Learning Environment (ALE) (Bellemare et al., 2013).
The action space of the Atari 2600 is determined by a digital joystick with three degrees of freedom: three positions for each axis of the joystick, plus a button.
This implies that these games can have a maximum of 18 discrete actions, with many not making full use of the joystick capacity.
To focus our compute resources on games with a more complex action space, we limited our tests to the 29 Atari 2600 games from the literature that feature 18 valid actions.
For the purpose of our control experiments, we combine our approach with deep Q-networks (DQN) (Mnih et al., 2015).
The resulting agent, which we dub hypergraph Q-networks (HGQN), deploys an architecture based on our action hypergraph networks, similar to that shown in Fig.
2b, and using the summation mixer.
Given that the action space has merely three dimensions, we instantiate our agent’s model based on a hypergraph including all seven possible hyperedges.
We realise this model by modifying the DQN’s ﬁnal hidden layer into a multi-head one, where each head implements a block from our framework for a respective hyperedge.
To achieve a fair comparison with DQN, we ensure that our agent’s model has roughly the same number of parameters as DQN by making the sum of the hidden units across its seven heads to match that of the ﬁnal hidden layer in DQN.
Speciﬁcally, we implemented HGQN by replacing the ﬁnal hidden layer of 512 rectiﬁer units in DQN with seven network heads, each with a single hidden layer of 74 rectiﬁer units.
Our agent is trained end-to-end using backpropagation in the same way as DQN.
Our tests are conducted on a stochastic version of the Atari 2600 using sticky actions (Machado et al., 2018).
Figure 4 shows the relative human-normalised score of HGQN ( random seeds) versus DQN (5 random seeds) for each game.
Figure 5 shows mean and median human-normalised scores across the 29 Atari games for HGQN and DQN.
Our results indicate improvements over DQN on the majority of the games (see Fig. 4) as well as in overall performance (see Fig. 5).
Notably, these improvements are both in terms of sample complexity and ﬁnal performance (see Fig. 5).
The consistency of improvements in these games has the promise of greater improvements in tasks with larger action spaces.
See Appendix B for details, including the learning curves.
To further demonstrate the versatility of our approach, we combine it with a simpliﬁed version of
Rainbow (Hessel et al., 2018).
We ran the resulting agent on the three best and worst-performing games from Fig.
We report our results in Appendix B.1.
Notably, our results show that where HGQN outperforms DQN most signiﬁcantly, its combination with the simpliﬁed version of Rainbow also outperforms the corresponding Rainbow baseline.
P HYSICAL CONTROL BENCHMARKS
To test our approach in problems with larger action spaces, we consider a suite of physical control benchmarks, simulated using PyBullet (Coumans & Bai, 2019).
These domains feature continuous action spaces of diverse dimensionality, allowing us to test on a range of combinatorial action spaces via discretisation.
We discretise to obtain ﬁve sub-actions per joint.
See Appendix C for experiment [Figure] details, including changes to the network architectures to reﬂect the non-pixel nature of states in these domains.
We ran HGQN with increasingly more complete hypergraphs, speciﬁed using Eq.
(6) by varying rank r from 1 to 3.
Figure 6 shows the learning curves for HGQN and DQN.
In lowdimensional Reacher and Hopper, we do not see any signiﬁcant difference.
However, the higherdimensional domains render DQN entirely ineffective.
This clearly demonstrates the utility of action representations learned by our approach.
In Ant, we only ran HGQN with a 1-complete model as the other variants imposed greater computational demands.
We provide the average ﬁnal performance of DDPG (Lillicrap et al., 2016) to give a sense of the performance achieved by a continuous control method, particularly one that is closely related to
DQN.
Interestingly, HGQN generally outperforms DDPG despite using the training routine of DQN, without involving policy gradients (Sutton et al., 2000).
This may be due to local optimisation issues in DDPG which can lead to sub-optimal policies in multi-modal value landscapes (Metz et al., 2017; Tessler et al., 2019).
Furthermore, we include the performance of DQN combined with prioritised replay, dueling networks, multi-step learning, and Double Q-learning (denoted Rainbow † ; see Hessel et al.
(2018) for an overview).
The signiﬁcant gap between the performances of this agent and vanilla HGQN supports the orthogonality of our approach with respect to these extensions.
In Walker2D, we see a clear advantage for including the 2-hyperedges, but adding the 3-hyperedges relatively degrades performance.
This suggests that a second-order hypergraph strikes the right balance between bias and variance, where including 3-hyperedges causes higher variance and, potentially, overﬁtting.
HalfCheetah has the same action space as Walker2D, but we see little difference across hypergraphs.
This suggests that the 1-complete model is sufﬁcient in this case for learning a good action-value estimator.
Figure 7 shows average per-hyperedge representations for the greedy action learned by our third-order ( r = 3 ) HGQN models.
Speciﬁcally, we evaluated three trained third-order HGQN models using a greedy policy for 10k steps in each case.
At each step, we collected the greedy action’s corresponding representations (i.e. one representation for each hyperedge) and averaged them per hyperedge over the 30k steps.
The error bars show the minimum-maximum ranges of these representations.
In HalfCheetah, all signiﬁcant representations are concentrated on the 1-hyperedges.
In Walker2D, while 1-hyperedges receive higher average representations overall, there is at least a 2-hyperedge that receives a comparatively signiﬁcant average representation.
Importantly, this 2-hyperedge also receives the highest variation of values across the steps.
This 2-hyperedge, denoted { 1, 4 } , corresponds to the left and right hips in the agent’s morphology (see Fig.
7b).
Intuitively, a good bipedal walking behaviour relies heavily on the hip joints acting in unison.
Therefore, modelling their joint interaction explicitly enables a way of obtaining coordinated representations.
Interestingly, all signiﬁcant representations learned on the 3-hyperedges correspond to those including the hip joints.
This perhaps suggests that the 2-hyperedge representing the hip joints is critical in achieving a good walking behaviour and the representations learned by the 3hyperedges are only learned due to lack of identiﬁability in value decomposition (see Sec. 3).
R ELATED WORK
Value decomposition has been studied in cooperative multi-agent RL under the paradigm of centralised training but decentralised execution.
In this context, the aim is to decompose the joint action values into agent-wise values such that acting greedily with respect to the local values yields the same joint actions as those which maximise the joint action values.
A sufﬁcient but not necessary condition for a decomposition that satisﬁes this property is the monotonicity of the mixing function (Rashid et al., 2018).
An instance is VDN (Sunehag et al., 2017) which learns to decompose the joint action values onto a sum of agent-wise values.
QMIX (Rashid et al., 2018) advances this by learning a monotonic nonlinear mixer.
These are multi-agent counterparts of our 1-complete models combined with the respective mixing functions.
To increase the representational capacity of these methods, higher-order interactions need to be represented.
In a multi-agent RL context this means that fully-localised maximisation of the joint action values is no longer possible.
By relaxing this requirement and allowing some communication between the agents, coordination graphs (Guestrin et al., 2002) provide a framework for expressing higher-order interactions in a multi-agent setting.
Recent works have combined coordination graphs with neural networks and studied them in such settings (Castellini et al., 2019; B ¨ ohmer et al., 2020).
Our work repurposes coordination graphs from cooperative multi-agent RL as a method for action representation learning in standard RL.
Using Q-learning in continuous action problems by discretisation has proven as a viable alternative to continuous control.
Existing methods deal with the curse of dimensionality by factoring the action space and predicting action values sequentially (Metz et al., 2017) or independently (Tavakoli et al., 2018) across the action vertices.
Others resort to a sampling-based maximisation of action values (de Wiele et al., 2020).
Our work introduces an alternative approach based on value decomposition which further enables a capacity for explicitly modelling higher-order interactions among the action vertices.
We remark that our 1-complete models can achieve linear-time complexity when used with a monotonic mixer (e.g. summation) by maximising locally on each 1-hyperedge.
Sharma et al.
(2017) proposed a model on par with a 1-complete one in our framework and evaluated it in multiple Atari 2600 games.
In contrast to HGQN, their model shows little improvement beyond
DQN.
This performance difference is due to not including higher-order hyperedges, which in turn limits the representational capacity of their model.
Graph networks (Scarselli et al., 2009) have been combined with policy gradient methods for training modular policies that generalise to new agent’s morphologies (Wang et al., 2018;
Pathak et al., 2019; Huang et al., 2020).
The global policy is expressed as a collection of graph policy networks that correspond to each of the agent’s actuators, where every policy is only responsible for controlling its corresponding actuator and receives information from only its local sensors.
The information is propagated, based on some assumed graph structure, between the policies before acting.
Our work differs from this literature in many ways.
Notably, our approach does not impose any assumptions on the structure of state and, thus, is applicable to any problem with a multidimensional discrete action space.
D ISCUSSION AND FUTURE WORK
The requirement to maximise over the set of possible actions limits the applicability of Q-learning in environments with high-dimensional action spaces. In the case of approximate Q-learning, this limitation is partly due to the cost of computing the possible actions’ values before an exact maximi- sation can be performed. Our approach can bypass these issues in certain cases (e.g. when using a 1-complete hypergraph with a monotonic mixer; see Sec. 7 for more information).
In order to more generally address such issues, we can perform maximisation over a sampled set of actions instead of the entire set of possible actions (de Wiele et al., 2020). An approximate maximisation as such will enable including higher-order hyperedges in environments with high-dimensional action spaces. The above discussion is speciﬁc to the combination of our approach with Q-learning. Nevertheless, our approach can be combined with other action-value methods.
For instance, it can be used to learn a critic in an actor-critic method where an action drawn from the actor only needs to be evaluated by the critic in a single forward pass through the model.
Interestingly, this also allows us to use a continuous action actor with a discrete action critic.
We showed in our bandit problems of Sec.
4 the potential beneﬁt of using a more generic mixing function.
However, in an informal study on a subset of our RL benchmarks, we did not observe any improvements beyond our non-parametric summation mixer.
Further exploration of more generic, potentially state-conditioned, mixing functions is an interesting direction for future work.
An intriguing direction for future research is exploring regularisation strategies for promoting the emergence of useful sparse hypergraph structures.
Another interesting direction is to formulate the problem of hypergraph speciﬁcation as architecture search in a meta-learning problem setting.
A I LLUSTRATIVE PREDICTION EXPERIMENT DETAILS A. H YPERPARAMETERS
Each predictor was trained by backpropagation using supervised learning.
In each training iteration, we sampled a minibatch of 32 rewards (i.e. action values in our deterministic bandit problems) from the reward function to update the parameters.
We used the Adam optimiser (Kingma & Ba, 2015) to minimise the mean-squared prediction error.
The learning curves were generated by calculating the RMS prediction error across the actions every 100 training iterations for each independent trial.
We used minimalist parameterised models that resemble tabular ones as closely as possible.
As such, our baseline corresponds to a standard tabular model in which each action value is estimated by a single unique parameter.
In contrast, our approach does not require a unique parameter for each action as it relies on mixing multiple action-representation terms to produce an action-value estimate.
Therefore, for our approach we instantiated as many unique parameters as the total number of outputs from each model’s hyperedges.
For any model using a universal mixer, we additionally use a single hidden layer of 10 rectiﬁer units for mixing the action-representation terms, initialised using the Xavier uniform method (Glorot & Bengio, 2010).
The number of hyperedges in our hypergraphs of interest (see Sec. 3.2) can be expressed in terms of binomial coefﬁcients as
[Equation]
where hypergraph H is speciﬁed by its rank r according to Eq.
As we discussed in Sec. 3, each action’s value estimator is formed by mixing as many action-representation terms as there are hyperedges in the model (see Eq.
In our simple models for the bandit problems, each such term corresponds to a parameter in our model.
As such, each learning update per action involves updating as many parameters as the number of hyperedges.
Therefore, to ensure fair comparisons, we adapt the learning rates across different models in our study based on the number of parameters involved in updating each action’s value estimate.
Concretely, we set a single effective learning rate across all models in our study.
We then obtain each model’s individual learning rate α via
[Equation]
We used an effective learning rate of 0.0007 in our study.
We remark that while this achieves the same effective learning rate for both the baseline and our models using the summation mixer, the same does not hold exactly for our models using a universal mixer.
Nonetheless, this still serves as a useful heuristic to obtain similar effective learning rates across all models.
Importantly, the baseline receives the largest individual learning rate across all other models—especially one that improved its performance with respect to any other learning rates used by the variants of our approach.
A. R EWARD FUNCTION
The reward functions in our illustrative bandit problems are deterministic but differ per random seed.
In each independent trial, a reward function is generated such that it features a good degree of decomposability with respect to the combinatorial structure of the action space.
That is to say, by design, there is always at least one possible decomposition that has non-zero values on all possible hyperedges.
To achieve this, we create each reward function by randomly initialising a hypergraph model that includes all possible hyperedges.
Explicitly, in each independent trial, we begin by sampling as many values as the total number of outputs across all possible hyperedges.
We do so by sampling values uniformly from [-10, 10] for the 1-hyperedges, from [-5, 5] for the 2-hyperedges, and from [-2.5, 2.5] for the 3-hyperedges.
Doing so results in structured reward functions that can be decomposed to a good degree on the lower-order hyperedges but still need the highest-order hyperedge for a precise decomposition.
Next, we generate a random mixing function by uniformly sampling the parameters of a single-hidden-layer neural network from [-1, 1].
The number of hidden units and activation functions are sampled uniformly from { 1 , 2 , . .
, 5 } and { ReLU, tanh, sigmoid, linear } , respectively.
Lastly, a deterministic reward for each action is generated by mixing the subset of these sampled values that corresponds to the action.
B A TARI 2600 EXPERIMENT DETAILS
We base our implementation of HGQN on the Dopamine framework (Castro et al., 2018).
Dopamine provides a reliable open-source code for DQN as well as enables standardised benchmarking in the ALE under best-known evaluation practices (see, e.g., Bellemare et al.
; Machado et al.
As such, our evaluations are without any modiﬁcations to the agent or environment parameters with respect to those outlined in Castro et al.
(2018), except for the network architecture change for HGQN (see Sec. 5).
Our DQN results are based on the published Dopamine baselines.
We limited our tests in the Atari 2600 to 29 games from Wang et al.
(2016) that feature 18 valid actions, excluding Defender for which DQN results were not provided as part of the Dopamine baselines.
The human-normalised scores reported in this paper are given by the formula (similar to van Hasselt et al.
; Dabney et al.
[Equation]
(9) where score agent , score human , and score random are the per-game scores (undiscounted returns) for the given agent, a reference human player, and random agent baseline.
We use Table 2 from Wang et al.
(2016) to retrieve the human player and random agent scores.
The relative human-normalised score of HGQN versus DQN in each game (Fig. 4) is given by the formula (similar to Wang et al.
[Equation]
(10) where score HGQN and score DQN are computed by averaging over their respective learning curves.
B. A DDITIONAL RESULTS IN A TARI 2600 GAMES
We combine our approach with a simpliﬁed version of Rainbow (Hessel et al., 2018) that includes prioritised replay (Schaul et al., 2016), dueling networks (Wang et al., 2016), multi-step learning (Hessel et al., 2018), and Double Q-learning (van Hasselt et al., 2016).
We do not include C (Bellemare et al., 2017) as it is not trivial how it can effectively be combined with our approach.
We also do not combine with noisy networks (Fortunato et al., 2018) which are not implemented in Dopamine.
We denote the simpliﬁed Rainbow by Rainbow † and the version combined with our approach by HG-Rainbow † .
This experiment mainly serves to demonstrate the technical feasibility of combining our approach with several DQN extensions.
However, as each extension in Rainbow † impacts the learning process in certain ways, we believe that extensive work is required to establish whether such extensions are theoretically sound when combined with the learning dynamics of value decomposition.
In fact, a recent study on the properties of linear value-decomposition methods in multi-agent RL could hint at a potential theoretical incompatibility with certain replay schemes (Wang et al., 2020).
Therefore, we leave such a study as future work.
We ran these agents on the three best-performing and worst performing games from Fig.
We conjecture that games in which HGQN outperforms DQN most signiﬁcantly should have a structure that is exploitable by our approach.
Therefore, given that our approach is notionally orthogonal to the extensions in Rainbow † , we can also expect to see improvements by HG-Rainbow † over
Rainbow † .
Figure 8 shows the relative human-normalised score of HG-Rainbow † versus Rainbow † (3 random seeds in each case) along with those of HGQN versus DQN from Fig.
4 for six games, sorted according to Fig.
4 (the blue bars).
We see that generally the signs of relative scores for
Rainbow † -based runs are aligned with those for DQN-based runs.
Remarkably, in Stargunner the relative improvements are on par in both settings.
See Fig.
10 for the learning curves.
[Figure]
C P HYSICAL CONTROL EXPERIMENT DETAILS
We tested our approach on a suite of physical control benchmarks, simulated using PyBullet (Coumans & Bai, 2019).
PyBullet benchmarks provide a free and open-source alternative to the standard MuJoCo (Todorov et al., 2012) benchmarks of OpenAI Gym (Brockman et al., 2016).
The ﬁve environments in our experiment feature all PyBullet benchmarks, excluding those with one-dimensional action spaces (i.e. InvertedPendulum and InvertedDoublePendulum) or without a stable implementation (i.e. Humanoid).
Table 1 shows dimensionality ( N v ) and size ( |A| ) of the action spaces in these environments, where the latter is obtained by discretising each action space with the granularity of ﬁve sub-actions per joint.
Generally, we used the same agent implementations as in our Atari 2600 experiment, with the exception of the following changes.
We simpliﬁed the network architectures by replacing the convolutional networks with two hidden layers of 600 and 400 rectiﬁer units to reﬂect the non-pixel nature of states.
Moreover, we adapted the ﬁnal hidden layer from 512 to 400 rectiﬁer units, applying the same heuristic of dividing them equally across the number of network heads for HGQN (as in our Atari 2600 experiment).
We outline any changes to the hyperparameters with respect to those used in our Atari 2600 experiment in Table 2 (see Extended Data Table 1 of Mnih et al.
for descriptions of hyperparameters).
The reported DDPG results are based on SpinningUp (Achiam, 2018) which provides a reliable open-source code for DDPG.
The ﬁnal performances at 3M environment steps are obtained by averaging over the last 100k steps (20 random seeds).
The network architecture and hyperparameters match those reported in Lillicrap et al.
Bootstrapping from timeout transitions can signiﬁcantly improve performance in environments which have short auxiliary time limits (Pardo et al., 2018).
As such, in our physical control environments which have relatively short time limits, we apply this technique to all agents
[Table]
[Table] [Table]