S OLVING C ONTINUOUS C ONTROL VIA Q- LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
While there has been substantial success for solving continuous control with actorcritic methods, simpler critic-only methods such as Q-learning ﬁnd limited application in the associated high-dimensional action spaces.
However, most actorcritic methods come at the cost of added complexity: heuristics for stabilisation, compute requirements and wider hyperparameter search spaces.
We show that a simple modiﬁcation of deep Q-learning largely alleviates these issues.
By combining bang-bang action discretization with value decomposition, framing singleagent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods when learning from features or pixels.
We extend classical bandit examples from cooperative MARL to provide intuition for how decoupled critics leverage state information to coordinate joint optimization, and demonstrate surprisingly strong performance across a variety of continuous control tasks.
Simply combining bang-bang action discretization with full value decomposition scales to highdimensional control tasks and recovers performance competitive with recent actor-critic methods.
Our Decoupled Q-Networks (DecQN) thereby constitute a concise baseline agent to highlight the power of simplicity and to help put recent advances in learning continuous control into perspective.
I NTRODUCTION
Reinforcement learning provides a powerful framework for autonomous systems to acquire complex behaviors through interaction.
Learning efﬁciency remains a central aspect of algorithm design, with a broad spectrum spanning sample-efﬁcient model-based off-policy approaches (Ha & Schmidhuber, 2018; Hafner et al., 2019) at one extreme and time-efﬁcient on-policy approaches leveraging parallel simulation at the other extreme (Rudin et al., 2022;
Xu et al., 2022).
Particularly in high-dimensional domains with complicated environment dynamics and task objectives, complex trade-offs between representational capacity, exploration capabilities, and optimization accuracy commonly arise.
Continuous state and action spaces yield particularly challenging exploration problems due to the vast set of potential trajectories they induce.
Signiﬁcant research effort has focused on improving efﬁciency through representation learning in the context of model-free abstraction or model-based planning (Ha & Schmidhuber, 2018; Srinivas et al., 2020; Wulfmeier et al., 2021), guided exploration via auxiliary rewards (Osband et al., 2016;
Pathak et al., 2017; Sekar et al., 2020; Seyde et al., 2022b), or constrained optimization particularly to stabilize learning with actor-critic approaches
(Schulman et al., 2015; Haarnoja et al., 2018; Abdolmaleki et al., 2018).
However, recent results have shown that competitive performance can be achieved with strongly reduced, discretized versions of the original action space (Tavakoli et al., 2018; Tang & Agrawal, 2020; Seyde et al., 2021).
This opens the question whether tasks with complex high-dimensional action spaces can be solved using simpler critic-only, discrete action-space algorithms instead.
A potential candidate is Qlearning which only requires learning a critic with the policy commonly following via (cid:15) -greedy or
Boltzmann exploration (Watkins & Dayan, 1992; Mnih et al., 2013).
While naive Q-learning struggles in high-dimensional action spaces due to exponential scaling of possible action combinations, the multi-agent RL literature has shown that factored value function representations in combination with centralized training can alleviate some of these challenges (Sunehag et al., 2017; Rashid et al., 2018), further inspiring transfer to single-agent control settings (Sharma et al., 2017; Tavakoli, 2021).
Other methods have been shown to enable application of critic-only agents to continuous action spaces but require additional, costly, sampling-based optimization (Kalashnikov et al., 2018).
We build on insights at the intersection of these methods to show that a surprisingly straightforward variation of deep Q-learning (Mnih et al., 2013), within the framework of Hypergraph Q-Networks (HGQN) (Tavakoli et al., 2021), can indeed solve various state- and pixel-based single-agent continuous control problems at performance levels competitive with state-of-the-art continuous control algorithms.
This is achieved by a combination of extreme action space discretization and full value decomposition with extensive parameter-sharing, requiring only small modiﬁcations of DQN.
To summarize, this work focuses on the following key contributions:
• The DecQN agent as a simple, decoupled version of DQN combining value decomposition with bang-bang action space discretization to achieve performance competitive with stateof-the-art continuous control actor-critic algorithms on state- and pixel-based benchmarks.
• The related discussion of which aspects are truly required for competitive performance in continuous control as bang-bang control paired with actuator decoupling in the critic and without an explicit policy representation appears sufﬁcient to solve common benchmarks.
• An investigation of time-extended collaborative multi-agent bandit settings to determine how decoupled critics leverage implicit communication and the observed state distribution to resolve optimisation challenges resulting from correlations between action dimensions.
R ELATED W ORKS
Discretized Control Continuous control problems are commonly solved with continuous policies (Schulman et al., 2017; Abdolmaleki et al., 2018;
Haarnoja et al., 2018; Hafner et al., 2019;
Yarats et al., 2021).
Recently, it has been shown that even discretized policies can yield competitive performance and favorable exploration in continuous domains with acceleration-level control (Tavakoli et al., 2018; Farquhar et al., 2020;
Neunert et al., 2020; Tang & Agrawal, 2020; Seyde et al., 2021; 2022a).
When considering discretized policies, discrete action-space algorithms are a natural choice (Metz et al., 2017; Sharma et al., 2017; Tavakoli, 2021).
Particularly Q-learning based approaches promise reduced model complexity by avoiding explicit policy representations (Watkins & Dayan, 1992), although implicit policies in the form of proposal distributions may be required for scalability (Van de Wiele et al., 2020).
We build on perspectives from cooperative multi-agent
RL to tackle complex single-agent continuous control tasks with a decoupled extension of Deep QNetworks (Mnih et al., 2013) over discretized action spaces to reduce agent complexity and to better dissect which components are required for competitive agents in continuous control applications.
Cooperative MARL Conventional Q-learning requires both representation and maximisation over an action space which exponentially grows with the number of dimensions and does not scale well to the high-dimensional problems commonly encountered in continuous control of robotic systems.
Signiﬁcant research in multi-agent reinforcement learning (MARL) has focused on improving scalability of Q-learning based approaches (Watkins & Dayan, 1992).
Early works considered independent learners (Tan, 1993), which can face coordination challenges particularly when operating under partial observability (Claus & Boutilier, 1998; Matignon et al., 2012).
Some methods approach these challenges through different variations of optimistic updates (Lauer & Riedmiller, 2000; Matignon et al., 2007; Panait et al., 2006), while importance sampling can assist with off-environment learn-
ing (Foerster et al., 2017).
Furthermore, different degrees of centralization during both training and execution can address non-stationarity with multiple actors in the environment (Busoniu et al., 2006; Lowe et al., 2017; B¨ohmer et al., 2019).
We take a cooperative multi-agent perspective, where individual actuators within a single robot aim to jointly optimize a given objective.
Value Decomposition One technique to avoid exponential coupling is value function factorization.
For example, interactions may be limited to subsets of the state-action space represented by a coordination graph (Guestrin et al., 2002; Kok & Vlassis, 2006).
Schneider et al.
(1999) consider a distributed optimization by sharing return information locally, while Russell & Zimdars (2003) share value estimates of independent learners with a centralized critic.
Similarly, Yang et al.
(2018) factorize value functions across interactions with neighbors to determine best response behaviors.
Sunehag et al.
(2017) introduced Value-Decomposition Networks (VDNs) to approximate global value functions as linear combinations of local utility functions under partial observability.
Rashid et al.
(2018) extend this to non-linear combinations in QMIX, where a centralized critic further leverages global state information.
Son et al.
(2019) propose QTRAN to factorizes a transformation of the original value function based on a consistency constraint over a joint latent representation.
These insights can further be transferred to the multi-agent actor-critic setting (Wang et al., 2020; Su et al., 2021; Peng et al., 2021).
Our approach scales to complex morphologies by leveraging a linear factorization over action dimensions within a centralized critic conditioned on the global robot state.
Parameter Sharing Efﬁciency of learning a factored critic can improve through parametersharing (Gupta et al., 2017; B¨ohmer et al., 2020; Christianos et al., 2021) and non-uniform prioritization of updates (Bargiacchi et al., 2021).
In particular, the Hybrid Reward Architecture (HRA) proposed by Van Seijen et al.
(2017) provides a compact representation by sharing a network torso between individual critics (see also (Chu & Ye, 2017)).
We achieve efﬁcient learning at reduced model complexity through extensive parameter-sharing within a joint critic that splits into local utility functions only at the output neurons, while biasing sampling towards informative transitions through prioritized experience replay (PER) on the temporal-difference error (Schaul et al., 2015).
Decoupled Policies Stochastic policies for continuous control are commonly represented by Gaussians with diagonal covariance matrix, e.g. (Schulman et al., 2017; Haarnoja et al., 2018; Abdolmaleki et al., 2018), while we leverage decoupled Categoricals as a discrete counterpart.
Fully decoupled discrete control via Q-learning was originally proposed by Sharma et al.
(2017) to separate dimensions in Atari.
Tavakoli et al.
(2021) extend this concept to hypergraph Q-networks (HGQN) to leverage mixing across higher-order action subspaces (see also Appendix F).
Here, we demonstrate that learning bang-bang controllers, leveraging only extremal actions at the action space boundaries and inclusion of the zero action for bang-off-bang control (Bellman et al., 1956; LaSalle, 1960), with full decoupling as a minimal modiﬁcation to the original DQN agent yields state-of-the-art performance across feature- and pixel-based continuous control benchmarks.
Our investigation provides additional motivation for existing work and proposes a straightforward if surprising baseline to calibrate continuous control benchmark performance by pushing simple concepts to the extreme.
B ACKGROUND
We describe the reinforcement learning problem as a Markov Decision Process (MDP) deﬁned by the tuple {S , A , T , R , γ } , where S ⊂ R N and A ⊂ R M denote the state and action space, respectively, T : S×A → S the transition distribution, R : S×A → R the reward mapping, and γ ∈ [0 , 1) the discount factor.
We deﬁne s t and a t to be the state and action at time t and represent the policy by π ( a t | s t ) .
The discounted inﬁnite horizon return is given by G t = (cid:80) ∞ τ = t γ τ − t R ( s τ , a τ ) , where s t +1 ∼ T ( ·| s t , a t ) and a t ∼ π ( ·| s t ) .
Our objective is to learn the optimal policy maximizing the expected inﬁnite horizon return E [ G t ] under unknown transition dynamics and reward mappings.
3. P OLICY R EPRESENTATION
Current state-of-the-art algorithms for continuous control applications commonly consider the actorcritic setting with the policy modelled as a continuous distribution π φ ( a t
| s t ) independently from the value estimator Q θ ( s t , a t ) , or V θ ( s t ) , both represented by separate neural networks with parameters φ and θ .
Recent results have shown that comparable performance can be achieved with these approaches when replacing the continuous policy distributions by discretized versions (Seyde et al., 2021), at the potential cost of principled ways to represent stochastic policies.
One may then ask whether we require the actor-critic setting or if simpler discrete control algorithms are sufﬁcient.
Q-learning presents itself as a lightweight alternative by only learning the value function and recovering the policy by (cid:15) -greedy or Boltzmann exploration, side-stepping explicit policy gradients.
3. D EEP Q-N ETWORKS
We consider the framework of Deep Q-Networks (DQN) (Mnih et al., 2013), where the state action value function Q θ ( s t , a t ) is represented by a neural network with parameters θ .
The parameters are updated in accordance with the Bellman equation by minimizing the temporal-difference (TD) error.
We leverage several modiﬁcations that accelerate learning and improve stability based on the
Rainbow agent (Hessel et al., 2018) as implemented in Acme Hoffman et al.
In particular, we leverage target networks for improved stability in combination with double Qlearning to mitigate overestimation (Mnih et al., 2015; Van Hasselt et al., 2016).
We further employ prioritized experience replay (PER) based on the temporal difference error to bias sampling towards more informative transitions and thereby accelerate learning (Schaul et al., 2015).
Additionally, we combine rewards over several consecutive transitions into multi-step return to improve stability of
Bellman backups (Sutton & Barto, 2018).
A more detailed discussion of these aspects is included in
Appendix G, while we provide ablation studies on individual components in Appendix E.
Combining these modiﬁcation of the original DQN agent, we optimize the following loss function
[Equation]
where action evaluation employs the target y t = (cid:80) n − 1 j =0 γ j r ( s t + j , a t + j ) + γ n Q θ − (cid:0) s t + n , a ∗ t + n (cid:1) , action selection uses a ∗ t +1 = arg max a Q θ ( s t +1 , a ) , L δ ( · ) is the Huber loss and the batch size is B .
D ECOUPLED Q-N ETWORKS
The Rainbow agent (Hessel et al., 2018) from the previous section provides a framework for learning high-quality discrete control policies.
However, conventional Q-learning methods enumerate the action space and can therefore become inefﬁcient when attempting to scale to high dimensions.
4. M ULTI -A GENT P ERSPECTIVE
Scaling Q-learning to high-dimensional action spaces has been studied extensively in MARL settings.
Particularly in cooperative MARL it is common to consider factorized value functions in combination with centralized training and decentralized execution (CTDE).
Factorization reduces the effective dimensionality from the perspective of individual agents by learning utility functions conditioned on local observations.
Centralized training considers global information by composing individual utilities and optimizing for consistent behavior among agents.
Decentralized execution then assumes that the local utility functions are sufﬁcient for optimal distributed action selection.
4. F ACTORIZED Q -F UNCTION
Inspired by the MARL literature, we consider our agent to be a team of individual actuators that aim to cooperatively solve the given objective.
We further assume that the overall state-action value function Q ( s t , a t ) can be locally decomposed along action dimensions into a linear combination of
M single action utility functions Q jθ ( s t , a jt ) .
This simply applies the principle of value decomposition (Sunehag et al., 2017) and ﬁts within the general concept of HGQN-type algorithms (Tavakoli et al., 2021) as a simpliﬁed variation of the base case without a hypergraph as was also leveraged by Sharma et al.
(2017) for Atari.
The overall state-action value function is then represented as
[Equation]
where each utility function is conditioned on the global robot state to facilitate actuator coordination via implicit communication as we observe in our experiments.
This is further assisted by a highdegree of parameter sharing within a uniﬁed critic (Van Seijen et al., 2017) that only splits to predict decoupled state-action utilities at the outputs.
The linear combination of univariate utility functions then allows for efﬁcient decomposition of the argmax operator
[Equation]
where each actuator only needs to consider its own utility function.
Global optimization over a t then simpliﬁes into parallel local optimizations over a jt .
Training still considers the entire robot state and all joint actions within the centralized value function, while online action selection is decoupled.
Inserting this decomposition into the Bellman equation yields a decoupled target representation
[Equation]
We can then insert the factorized value function of Eq.
2 and the decoupled target of Eq.
4 into Eq.
We also considered a composition based on learned weights inspired by QMIX (Rashid et al., 2018).
Our ﬁndings suggest that access to the robot state throughout local utility functions outweighs the potential gains of more complex combinations, see also (Tavakoli et al., 2021) for a more detailed discussion on monotonic mixing functions.
The following section further discusses the importance of shared observations and state-space coordination based on illustrative matrix games.
E XPERIMENTS
The DecQN agent solves various continuous control benchmark problems at levels competitive with state-of-the-art continuous control algorithms.
First, we provide intuition for how DecQN reacts to coordination challenges based on illustrative matrix games.
Then, we show results for learning state-based control on tasks from the DeepMind Control Suite (Tunyasuvunakool et al., 2020) and
MetaWorld (Yu et al., 2020) - including tasks with action dimension dim ( A ) = 38 - and compare to the state-of-the-art DMPO and D4PG agents (Abdolmaleki et al., 2018;
Bellemare et al., 2017;
Barth-Maron et al., 2018).
We further provide results for pixel-based control on Control Suite tasks and compare to the state-of-the-art DrQ-v2 and Dreamer-v2 agents (Yarats et al., 2021; Hafner et al., 2020).
Lastly, we discuss qualitative results for learning a velocity-tracking controller on a simulated
Mini Cheetah quadruped in Isaac Gym (Makoviychuk et al., 2021) highlighting DecQN’s versatility.
5. I LLUSTRATIVE E XAMPLES : C OORDINATION IN M ATRIX G AMES
Representing the critic as a linear decomposition across agents enables scaling to high-dimensional tasks while limiting the ability to encode the full value distribution (Sunehag et al., 2017; Rashid et al., 2018).
This can induce coordination challenges (Lauer & Riedmiller, 2000; Matignon et al., 2012).
Here, we consider cooperative matrix games proposed by Rashid et al.
(2018) and Claus &
Boutilier (1998) to illustrate how DecQN still achieves coordination in a variety of settings.
Two-step game: We look at a 2-player game with three states proposed by Rashid et al.
In state 1, agent 1 playing action A/B transitions both agents to state 2/3 without generating reward.
In states 2 or 3, the actions of both agents result in the cooperative payout described in Figure 3 (left).
Rashid et al.
(2018) showed that a VDN-type decomposition is unable to learn the correct value distribution under a uniform policy, which our results conﬁrm for DecQN with (cid:15) =
(middle).
However, DecQN solves the task for non-uniform sampling with (cid:15) = 0 .
5 by learning an accurate value distribution around the optimal policy (right).
In online RL settings, agents actively shape the data distribution and learning to differentiate behavior quality is often sufﬁcient for solving a task.
In the following we consider cooperative 2-player matrix games based on Claus & Boutilier (1998).
We further augment the tasks by point-mass dynamics, where each agent selects acceleration input along the x or y-axis.
We evaluate three variations: ”1-step” resets after 1 step (no dynamics); ”1K-step” resets after 1000 steps (dynamics); ”Optimism” runs ”1K-step” with optimistic updating inspired by Lauer & Riedmiller (2000).
Each agent selects from three actions and we set (cid:15) =
Optimism helps
DecQN in action space, while
DecQN matches DQN performance under state space reward.
Penalty game: The reward structure is deﬁned in actionspace and provided in Figure 3 (right).
The two maxima at ( a 1 , a 2 ) =
{ ( − 1 , − 1) , (+1 , +1) } and minima at ( a 1 , a 2 ) = { ( − 1 , +1) , (+1 , − 1) } induce a Pareto-selection problem that requires coordination.
DecQN is unable to consistently solve the 1-step game for high costs k while it does solve the multi-step variation.
In the latter, we ﬁnd that DecQN coordinates actions (acceleration) based on state information (velocity).
Computing action outputs over varying velocity inputs indicates a clear correlation in directionality across the unambiguous velocity pairings (middle).
This underlines the importance of shared observations to achieve coordination with decoupled agents.
Climbing game: The reward structure is deﬁned in Figurewith action (top) and state space (bottom) rewards.
In action space, Nash equilibria at ( a 1 , a 2 ) =
{ ( − 1 , − 1) , (0 , 0) } are shadowed by action ( a 1 , a 2 ) =
{ (1 , 1) } , which has a higher minimum gain should one agent deviate (Matignon et al., 2012).
DecQN often settles at the sub-optimal action ( a 1 , a 2 ) =
(0 , 0) to avoid potential cost of unilateral deviation, which can be resolved by optimistic updating.
However, this can impede learning in domains that require state-space exploration as highlighted for state-space reward (bottom).
We there ﬁnd DecQN and DQN to achieve similar performance.
In fact, the tasks considered in this paper do not require coordination at the action level but rather in higher-level integrated spaces.
Generally, our objective is learning control of articulated agents, which requires temporarilyextended coordination in state space.
We ﬁnd that while a decoupled critic may not be able to represent the full action-value distribution, it is often sufﬁcient to differentiate action quality around the optimal policy.
In particular, centralized learning combined with shared observations enables actuator coordination even in complex, high-dimensional domains as we observe in the next section.
5. B ENCHMARKING : C ONTINUOUS C ONTROL FROM S TATE I NPUTS
We evaluate performance of the DecQN agent on several continuous control environments from the
DeepMind Control Suite as well as manipulation tasks for the Sawyer robot from MetaWorld.
We consider a Bang-Off-Bang policy encoded via a 3 -bin Categorical distribution and use the same set of hyperparameters across all tasks.
Performance mean and standard deviation across 10 seeds are provided in Figure 5 and we compare to the recent DMPO and D4PG agents, with additional training details provided in Appendix A.
We note that DecQN exhibits very strong performance with respect to the continuous baselines and sometimes even improves upon them, despite operating over a heavily-constrained action subspace.
This extends even to the high-dimensional Humanoid and
Dog domains with action dimensions dim ( A ) = 21 and dim ( A )
= 38 , respectively.
Signiﬁcant research effort has produced highly capable continuous control algorithms through advances in representation learning, constrained optimization and targeted exploration.
It is then surprising that a simple decoupled critic operating over a discretized action space and deployed with only an (cid:15) -greedy policy is able to provide highly competitive performance.
5.2. S CALABILITY OF THE D ECOUPLED R EPRESENTATION
To put the benchmarking results into perspective we compare DecQN to the non-decoupled DQN agent for a 3 -bin discretization each.
DQN conventionally enumerates the action space and scales exponentially in the action dimensionality, whereas DecQN only scales linearly (see also Figure 2).
According to Figure 6, performance of DecQN and DQN remains comparable on low-dimensional tasks, however, for high-dimensional tasks the DQN agent quickly exceeds memory limits due to the large number of output parameters required.
This contrast is ampliﬁed when signiﬁcantly increasing granularity of DecQN’s discretization to 21 bins.
Generally, increasing granularity increases the action space and in turn yields more options to explore.
While exploration over ﬁner discretizations is likely to delay convergence, we observe highly competitive performance with both 3 and 21 bins.
5. B ENCHMARKING : C ONTINUOUS C ONTROL FROM P IXEL I NPUTS
[Table]
We further evaluate on a selection of visual control tasks to see whether DecQN’s decoupled discrete approach is compatible with learning latent representation from visual input.
To this end, we combine DecQN with a convolutional encoder trained under shiftaugmentations as proposed by Yarats et al.
Figure 7 compares DecQN with a bang-bang parameterization ( 2 -bin) to the state-of-the-art model-free DrQ-v2 and model-based Dreamer-v agents.
DecQN and Dreamer-v2 employ a ﬁxed set of hyperparameters across all tasks, while DrQ-v2 leverages a few task-speciﬁc variations.
We note that
DecQN is highly competitive across several environments while relinquishing some performance on the complex Humanoid task.
Scaling DecQN to efﬁcient learning of latent representation for high-dimensional systems under action penalties will be an interesting challenge.
As the highest performing runs of DecQN on Humanoid Walk reach competitive performance, we are optimistic that this will be viable under potential parameter adjustments or more sophisticated exploration strategies.
As both DrQ-v2 and Dreamer-v2 provide limited data on Quadruped Run (3M and 2M timesteps, respectively), we further re-ran the authors’ code on our hardware and record average timings for one cycle of data collection and training.
We observe favorable runtime results for DecQN in Table 1 and estimate that code optimization could further reduce timings as observed for DrQ-v2.
5. C OMMAND - CONDITIONED L OCOMOTION C ONTROL
We additionally provide qualitative results for learning a command-conditioned locomotion controller on a simulated Mini Cheetah quadruped to demonstrate the broad applicability of decoupled discretized control.
We consider a minimal implementation of DecQN in PyTorch simulated with
Isaac Gym.
The agent aims to track velocity commands by predicting position targets for the leg joints, which are then provided to a low-level proportional-derivative (PD) controller.
As targets are generated in joint angle space, we consider a 31 -bin discretization.
We evaluate the resulting command-conditioned policy based on tracking performance when following a reference position trajectory.
To this end, velocity commands are generated by a simple proportional controller on the position error between the reference and the robot base.
Figure 8 showcases tracking for two references.
The circular trajectory has a diameter of 5 m and is tracked at a velocity of 1 m / s (middle), while the step trajectory introduces lateral jumps of 0 .
5 m
and is tracked at a velocity of 0 .
7 m / s (right).
In both scenarios we observe good tracking performance at levels competitive with the commonly used PPO agent (Rudin et al., 2022).
The system reacts well even to strong jumps in the reference with quick recovery in response to slight overshooting.
This underlines the versatility of simple decoupled discrete control, learning policies for various control types including application of acceleration or velocity control and generating position targets for downstream control systems.
C ONCLUSION
Recent successes in data-efﬁcient learning for continuous control have been largely driven by actorcritic algorithms leveraging advances in representation learning, constrained optimization, and targeted exploration.
In this paper, we take a step back and investigate how a simple variation of
Q-learning with bang-bang action discretization and (cid:15) -greedy exploration compares to current stateof-the-art algorithms on a variety of continuous control tasks.
Decoupled Q-Networks (DecQN) discretizes the action space, fully decouples individual action dimensions, and frames the singleagent control problem as a cooperative multi-agent optimization.
This is then solved via centralized training with decentralized execution, where the decoupled critic can be interpreted as a discrete analog to the commonly used continuous diagonal Gaussian policies.
We observe highly competitive performance of DecQN on state-based control tasks while comparing to the state-of-the-art
D4PG and DMPO agents, extend the approach to pixel-based control while comparing to the recent
DrQ-v2 and Dreamer-v2 agents, and demonstrate that the method solves more complex commandconditioned tracking control on a simulated Mini Cheetah quadruped.
DecQN is highly versatile, learning capable policies across a variety of tasks, input and output modalities.
This is intriguing, as DecQN does not rely on many of the ingredients associated with state-of-the-art performance in continuous control and completely decouples optimisation and exploration across the agent’s action dimensions.
We provide intuition for how the decoupled critic handles coordination challenges by studying variations of matrix games proposed in prior work.
We observe that shared access to the global observations facilitates coordination among decoupled actuators.
Furthermore, it is often sufﬁcient for a critic to accurately reﬂect relative decision quality under the current state distribution to ﬁnd the optimal policy.
Promising future work includes the interplay with guided exploration and the investigation of these concepts on physical systems.
There is great value in contrasting complex state-of-the-art algorithms with simpler ones to dissect the importance of individual algorithmic contributions.
Identifying the key ingredients necessary to formulate competitive methods aides in improving the efﬁciency of existing approaches, while providing better understanding and foundations for future algorithm development.
To this end, the
DecQN agent may serve as a straightforward and easy to implement baseline that constitutes a strong lower bound for more sophisticated methods, while raising interesting questions regarding how to assess and compare continuous control benchmark performance.
[-= (cid:80) Mj =
-]
[-Particularly for robotics applications, many common reward structures depend (approximately) linearly on the next state observation while the system dynamics are control-afﬁne such that s t +
-]