M ULTI -C RITIC A CTOR L EARNING : T EACHING RL P OLICIES TO A CT WITH S TYLE
A BSTRACT
Using a single value function (critic) shared over multiple tasks in Actor-Critic multi-task reinforcement learning (MTRL) can result in negative interference between tasks, which can compromise learning performance.
Multi-Critic Actor
Learning (MultiCriticAL) proposes instead maintaining separate critics for each task being trained while training a single multi-task actor.
Explicitly distinguishing between tasks also eliminates the need for critics to learn to do so and mitigates interference between task-value estimates.
MultiCriticAL is tested in the context of multi-style learning, a special case of MTRL where agents are trained to behave with different distinct behavior styles, and yields up to 56% performance gains over the single-critic baselines and even successfully learns behavior styles in cases where single-critic approaches may simply fail to learn.
In a simulated real-world use case, MultiCriticAL enables learning policies that smoothly transition between multiple ﬁghting styles on an experimental build of EA’s UFC game.
I NTRODUCTION
Reinforcement Learning (RL) offers an interesting means by which to develop interesting behaviors in a variety of controls settings.
The work presented in this paper is primarily contextualized by our efforts to develop RL techniques for teaching RL-policies to behave with multiple distinct and speciﬁc styles for game-play.
RL can be useful in developing control policies for game-play and has been demonstrated as being capable of learning human-like or even superhuman play, most notably beating top-ranked human players in games such as Dota 2 (Berner et al., 2019), StarCraft (Vinyals et al., 2019), Chess (Silver et al., 2017), and Go (Silver et al., 2016).
RL-based control in games offers a wide array of potential applications, including testing (Zhao et al., 2020;
Ariyurek et al., 2021; Gordillo et al., 2021), game design (Gissl ´ en et al., 2021), providing competition for human players (Berner et al., 2019), or simply as a means to develop more interesting game-play AI behaviors (Zhao et al., 2020; Alonso et al., 2020).
RL algorithms can be notorious however for training aesthetically unappealing policies, but prior work has demonstrated that, with careful shaping of the reward functions, it is possible to derive highly speciﬁc desirable behavior (Peng et al., 2018; Mysore et al., 2021b).
Machine learning (ML) solutions to game-play AI can however represent a signiﬁcant increase in compute requirements over their heuristic-based counterparts, especially when using deep learning techniques.
If a learned control policy could represent multiple desirable styles and offered a controllable way to transition between the learned styles, this could reduce the total compute burden of the ML-based controller, making it more practically viable.
Ideally, developing an
ML model would constitute learning a single, low-complexity policy that would model and transition smoothly between multiple behavior styles.
Reduced model complexity and deploying just a single policy network enables greater compute efﬁciency through hardware parallelism and reduced memory costs, which is beneﬁcial in resource-constrained applications, such as videogames.
We explore the problem of mutli-style RL, a special case of the more commonly explored problem of multi-task RL (MTRL).
Whereas MTRL techniques typically seek to solve a wide array of
RL problems under a single learning campaign, multi-style RL adds the extra constraint that the ‘tasks’ in question all have identical system dynamics but would be characterized by different re-
[Figure]
ward signals corresponding to the different desired styles of behavior. We focus speciﬁcally on applications of Actor-Critic RL techniques, which, in addition to being compatible with continuous action-space control, allow for reduced runtime computational cost as only the actor functions are required for inference, while the critics can often be disregarded after training.
Mysore et al.
(2021a)
and Andrychowicz et al.
(2021) have also demonstrated that it is possible to train compact and performant policy networks through careful consideration of the actor and critic network architectures.
A common baseline approach to MTRL is to employ one-hot task encoding to delineate between tasks.
However, in a number of multi-style cases considered this was not sufﬁcient to enable successful multi-style policy-learning.
We suspected that the similarity of game-play states visited between each trained style interfered with each style’s value optimization.
In surveying existing MTRL literature for advancements in the ﬁeld of multi-task Actor-Critic RL methods, we noticed a surprising hole.
There are studies that employ single actors and single critics (Finn et al., 2017;
Yang et al., 2020; Sodhani et al., 2021; Nichol et al., 2018), multiple actors and multiple critics (Andreas et al., 2017;
Teh et al., 2017; Rusu et al., 2015; 2016; Huang et al., 2017), or multiple actors and a single critic (Yang et al., 2017; Dewangan et al., 2018).
Curiously, seemingly no prior work explores using a single policy with distinct, per-task value estimation, i.e. a single actor with multiple critics.
This paper proposes Multi-Critic Actor Learning (MultiCriticAL), a single-actor, multi-critic framework for multi-task Actor-Critic optimization, and evaluates it on a variety of multi-style learning problems.
The core idea of MultiCriticAL is to maintain separate per-style (or task) critic functions for each style being learned.
By separating the critic MultiCriticAL would be able to avoid negative interference in the learned values for different styles.
Our results show that MultiCriticAL consistently trains more performant policies, compared to its single-critic counterparts, succeeds in cases where the single-critic methods fail and achieves between 20-45% improvement in more traditional multi-task settings involving multi-level game-play.
Additionally, we demonstrate the utility of MultiCriticAL in a use-case more reﬂective of real-world application, in an experimental build for EA’s
UFC Knockout Mode, where we train policies to act with multiple speciﬁc and distinct behavior styles.
This work is also amongst the ﬁrst to study the efﬁcacy of MTRL algorithms across a broad range of multi-style problems, an aspect of MTRL receiving limited prior attention.
P RELIMINARIES
The core reinforcement learning (RL) problem is framed as a ﬁnite-horizon Markov decision process, (cid:104) S, A, P, r, γ (cid:105) , with state transition dynamics P : S, A → S , for states s ∈ S and actions a ∈ A , and reward function r : S, A → R .
The RL objective is to train a policy, π : S
→ A such that the expected sum of discounted rewards, E τ π (cid:2)(cid:80) i γ i r
( s i , a i ) (cid:3) , with discount factor γ , is maximized for all trajectories, τ π , sampled on policy π .
RL typically assumes that both the transition dynamics and reward dynamics are encompassed by a black-box environment.
RL agents interact with the environment to observe the state response and generated rewards, but where speciﬁc reward functions and transition dynamics are not assumed known.
Solving the RL optimization problem requires estimating expected rewards and using these approximations to optimize the action policies.
Among the different techniques for tackling RL optimization, we mainly consider Actor-Critic methods, whose key characteristic is their separation of the ‘actors’, representing the RL policy, and the ‘critic’, representing the value-function estimator that learns to estimate future rewards.
This sep- aration allows Actor-Critic RL to take advantage of the improved sample-efﬁciency of value-based techniques, such as Q-learning (Watkins & Dayan, 1992) while taking advantage of policy-based techniques, such as policy gradients (Sutton et al., 2000), for improved robustness to stochasticity and to enable learning in continuous action domains.
Examples of prominent contemporary algorithms in this class of RL methods include TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017), DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018).
In algorithms based on Q-learning, such as DDPG, TD3, and SAC, the policy optimization criteria
J π θ for a policy, π , parameterized by θ , is proportional to the Q-value:
[Equation]
where s (cid:48) is the next state reached from state s in response to action a , and R ( τ ) is the discounted sum of rewards over trajectory τ .
For algorithms based on a more traditional policy gradient optimization, such as TRPO and PPO, policies are optimized proportional to the advantage function, A π θ and a measure of policy divergence:
[Equation]
where the value-function V π ( s ) =
E τ π
[ R ( τ π ) | s
= s ]
=
E s [Equation]
[ r ( s, π ( s )) + γV π ( s (cid:48) )] .
For critic networks parameterized by φ , either a state-value estimator, V φ , or Q-value (state-action-value) estimator, Q φ are optimized with optimization criteria J V πφ or J Q πφ respectively:
[Equation]
While speciﬁc details around how actors and critics are optimized vary between algorithms, the above represents the fundamentals of optimization criteria upon which we shall build our discussion.
R ELATED W ORK
We broadly categorize multi-task RL (MTRL) techniques as either single-policy , if the same policy function activations are used to determine actions over all tasks, or multi-policy , if more separate activations (in the case of multi-headed policies) or policy functions are used.
Multi-policy techniques range from training multiple policies and selecting or mixing between them (Andreas et al., 2017;
Peng et al., 2018; Dewangan et al., 2018; Yang et al., 2017; Huang et al., 2017), to simultaneously or iteratively training additional policies while sharing representation between policies (Rusu et al., 2015; Teh et al., 2017; Rusu et al., 2016).
Techniques requiring multiple policy networks are limited by their storage and computational costs during inference, and multi-headed policies can limit our ability to smoothly transition between behaviors, which makes them less attractive in the context of
RL for game-play.
Therefore, we focus instead on single-policy learning.
Single-policy MTRL techniques train a single policy function to solve multiple tasks, without relying on information from previously trained policies.
Model-free meta-learning techniques have been proposed to improve multi-task generalization, in the absence of task speciﬁcity (Finn et al., 2017;
Nichol et al., 2018), but where explicit task distinction can be exploited, a common approach has been to exploit task encoding to distinguish between tasks.
A simple, yet effective approach to this problem has been to extend state representations with one-hot task encoding (Peng et al., 2018; Yu et al., 2020; Yang et al., 2020), thus allowing policies to distinguish between individual tasks and decide actions in a task-appropriate way.
Recent work has also sought to enhance the performance and generalizability of such single-policy techniques through automated task encoding (Sodhani et al., 2021) or through soft-modularization (Yang et al., 2020), which encourages the development of distinct information pathways per task within actor networks.
Our work considers the special MTRL case of multi-style learning.
Multi-style learning requires agents to learn different behaviors for highly similar states while providing potentially highly dissimilar rewards for each style.
This variability in the reward signals can have an adverse impact on the quality of learned value-functions when using a single value estimator if the estimator cannot sufﬁciently distinguish between styles.
We hypothesize that negative interference more strongly impacts value-function learning in multi-style tasks as task similarity may make speciﬁc style-values harder to distinguish.
Multi-style learning is however an aspect of MTRL that has not received much prior consideration.
Peng et al.
(2018) explore the most similar setting to our work – where agents were required to learn different movement styles in animation tasks.
Their work tested single-policy
MTRL and found that policies were only able to learn a limited number of styles, requiring the authors to resort to a multi-policy approach.
Most works considering the Actor-Critic MTRL problem focus predominantly on how the actors, i.e the policy-functions, are optimized, with less attention given to critics.
The problem of negative interference between tasks in the broader context of MTRL is one that has been highlighted and studied before (Yang et al., 2020; Parisotto et al., 2015;
Teh et al., 2017; Standley et al., 2020) however, but has similarly been approached as a policy optimization problem.
A number of multipolicy techniques were noted to maintain separate valuations per policy (Teh et al., 2017; Andreas et al., 2017; Peng et al., 2018), though some work has also suggested that sharing critics can allow optimization to better capture useful shared knowledge to improve policy optimization (Dewangan et al., 2018; Yang et al., 2017).
A similar idea dominates single-policy techniques, where all the studied methods maintain a single value-function estimator across all tasks (Nichol et al., 2018;
Yu et al., 2020; Peng et al., 2018; Yang et al., 2020;
Sodhani et al., 2021).
While task-encoding may allow separate task-values to be learned implicitly by the critics, this was never found to be explicitly designed.
Our work alternatively frames the multi-style learning framework as one that explicitly learns distinct per-style values to propose a value-learning optimization scheme that is better able to accommodate disparate reward functions.
M ULTI -C RITIC A CTOR L EARNING
The policy functions, i.e. the ‘actor’ part of Actor-Critic frameworks, represent the core decisionmaking processes of RL agents and the success or failure of any algorithm rests on the policies’ abilities to learn useful mappings.
The learned critics have limited utility after training and learned value-function estimators are also known for being poor approximations of the true value-functions they attempt to capture (Ilyas et al., 2020), so it makes sense that prior work has focused more on how policies are constructed and trained.
The limitations of value learning are however exactly why the speciﬁcs of value-network constructions and optimization are important to consider.
A critical element in the performance of RL algorithms can be the representational capacity of the learned value-functions (Mysore et al., 2021a; Andrychowicz et al., 2021).
It, therefore, stands to reason, that better structuring how value-networks represent the learned values can be an important component of improving value-function learning and therefore policy performance.
A common formulation of single-critic MTRL optimization involves including a task-encoding, Ω , to distinguish between learned tasks and to specify task rewards and updating the state representation
¯ s = (cid:104) s, Ω (cid:105) .
The optimization criteria in Equations 1-4 are correspondingly updated to use ¯ s in place of s .
The resulting MTRL optimization criteria typically represent rewards and learned values by scalar values.
Practically, the value-functions are learned by a single network with a single output node, regardless of the number of tasks.
A single learned value-function representing multiple values may enforce continuity between the learned values, which may not necessarily reﬂect the true value functions and could compromise the quality of learned values and resulting policies.
Value Optimization for Multi-Critic Actor Learning By separating the learned values per style, assumptions of continuity between style-values can be dismissed and style (or task) values can be learned explicitly, without relying on the critics to inherently learn the distinctions.
We propose
Multi-Critic Actor Learning (MultiCriticAL) with a style-conditioned value optimization criteria:
[Equation]
[Equation]
where, for M styles, the task/style encoding, Ω = [ ω 0 , . .
. , ω M ] and ω i ∈
The policy optimization criteria requires the value functions to be continuous with respect to the states and task encoding, and this property can be maintained by appropriately selecting the value functions when computing the policy optimization criteria.
The speciﬁcs of the exact formulation of the optimization criteria would vary in accordance with the base Actor-Critic algorithm employed, with Equa- tions 1 and 2 respectively being updated as:
[Equation]
Practically, multi-valued critics may be represented by separate independent critic networks – multinetwork (MN) MultiCriticAL – or a single base critic network with multiple heads, where each head represents a different learned value – multi-headed (MH) MultiCriticAL.
While the latter allows for some learned representations to be shared between the value functions, the former may be more beneﬁcial when sharing is likely to lead to undesirable interference.
This is visually shown in Figureand extended in Figure 14 in Appendix C.
Relationship to MTRL and MORL DQNs MultiCriticAL shares theoretical underpinnings with existing DQN techniques to tackle MTRL and multi-objective RL (MORL) by learning task/objective-speciﬁc values (Parisotto et al., 2015; Moffaert & Now´e, 2014; D’Eramo et al., 2020;
Rusu et al., 2016).
Learned DQN policies and value functions share a single combined architectural backbone, forcing value estimation machinery to be retained at inference time.
By extending the theory to Actor-Critic methods, we both enable utility in continuous control and the practical beneﬁts of disentangling the representational complexity of the actors from the often greater representational burden placed on the critics (Mysore et al., 2021a).
(Additional discussion in Appendix B).
E VALUATION
We evaluate the proposed MultiCriticAL method in a series of multi-style RL problems.
We ﬁrst test our method on two basic multi-style learning environments designed as simple and light-weight demonstrations of the types of problems we seek to address: (i) learning to follow simple shapes, where different styles correspond to different shapes, (ii) playing a modiﬁed game of Pong where the player’s paddle is allowed to rotate and agents are trained to play aggressively or defensively against a stock AI.
While relatively simple, these environments help highlight the efﬁcacy of MultiCriticAL over the more typical single-critic MTRL framework.
Additionally, we test MultiCriticAL on learning to play multiple levels of the Sega genesis games Sonic the Hedgehog, and Sonic the Hedgehog 2, where the agent is trained to play the different levels of each game, following the reward structure devised for the Gym Retro contest (Nichol et al., 2018).
Finally, we demonstrate a practical use-case for MultiCriticAL in an experimental build of EA’s UFC Knockout Mode ﬁghting game where RL agents are integrated into the game controls and trained with different ﬁghting styles.
Appendix A provides learning curves and additional visualizations for the results discussed in this section.
Baselines We train MultiCriticAL with two popular contemporary actor-critic algorithms:
PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018), which represent Actor-Critic algorithms based on both value iteration and Q-learning respectively.
These are compared against
Multi-task PPO (MTPPO) and Multi-task SAC (MTSAC), which augment the state information with a one-hot task encoding and were also proposed as baselines by Yu et al.
While we recognize that these are not state-of-the-art MTRL algorithms, they share a common critic optimization structure with more recent advancements in policy (actor) optimization (Sodhani et al., 2021;
Yang et al., 2020), and we aim to demonstrate the fundamental utility of MultiCriticAL in the absence of additional policy optimization tricks.
Training code is based on OpenAI’s Spinning
Up (Achiam, 2018) and is provided in the Supplementary Material for the multi-style environments.
As the Sonic the Hedgehog games require paid access and the development environment for UFC is proprietary, game code is not provided, though Appendix D details how they were adapted.
Seeding To account for the variability in RL, performance is averaged over 15 random seeds.
5. P ATH F OLLOWING
Environment Design and Reward Shaping
In the path following environment, given its current position in space, the current phase of motion ∈ [0 , 1] , and an indicator on which shape to reproduce, the agent is expected to follow a trajectory that generates the desired shape.
Three shapes are tested:
[Figure]
Circle, Square and Triangle.
This simple problem adds no additional complexity related to generalization or complex system dynamics and simply expects policies to learn a mapping function that encodes the desired shape.
The multi-style objective is for the agents to accurately produce any of the three required shapes, given the task encoding.
Agents are rewarded based on the absolute error between the agent’s current and desired position in space.
The rewards are cast to be positive in the range of [0 , 1] per step.
Appendix D offers further details, with code in Supplementary Material.
Results The pathing behavior learned for each algorithm over the three shapes is shown in Figure 5.1.
While single-style SAC and PPO are able to learn to follow each shape, performances of
MTSAC and MTPPO in the multi-style setting are compromised.
This is most prominent with MTSAC.
MTPPO fares better but performance in the multi-style setting is still notably worse.
Adapted to both algorithms, MultiCriticAL more consistently learns to follow all three shapes.
It should also be noted that larger networks were needed for the PPO algorithm to solve this task, using two hidden layers with 32 neurons each, as opposed to SAC’s 8 each.
This likely contributes to the improved performance of PPO in the multi-task setting as the networks being used have higher relative representational capacity, but are still outperformed by MultiCriticAL.
5. P ONG WITH ROTATABLE PADDLES
[Figure]
Environment Design and Reward Shaping
We use a modiﬁed game of Pong to explore a multi-style learning problem where one style is represented by a sparse reward, and the other a dense reward, anticipating that this might be a case where single-critc value-function estimation may fail.
Pong typically allows only a single degree of freedom for paddle movement and does not normally command nuanced strategies.
Adding an extra degree of freedom by allowing the player’s paddles to rotate introduces increased complexity to the game and facilitates the training of different behavior styles.
For this game, we consider two primary styles of play: (i) aggressive – with the objective of winning the game as quickly as possible, and (ii) defensive – where the objective is to avoid losing while prolonging the game as long as possible.
In a real game setting, these could be analogous to hard- and easymode AI respectively, where the former tries to give the opponent a challenge and the latter may help the opponent learn to play.
Our implementation is based on the Pong environment provided by the Pygame Learning Environment (PLE)
(Tasﬁ, 2016).
For simplicity, the basic opponent AI cannot rotate its paddle and uses a heuristic-based Pong controller provided by PLE.
The Aggressive style rewards winning as quickly as possible while the Defensive style encourages avoiding losing and ensuring that the opponent can keep receiving the ball to prolong the game (additional details in Appendix D).
Results We ﬁrst established baseline performance by training SAC and PPO on the Aggressive and Defensive play-styles.
Figure 3 shows samples of play strategies adopted for each style, where the Aggressive agent exploits paddle rotation to bounce the ball off the side-wall to outmaneuver the opponent.
Conversely, Defensive agents attempt to return the ball directly to the opponent, exploiting the paddle rotation to angle the ball back towards the opponent.
As desired, aggressive play yields high win-rates, with plays lasting signiﬁcantly shorter than defensive play, which would win approximately half of the games but play longer games (videos in Supplementary Material). [Table] [Table]
Multi-style performance is tested by introducing a one-hot style encoder and randomly uniformly sampling either defensive or aggressive play for each new training episode.
We observed that both MTSAC and MTPPO failed to learn to properly distinguish between play-styles.
MTSAC agents often defaulted to highly defensive play (likely due to the over-representation of the dense defensive rewards in the replay buffers) while MTPPO agents tended to learn intermediate behavior that was neither aggressive nor defensive.
MultiCriticAL agents were however able to successfully learn to distinguish between the styles with both base algorithms, with performance comparable to singlestyle agents.
Table 1 and Figure A.2.1, show performance statistics averaged over 20 test episodes per seed for each algorithm.
We also test how MultiCriticAL agents would respond to changes in the style encoding at test time.
As shown in Table 2, when provided an intermediate (0.5,0.5) encoding, the agents presented with behavior statistics that were almost halfway between that of their aggressive and defensive statistics.
This was even without explicit conditioning, when agents were trained with purely binary style indicators.
When explicitly interpolating between styles in training (by providing a multidimensional style-reward, with aggressive and defensive rewards scaled by their encoding weight), both MultiCriticAL SAC and PPO agents yielded performance statistics that more closely matched the mid-points between their aggressive and defensive behaviors.
5. S ONIC THE H EDGEHOG
Training Setup To test MultiCriticAL in a more established multi-task setting, we tested two
Sonic the Hedgehog games, using the Gym-retro wrappers designed for the games during the Gymretro contest (Nichol et al., 2018), using the same setup conﬁguration developed for the competition.
MultiCriticAL is set up to maintain a separate critic for each trained level.
Sufﬁcient experimentation for statistical rigor on this benchmark required hardware parallelism, which our code only supports for PPO.
SAC is therefore omitted for this benchmark.
Results Performance on 17 of the main levels for each game, averaged over 10 tests per agent, per level, are presented in Figure A.2.2, with per-level tabulated numerical rewards provided in Appendix A. Sonic the Hedgehog appears to be a game where learning shared representations over multiple levels helps overall performance, for both MultiCriticAL and MTPPO, though this observation does not hold for Sonic the Hedgehog 2, where MTPPO agents perform worse on average than agents trained on single-levels.
MultiCriticAL agents tend to perform comparable or better than independently trained and MTPPO agents in almost all cases, however, with 35% and 20% improvements on Sonic the Hedgehog and 56% and 40% improvement on Sonic the Hedgehog 2 of
MN-MultiCriticAL and MH-MultiCriticAL over MTPPO respectively.
#$%&'&()*+,-* !" #$%&.&()*+,-* / * 0 , +! "
#! #" $! $" %! %" !"
{+#$%&'(&)&%*+,%"-. /0++1 /2'/3%4"56"4"-
+}
S o n i c 1 L e v e l S a m p l e s
S o n i c 2 L e v e l S a m p l e s
[Figure]
5. UFC E NVIRONMENT
For a use case more reﬂective of real-world application we trained AI ﬁghters in an experimental build of UFC Knockout Mode.
[-Average R e w a r d 
 [Figure] GHZ-]A gym-like (Brockman et al., 2016) wrapper was built for the game environment to enable an RL agent to play against the game’s stock AI.
Note that this is not a feature of the commercially released game but is a modiﬁcation made for research purposes.
Environment design and Reward Shaping The RL agent observes information about the player’s and opponent’s position, velocity, health, and actions being taken, stacked over 8 steps.
The RLagent is able to control 11 basic discrete high-level actions including forward and backward movement, high or low blocking, left or right attacking (with straights, uppercuts, or hooks), and is also allowed to take no action.
The RL-agent is conditioned over 2 sets of 2 playing styles: aggressive or defensive and blocking or moving.
During training, aggression vs. defensiveness, and blocking vs. moving are set on [0 , 1] scales with (0 . 5 , 0 . 5) representing a neutral style which is meant to balance behavior from both scales.
For all styles, the agent is rewarded for winning and is penalized for every strike request.
For the aggressive style, the agent is additionally rewarded for reducing opponent health.
For the defensive style, the agent gets a small positive reward for each step that it remains alive, is penalized for losing health, and is rewarded for reducing opponent health (though not as much as with the aggressive style).
For the blocking style, the agent is rewarded for every successful block, and for the moving style, the agent is rewarded for maintaining a positive velocity.
Details are provided on the state and action spaces, reward tuning, and training in Appendix D.4.
[Table]
Results MultiCriticAL allows PPO agents to successfully learn and smoothly transition between the different ﬁghting styles.
The behavior styles are distinguished visually and Figure 5 attempts to capture this through frames from captured gameplay with videos included in the Supplementary
Material.
Reward statistics for each style are presented in Table 3 and Figure A.2.3.
Aggressive and defensive plays are the most distinct but there are also observable differences between control that prioritizes blocking and moving, with the blocking-style conditioning agents to hold their ground while moving prompts agents to move around the ring more.
When comparing MultiCriticAL to
MTPPO, we note that, while rewards at the extremes of style distinction are similar, MultiCriticAL appears to be better at more consistently maintaining intermediate styles (when a setting is at 0.5).
Visually, it appears that MTPPO also tended to favor blocking and defense over learning more nuanced behavior (Table 6).
5. D ISCUSSION
While explicit separation of the learned value functions per style under the MultiCriticAL framework offers a demonstrable and consistent improvement over the single-critic baselines, it must be noted that MTSAC and MTPPO baselines do not necessarily reﬂect the state-of-the-art in MTRL, but they do reﬂect how value optimization is typically handled in contemporary literature.
It is likely that utilizing some of the improvements to policy optimization techniques discussed in Section 3, may contribute to improved performance of policies trained using single-critic value-optimization, however the same may also further improve the performance of agents trained with MultiCriticAL and remains to be studied.
Curiously, MN-MultiCriticAL consistently outperformed MH-MultiCriticAL on all the tested tasks, which calls into question the idea that shared representation is important and/or useful (Dewangan et al., 2018; Yang et al., 2017) and is an aspect that may warrant further study.
An interesting extension could be to extend modularization (Yang et al., 2020; Andreas et al., 2017) to value-function optimization.
Learning with curricula (Bengio et al., 2009; Graves et al., 2017; Mysore et al., 2019; Andreas et al., 2017) has also been demonstrated to (sometimes dramatically) improve RL performance and multi-style/task curricula also seem like a potential natural ﬁt for multi-critic optimization.
Recent work (Mysore et al., 2021a; Andrychowicz et al., 2021; Ilyas et al., 2020) has also demonstrated that value function capacity can be a key bottleneck in RL performance.
Increasing the network capacity of a single-critic value-function may improve the efﬁcacy of the critic, but this is not guaranteed and there are few principled ways to determine what a sufﬁcient network capacity may be.
Signiﬁcantly increased computational resources may be required for the learnability of the multi-style problems under a single-critc framework.
MultiCriticAL also requires increased compute during training, particularly MN-MultiCriticAL, but MH-MultiCriticAL allows for a bounded increase to computational load ( < 3% as tested - see Appendix E) with an acceptable compromise in relative performance while still outperforming single-critic value learning.
C ONCLUSION
We study Actor-Critic optimization in multi-style RL problems, where agents are required to behave with different styles in the same environments, our work represents one of the ﬁrst explorations of the application of RL to a wider array of multi-style problems.
We hypothesized that limitations of single-actor, single-critic MTRL frameworks may result in interference between the reward signals for different styles due to high similarity in visited states and to mitigate this, we proposed
Multi-Critic Actor Learning (MultiCriticAL), which separates critics into per-task value-function estimators, thus avoiding interference between style rewards and reduces learning complexity by not requiring critics to learn to distinguish between tasks.
MultiCriticAL consistently outperforms the single-critic MTRL baselines, achieving up to 56% improvement in multi-task performance and successfully learning multiple behavior styles where single-critic methods would fail.
R EPRODUCIBILITY S TATEMENT
As noted in Section training code is based upon the open-source OpenAI Spinning Up codebase (Achiam, 2018), with modiﬁcations made to support MultiCriticAL for PPO and SAC.
The modiﬁed code-base is included in the Supplementary Material for this paper, along with the code for the custom environments for Path
Following (presented in Section 5.1) and Pong with rotatable paddles (presented in Section 5.2).
The README ﬁles included with the code detail installation and use of the code.
For the Sonic the Hedgehog games, reproducing results would require acquiring the games’ ROM ﬁles and linking them to an install of gym-retro, as detailed by Nichol et al.
Also included in Appendices D and E are additional details for the setup of the training environments discussed in the main body of the paper as well as training conﬁgurations used.
We recognize that tests on the UFC environment may not be independently reproducible due to requiring access to proprietary development software and we therefore mainly discuss it as a demonstration of the practical utility for our proposed MultiCriticAL method, rather than as a benchmark task
A. A LTERNATIVE R ESULTS R EPRESENTATION A.2.
P ONG
[Figure]
A.2. S ONIC THE H EDGEHOG
[Table]
ScrapBrainZone
[Table]
ScrapBrainZone
[Table]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
A CTION S TATISTICS
[Table]
[Table] We observe from Figure A.2.3 and Table 6 that MultiCriticAL more consistently learns delineated styles that are consistent with the behaviors that the rewards try to encourage. Note that, with our method [Table]
it is, in fact, the aggressive style that wins most ﬁghts consistently and the defensive style learns to block and move to not lose as quickly [Table] whereas MTPPO tends seems to equate defensive play with just blocking [Table]
and this also contributes to a higher rate of losing.
We see as well that the intermediate settings correspond to a more balanced blend of action probabilities between extremes when compared to MTPPO.
[Figure]
A. P ONG A CTION I NTERPOLATION WITH M ULTI -P OLICY C OMPOSITION
As an exercise in curiosity, we also consider what would happen if we attempted to derive intermediate styles by interpolating between actions in the Pong game.
This was meant to test an approach similar to the multi-policy action composition technique used by Peng et al.
Unlike in their case where actions were weighted according to each actor’s estimated critic value, we instead directly interpolate between actions proposed by pre-trained aggressive and defensive agents.
We take this approach as the training for either style would not offer the value functions any meaningful information about the other unseen style(s), prohibiting the form of value comparison employed by
Peng et al.
We hypothesized that it is possible and even likely that policies would destructively interfere with each other if the longer-term goals of constituent policies conﬂict with each other and this is exactly what was observed in testing.
Except for intermediate styles very close to the learned extremes, interpolating between actions proposed by the aggressive and defensive agents caused a near-complete failure in the composite policy’s control.
Composite policies were built by pairing a randomly selected pre-trained aggressive policy with a randomly selected defensive policy.
On a (0.5,0.5) ‘middle’ setting, SAC composite policies achieved an average win-rate of 30% ± 5% (compared to the approximately 95% and 50% of the constituent policies) with an average Play-time of 213 ± 89 steps (often losing the game quickly).
Similarly, composite PPO also achieves a low average win-rate of 15% ± 3% and similarly lose the game quickly.
A high win-rate would not be expected, as the defensive policies are explicitly trained to avoid winning, but both policy classes are trained to avoid losing.
Given the high loss-rate, composing policies in this way, for this multi-style task, can only be regarded as a failure.
By contrast however, as we showed in Table 2, MultiCriticAL is capable of learning to interpolate between styles, both when and not explicitly trained to do so.
This experiment, by no means, constitutes exhaustive testing, and cannot be used to claim that all, or even most, forms of multi-policy policy composition would be ineffective and is therefore not included in the main body of the paper.
We do believe however that it helps highlight a circumstance where policy composition can fail in cases where long-term strategies for different target behavior styles may interfere.
Considered more generally, there are no guarantees that actions proposed by distinct policies would be similar enough, that a weighted sum over them would even make sense, as cooperation between policies is not something that is explicitly trained for.
One might beneﬁt from explicit conditioning, though this was outside the scope of our work.
B R ELATING M ULTI C RITIC AL TO DQN S AND M ULIT -Q LEARNING
The idea of using multiple Q functions to guide the training of a single policy is not strictly new to the broader context of RL and has been used in notably single-task RL for improving the stability of learned value functions.
These include techniques such as Double Q-learning (Hasselt, 2010;
Van Hasselt et al., 2016), Dueling Q-networks (Wang et al., 2016), and Bootstrapped DQNs (Osband et al., 2016; Steckelmacher et al., 2019).
There are also applications in multi-task learning, though we only encountered them in works considering DQN frameworks that, in effect, learn taskspeciﬁc Q values for a single policy in DQN architectures (Rusu et al., 2016; 2015; D’Eramo et al., 2020).
As DQNs jointly encode both the action policies and the value representation in the same network, however, they do not share the practical beneﬁts offered by Actor-Critic methods (and consequently MultiCriticAL) in separating the actors and critics.
We believe that, practically, this is a meaningful distinction as DQN methods are (i) limited to discrete action spaces and (ii) need the full representational and computational complexity of the value estimation to be preserved at inference time after training, which is computationally inefﬁcient.
As shown by Andrychowicz et al.
(2021) and Mysore et al.
(2021a), oftentimes, it appears that the value function often bears the higher burden of required complexity to enable successful policy learning.
Noting this, there is a signiﬁcant practical advantage to being able to just train smaller, less computationally expensive actor-networks of actor-critic methods and disregard the value estimation networks at inference time.
Given that our work is largely motivated by application to video games, where runtime resource use is an important consideration, this was an important distinction.
C M ULTI -N ETWORK VS . M ULTI -H EADED T RAINING F RAMEWORK O VERVIEW
Critic O b s e r v
[Figure]
t
i
[Figure]
a
[Figure]
D A DDITIONAL E NVIRONMENT S ETUP D ETAILS D. P ATH F OLLOWING
Observed State Agents observe a 4-dimensional state with the current ( x, y ) coordinates of the agent in space and the current phase ρ ∈ [0 , 1] represented by [ sin(2 ρπ ) , cos(2 ρπ ) ]
Goal generation Goals, g are generated based on the current phase of motion.
They are as follows:
• Circle: g ( ρ ) =
[sin(2 ρπ ) , cos(2 ρπ )]
∀ ρ ∈ [0 , 1] • Square: g ( ρ ) =   [4 ρ − 0 .
5] ρ ≤ 0 .[0 .
5 , 1 . 5 − 4 ρ ] 0 .
25 ≤ ρ ≤ 0 .[2 .
5 − 4 ρ, − 0 .
5 ≤ ρ ≤ 0 .
[ − 0 .
5 , 4 ρ − 3 . 5] 0 .
75 ≤ ρ ≤• Triangle: g ( ρ ) =
  [1 . 5 ρ, 0 .
5 − 3 ρ ] ρ ≤ 1 /[1 .
5 − 3 ρ, − 0 .
{+5] 1 / 3 ≤ ρ ≤ 2 /[ − 1 . 5 + 1 . 5 ρ, 1 . 
 ρ − 1 . 5] 2 / 3 ≤ ρ ≤+}
Reward design For current state s and desired state g : r = 1 − log( | s − g | ) / log(
max (∆ s )) where max(∆ s ) is the difference between the extents of the space bounds.
D. P ONG
Observed State Agents observe an 8-dimensional state with the current vertical y positions of both the player and opponent, the player’s velocity, the paddle angle, and the ball’s ( x, y ) position and velocity.
Reward design Aggressive play is conditioned with a sparse reward: +1 on a win and -1 on loss.
Agents are meant to learn that the best way to maximize cumulative discounted rewards is to win as quickly as possible.
Defensive play is conditioned with dense rewards: -1 on loss, -1 for bouncing the ball against walls, +0.5 for receiving the ball, +0.1 if the opponent successfully receives the ball, and +0.1 for every time-step of play.
Defensive rewards encourage avoiding losing and ensuring that the opponent can receive the ball in order to prolong the game.
D. S ONIC THE H EDGEHOG
Reward scaling As recommended by Nichol et al.
(2018), we scale rewards by a factor of 0.01 in order to encourage more stable learning with PPO.
D. EA UFC D EVELOPMENT E NVIRONMENT
Environment The UFC environment is an experimental build of UFC.
During training simulation, superﬂuous elements of the game, such as audio, spectators, etc. are disabled.
Socket communication is used to interface between the game-code and the python-based RL algorithms.
The environment wrapper controls the game simulation and extracts and processes state information before communicating it through the socket to the python RL code and interprets requests from the python code to trigger inputs in the game.
During training, agents are trained for 240 epochs with 4000 sample interactions collected per epoch.
Training is run in increments of 30 epochs to improve simulation and memory management stability.
Episodes are terminated when the agent runs out the max episode length or if the health differential between the agent and opponent is over 40 units.
State and Action Space for the UFC environment The environment returns the following data as 30-dimensional observations for the RL agent:
• Position: (X,Z) coordinates in the ring • Velocity: scalar of trajectory velocity • Health: [0,100] ﬂoat value • Action active: high-level action indicator • Strike type: indicator of type of strike performed • Strike segment: indicates the current stage of striking • Ticks to event: time until next event in game ticks • Target hit location: the region of the opponent’s body the agent is targeting (upper or lower) • Block successful: indicates when a strike is successfully blocked
The agent is allowed to take the following actions:
• No action • (Strike) Straight: left or right, single request • (Strike) Uppercut: left or right, single request • (Strike) Hook: left or right, single request • Block: high or low, needs to be held to sustain blocking • Move: forward or backward relative to current opponent position
Reward Design The reward breakdown for the environment is as follows:
• Standard to all styles – Victory reward: +1; Strike request penalty: -0.
• Aggressive style – successful attack reward: + | ∆ Health opponent | /•
Defensive style – survival reward: +0.02 per step; attack reward: + | ∆ Health opponent | / 200 , health loss penalty: − ∆ Health agent /•
Blocking style – block reward: +0.01 per successful block • Movement style – velocity reward: + 0 .
05 v
E T RAINING C ONFIGURATION
Training code is based on OpenAI’s
Spinning Up (Achiam, 2018) and is provided in the zip folder included as additional supplementary material with our main paper submission.
With the exception of hidden layer conﬁguration, we use the same default hyperparameters for each of the algorithms used.
Different hidden layer conﬁgurations were selected for each environment such that they would not be needlessly large but would consistently solve the problem in the singlestyle cases
Network Conﬁguration Hidden layer conﬁgurations for both the actors and critics per environment:
• Path following: PPO [64,64], SAC [8,8] • Pong: PPO [64,64], SAC [32,32] • Sonic the Hedgehog 1 and 2: PPO [256,256].
Note additionally that the CNN feature extractor is shared between the actors and all critics in any particular training run, though they are not shared between separate training runs.
The CNN feature extractor was conﬁgured based on the the extractor used by Mnih et al.
(2015) as it is the same conﬁguration used in the Retro-contest (Nichol et al., 2018).
The above network conﬁgurations correspond to the following numbers of trainable weights for each of the critic network(s).
[Table]
Misc.
details Additionally, a frame wrapper is used with Pong and the Sonic games to augment state information with information from the last 4 observed states.
This wrapper is based on the
FrameStack wrapper provided in OpenAI Baselines (Dhariwal et al., 2017).
Training time was determined by running training until performance plateaued, with an additional buffer given to multi-task training to account for the multi-task problem setting, with initial tests conducted to ensure that performance indeed plateaued.
Run conﬁgurations Conﬁgurations for training per environment are as follows:
Path following
• Single-style PPO – Trained for 100 epochs with 4000 steps per epoch and 90 steps per episode • Single-style SAC – Trained for 50 epochs with 4000 steps per epoch and 90 steps per episode • Multi-style MTPPO – Trained for 300 epochs with 4000 steps per epoch and 90 steps per episode • Multi-style MTSAC – Trained for 150 epochs with 4000 steps per epoch and 90 steps per episode • Multi-style MultiCriticAL-PPO – Trained for 300 epochs with 4000 steps per epoch and 90 steps per episode • Multi-style MultiCriticAL-SAC – Trained for 150 epochs with 4000 steps per epoch and 90 steps per episode Pong with rotatable paddles • Single-style PPO – Trained for 150 epochs with 4000 steps per epoch and variable length episodes • Single-style SAC – Trained for 50 epochs with 4000 steps per epoch and variable length episodes • Multi-style MTPPO – Trained for 250 epochs with 4000 steps per epoch and variable length episodes • Multi-style MTSAC – Trained for 150 epochs with 4000 steps per epoch and variable length episodes • Multi-style MultiCriticAL-PPO – Trained for 250 epochs with 4000 steps per epoch and variable length episodes • Multi-style MultiCriticAL-SAC – Trained for 150 epochs with 4000 steps per epoch and variable length episodes
Sonic the Hedgehog (both 1 and 2)
• Single-style PPO – Trained for 30 epochs with 10000 steps per epoch and up to 2500 steps per episode, or until Sonic’s death, whichever occurs ﬁrst • Multi-style MTPPO – Trained for 300 epochs with 10000 steps per epoch and up tosteps per episode, or until Sonic’s death, whichever occurs ﬁrst • Multi-style MultiCriticAL-PPO – Trained for 300 epochs with 10000 steps per epoch and up to 2500 steps per episode, or until Sonic’s death, whichever occurs
ﬁrst