L EARNING TO S HAPE R EWARDS USING A G AME OF T WO P ARTNERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Reward shaping (RS) is a powerful method in reinforcement learning (RL) for overcoming the problem of sparse or uninformative rewards.
However, RS typically relies on manually engineered shaping-reward functions whose construction is time-consuming and error-prone.
It also requires domain knowledge which runs contrary to the goal of autonomous learning.
We introduce Reinforcement Learning
Optimising Shaping Algorithm (ROSA), an automated RS framework in which the shaping-reward function is constructed in a novel Markov game between two agents.
A reward-shaping agent ( Shaper ) uses switching controls to determine which states to add shaping rewards and their optimal values while the other agent ( Controller ) learns the optimal policy for the task using these shaped rewards.
We prove that
ROSA, which easily adopts existing RL algorithms, learns to construct a shapingreward function that is tailored to the task thus ensuring efﬁcient convergence to high performance policies.
We demonstrate ROSA’s congenial properties in three carefully designed experiments and show its superior performance against state-of-the-art RS algorithms in challenging sparse reward environments.
I NTRODUCTION
Reinforcement learning (RL) offers the potential for autonomous agents to learn complex behaviours without the need for human intervention [39].
Despite the notable success of RL in a variety domains [8, 31, 35, 24], enabling RL algorithms to learn successfully in numerous real-world tasks remains a challenge [41].
A key obstacle to the success of RL algorithms is the requirement of a rich reward signal that can guide the agent towards an optimal policy [7].
In many settings of interest such as physical tasks and video games, rich informative signals of the agent’s performance are not readily available [14].
For example, in the video game Super Mario [33], the agent must perform sequences of hundreds of actions while receiving no rewards for it to successfully complete its task.
In this setting, the sparse reward provides infrequent feedback of the agent’s performance.
This leads to
RL algorithms requiring large numbers of samples (and high expense) for solving problems [14].
Consequently, there is great need for RL techniques that solve these problems efﬁciently.
Various biological systems have mechanisms that produce rewards for activities that present little or no direct biological utility.
Such mechanisms, which have been fashioned over millions of years are designed to provide intrinsic rewards to promote behaviours that improve chances of outcomes with biological utility [1].
In RL, reward shaping (RS) is a tool to introduce intrinsic rewards known as shaping rewards that supplement environment reward signals.
These rewards can encourage exploration and insert structural knowledge in the absence of informative environment rewards which can improve learning outcomes [10].
In general however, RS algorithms assume hand-crafted and domainspeciﬁc shaping functions whose construction is typically highly labour intensive.
This runs contrary to the aim of autonomous learning.
Moreover, poor choices of shaping rewards can worsen the agent’s performance [9].
To resolve these issues, a useful shaping reward must be obtained autonomously.
Inspired by naturally occurring systems, we tackle the problem of sparse and uninformative rewards by developing a framework that autonomously constructs shaping rewards during learning.
Our framework, ROSA, works by introducing an additional RL agent, Shaper , that adaptively learns to construct shaping rewards by observing Controller (the agent whose goal is to solve the environment task) while Controller learns to solve its task.
This generates tailored shaping rewards without the need for domain knowledge or manual engineering.
The shaping rewards supplement the environment reward and promote effective learning, our framework therefore addresses the key challenges in RS.
The resulting framework is a two-player nonzero-sum Markov game (MG) [34] – an extension of
Markov decision process (MDP) that involves two independent learners with distinct objectives.
In our framework, the two agents that have distinct learning agendas, cooperate to achieve Controller ’s objective.
This MG formulation confers various advantages:
The shaping-reward function is constructed fully autonomously.
The game also ensures the shaping reward improves Controller ’s performance unlike RS methods that can lower performance.
By learning the shaping-reward function while Controller learns its optimal policy, Shaper learns to adaptively facilitate Controller ’s learning and improve Controller ’s performance.
Both learning processes converge so Controller learns the optimal value function for its task.
4) ROSA learns subgoals [28] to decompose complex tasks and promote complex exploration patterns.
An integral component of ROSA is a novel combination of RL and switching controls [2, 22].
This enables Controller to quickly determine useful states to learn to add and calibrate shaping rewards (i.e. the states in which adding shaping rewards improve Controller ’s performance) and disregard others.
This is in contrast with an RL controller (i.e. Controller ) which must learn its best actions at every state.
This leads to Shaper quickly ﬁnding shaping rewards that guide Controller ’s learning process toward optimal trajectories (and away from suboptimal trajectories, c.f. Experiment 1).
For our two-player framework to succeed we have to overcome several obstacles.
Solving MGs involves ﬁnding a stable point in which each player responds optimally to the actions of the other.
In our MG, this stable point describes a pair of policies for which Shaper introduces a shaping reward that improves performance and, with that, Controller executes an optimal policy for the task.
Tractable methods for solving MGs are rare with convergence of MG methods being seldom guaranteed except in a few special cases [42].
Nevertheless, using special features in the design of our game, we prove the existence of a stable point solution of our MG.
We then prove the convergence of our learning method to the solution of the game and show that the solution coincides with the solution of the Controller ’s problem.
This ensures Shaper learns a shaping-reward function that improves
Controller ’s performance and that Controller learns the optimal value function for the task.
R ELATED W ORK
Reward Shaping (RS) adds a shaping function F to supplement the agent’s reward to boost learning.
RS however has some critical limitations.
First, RS does not offer a means of ﬁnding F .
Second, poor choices of F can worsen the agent’s performance [9].
Last, adding shaping rewards can change the underlying problem therefore generating policies that are completely irrelevant to the task [20].
In [27] it was established that potential-based reward shaping (PBRS) which adds a shaping function of the form F ( s t +1 , s t ) =
γφ ( s t +1 ) − φ ( s t ) preserves the optimal policy of the problem.
Recent variants of PBRS include potential-based advice which deﬁnes F over the state-action space [13] and approaches that include time-varying shaping functions [11].
Although the last issue can be addressed using potential-based reward shaping (PBRS) [27], the ﬁrst two issues remain.
To avoid manual engineering of F , useful shaping rewards must be obtained autonomously.
Towards this [45] introduce an RS method that adds a shaping-reward function prior which ﬁts a distribution from data obtained over many tasks.
Recently, [16] use a bilevel technique to learn a scalar coefﬁcient for an already-given shaping-reward function.
Nevertheless, constructing F while training can produce convergence issues since the reward function now changes during training.
Moreover, while F is being learned the reward can be corrupted by inappropriate signals that hinder learning.
Curiosity based reward shaping aims to encourage the agent to explore states by rewarding the agent for novel state visitations using exploration heuristics.
One approach is to use state visitation counts [29].
More elaborate approaches such as [6] introduce a measure of state novelty using the prediction error of features of the visited states from a random network.
[30] use the prediction error of the next state from a learned dynamics model and [15] maximise the information gain about the agent’s belief of the system dynamics.
In general, these methods provide no performance guarantees nor do they ensure the optimal policy (of the underlying MDP) is preserved.
Moreover, they naively reward exploration to unvisited states without consideration of the environment reward.
This can lead to spurious objectives being maximised (see Experiment 3 in §6).
Within these two categories, closest to our work are bilevel approaches for learning the shaping function [16, 36].
Unlike [16] which requires a useful shaping reward to begin with, ROSA constructs a shaping reward function from scratch leading to a fully autonomous method.
Moreover, in [16, 36], the agent’s policy and shaping rewards are learned with consecutive updates.
In contrast, ROSA performs these operations concurrently leading to a faster, more efﬁcient procedure.
Also in contrast to [16, 36], ROSA learns shaping rewards only at relevant states, this confers high computational efﬁciency (see Experiment 2, §6)).
As we describe, ROSA, which successfully learns the shapingreward function F , uses a similar form as PBRS.
However in ROSA, F is augmented to include the actions of another RL agent to learn the shaping rewards online.
Lastly, unlike curiosity-based methods e.g., [6, 30], our method preserves the agent’s optimal policy for the task (see Experiment 3, §6) and introduces intrinsic rewards that promote complex learning behaviour (see Experiment 1, §6) .
P RELIMINARIES & N OTATIONS
In RL, an agent sequentially selects actions to maximise its expected returns.
The underlying problem is typically formalised as a MDP (cid:104)S , A , P, R, γ (cid:105) where S is the set of states, A is the discrete set of actions, P : S ×
A × S → [0 , 1] is a transition probability function describing the system’s dynamics,
R : S ×
A → R is the reward function measuring the agent’s performance, and the factor γ ∈ [0 , 1) speciﬁes the degree to which the agent’s rewards are discounted over time [39].
At time t the system is in state s t ∈ S and the agent must choose an action a t ∈
A which transitions the system to a new state s t +1 ∼ P ( ·| s t , a t ) and produces a reward R ( s t , a t ) .
A policy π : S ×
A → [0 , 1] is a probability distribution over state-action pairs where π ( a | s ) represents the probability of selecting action
a ∈ A in state s ∈ S .
The goal of an RL agent is to ﬁnd a policy π (cid:63) ∈ Π that maximises its expected returns given by the value function: v π ( s ) =
E [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | a t ∼ π ( ·| s t )] where Π is the agent’s policy set.
We denote this MDP by M .
A two-player Markov game (MG) is an augmented MDP involving two agent that simultaneously take actions over many rounds [34].
In the classical MG framework, each agent’s rewards and the system dynamics are now inﬂuenced by the actions of both agents.
Therefore, each agent i ∈ { 1 , 2 } has its reward function R i : S
×
A 1 ×
A 2 → R and action set A i and its goal is to maximise its own expected returns.
The system dynamics, now inﬂuenced by both agents, are described by a transition probability P : S ×
A 1 × A 2 × S → [0 , 1] .
As we discuss in the next section, ROSA induces a speciﬁc MG in which the dynamics are inﬂuenced by only Controller .
In RS the question of which φ to insert has not been addressed.
Moreover it has been shown that poor choices of φ hinder learning [9].
Consequently, in general RS methods rely on hand-crafted shaping-reward functions that are constructed using domain knowledge (whenever available).
In the absence of a useful shaping-reward function F , the challenge is to learn a shaping-reward function that leads to more efﬁcient learning while preserving the optimal policy.
Naturally, we can formalise the problem of learning such an F by constructing F as a parametric function of θ ∈ R m : ˆ F ( s t +1 , s t ; θ ) :
= γ ˆ φ ( s t +1 , θ ) − ˆ φ ( s t , θ ) .
Now the problem is to ﬁnd θ (cid:63) ∈ R m for φ ( s )
=
ˆ φ ( s, θ (cid:63) ) such that F ( s t +1 , s t )
=
ˆ F ( s t +1 , s t ; θ (cid:63) ) , i.e., we aim to ﬁnd θ (cid:63) that yields a useful shaping-reward function.
Determining this function is a signiﬁcant challenge; poor choices can hinder the learning process, moreover attempting to learn the shaping-function while learning the RL agent’s policy presents convergence issues given the two concurrent learning processes [44].
Another issue is that using an optimisation procedure to ﬁnd θ (cid:63) directly does not make use of information generated by intermediate state-action-reward tuples of the RL problem which can help to guide the optimisation.
O UR F RAMEWORK
We now describe the problem setting, details of our framework, and how it learns the shaping-reward function.
We then describe Controller ’s and Shaper ’s objectives.
We also describe the switching control mechanism used by Shaper and the learning process for both agents.
To tackle the challenges described above, we introduce Shaper an adaptive agent with its own objective that determines the best shaping rewards to give to Controller .
Using observations of the actions taken Controller , Shaper ’s goal is to construct shaping rewards (which the RL learner
Controller cannot generate itself) to guide the Controller towards quickly learning its optimal policy.
To do this, Shaper learns how to choose the values of an shaping-reward at each state.
Simultaneously,
Controller performs actions to maximise its rewards using its own policy.
Crucially, the two agents tackle distinct but complementary set problems.
The problem for Controller is to learn to solve the task by ﬁnding its optimal policy, the problem for Shaper is to learn how to add shaping rewards to aid Controller .
The objective for Controller is given by:
[Equation]
where a ∼ π is Controller ’s action, ˆ F is the shaping-reward function which is given by ˆ F ( · ) ≡ φ ( s t , a 2 t ) − γ − 1 φ ( s t − 1 , a 2 t − 1 ) for any s t , s t − 1 ∈ S and a 2 t ∼ π 2 is chosen by Shaper (and a 2 t ≡ 0 , ∀ t < 0 ) using the policy
π 2 : S ×
A 2 → [0 , 1] where A 2 ⊂ R p is the action set for Shaper .
The function φ : S ×
A 2 → R is a continuous map that satisﬁes the condition φ ( s, 0) ≡ 0 for any s ∈ S ( φ can be, for example, a neural network with ﬁxed weights with input ( s, a 2 ) , A 2 can be for example a set of integers { 1 , . .
. , K } ).
Therefore, Shaper determines the output of ˆ F (which it does through its choice of a 2 t ).
With this, Shaper constructs a shaping-reward function which is tailored for the speciﬁc setting.
The transition probability P : S×A×S → [0 , 1] takes the state and only Controller ’s actions as inputs.
Formally, the MG is deﬁned by a tuple G =
(cid:104)N , S , A , A 2 , P, ˆ R 1 , ˆ R 2 , γ (cid:105) where the new elements are N = { 1 , 2 } which is the set of agents, ˆ R 1 :=
R + ˆ F is the new Controller reward function which now contains a shaping reward ˆ F , the function ˆ R 2 : S ×
A ×
A 2 → R is the one-step reward for Shaper (we give the details of this function later) and lastly the transition probability P : S ×
A × S → [0 , 1] takes the state and only Controller action as inputs.
As Controller ’s policy can be learned using any RL method, ROSA easily adopts any existing RL algorithm for Controller .
Note that unlike reward-shaping methods e.g. [27], the function φ now contains an action term a 2 which is chosen by Shaper which enables the best shaping-reward function to be learned online.
The presence of an action term may spoil the policy invariance result in [27].
We however prove an policy invariance result (Prop. 1) analogous to that in [27] and show RIGA preserves the optimal policy for M .
4. S WITCHING C ONTROLS
So far Shaper ’s problem involves learning to construct shaping rewards at every state including those that are irrelevant for guiding Controller .
In order to increase the (computational) efﬁciency of Shaper ’s learning process, we now replace the policy space for Shaper with a form of policies known as switching controls .
This enables Shaper to decide at which states to learn the value of shaping rewards it would like to add.
Therefore, now Shaper is tasked with learning how to shape
Controller ’s rewards only at states that are important for guiding Controller to its optimal policy.
This enables Shaper to quickly determine its policy
π 2 and how to choose the values of F unlike
Controller whose policy must learned for all states.
Now at each state Shaper ﬁrst makes a binary decision to decide to switch on its shaping reward
F for Controller affecting a switch I t which takes values in { 0 , 1 } .
This leads to an MG in which, unlike classical MGs, Shaper now uses switching controls to perform its actions.
The new
Controller objective is:
[Equation]
, where I τ k + = 1 −
I τ k , which is the switch for the shaping rewards which is 0 or 1 (and I t ≡ 0 , ∀ t ≤ 0 ) and { τ k } are times that a switch takes place for example if the switch is ﬁrst turned on at state s 5 then turned off at s 7 , then τ 1 =and τ 2 = 7 (we will shortly describe these in detail).
The switch I t is managed by Shaper , therefore by switching I t between 0 or 1 , Shaper activates or deactivates the shaping reward.
We now describe how at each state both the decision to activate a shaping reward and their magnitudes are determined.
Recall that a 2 t ∼ π 2 determines the shaping reward through F .
At any s t , the decision to turn on I t and shape rewards is decided by a (categorical) policy g 2 : S → { 0 , 1 } .
Therefore, gdetermines whether a (or no) Shaper policy π 2 should be used to execute an action a 2 t ∼ π 2 .
It can now be seen the sequence of times τ k = inf { t > τ k − 1 | s t ∈ S , g 2 ( s t )
= 1 } are rules that depend on the state.
Hence, by learning an optimal g 2 , Shaper learns the best states to activate F .
Summary of events:
At a time t ∈ 0 , 1 . . .
• Both players make an observation of the state s t ∈ S .
• Controller takes an action a t sampled from its policy π
. • Shaper decides whether or not to activate the shaping reward using g 2 : S → { 0 , 1 } • If g 2 ( s t )
=
: X ◦ The switch is not activated ( I t = 0 ).
Controller receives a reward r ∼ R ( s t , a t ) and the system transitions to the next state s t
+1 .
• If g 2 ( s t ) = 1 : X ◦ Shaper takes an action a 2 t sampled from its policy π 2 .
X ◦
The switch is activated ( I t = 1 ), Controller receives a reward R ( s t , a t )
+ ˆ F ( s t , a 2 t ; s t − 1 , a 2 t − 1 ) ×
and the system transitions to the next state s t +1 .
We set τ k ≡ 0 ∀ k ≤ 0 and a 2 τ k ≡ 0 , ∀ k ∈ N ( a 2 τ k +1 , . . .
, a 2 τ k + 1 − 1 remain non-zero) and a 2 k ≡ 0 ∀ k ≤ 0 and use the shorthand I ( t ≡ I t .
4. T HE S HAPER ’ S O BJECTIVE
The goal of Shaper is to guide Controller to efﬁciently learn to maximise its own objective (given in M ).
The shaping reward F is activated by switches controlled by Shaper .
As we later describe, the termination times occur according to some external (probabilistic) rule.
To induce Shaper to selectively choose when to switch on the shaping reward, each switch activation incurs a ﬁxed cost for Shaper .
The cost has two effects: ﬁrst it reduces the complexity of Shaper problem since its decision space is to determine which subregions of S it should activate the shaping rewards (and their magnitudes).
Second, it ensures that the information-gain from Shaper encouraging Controller to explore a given set of states is sufﬁciently high to merit activating the stream of rewards.
Given these remarks the objective for Shaper is given by
[Equation] ) =
E π,π  ∞ (cid:88) t
= γ t   ˆ R 1 ( s t , I t , a t , a 2 t , a 2 t − 1 ) + ∞ (cid:88) k ≥c ( I t , I t − 1 )
δ tτ 2 k − 1 + L ( s t ) 
 
 .
The objective encodes Shaper agenda, namely to maximise the expected return.
Therefore, using its shaping rewards, Shaper seeks to guide Controller towards optimal trajectories (potentially away from suboptimal trajectories, c.f. Experiment 1) and enable Controller to learn faster (c.f. Cartpole experiment in Sec. 6).
The function c : { 0 , 1 } 2 → R < 0 is a strictly negative cost function which imposes a cost for each switch and is modulated by the Kronecker-delta function δ tτ 2 k − 1 which iswhenever t
= τ 2 k − 1 and 0 otherwise (this restricts the costs to only the points at which the shaping reward is activated).
Lastly, the term L : S → R is a Shaper bonus reward for when Controller visits infrequently visited states and tends to 0 as the states are revisited.
For this there are various possibilities e.g. model prediction error [37], count-based exploration bonus [38].
With this, Shaper constructs a shaping-reward function that supports Controller ’s learning which is tailored for the speciﬁc setting.
This avoids inserting hand-designed exploration heuristics into
Controller ’s objective as in curiosity-based methods [6, 30] and classical reward shaping [27].
We later prove that with this objective, Shaper ’s optimal policy maximises Controller ’s (extrinsic) return (Prop. 1).
Additionally, we show that the framework preserves the optimal policy of M .
There are various possibilities for the termination times { τ 2 k } (recall that { τ 2 k +1 } are the times which the shaping reward F is switched on using g 2 ).
One is for Shaper to determine the sequence.
Another is to build a construction of { τ 2 k } that directly incorporates the information gain that a state visit provides — we defer the details of this arrangement to Sec.
10 of the Appendix.
4. T HE O VERALL L EARNING P ROCEDURE
The game G is solved using our multi-agent RL algorithm (ROSA).
In the next section, we show the convergence properties of ROSA.
Here, we ﬁrst give a description of ROSA (the full code is in Sec.of the Appendix).
The ROSA algorithm consists of two independent procedures: Controller learns its own policy while Shaper learns which states to perform a switch and the shaping reward magnitudes.
In our implementation, we used proximal policy optimization (PPO) [32] as the learning algorithm for all policies: Controller ’s policy, switching control policy, and the reward magnitude policy.
For
Shaper L term we used L ( s t ) :
= (cid:107)
ˆ h ( s t ) − h ( s t ) (cid:107) 22 as in RND [6] where h is a random initialised, ﬁxed target network while ˆ h is the predictor network that seeks to approximate the target network.
We constructed ˆ F using a ﬁxed neural network f : R d (cid:55)→ R m and a one-hot encoding of the action of Shaper .
Speciﬁcally, ˆ φ ( s t , a 2 t ) :
= f
( s t ) · i ( a 2 t ) where i ( a 2 t ) is a one-hot encoding of the action a 2 t picked by Shaper .
Thus, ˆ F ( s t , a 2 t ; s t − 1 , a 2 t − 1 ) =
f ( s t ) · i ( a 2 t ) − γ − 1 f ( s t − 1 ) · i ( a 2 t − 1 ) .
The action set of Shaper is thus A 2 := { 0 , 1 , ..., m } where each element is an element of N , and π 2 is a MLP π 2 : R d (cid:55)→ R m .
Precise details are in the Supplementary Materials Section 8.
Input: Initial Controller policy π 0 , Shaper policies g 2 0 , π 20 , RL learning algorithm ∆
Output: Optimised Controller policy π ∗ 1 for t = 1 , T do
Given environment state s t , sample a t from π ( s t ) and obtain s t +1 , r t +1 by applying a t to environment
Evaluate g 2 ( s t ) according to Prop.
if g 2 ( s t ) = 1 then
Shaper samples an action a 2 t +1 ∼ π 2 ( ·| s t +1 )
Shaper computes r it +
=
ˆ F ( s t , a 2 t , s t +1 , a 2 t +1 ) , Set shaped reward r = r t +1 + r it +  else Set r = r t +  Update π, g 2 , π 2 using ( s t , a t , r, s t +1 ) and ∆ //
Learn the individual policies
C ONVERGENCE AND O PTIMALITY OF O UR M ETHOD
The ROSA framework enables Shaper to learn a shaping-reward function with which Controller can learn the optimal policy for the task.
The interaction between the two RL agents induces two concurrent learning processes which can occasion convergence issues [44].
We now show that our method converges and the solution ensures higher performing Controller policy than would be achieved by solving M directly.
To do this, we ﬁrst study the stable point solutions of G .
Unlike MDPs, the existence of a stable point solution in Markov policies is not guaranteed for MGs [5] and is rarely computable.
2 MGs also often have multiple stable points that can be inefﬁcient [25]; in G the outcome of such stable point proﬁles would be a poor performing Controller policy.
To ensure the framework is useful, we must verify that the solution of G corresponds to M .
We solve these challenges with the following scheme: [A] The method preserves the optimal policy of M .
[B]
A stable point of the game G in Markov policies exists and is the convergence point of ROSA.
[C]
The convergence point of ROSA yields a payoff that is (weakly) greater than that from solving M directly.
[D] ROSA converges to the stable point solution of G .
We now prove [A] which shows the solution to M is preserved under the inﬂuence of Shaper :
Proposition 1 The following statements hold:
i) max π ∈ Π v π,π 2( s ) = max π ∈ Π v π 1 ( s ) , ∀ s ∈ S , ∀ π 2 ∈ Π 2 , where v π 1 ( s ) =
E [ (cid:80) ∞ t =0 γ t R ( s t , a t )] .
ii)
The Shaper ’s optimal policy maximises v π 1 ( s ) for any s ∈ S .
Result (i) says that the Controller ’s problem is preserved under the inﬂuence of the Shaper .
Moreover the (expected) total return received by the agents is that from the environment (extrinsic rewards).
Result (ii) establishes that Shaper ’s optimal policy induces Shaper to maximise its ( Controller ’s) extrinsic total return.
The result is established by a careful adaptation of the policy invariance result in [27] to our multi-agent switching control framework in which the shaping reward is no longer added at all states.
Building on Prop. 1, we deduce the following result:
Corollary 1 ROSA preserves the MDP M .
In particular, let (ˆ π 1 , ˆ π 2 ) be a stable point policy proﬁleof the MG induced by ROSA G then ˆ π 1 is a solution to the MDP, M .
Therefore, the introduction of Shaper does not alter the fundamentals of the problem.
We now show that G belongs to a special class of MGs which possess a stable point Markov policies which can be computed as a limit point of a sequence of Bellman operations.
We later exploit this result to prove the convergence of ROSA.
We begin by deﬁning some objects which are central to the analysis.
For any π ∈ Π and π 2 ∈ Π 2 , given a function V π,π: S × N → R , we deﬁne the intervention operator M π,π 2 by M π,π 2 V π,π 2 ( s τ k , I ( τ k )) :
=
ˆ R 1 ( s τ k , I ( τ k ) , a τ k , a 2 τ k , · ) + c ( I k , I k − 1 ) + γ (cid:80) s (cid:48) ∈S P ( s (cid:48) ; a τ k , s ) V π,π 2 ( s (cid:48) , I ( τ k +1 )) for any s τ k ∈ S and ∀ τ k where a τ k ∼ π ( ·| s τ k ) and where a 2 τ k ∼ π 2 ( ·| s τ k ) .
We deﬁne the Bellman operator T of the game G by TV π,π 2 ( s τ k , I ( τ k )) :
=
max (cid:110)
M π,π 2 Λ( s τ k , I ( τ k )) , R ( s τ k , a )
+ γ max a ∈A (cid:80) s (cid:48) ∈S P ( s (cid:48) ; a, s τ k )Λ( s (cid:48) , I ( τ k ))
(cid:111) .
Given a value function { V i } i ∈{ 1 , 2 } , the quantity M V i measures the expected future stream of rewards for agent i after an immediate switch minus the cost of switching.
We now show that a stable point of G can be computed using dynamic programming:
Theorem 1 Let V : S× N → R then the game G has a stable point which is a given by lim k →∞ T
k V π
= sup ˆ π ∈ Π V
ˆ π =
V π (cid:63) , where ˆ π is a stable policy proﬁle for the MG, G .
Theorem 1 proves that the MG G (which is the game that is induced when Shaper inﬂuences Controller ) has a stable point which is the limit of a dynamic programming method.
In particular, it proves the that the stable point of G is the limit point of the sequence T 1 V, T 2 V, . . .
Crucially, (by Corollary 1) the limit point corresponds to the solution of the MDP M .
Theorem 1 is proven by ﬁrstly proving that G has a dual representation as an MDP whose solution corresponds to the stable point of the MG.
Theorem 1 enables a distributed Q-learning method [4] to tractably solve the MG.
Having constructed a procedure to ﬁnd the optimal Controller policy, our next result characterises
Shaper policy g 2 and the optimal times to activate F .
Proposition 2 The policy g 2 is given by the following expression: g 2 ( s t ) =
H ( M π,π 2 V π,π 2 −
V π,π 2 )( s t , I t ) , ∀ ( s t , I t )
∈ S ×{ 0 , 1 } , where V is the solution in Theorem 1 and H is the Heaviside function, moreover Shaper ’s switching times are τ k = inf { τ > τ k − 1 |M π,π 2 V π,π 2 =
V π,π 2 } .
Hence, Prop. 2 also characterises the (categorical) distribution g 2 .
Moreover, given the function V , the times { τ k } can be determined by evaluating if M V = V holds.
In general, introducing shaping rewards may undermine learning and worsen overall performance.
We now prove that the ROSA framework introduces an shaping rewards that yield higher total (environment) returns for Controller as compared to solving M directly ([C]).
Proposition 3 Controller ’s (extrinsic) expected return v π,π 2whilst playing G is (weakly) higher than v π 1 , the (extrinsic) expected return for M i.e. v π,
π
[Equation]
Prop. 3 shows that the stable point of G improves outcomes for Controller .
Unlike reward shaping methods in general, the stable points generated never lead to a reduction to the total (environment) return for Controller as compared to its total return without F .
Note that by Prop. 1, Theoremcompares the environment (extrinsic) rewards accrued by the agents so that the presence of Shaper increases the total expected environment rewards.
We now prove the convergence to the solution with (linear) function approximators.
In what follows, we deﬁne a projection Π on a function Λ by: ΠΛ :=
[Equation]
Theorem 2 ROSA converges to the stable point of G , moreover, given a set of linearly independent basis functions
Ψ
= { ψ 1 , . . .
, ψ p } with ψ k ∈ L 2 , ∀ k , ROSA converges to a limit point r (cid:63) ∈ R p which is the unique solution to Π F (Ψ r (cid:63) ) =
Ψ r (cid:63) where F is deﬁned by: F Λ := ˆ R 1 + γP max {M Λ , Λ } where r (cid:63) satisﬁes: (cid:107) Ψ r
(cid:63) − Q (cid:63) (cid:107) ≤ (1 − γ 2 ) − 1 / 2 (cid:107) Π Q (cid:63) − Q (cid:63) (cid:107) .
Theorem 2 establishes the solution to G can be computed using ROSA.
This means that Shaper converges to a shaping-reward function that necessarily improves Controller ’s performance and
Controller learns the optimal value function for the task.
Secondly, the theorem establishes the convergence of ROSA to the solution using (linear) function approximators.
Lastly, the approximation error is bounded by the smallest error that can be achieved given the basis functions.
E XPERIMENTS
We performed a series of experiments to test if ROSA (1) learns a beneﬁcial shaping-reward function (2) decomposes complex tasks into sub-goals, and (3) tailors shaping rewards to encourage Controller to capture environment rewards (as opposed to merely pursuing novelty).
In these tasks, we compared the performance of our method to random network distillation (RND) [6], intrinsic curiosity module (ICM) [30], learning intrinsic reward policy gradient (LIRPG) [43], bi-level optimization of parameterized reward shaping (BiPaRS-IMGL) [16] 4 and vanilla PPO [32].
We then compared our method against these baselines on performance benchmarks including Sparse Cartpole, Gravitar,
Solaris, and Super Mario.
Lastly, we ran a detailed suite of ablation studies (supplementary material).
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
a t S u
[Figure]
p t i
[Figure]
a l
[Figure]
p t i
[Figure]
Beneﬁcial shaping reward.
Our method is able to learn a shapingreward function that leads to improved
Controller performance.
In particular, it is able to learn to shape rewards that encourage the RL agent to avoid suboptimal – but easy to learn – policies in favour of policies that attain the maximal return.
To demonstrate this, we designed a Maze environment with two terminal states: a suboptimal goal state that yields a reward of 0 .and
an optimal goal state which yields a reward of 1 .
In this maze design, the sub-optimal goal is more easily reached.
A good shaping-reward function discourages the agent from visiting the sub-optimal goal.
As shown in Fig.
1 5 our method achieves this by learning to place high shaping rewards (dark green) on the path that leads to the optimal goal.
[Figure]
Subgoal discovery.
We used the Subgoal Maze introduced in [21] to test if ROSA can discover subgoals.
The environment has two rooms separated by a gateway.
To solve this, the agent has to discover the subgoal of reaching the gateway before it can reach the goal.
Rewards are − 0 .
01 everywhere except at the goal state where the reward is 1 .
As shown in Fig.
2, our method successfully solves this environment whereas other methods fail.
Our method assigns importance to reaching the gateway, depicted by the heatmap of added shaped rewards.
Ignoring non-beneﬁcial shaping reward.
Switching control gives our method the power to learn when to attend to shaping rewards and when to ignore them.
This allows us to learn to ignore “red-herrings”, i.e., unexplored parts of the state space where there is no real environment reward, but where surprise or novelty metrics would place high shaping reward.
To verify this claim, we use a modiﬁed Maze environment called Red-Herring Maze which features a large part of the state space that has no environment reward, but with the goal (and accompanying real reward) in a different part of the state space.
Ideally, we expect that the reward shaping method can learn to quickly ignore the large part of the state space.
Fig.
3 shows that our method outperforms all other baselines.
Moreover, the heatmap shows that while RND is easily dragged to reward exploring novel but non rewarding states our method learns to ignore them [Figure]
[Figure]
Learning Performance.
We compared our method with the baselines in four challenging sparse rewards environments: Cartpole, Gravitar, Solaris, and Super Mario.
These environments vary in state representation, transition dynamics and reward sparsity.
In Cartpole, a penalty of − 1 is received only when the pole collapses; in Super Mario Brothers the agent can go for 100s of steps without encountering a reward.
Fig.
4 shows learning curves.
In terms of performance,
ROSA either markedly outperforms the best competing baseline (Cartpole,
Gravitar) or is on par with them (Solaris, Super Mario) showing that it is robust to the nature of the environment and underlying sparse reward.
Moreover, ROSA does not exhibit the failure modes where after good initial performance it deteriorates.
E.g., in Solaris both ICM and RND have good initial performance but deteriorate sharply while ROSA’s performance remains satisfactory.
C ONCLUSION
In this paper, we presented a novel solution method to solve the problem of reward shaping.
Our
Markov game framework of a primary Controller and a secondary reward shaping agent is guaranteed to preserve the underlying learning task for Controller whilst guiding Controller to higher performance policies.
Moreover, our method is able to decompose complex learning tasks into subgoals and to adaptively guide Controller by selectively choosing the states to add shaping rewards.
By presenting a theoretically sound and empirically robust approach to solving the reward shaping problem, our method opens up the applicability of RL to a range of real-world control problems.
The most signiﬁcant contribution of this paper, however, is the novel construction that marries RL, multi-agent RL and game theory which leads to new solution method in RL.
We believe this powerful approach can be adopted to solve other open challenges in RL. Appendix Table of Contents
8 Algorithm 9 Further Implementation Details 10 Shaper Termination Times 11 Experimental Details 11.1 Environments & Preprocessing Details . . . . . . . . . . . . . . . . . . . . . .
11.2 Hyperparameter Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Ablation Studies 13 Flexibility of ROSA to Accommodate different Exploration Bonus Terms L 14 Robustness to choices of φ Parameters 15 Notation & Assumptions 16 Proof of Technical Results 17 Additional Experiment 1 - Replacing RND for Bonus Reward 18 Additional Experiment 2 - Robustness to initialisation of φ  A LGORITHM : Environment E
Initial Shaper Initial Shaper
Randomly initialised ﬁxed
Neural networks (ﬁxed)
RND parameter
Number rollouts N length T
Number mini
N u Switch cost c ( · )
Discount factor γ learning rate α
Output: Optimised Controller policy π ∗ 1 π r do
Collect rollouts do
Sample
Calculate
else
Append
B do
Sample data s t a t g t , 2 t , t it t 1 ) from B then
Set r it else
RND
|| h ( s t ) − ˆ h
( s t ) ||
← θ
− α ∇ l t = ||
h ( s t ) − ˆ h ( s t )
||
Compute L ( s t ) c t c ( · )
F URTHER I MPLEMENTATION D ETAILS Details of Shaper and F (shaping reward)
Object
Description f
Fixed feed forward NN that maps R d (cid:55)→ R m [512, ReLU , 512, ReLU , 512, m ]
A
Discrete integer action set which is size of output of f , i.e., A 2 is set of integers { 1 , ..., m } π
Fixed feed forward NN that maps R d (cid:55)→ R m [512, ReLU , 512, ReLU , 512, m ]
Potential function φ
[Equation] F
[-a F γφ ( s t +1 , a 2 t +1 ) - φ ( s t , a 2 t ) , γ = 
-][Equation]
.d
=
Dimensionality of states; m ∈ N - tunable free parameter.
In all experiments we used the above form of F as follows: a state s t is input to the π 2 network and the network outputs logits p t .
We softmax and sample from p t to obtain the action a 2 t .
This action is one-hot encoded.
Then, the action a 2 t is multiplied with f ( s t ) to compute the second term of F .
A similar process is used to compute the ﬁrst term.
In this way the policy of Shaper chooses the shaping reward.
S HAPER T ERMINATION T IMES
There are various possibilities for the termination times { τ 2 k } (recall that { τ 2 k +1 } are the times which the shaping reward F is switched on using g 2 ).
One is for Shaper to determine the sequence.
Another is to build a construction of { τ 2 k } that directly incorporates the information gain that a state visit provides: let w : Ω → { 0 , 1 } be a random variable with Pr( w = 1)
=
p and Pr( w = 0)
= 1 − p where p ∈ ]0 , 1] .
Then for any k = 1 , 2 , . . .
, and denote by ∆ L ( s τ k ) :
= L ( s τ k ) − L ( s τ k − 1 ) , then we can set:
[Equation]
To explain, since { τ 2 k } k ≥ 0 are the times at which F is switched off then if F is deactivated at exactly after j time steps then I ( s τ 2 k +1 + l )
=
I ( s τ 2 k +1 ) for any 0 ≤ l < j and I ( s τ 2 k
+1 + j )
=
I ( s τ 2 k +2 ) .
We now see that (2) terminates F when either the random variable w attains a 0 or when ∆ L ( s τ k + j ) ≤ 0 which occurs when the exploration bonus in the current state is lower than that of the previous state.
E XPERIMENTAL D ETAILS 11. E NVIRONMENTS & P REPROCESSING D ETAILS
The table below shows the provenance of environments used in our experiments.
Atari & Cartpole https://github.com/openai/gym
Maze https://github.com/MattChanTK/gym-maze
Super Mario Brothers https://github.com/Kautenja/gym-super-mario-bros
Furthermore, we used preprocessing settings as indicated in the following table.
Setting
Value
Max frames per episode
Atari & Mario → 18000 / Maze & Cartpole →
Observation concatenation
Preceding 4 observations
Observation preprocessing
Standardization followed by clipping to [-5, 5]
Observation scaling
Atari & Mario → (84, 84, 1) / Maze & Cartpole →
None
Reward (extrinsic and intrinsic) preprocessing
Standardization followed by clipping to [-1, 1]
11. H YPERPARAMETER S ETTINGS
In the table below we report all hyperparameters used in our experiments.
Hyperparameter values in square brackets indicate ranges of values that were used for performance tuning.
Clip Gradient Norm γ E 0.
λ 0.
Learning rate 1 x 10 −
Number of minibatches
Number of optimization epochs
Policy architecture
CNN (Mario/Atari) or MLP (Cartpole/Maze)
Number of parallel actors 2 (Cartpole/Maze) or 20 (Mario/Atari)
Optimization algorithm
Adam
Rollout length
Sticky action probability 0.
Use Generalized Advantage Estimation
True
Coefﬁcient of extrinsic reward [1, 5]
Coefﬁcient of intrinsic reward [1, 2, 5, 10, 20, 50] γ I 0.
Probability of terminating option [0.5, 0.75, 0.8, 0.9, 0.95]
RND output size [2, 4, 8, 16, 32, 64, 128, 256] [Figure]
A BLATION S TUDIES
Our reward-shaping method features a mechanism to selectively pick states to which intrinsic rewards are added.
It also adapts its shaping rewards according to Controller ’s learning process.
In this section, we present the results of experiments in which we ablated each of these components.
In particular, we test the performance of our method in comparison to a version of our method with the switching mechanism removed.
We then present the result of an experiment in which we investigated the ability of our method to adapt to different behaviour of Controller .
A BLATION S TUDY 1: S WITCHING C ONTROLS
Switching controls enable our method to be selective of states to which intrinsic rewards are added.
This improves learnability (speciﬁcally, by reducing the computational complexity) of the learning task for Shaper as there are fewer states where it must learn the optimal intrinsic reward to add to
Controller objective.
To test the effect of this feature on the performance of our method, we compared our method to a modiﬁed version in which Shaper must add intrinsic rewards to all states.
That is, for this version of our method we remove the presence of the switching control mechanism for Shaper .
Figure(a) shows learning curves on the Maze environment used in the "Optimality of shaping reward" experiments in Section 6.
As expected, the agent with the version of our method with switching controls learns signiﬁcantly faster than the agent that uses the version of our method sans the switching control mechanism.
For example, it takes the agent that has no switching control mechanism almost 50,000 more steps to attain an average episode return of 0.5 as compared against the agent that uses the version of our algorithm with switching controls.
This illustrates a key beneﬁt of switching controls which is to reduce the computational burden on
Shaper (as it does not need to model the effects of adding intrinsic rewards in all states) which in turn leads to both faster computation of solutions and improved performance by Controller .
Moreover,
Maze is a relatively simple environment, expectedly the importance of the switching control is ampliﬁed in more complex environments.
A BLATION S TUDY 2: A DAPTION OF OUR M ETHOD TO D IFFERENT C ONTROLLER P OLICIES
We claimed Shaper can design a reward-shaping scheme that can adapt its shaping reward guidance of Controller (to achieve the optimal policy) according to Controller ’s (RL) policy.
To test this claim, we tested two versions of our agent in a corridor Maze.
The maze features two goal states that are equidistant from the origin, one is a suboptimal goal with a reward of 0 .
5 and the other is an optimal goal which has a reward 1 .
There is also a ﬁxed cost for each non-terminal transition.
We tested this scenario with two versions of our controller: one with a standard RL
Controller policy and another version in which the actions of Controller are determined by a high entropy policy, we call this version of Controller the high entropy controller .
The high entropy policy induces actions that may randomly push Controller towards the suboptimal goal.
Therefore, in order to guide Controller to the optimal goal state, we expect Shaper to strongly shape the rewards of
Controller to guide Controller away from the suboptimal goal (and towards the optimal goal).
Figure 5 (b) shows heatmaps of the added intrinsic reward (darker colours indicate higher intrinsic rewards) for the two versions of Controller .
With the standard policy controller, the intrinsic reward is maximal in the state to the right of the origin indicating that Shaper determines that these shaping rewards are sufﬁcient to guide Controller towards the optimal goal state.
For the high entropy controller, Shaper introduces high intrinsic rewards to the origin state as well as states beneath the origin.
These rewards serve to counteract the random actions taken by the high-entropy policy that lead Controller towards the suboptimal goal state.
It can therefore be seen that Shaper adapts the shaping rewards according to the type of Controller it seeks to guide.
To demonstrate the robustness of our method to different choices of exploration bonus terms in
Shaper ’s objective, we conducted an Ablation study on the L -term (c.f. Equation 1) where we replaced the RND L term with a basic count-based exploration bonus. To exemplify the high degree of ﬂexibility, we replaced the RND with a simple exploration bonus term L ( s ) = 1 Count(s)
+1 for any given state s ∈ S where Count ( s ) refers to a simple count of the number of times the state s has been visited.
We conducted the Ablation study in the environment in Experiment 1 presented in Sec. 6. We note that despite the simplicity of the count-based measure, generally the performance of both versions of ROSA is comparable and in fact the count-based variant is slightly superior to the RND version.
[Figure]
R OBUSTNESS TO CHOICES OF φ P ARAMETERS
To demonstrate the robustness of φ to different choices of weight parameters, we conducted a study with 3 sets of randomly sampled values of weight parameters for the feed forward NN that constructs the φ function (c.f.
Sec. 9).
As is shown in Fig. 7, the performance across all 3 values is very comparable demonstrating the robustness of ROSA to different values of weight parameters for the φ function.
[Figure]
N OTATION & A SSUMPTIONS
We assume that S is deﬁned on a probability space (Ω , F , P ) and any s ∈ S is measurable with respect to the Borel σ -algebra associated with R p .
We denote the σ -algebra of events generated by { s t } t ≥ 0 by F t ⊂ F .
In what follows, we denote by ( V , (cid:107)(cid:107) )
any ﬁnite normed vector space and by H the set of all measurable functions.
The results of the paper are built under the following assumptions which are standard within RL and stochastic approximation methods:
Assumption 1 The stochastic process governing the system dynamics is ergodic, that is the process is stationary and every invariant random variable of { s t } t ≥ 0 is equal to a constant with probability 1 .
Assumption 2 The constituent functions of the players’ objectives R , F and L are in L 2 .
Assumption 3 For any positive scalar c , there exists a scalar µ c such that for all s ∈ S and for any t ∈ N we have: E [1 + (cid:107) s t (cid:107) c | s
=
s ] ≤ µ c (1 + (cid:107) s (cid:107) c ) .
Assumption 4 There exists scalars C 1 and c 1 such that for any function J satisfying | J ( s ) | ≤
C 2 (1 + (cid:107) s (cid:107) c 2 ) for some scalars c 2 and C 2 we have that: (cid:80) ∞
t =0 | E [ J ( s t ) | s 0 = s ] − E
[ J ( s 0 )] | ≤
C 1 C 2 (1 + (cid:107) s t (cid:107) c
1 c 2 ) .
Assumption 5 There exists scalars c and C such that for any s ∈ S we have that: | J ( z, · ) | ≤
C (1 + (cid:107) z (cid:107) c ) for J ∈ { R, F, L } .
We also make the following ﬁniteness assumption on set of switching control policies for Shaper :
Assumption 6 For any policy g c , the total number of interventions is given by K < ∞ .
We lastly make the following assumption on L which can be made true by construction:
Assumption 7 Let n ( s ) be the state visitation count for a given state s ∈ S .
For any a ∈ A , the function L ( s, a ) =
for any n ( s ) ≥ M where 0 < M ≤ ∞ .
P ROOF OF T ECHNICAL R ESULTS
We begin the analysis with some preliminary lemmata and deﬁnitions which are useful for proving the main results.
Deﬁnition 1 A.1 An operator T : V → V is said to be a contraction w.r.t a norm (cid:107) · (cid:107) if there exists a constant c ∈ [0 , 1[ such that for any V 1 , V 2 ∈ V we have that:
[Equation]
Deﬁnition 2 A.2 An operator T : V → V is non-expansive if ∀ V 1 , V 2 ∈ V we have:
[Equation]
Lemma
For any f : V → R , g : V → R , we have that:
[Equation]
[Equation] [Equation] Proof 1 We restate the proof given in [23]:
[Equation]
Deducting max a ∈V g ( a ) from both sides of (7) yields:
[Equation]
After reversing the roles of f and g and redoing steps (6) - (7), we deduce the desired result since the
RHS of (8) is unchanged.
Lemma
A.4 The probability transition kernel P is non-expansive, that is:
[Equation]
Proof
The result is well-known e.g. [40].
We give a proof using the Tonelli-Fubini theorem and the iterated law of expectations, we have that:
[Equation]
where we have used Jensen’s inequality to generate the inequality.
This completes the proof.
P ROOF OF P ROPOSITION
Proof 3 (Proof of Prop 1) To prove (i) of the proposition it sufﬁces to prove that the term (cid:80)
Tt =
γ t F ( θ t , θ t − 1 ) I ( t ) converges to 0 in the limit as T → ∞ .
As in classic potential-based reward shaping [27], central to this observation is the telescoping sum that emerges by construction of F .
First recall v π,π 2( s, I 0 ) , for any ( s, I 0 )
∈ S × { 0 , 1 } is given by:
[Equation]
Hence it sufﬁces to prove that E π,π 2 (cid:104)(cid:80)
[Equation]
=
[Equation]
Recall there a number of time steps that elapse between τ k and τ k +1 , now ∞ (cid:88)
[Equation]
t
[Equation] [Equation]
where we have used the fact that by construction a 2 t ≡
whenever t = τ 1 , τ 2 , . .
and by construction φ ( s, 0) ≡ 0 for any s .
With this we readily deduce that v π,π 2 i
[Equation]
] which is a measure of environment rewards only from which statement (i) can be readily deduced.
For part (ii) we note ﬁrst that it is easy to see that v π,π 2( s 0 , I 0 ) is bounded above, indeed using the key result in the proof of part (i) and the properties of c we have that
[Equation]
using the triangle inequality [Equation] the deﬁnition of ˆ R and the (upper-)boundedness of L and R (Assumption 5).
We now note that by the dominated convergence theorem we have that ∀ ( s 0 , I 0 )
∈ S ×
[Equation] [Equation] [Equation] [Equation]
[Equation]
again using the key result in the proof of (i) and Assumption 6 in the last step, after which we deduce (ii).
Note that by (ii) we heron may consider the quantity for the Shaper expected return:
[Equation]
P ROOF OF T HEOREM
Proof 4 Theorem 1 is proved by ﬁrstly showing that when the players jointly maximise the same objective there exists a ﬁxed point equilibrium of the game when all players use Markov policies and
Shaper uses switching control.
The proof then proceeds by showing that the MG G admits a dual representation as an MG in which jointly maximise the same objective which has a stable point that can be computed by solving an MDP.
Thereafter, we use both results to prove the existence of a ﬁxed point for the game as a limit point of a sequence generated by successively applying the Bellman operator to a test function.
Therefore, the scheme of the proof is summarised with the following steps:
I) Prove that the solution to Markov Team games (that is games in which both players maximise identical objectives) in which one of the players uses switching control is the limit point of a sequence of Bellman operators (acting on some test function).
II) Prove that for the MG G that is there exists a function B π,π 2 : S × { 0 , 1 } → R such that
[Equation]
III) Prove that the MG G has a dual representation as a Markov Team Game which admits a representation as an MDP.
P ROOF OF P ART I
Our ﬁrst result proves that the operator T is a contraction operator.
First let us recall that the switching time τ k is deﬁned recursively τ k = inf { t > τ k − 1 | s t ∈ A, τ k ∈ F t } where
A = { s ∈
S , m ∈ M | g 2 ( m | s t ) > 0 } .
To this end, we show that the following bounds holds:
Lemma
The Bellman operator T is a contraction, that is the following bound holds:
[Equation]
Proof 5 Recall we deﬁne the Bellman operator T ψ of G acting on a function Λ : S × N → R by T ψ Λ
[Equation] s [Equation] k ,
[Equation]
In what follows and for the remainder of the script, we employ the following shorthands:
[Equation]
To prove that T is a contraction, we consider the three cases produced by (25) , that is to say we prove the following statements:
[Equation]
[Table]
and hence M is a contraction)
[Table]
[Equation]
z [Equation] ≡
[Equation]
s
[Equation]
S × { 0 , 1 } .
We begin by proving i)
[Equation]
Indeed, for any a ∈ A and ∀ z t ∈ S × { 0 , 1 } , ∀ θ t , θ t − 1 ∈ Θ , ∀ s (cid:48) ∈ S we have that
[Equation]
again using the fact that P is non-expansive and Lemma 1.
We now prove ii).
For any τ ∈ F , deﬁne by τ (cid:48)
=
inf { t > τ | s t ∈ A, τ ∈ F t } .
Now using the deﬁnition of M we have that for any s τ ∈ S
[Equation]
using the fact that P is non-expansive.
The result can then be deduced easily by applying max on both sides.
We now prove iii).
We split the proof of the statement into two cases:
Case 1:
[Equation]
We now observe the following:
[Equation]
[Table] [Table]
[Equation]
where we have used the fact that for any scalars a, b, c we have that | max { a, b } − max { b, c }| ≤ | a − c | and the non-expansiveness of P
[Equation]
Case 2:
[Equation]
For this case, ﬁrst recall that for any τ ∈ F , − c
( I τ , I τ − 1 ) > λ for some λ > 0 .
[Equation]
again using the fact that P is non-expansive.
Hence we have succeeded in showing that for any Λ ∈ L 2 we have that
[Equation]
Gathering the results of the three cases gives the desired result.
P ROOF OF P ART II
To prove Part II
[Equation]
we prove the following result:
Proposition 4 For any π ∈ Π and for any Shaper policy π 2 , there exists a function B π,π 2 : S × { 0 , 1 } → R such that
[Equation]
where in particular the function B is given by:
[Equation]
for any ( s 0 , I 0 ) ∈ S × { 0 , 1 } .
Proof 6 Note that by the deduction of (ii) in Prop 1, we immediately observe that
[Equation]
We therefore immediately deduce that for any two Shaper policies π 2 and π (cid:48) 2 the following expression holds ∀ ( s 0 , I 0 )
∈ S × { 0 , 1 } :
[Equation]
Our aim now is to show that the following expression holds ∀ ( s 0 , I 0 ) ∈ S × {
[Equation] [Equation]
For the ﬁnite horizon case, the result is proven by induction on the number of time steps until the end of the game.
Unlike the inﬁnite horizon case, for the ﬁnite horizon case the value function and policy have an explicit time dependence.
We consider the case of the proposition at time T − 1 that is we evaluate the value functions at the penultimate time step.
In this case, we have that:
[Equation]
We now observe that for any π ∈ Π and for any π 2 we have that B π,π 2 T ( I T [Equation]
s T ) = E (cid:104)(cid:80) a T ∈A π ( a T ; s T ) (cid:104) R ( s T , a T ) + (cid:80) k ≥ 0 (cid:80) ∞
j = T c ( I j , I j − 1 ) δ j τ k (cid:105)(cid:105) , moreover we have that for any π ∈ Π and for any π
[Equation]
Hence we ﬁnd that
[Equation]
[Equation] [Equation] [Equation]
[Equation] [Equation]
Now
[Equation] [Equation] [Equation]
Hence
[Equation]
we ﬁnd that
[Equation]
Hence
[Equation]
we have succeeded in proving that the expression (28) holds for T − k when k = 1 .
Our next goal is to prove that the expression holds for any 0 < k ≤ T .
Note that for any T ≥ k > [Equation] we can write B π,π 2 T − k as
[Equation]
Now we consider the case when we evaluate the expression (28) for any 0 < k ≤ T .
Our inductive hypothesis is the the expression holds for some 0 < k ≤ T [Equation] that is for any 0 < k ≤ T we have that:
[Equation]
It remains to show that the expression holds for k + 1 time steps prior to the end of the horizon.
The result can be obtained using the dynamic programming principle and the base case ( k = 1 ) result, indeed we have that [Equation] ) ∈A
[Equation]
a
[Equation]
[Equation]
T −
[Equation]
[Equation]
[Equation]
a
[Equation] − [Equation] [Equation]
a
[Equation]
[Equation]
a
[Equation]
k
[Equation]
s T
[Equation]
k
[Equation]
[Equation]
k
[Equation]
[Equation]
+
[Equation]
(cid:34)
[Equation]
a
[Equation]
[Equation] [Equation] (56) = E s T − ( k +1) ∼ d θ (cid:34) (cid:88) a T − ( k +1) ∈A
[Equation]
a
[Equation]
k +1)
[Equation]
T −
[Equation]
k +1) )
[Equation]
( s T
[Equation]
k +1)
[Equation]
a T − ( k +1) ) − (cid:88)
a
[Equation]
k +1)
[Equation]
a (cid:48) T − ( k +1)
[Equation]
s T − ( k +1) )
R ( s T − ( k +1) , a (cid:48)
T − ( k +1) )
+ γ (cid:34)
[Equation]
[Equation]
[Equation]
using the inductive hypothesis and where we have used the fact that
[Equation]
[Equation]
via similar reasoning as before and after which which we deduce the result in the ﬁnite case.
For the inﬁnite horizon case, we must prove that there exists a measurable function B : Π × S → R such that the following holds for any i ∈ N and ∀ π, π (cid:48)
i π (cid:48) ∈ Π , ∀ π − i π (cid:48) ∈
Π − i and ∀ s ∈ S :
[Equation]
The result is proven by contradiction.
To this end, let us ﬁrstly assume ∃ c (cid:54) =
such that
[Equation]
Let us now deﬁne the following quantities for any s ∈ S and for each ππ (cid:48) ∈ Π and π − i π (cid:
) ∈
Π − i and ∀ i ∈ N :
[Equation]
j
[Equation]
s j ) R i [Equation] z j , a j ) , and
B [Equation]
[Equation]
so that the quantity v πi,T (cid:48) ( s ) measures the expected cumulative return until the point T (cid:48) < ∞ .
Hence, we deduce that
[Equation]
Next we observe that:
[Equation]
[Table]
Considering the last expectation and its coefﬁcient and denoting the product by κ , using the fact that by the Cauchy-Schwarz inequality we have (cid:107) AX − BY (cid:107) ≤ (cid:107) A (cid:107)(cid:107) X (cid:107) + (cid:107) B (cid:107)(cid:107) Y (cid:107) , moreover whenever A, B are non-expansive we have that (cid:107) AX − BY (cid:107) ≤ (cid:107) X (cid:107) + (cid:107) Y (cid:107) , hence we observe the following κ ≤ (cid:107) κ (cid:107) ≤ 2 γ T (cid:48) ( (cid:107) v i (cid:107) + (cid:107) B (cid:107) ) .
Since we can choose T (cid:48) freely and γ ∈ ]0 , 1[ , we can choose T (cid:48) to be sufﬁciently large so that γ T (cid:48) ( (cid:107) v i (cid:107) + (cid:107) B (cid:107) ) <
14 | c | .
This then implies that
[Equation]
which is a contradiction since we have proven that for any ﬁnite T (cid:48)
it is the case that
[Equation]
and hence we deduce the result in the inﬁnite horizon case.
P ROOF OF P ART III
We begin by recalling that a Markov strategy is a policy
π
i : S ×
A i → [0 , 1] which requires as input only the current system state (and not the game history or the other player’s action or strategy [22]).
With this, we give a formal description of the stable points of G in Markov strategies.
Deﬁnition
A policy proﬁle ˆ
π =
(ˆ π 1 , ˆ π 2 ) ∈
Π is a Markov perfect equilibrium (MPE) if the following holds ∀ i (cid:54) =
j ∈ { 1 , 2 } , ∀ ˆ π
[Equation]
∈ Π [Equation] [Equation] ˆ
[Equation]
ˆ
[Equation]
S × { 0 , 1 } . The MPE describes a conﬁguration in policies in which no player can increase their payoff by changing (unilaterally) their policy.
Crucially, it deﬁnes the stable points to which independent learners converge (if they converge at all).
P ROOF OF P ROPOSITION
Proof 7 We do the proof by contradiction.
Let σ =
( π, g ) ∈ arg sup π (cid:48)
∈ Π ,g (cid:48)
B π (cid:48) ,g (cid:48) ( s ) for any s ∈ S .
Let us now therefore assume that σ / ∈ NE {G} , hence there exists some other strategy proﬁle ˜
σ
= ( g, ˜ π ) for which Controller has a proﬁtable deviation where π (cid:48)
(cid:54) =
π i.e. v π (cid:48) ,π 2( s ) > v π,π 2( s ) (using the preservation of signs of integration).
Prop. 4 however implies that B π (cid:48) ,π 2 ( s ) − B π,π 2 ( s ) > 0 which is a contradiction since σ =
( π, g ) is a maximum of B .
The proof can be straightforwardly adapted to cover the case in which the deviating player is Shaper after which we deduce the desired result.
The last result completes the proof of Theorem 1.
P ROOF OF P ROPOSITION
Proof 8 (Proof of Prop. 2)
The proof is given by establishing a contradiction.
Therefore suppose that M π,π 2 ψ ( s τ k , I ( τ k )) ≤ ψ ( s τ k , I ( τ k )) and suppose that the intervention time τ (cid:48) 1 > τ 1 is an optimal intervention time.
Construct the Player 2 π (cid:48) 2 ∈ Π 2 and ˜ π
2 policy switching times by ( τ (cid:48) 0 , τ (cid:48) 1 , . . .
and π (cid:48) 2 ∈ Π 2 policy by ( τ (cid:48) 0 , τ 1 , . . . ) respectively.
Deﬁne by l = inf { t > 0; M π,π 2 ψ ( s t , I 0 ) =
ψ ( s t , I 0 ) } and m = sup { t ; t < τ (cid:48) 1 } .
By construction we have that v π 1 ,π (cid:48) {+
+}
( s
[Equation]
I 0 )
[Equation]
[Equation]
We now use the following observation
[Equation]
Using this we deduce that
[Equation]
where the ﬁrst inequality is true by assumption on M .
This is a contradiction since π (cid:48) 2 is an optimal policy for Player 2.
Using analogous reasoning, we deduce the same result for τ (cid:48)
k < τ k after which deduce the result.
Moreover, by invoking the same reasoning, we can conclude that it must be the case that ( τ 0 , τ 1 , . . .
, τ k − 1 , τ k , τ k +1 , . . .
, ) are the optimal switching times.
P ROOF OF T HEOREM
To prove the theorem, we make use of the following result:
Theorem 3 (Theorem 1, pg 4 in [18]) Let Ξ t ( s ) be a random process that takes values in R n and given by the following: [Equation]
[Equation]
then Ξ t ( s ) converges to 0 with probability 1 under the following conditions:
i)
≤ α t ≤ 1 , (cid:80) t α t = ∞ and (cid:80) t
α t < ∞ ii) (cid:107) E
[ L t |F t ] (cid:107) ≤ γ (cid:107)
Ξ t (cid:107) , with γ < 1 ; iii)
Var [ L t |F t ] ≤ c (1 + (cid:107) Ξ t (cid:107) 2 ) for some c > 0 .
Proof 9 To prove the result, we show (i) - (iii) hold.
Condition (i) holds by choice of learning rate.
It therefore remains to prove (ii) - (iii).
We ﬁrst prove (ii).
For this, we consider our variant of the
Q-learning update rule:
[Equation]
After subtracting Q (cid:63) ( s t , I t , a t ) from both sides and some manipulation we obtain that:
[Equation]
s
[Equation]
where Ξ t ( s t , I t , a t ) :
=
Q t ( s t , I t , a t ) − Q (cid:63) ( s t , I t , a t ) .
Let us now deﬁne by
[Equation]
(cid:48) ) (cid:27) − Q (cid:63) ( s t , I t , a ) .
Then
[Equation]
We now observe that
[Equation]
Now, using the ﬁxed point property that implies Q (cid:63)
=
T φ Q (cid:63) , we ﬁnd that
[Equation]
using the contraction property of T established in Lemma 3.
This proves (ii)
[Equation]
We now prove iii)
[Equation]
that is
[Equation]
Now by (65) we have that
Var [ L t |F t ] =
Var
[Equation]
for some c > 0 where the last line follows due to the boundedness of Q (which follows from
Assumptions 2 and 4).
This concludes the proof of the Theorem.
P ROOF OF C ONVERGENCE WITH L INEAR F UNCTION A PPROXIMATION
First let us recall the statement of the theorem:
Theorem 3 ROSA converges to a limit point r (cid:63) which is the unique solution to the equation:
[Equation]
where we recall that for any test function Λ ∈ V , the operator F is deﬁned by F Λ := Θ + γP max {M Λ , Λ } .
Moreover, r (cid:63) satisﬁes the following:
[Equation]
[Equation]
The theorem is proven using a set of results that we now establish.
To this end, we ﬁrst wish to prove the following bound:
Lemma 4 For any Q ∈ V we have that
[Equation]
so that the operator F is a contraction.
Proof 10 Recall, for any test function ψ , a projection operator Π acting Λ is deﬁned by the following
[Equation]
Now, we ﬁrst note that in the proof of Lemma 3, we deduced that for any Λ ∈ L 2 we have that
[Equation]
c.f. Lemma 3).
Setting Λ = Q and ψ =
Θ , it can be straightforwardly deduced that for any Q, ˆ Q ∈ L 2 : (cid:13)
[Equation]
γ [Equation] (cid:13) (cid:13) Q −
[Equation]
Q
[Equation]
(cid:13) (cid:13) .
Hence, using the contraction property of M , we readily deduce the following bound:
[Equation]
We now observe that F is a contraction.
Indeed, since for any Q, Q (cid:48) ∈
L 2 we have that:
[Equation]
using (71) and again using the non-expansiveness of P .
We next show that the following two bounds hold:
Lemma
For any Q ∈ V we have that
i
[Equation]
Proof
The ﬁrst result is straightforward since as Π is a projection it is non-expansive and hence:
[Equation]
using the contraction property of F .
This proves i).
For ii)
[Equation]
we note that by the orthogonality property of projections we have that (cid:104)
Φ r (cid:63) [Equation] [Equation] r
[Equation]
(cid:63)
[Equation]
, hence we observe that:
[Equation]
after which we readily deduce the desired result. Lemma 6 Deﬁne the operator H by the following: HQ ( z ) =
(cid:26) M Q ( z ) , if M Q ( z ) > Φ r (cid:63) ,
Q ( z ) , otherwise , and ˜ F by: ˜ F Q := Θ + γPHQ .
For any Q, ¯ Q ∈ L 2 we have that
[Equation]
(72) and hence ˜ F is a contraction mapping.
Proof 12 Using (71) , we now observe that
[Equation]
again using the non-expansive property of P .
Lemma 7 Deﬁne by ˜ Q :=
Θ + γPv ˜ π where
[Equation]
then ˜ Q is a ﬁxed point of ˜ F ˜ Q , that is ˜ F ˜ Q = ˜ Q .
Proof 13 We begin by observing that
[Equation]
Hence, ˜ F ˜
[Equation]
= [Equation] ˜
[Equation]
˜ Q.
(74) which proves the result.
Lemma
The following bound holds:
[Equation]
Proof 14 By deﬁnitions of v ˆ π and v ˜ π
(c.f (73) ) and using Jensen’s inequality and the stationarity property we have that,
[Equation]
Now recall that ˜ Q :
=
Θ + γPv ˜ π and Q (cid:63) :=
Θ + γPv π (cid:63) , using these expressions in (76) we ﬁnd
that
[Equation]
Moreover, by the triangle inequality and using the fact that F (Φ r (cid:63) ) =
˜ F (Φ r (cid:63) ) and that F Q (cid:63) = Q (cid:63) and F ˜ Q =
˜ Q (c.f. (75) )
we have that
[Equation]
which gives the following bound:
[Equation]
from which, using Lemma 5, we deduce that
[Equation]
after which by (77) , we ﬁnally obtain
[Equation]
as required.
Let us rewrite the update in the following way:
[Equation]
where the function Ξ : R 2 d × R p
→ R p is given by:
[Equation]
} − [Equation] Φ r )( z )) , for any w ≡ ( z, z (cid:48) ) ∈ ( N × S ) 2 where z
=
( t, s ) ∈ N × S and z (cid:48)
=
( t, s (cid:48) )
∈ N × S and for any r ∈ R p .
Let us also deﬁne the function Ξ : R p
→ R p by the following:
[Equation]
Lemma
The following statements hold for all z ∈ { 0 , 1 } × S :
[Equation]
Proof 15 To prove the statement, we ﬁrst note that each component of Ξ k ( r ) admits a representation as an inner product, indeed:
[Equation]
using the iterated law of expectations and the deﬁnitions of P and F .
We now are in position to prove i).
Indeed, we now observe the following:
[Equation]
where in the last step we used the orthogonality of ( 1 − Π) .
We now recall that Π F Φ r (cid:63) =
Φ r (cid:63) since Φ r (cid:63) is a ﬁxed point of Π F .
Additionally, using Lemma 5 we observe that (cid:107) Π F Φ r − Φ r (cid:63) (cid:107) ≤ γ (cid:107) Φ r − Φ r (cid:63) (cid:107) .
With this we now ﬁnd that
[Equation]
[Equation]
which is negative since γ < 1 which completes the proof of part i).
The proof of part ii) is straightforward since we readily observe that
[Equation]
as required and from which we deduce the result.
To prove the theorem, we make use of a special case of the following result:
Theorem 4 (Th. 17, p. 239 in [3]) Consider a stochastic process r t : R × {∞} × Ω
→ R k which takes an initial value r 0 and evolves according to the following:
[Equation]
for some function s : R 2 d × R k
→ R k and where the following statements hold:
1. { s t | t = 0 , 1 , . . .
} is a stationary, ergodic Markov process taking values in R 2 d 2.
For any positive scalar q , there exists a scalar µ q such that E [1 + (cid:107) s t (cid:107) q | s ≡ s 0 ] ≤ µ q (1 + (cid:107) s (cid:107) q ) 3.
The step size sequence satisﬁes the Robbins-Monro conditions, that is (cid:80) ∞ t =
α t = ∞ and (cid:80) ∞ t =
α 2 t < ∞ 4.
There exists scalars c and q such that (cid:107) Ξ( w, r ) (cid:107) ≤ c (1 + (cid:107) w (cid:107) q ) (1 + (cid:107) r (cid:107) ) 5.
There exists scalars c and q such that (cid:80) ∞ t =
(cid:107) E
[Ξ( w t , r )
| z 0 ≡ z ] − E [Ξ( w 0 , r )] (cid:107) ≤ c (1 + (cid:107) w (cid:107) q ) (1 + (cid:107) r (cid:107) ) 6.
There exists a scalar c > 0 such that (cid:107) E
[Ξ( w 0 , r )] − E [Ξ( w 0 , ¯ r )] (cid:107) ≤ c (cid:107) r − ¯ r (cid:107) 7.
There exists scalars c > and q > such that (cid:80) ∞ t =0 (cid:107) E
[Ξ( w t , r ) | w 0 ≡ w ] − E [Ξ( w 0 , ¯ r )] (cid:107) ≤ c (cid:107) r − ¯ r (cid:107) (1 + (cid:107) w (cid:107) q ) 8.
There exists some r (cid:63) ∈ R k such that Ξ ( r )( r − r (cid:63) ) < 0 for all r (cid:54)
=
r (cid:63) and ¯ s ( r (cid:63) )
= 0 .
Then r t converges to r (cid:63) almost surely.
In order to apply the Theorem 4, we show that conditions 1 - 7 are satisﬁed.
Proof 16 Conditions 1-2 are true by assumption while condition 3 can be made true by choice of the learning rates.
Therefore it remains to verify conditions 4-7 are met.
To prove 4, we observe that
[Equation]
Now using the deﬁnition of M , we readily observe that (cid:107)M Φ( z (cid:48) )
(cid:107) ≤ (cid:107) Θ (cid:107) + γ (cid:107)P πs [Equation] s
[Equation] Φ [Equation] ≤
[Equation] Θ
[Equation]
+ γ (cid:107) Φ (cid:107) using the non-expansiveness of P . Hence, we lastly deduce that
[Equation]
φ
[Equation]
Now we observe the following Lipschitz condition on Ξ :
[Equation]
using Cauchy-Schwarz inequality and that for any scalars a, b, c we have that | max { a, b } − max { b, c }| ≤ | a − c | .
Using Assumptions 3 and 4, we therefore deduce that ∞ (cid:88) t =
[Equation]
Part 2 is assured by Lemma 5 while Part 4 is assured by Lemma 8 and lastly Part 8 is assured by
Lemma 9.
[Figure]
ROSA (L = RND)
ROSA (L = Count-based)
Rebuttal
A DDITIONAL E XPERIMENT 1 - R EPLACING RND FOR B ONUS R EWARD
In this experiment we sought to ascertain if ROSA is robust to the component used to generate exploration bonus L .
We ran two versions of ROSA, one where L is computed using RND and one where L is computed using a simple count-based measure 1 Count ( s ) where s ∈ S (the function
Count ( s ) simply tallies the number of times state s has been visited).
Figure 8 shows performance of these two versions of ROSA on the Maze environment shown in Figure 1.
ROSA performs equally well with both components, indicating that it is not dependent on a ﬁne-tuned exploration bonus.
The additional machinery of switching controls and choices of intrinsic rewards to add mean that
ROSA can work with basic exploration bonuses.
Note that we generally use RND since it is simple to implement and works equally well on discrete and continuous state spaces.
A DDITIONAL E XPERIMENT 2 - R OBUSTNESS TO INITIALISATION OF φ
Some reviewers raised important questions about the requirement of a carefully initialised phi function.
Here, we show performance of individual runs of ROSA on the Maze shown in Figure 1.
In each run, φ is randomly initialised using Xavier initialisation, and as a consequence ROSA works with a different φ function in each run.
As can be seen in Figure 9, despite the randomness in φ the performance of ROSA does not vary signiﬁcantly over the runs.
This suggests that ROSA can adapt to the φ function it is presented with, and still come up with good reward shaping.
[Figure]
Run
Run
Run