LIGS: L EARNABLE I NTRINSIC -R EWARD G ENERATION S ELECTION FOR M ULTI -A GENT L EARNING Anonymous authors
Paper under double-blind review
A BSTRACT
Efﬁcient exploration is important for reinforcement learners (RL) to achieve high rewards.
In multi-agent systems, coordinated exploration and behaviour is critical for agents to jointly achieve optimal outcomes.
In this paper, we introduce a new general framework for improving coordination and performance of multi-agent reinforcement learners (MARL).
Our framework, named Learnable Intrinsic-Reward
Generation Selection algorithm (LIGS) introduces an adaptive learner, Generator that observes the agents and learns to construct intrinsic rewards online that coordinate the agents’ joint exploration and joint behaviour.
Using a novel combination of reinforcement learning (RL) and switching controls, LIGS determines the best states to learn to add intrinsic rewards which leads to a highly efﬁcient learning process.
LIGS can subdivide complex tasks making them easier to solve and enables systems of RL agents to quickly solve environments with sparse rewards.
LIGS can seamlessly adopt existing multi-agent RL algorithms and our theory shows that it ensures convergence to joint policies that deliver higher system performance.
We demonstrate the superior performance of the LIGS framework in challenging tasks in Foraging and StarCraft II.
I NTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) has emerged as a powerful tool to enable autonomous agents to solve various tasks such as ride-sharing (Zhou, Luo, et al., 2020) and swarm robotics (Hüttenrauch et al., 2017; Mguni et al., 2018).
In multi-agent systems (MAS), maximising system performance often requires agents to coordinate during exploration and learn coordinated joint actions (Matignon et al., 2012).
However, in many MAS, the reward signal provided by the environment is not sufﬁcient to guide the agents towards coordinated behaviour (Matignon et al., 2012).
Consequently, relying on solely the individual rewards received by the agents may not lead to optimal outcomes (Mguni et al., 2019).
This problem is worsened by the fact that MAS can have many stable points some of which lead to arbitrarily bad outcomes (Roughgarden & Tardos, 2007).
As in single agent RL, in MARL inefﬁcient exploration can dramatically decrease sample efﬁciency.
In MAS, a major challenge is how to overcome sample inefﬁciency from poorly coordinated exploration .
Unlike single agent RL, in MARL, the collective of agents is typically required to coordinate its exploration to ﬁnd their optimal joint policies 1 .
A second issue is that in many MAS settings of interest, such as video games and physical tasks, rich informative signals of the agents’ joint performance are not readily available (Hosu & Rebedea, 2016).
For example, in StarCraft Micromanagement (Samvelyan et al., 2019), the sparse reward alone (win, lose) gives insufﬁcient information to guide agents toward their optimal joint policy.
Consequently, MARL requires large numbers of samples producing a great need for MARL methods that can solve such problems efﬁciently.
To aid coordinated learning, algorithms such as QMIX (Rashid et al., 2018), MAVEN (Mahajan et al., 2019) and COMA (Foerster et al., 2018), so-called centralised critic and decentralised execution (CT-DE) methods use a centralised critic whose role is to estimate the agents’ expected returns.
The critic makes use of all available information generated by the system, speciﬁcally the global state and the joint action.
To enable effective CT-DE, it is critical that the joint greedy action should be equivalent to the collection of individual greedy actions of agents, which is called
the
IGM (Individual-Global-Max) principle (Son et al., 2019).
CT-DE methods are however, prone to convergence to suboptimal joint policies (Wang, Ren, Han, et al., 2020).
In particular, existing value factorisations, e.g. QMIX and VDN (Sunehag et al., 2017), cannot ensure an exact guarantee of
IGM consistency (Wang, Ren, Liu, et al., 2020).
Moreover, CT-DE methods such as QMIX require a monotonicity condition which is violated in scenarios where multiple agents must coordinate but are penalised if only a subset of them do so (see Exp. 2, Sec. 6.1).
To tackle these issues, in this paper we introduce a new MARL framework, LIGS that constructs intrinsic rewards online which guide MARL learners towards their optimal joint policy.
LIGS involves an adaptive intrinsic reward agent, Generator that selects intrinsic rewards to add according to the history of visited states and joint actions performed by the agents.
Generator adaptively guides the agents’ exploration and behaviour towards coordination and maximal joint performance.
A pivotal feature of LIGS is the novel combination of RL and switching controls (Bayraktar & Egami, 2010;
Mguni, 2018) which enables it to determine the best set of states to learn to add intrinsic rewards while disregarding less useful states.
This enables Generator to quickly learn how to set intrinsic rewards that guide the agents during their learning process.
Moreover, the intrinsic rewards added by Generator can signiﬁcantly deviate from the environment rewards.
This enables LIGS to both promote complex joint exploration patterns and decompose difﬁcult tasks.
Despite this ﬂexibility, special features within LIGS ensure the underlying optimal policies are preserved so that the agents learn to solve the task at hand.
Overall, LIGS has several advantages:
• LIGS has the freedom to introduce rewards that vastly deviate from the environment rewards.
With this, LIGS promotes coordinated exploration (i.e. visiting unplayed state-joint actions) among the agents enabling them to ﬁnd joint policies that maximise the system rewards and generates intrinsic rewards to aid solving sparse reward MAS.
• LIGS selects which best states to add intrinsic rewards adaptively in response to the agents’ behaviour while the agents learn leading to an efﬁcient learning process.
• LIGS’s intrinsic rewards preserve the agents’ optimal joint policy and ensure that the total environment return is (weakly) increased.
To enable the framework to perform successfully, we overcome several challenges: i)
Firstly, constructing an intrinsic reward can change the underlying problem leading to the agents solving irrelevant tasks (Mannion et al., 2017).
We resolve this by endowing the intrinsic reward function with special form which both allows a rich spread of intrinsic rewards while preserving the optimal policy.
ii)
Secondly, introducing intrinsic reward functions can worsen the agents’ performance (S. Devlin & Kudenko, 2011) and doing so while training can lead to convergence issues.
We prove
LIGS leads to better performing policies and that LIGS’s learning process converges and preserves the MARL learners’ convergence properties.
iii)
Lastly, adding an agent Generator with its own goal leads to a Markov game (MG) with N + 1 agents (Fudenberg & Tirole, 1991).
Tractable methods for solving MGs are extremely rare with convergence only in special cases (Yang & Wang, 2020).
Nevertheless, using a special set of features in LIGS’s design, we prove LIGS converges to a solution in which it learns an intrinsic reward function that improves the agents’ performance.
R ELATED W ORK
Multi-agent exploration methods seek to promote coordinated exploration among MARL learners.
Maven et.
al (Mahajan et al., 2019) propose a hybridisation of value and policy-based methods that uses mutual information to learn a diverse set of behaviours between agents.
Though this approach promotes coordinated exploration, it does not encourage exploration of novel states.
Other approaches to promote exploration in MARL while assuming aspects of the environment are known in advance and agents can perform perfect communication between themselves (Viseras et al., 2016).
Similarly, to promote coordinated exploration in partially observable settings, (Pesce & Montana, 2020) propose end-to-end learning of a communication protocol through a memory device.
In general, exploration-based methods provide no performance guarantees nor do they ensure the optimal policy (of the underlying dec-MDP) is preserved.
Moreover, many employ heuristics that naively reward exploration to unvisited states without consideration of the environment reward.
This can lead to spurious objectives being maximised.
Reward shaping (Harutyunyan et al., 2015) (RS) is a technique which aims to alleviate the problem of sparse and uninformative rewards by supplementing the agent’s reward with a preﬁxed term
F .
In (Ng et al., 1999) it was established that adding a shaping reward function of the form
F ( s t +1 , s t )
=
γφ ( s t +1 ) − φ ( s t ) preserves the optimal policy and in some cases can aid learning.
RS has been extended to MAS (S. Devlin & Kudenko, 2011, 2016; S. Devlin et al., 2011; S. M. Devlin & Kudenko, 2012; Mannion et al., 2018; Sadeghlou et al., 2014) in which it is used to promote convergence to efﬁcient social welfare outcomes.
Poor choices of F in a MAS can slow the learning process and can induce convergence to poor system performance (S. Devlin & Kudenko, 2011).
In
MARL, the question of which shaping function to use remains unaddressed.
Typically, RS algorithms rely on hand-crafted shaping reward functions that are constructed using domain knowledge, contrary to the goal of autonomous learning (S. Devlin & Kudenko, 2011).
As we later describe LIGS, which successfully learns an instrinsic reward function F , uses a similar form as PBRS however, F is now augmented to include the actions of another RL agent to learn the intrinsic rewards online.
In (Du et al., 2019) an approach towards learning intrinsic rewards is proposed in which a parameterised intrinsic reward is learned using a bilevel approach through a centralised critic.
Loosely related are single-agent methods (Dilokthanakul et al., 2019;
Kulkarni et al., 2016; Pathak et al., 2017; Zheng et al., 2018) which, in general, introduce heuristic terms to generate intrinsic rewards.
Within these categories, closest to our work is the intrinsic reward approach in (Du et al., 2019).
There, the agents’ policies and intrinsic rewards are learned with a bilevel approach.
In contrast,
LIGS performs these operations concurrently leading to a faster, more efﬁcient procedure.
A crucial point of distinction is that in LIGS, the intrinsic rewards are constructed by an RL agent ( Generator ) with its own reward function.
Consequently, LIGS can generate complex patterns of intrinsic rewards, encourage joint exploration .
Additionally, LIGS learns intrinsic rewards only at relevant states, this confers high computational efﬁciency.
Lastly, unlike exploration-based methods e.g., (Mahajan et al., 2019), LIGS ensures preservation of the agents’ joint optimal policy for the task.
P RELIMINARIES
A fully cooperative MAS is modelled by a decentralised-Markov decision process (dec-MDP)
(Yang & Wang, 2020).
A dec-MDP is an augmented MDP involving a set of N ≥ 2 agents denoted by
N that independently decide actions to take which they do so simultaneously over many rounds.
Formally, a dec-MDP is a tuple M =
(cid:104)N , S , ( A i ) i ∈N , P, R, γ (cid:105) where S is the ﬁnite set of states,
A i is an action set for agent i ∈ N and R : S ×
A → P ( D ) is the reward function that all agents jointly seek to maximise where D is a compact subset of R and lastly, P : S ×
A × S → [0 , 1] is the probability function describing the system dynamics where A :=
× Ni =1 A i .
Each agent i ∈ N uses a
Markov policy π i : S
×
A i → [0 , 1] to select its actions.
At each time t ∈ 0 , 1 , . . . , the system is in state s t ∈ S and each agent i ∈ N takes an action a i t ∈
A i .
The joint action a t = ( a 1 t , . .
. , a N t ) ∈
A produces an immediate reward r i ∼ R ( s t , a t ) for agent i ∈ N and inﬂuences the next-state transition which is chosen according to P .
The goal of each agent i is to maximise its expected returns measured by its value function v π i ,π − i ( s ) =
E π
[Equation]
,π − i [ (cid:80) ∞ t =0 γ t R ( s t , a t )] , where Π i is
a compact
Markov policy space and − i denotes the tuple of agents excluding agent i .
Intrinsic rewards can strongly induce more efﬁcient learning (and can promote convergence to higher performing policies) (S. Devlin & Kudenko, 2011).
We tackle the problem of how to learn intrinsic rewards produced by a function F (cid:63) that leads to the agents learning policies that jointly maximise the system performance (through coordinated learning).
Naturally, the problem of learning F (cid:63) can be approached by optimising the input θ ∈ R m of a parameteric function:
ˆ F ( s, a ; θ ) ; that is, ﬁnding θ (cid:63) ∈ R m for which F (cid:63) ( s, a )
=
ˆ F ( s, a , θ (cid:63) ) given an intrinsic reward ˆ F where s is the state and a is the agents’ joint action.
Determining this function is a signiﬁcant challenge since poor choices can hinder learning and the concurrency of multiple learning processes presents potential convergence issues in a system already populated by multiple learners (Zinkevich et al., 2006).
Additionally, we require that the method preserves the optimal joint policy and the underlying dec-MDP.
Note that using an optimisation procedure to ﬁnd θ (cid:63) directly does not make use of information generated by intermediate state-joint-action-reward tuples of the RL problem which can guide the optimisation.
T HE LIGS F RAMEWORK
We now describe the details of the LIGS framework and how it learns intrinsic rewards that improve learning and team performance.
We then describe the agents’ objectives and their learning processes.
To tackle the challenges described above, we introduce Generator an adaptive agent with its own objective that determines the best intrinsic rewards to give to the agents at each state.
Using observations of the joint actions played by the N agents, the goal of Generator is to construct intrinsic rewards (which the N MARL learners cannot generate themselves) to coordinate exploration and guide the agents towards learning joint policies that maximise their shared rewards.
To do this, Generator learns how to choose the values of an intrinsic reward function F θ at each state.
Simultaneously, the N agents perform actions to maximise their rewards using their individual policies.
The objective for each agent i ∈ { 1 , . . .
, N } is given by:
[Equation]
where θ is determined by Generator using the policy g : S × Θ → [0 , 1] and Θ ⊂ R p is Generator ’s action set.
The intrinsic reward function is given by F θ ( · ) ≡ φ ( s t , θ c t ) − γ − 1 φ ( s t − 1 , θ c t − 1 ) for any s t , s t − 1 ∈ S where θ ct ∼ g is the action chosen by Generator and θ ct ≡ 0 , ∀ t < 0 .
The function φ : S × Θ → R is a continuous map that satisﬁes the condition φ ( s, 0) ≡ 0 , ∀ s ∈ S (for example, φ can be a neural network with ﬁxed weights with input ( s, θ c ) and Θ can be a set of integers { 1 , . . .
, K } ).
Therefore, Generator determines the output of F θ (which it does through its choice of θ c ).
With this, Generator constructs intrinsic rewards that are tailored for the speciﬁc setting.
As the agent i ∈ N policy can be learned using any MARL method, LIGS freely adopts any
MARL algorithm for the N agents (see Sec.
9 in the Supp.
Material).
The transition probability
P : S ×
A × S → [0 , 1] takes the state and only the actions of the N agents as inputs.
Note that unlike reward-shaping methods e.g. (Ng et al., 1999), the function φ now contains an action term θ c which is chosen by Generator which enables the intrinsic reward function to be learned online.
The presence of the action θ c term may spoil the policy invariance result in (Ng et al., 1999).
We however prove a policy invariance result (Prop. 1) analogous to that in (Ng et al., 1999) which shows LIGS preserves the optimal policy of M .
Generator is an RL agent whose objective takes into account the history of states and N agents’ joint actions.
Deﬁne by R θ
( s, a ) :
= R ( s, a ) + F θ Generator ’s objective is:
[Equation]
The objective encodes Generator ’s agenda, namely to maximise the agents’ joint expected return.
Therefore, using its intrinsic rewards, Generator seeks to guide the set of agents toward optimal joint trajectories (potentially away from suboptimal trajectories, c.f.
Experiment 2) and enables the agents to learn faster (c.f. StarCraft experiments in Sec. 6).
Lastly, L : S ×
A → R rewards Generator when the agents jointly visit novel state-joint-action tuples and tends to 0 as the tuples are revisited.
We later prove that with this objective, Generator ’s optimal policy (for constructing the intrinsic rewards) maximises the expected team (extrinsic) return (Prop. 1).
Since Generator has its own (distinct) objective, the resulting setup is an MG, G = (cid:104)N × { c } , S , ( A i ) i ∈N , Θ , P, R θ , R c , γ (cid:105) where the new elements are { c } , Generator agent, R θ := R + F θ , the new team reward function which contains the intrinsic reward F θ , R c : S ×
A × Θ → R , the one-step reward for Generator (we give the details of this later).
The transition probability
P : S ×
A × S → [0 , 1] takes the state and only the N agents joint action as inputs.
Switching Control Mechanism
So far Generator ’s problem involves learning to construct intrinsic rewards at every state which can be computationally expensive.
We now introduce an important feature to the framework with which it only learns the best intrinsic reward in a subset of states in which intrinsic rewards are most useful.
This is in contrast to the problem tackled by the N agents who must compute their optimal actions at all states.
To achieve this, we now replace Generator ’s policy space with a form of policies known as switching controls .
These policies enable Generator to decide at which states to learn the value of intrinsic rewards.
This enables Generator to learn quickly both where to add intrinsic rewards and the magnitudes that improve performance since Generator ’s magnitude optimisations are performed only at a subset of states.
Crucially, with this Generator can learn its policy rapidly enabling it to guide the agents toward coordination and higher performing policies while they train.
At each state Generator ﬁrst makes a binary decision to decide to switch on its F for agent i ∈ N using a switch I t which takes values in { 0 , 1 } .
Crucially, now Generator is tasked with learning how to construct the N agents’ intrinsic rewards only at states that are important for guiding the agents to their joint optimal policy.
Both the decision to activate the function F and its magnitudes is determined by Generator .
With this, the agent i ∈ N objective becomes:
[Equation]
where I τ k +
= 1 − I τ k , which is the switch for the function F which is 0 or 1 and { τ k } k> 0 are times that a switch takes place 2 so for example if the switch is ﬁrst turned on at the state s 5 then turned off at s 7 , then τ 1 = 5 and τ 2 =
(we will shortly describe these in more detail).
At any state, the decision to turn on I is decided by a (categorical) policy g c : S → { 0 , 1 } which acts according to
Generator ’s objective.
In particular, ﬁrst, Generator makes an observation of the state s k ∈ S and the joint action a k and using g c , Generator decides whether or not to activate the policy g to provide an intrinsic reward whose value is determined by θ ck ∼ g .
With this it can be seen the sequence of times { τ k } is τ k = inf { t > τ k − 1 | s t ∈ S , g c ( s t )
= 1 } so the switching times.
{ τ k } are rules that depend on the state.
Therefore, by learning an optimal g c , Generator learns the useful states to switch on F .
As we later describe, the termination times occur according to some external (probabilistic) rule.
To induce Generator to selectively choose when to switch on the additional rewards, each switch activation incurs a ﬁxed cost for Generator .
In this case, the objective for Generator is:
[Equation]
where the new term c : { 0 , 1 } 2 → R < 0 is a strictly negative cost function which imposes a cost for each switch activation and is modulated by the Kronecker-delta function δ tτ 2 k − 1 which is 1 whenever t = τ 2 k − 1 and 0 otherwise.
The cost has two effects: ﬁrst, it reduces the computational complexity of Generator ’s problem since Generator now determines subregions of S it should learn the values of F .
Second, it ensures the information-gain from encouraging the agents to explore a given set of state-action tuples is sufﬁciently high to merit activating a stream of intrinsic rewards.
We set τ 0 ≡ 0 , θ τ k ≡ 0 , ∀ k ∈ N ( θ τ k +1 , . . .
, θ τ k +1 − 1 remain non-zero), θ ck ≡ 0 ∀ k ≤ 0 and denote by I ( t ) ≡ I t .
There are various possibilities for the termination times { τ 2 k } (recall that { τ 2 k +1 } are the times which the intrinsic reward F is switched on) using g c .
One is for the terminations to occur randomly.
Another is to build a construction of { τ 2 k } that directly incorporates the information gain that a state visit provides — we defer the details of this arrangement to Sec.
14 of the Appendix.
Summary of Events
At a time t ∈ 0 , 1 , . . .
The N agents makes an observation
of the state s t ∈ S .
The N agents perform a joint action a t =
( a 1 t , . .
. , a Nt ) sampled from π =
( π 1 , . . .
, π N ) .
Generator makes an observation of s t and a t and draws samples from its polices ( g c , g ) .
If g c ( s t ) = 0 : X ◦ Each agent i ∈ N receives a reward r i ∼ R ( s t , a t ) and the system transitions to the next state s t +1 and steps 1 - 3 are repeated.
If g c ( s t ) = 1 : X ◦ F θ is computed using s t , a t and the Generator action θ c ∼ g .
X ◦
Each agent i ∈ N receives a reward r i + F θ and the system transitions to s t +1 .
At time t +1 if the intrinsic reward terminates then steps 1 - 3 are repeated or if the intrinsic reward has not terminated then step 5 is repeated.
4. T HE L EARNING P ROCEDURE
In Sec. 5, we provide the convergence properties of the algorithm, and give the full code of the algorithm in Sec.
8 of the Appendix.
The algorithm consists of the following procedures: Generator updates its policy that determines the values θ at each state and the states to perform a switch while the agents { 1 , . .
, N } learn their individual policies { π 1 , . .
. , π N } .
In our implementation, we used proximal policy optimization (PPO)
(Schulman et al., 2017) as the learning algorithm for both
Generator ’s intervention policy g c and Generator ’s policy g .
For the N agents we used MAPPO (Yu et al., 2021).
For Generator L term we use 3 L ( s t , a t ) :
= (cid:107) ˆ h − h (cid:107) 22 where ˆ h is a random initialised network which is the target network which is ﬁxed and h is the prediction function that is consecutively updated during training.
We constructed ˆ F using a ﬁxed neural network f : R d (cid:55)→ R m and a one-hot encoding of the action of Generator .
Speciﬁcally, φ ( s t , θ ct ) :
= f
( s t ) ·
i ( θ ct ) where i ( θ ct ) is a one-hot encoding of the action θ ct picked by Generator .
Thus, ˆ F ( s t , θ ct ; s t − 1 , θ ct − 1 ) =
f ( s t ) · i ( θ ct ) − γ − 1 f ( s t − 1 ) · i ( θ ct − 1 ) .
The action set of Generator is Θ := { 0 , 1 , ..., m } where each element is drawn from N , and g is an MLP g : R d (cid:55)→ R m .
Extra details are Sec.
8 of the Appendix.
C ONVERGENCE AND O PTIMALITY OF LIGS
We now show that LIGS converges and that the solution ensures a higher performing agent policies than what would be achieved by solving M directly.
The addition of Generator ’s RL process which modiﬁes N agents’ rewards during learning can produce convergence issues (Zinkevich et al., 2006).
Also to ensure the framework is useful, we must verify that the solution of G corresponds to solving the MDP, M .
To resolve these issues, we ﬁrst study the stable point solutions of G .
Unlike MDPs, the existence of a solution in Markov policies is not guaranteed for MGs (Blackwell & Ferguson, 1968) and is rarely computable (except for special cases such as team and zero-sum MGs (Shoham &
Leyton-Brown, 2008)).
MGs also often have multiple stable points that can be inefﬁcient (Mguni et al., 2019); in G such stable points would lead to a poor performing agent joint policy.
We resolve these challenges with the following scheme:
[I] LIGS preserves the optimal solution of M .
[II] The MG induced by LIGS has a stable point which is the convergence point of MARL.
[III] LIGS yields a team payoff that is (weakly) greater than that from solving M directly.
[IV] LIGS converges to the solution with a linear function approximators.
In what follows, we denote by Π :=
× i ∈N Π i .
The results are built under Assumptions 1 - 7 (Sec.
15 of the Appendix) which are standard in RL and stochastic approximation theory.
We now prove the result [I] which shows the solution to M is preserved under the inﬂuence of LIGS:
Proposition 1 The following statements hold:
i) max π ∈ Π v π ,g ( s, · ) = max π ∈ Π v π ( s ) , ∀ s ∈ S , ∀ i ∈ N , ∀ g where v π ( s ) =
E π [ (cid:80) ∞ t =
γ t R ] .
ii)
The Generator ’s optimal policy maximises v π ( s ) =
E [ (cid:80) ∞ t =0 γ t R ( s t , a t )] for any s ∈ S .
Result (i) says that the agents’ problem is preserved under Generator ’s inﬂuence.
Moreover the agents’ (expected) total return is that from the environment (extrinsic rewards).
Result (ii) establishes that Generator ’s optimal policy induces it to maximise the agents’ joint (extrinsic) total return.
The result is proven by a careful adaptation of the policy invariance result in (Ng et al., 1999) to our
MARL switching control setting where the intrinsic-reward is not added at all states.
Building on Prop. 1, we deduce the following result:
Corollary 1 LIGS preserves the dec-MDP played by the agents.
In particular, let ( ˆ π , g ) be a stable point policy proﬁle 4 of the MG induced by LIGS, G then ˆ π is a solution to the dec-MDP, M .
Therefore, the introduction of Generator does not alter the fundamentals of the problem.
Our next task is to prove the existence of a stable point of the MG induced by LIGS and show it is a limit point of a sequence of Bellman operations.
To do this we prove that a stable solution of G exists and that
G has a special property that permits its stable point to be found using dynamic programming.
We begin by deﬁning key objects for the analysis:
Given a V π ,g : S × N → R , ∀ π ∈ Π and g , deﬁne by M π ,g V π
[Equation]
( s
[Equation]
, I
[Equation] [Equation]
where a τ k ∼ π ( ·| s τ k ) , θ τ k ∼ g ( ·| s τ k ) and τ k is a Generator switching time.
We deﬁne the Bellman operator T of G by TV π ,g ( s τ k , I τ k ) :
= max (cid:110)
M
[Equation]
g
[Equation]
π
[Equation] [Equation]
[Equation]
The following result establishes that the solution of the MG G , can be computed using RL methods:
Theorem 1 Let V : S ×
A → R , then for any s ∈ S the game G has a stable point given by
[Equation]
where π
[Equation]
∈ Π is a stable joint policy of G .
Theorem 1 proves that the MG G (which is the game that is induced when Generator plays with the
N agents) has a stable point which is the limit of a dynamic programming method.
In particular, it proves the that the stable point of G is the limit point of the sequence T 1 V, T 2 V, . . .
Crucially, (by
Corollary 1) the limit point corresponds to the solution of the dec-MDP M .
Theorem 1 is proven by ﬁrstly proving that G has a dual representation as an MDP whose solution corresponds to the stable point of the MG.
Theorem 1 enables us to tackle the problem of ﬁnding the solution to G using distributed learning methods i.e. MARL to solve G .
Moreover, Prop. 1 indicates by computing the stable point of G leads to a solution of M .
These results combined prove [II] .
Our next result characterises Generator policy g c and the optimal times to activate F .
The result yields a key aspect of our algorithm for executing Generator activations of intrinsic rewards:
Proposition 2 The policy g c is given by the following:
[Equation]
V π ,g )( s t , I t ) , ∀ ( s t , I t )
∈ S × { 0 , 1 } , where V π ,g is the solution in Theorem 1 and H is the Heaviside function, moreover Generator ’s switching times are τ k = inf { τ > τ k − 1 |M π ,g V π ,g
= V π ,g } .
In general, introducing intrinsic rewards or shaping rewards may undermine learning and worsen overall performance.
We now prove that the LIGS framework introduces an intrinsic reward which yields better performance for the N agents as compared to solving M directly ( [III] ).
Theorem 2 Each agent’s expected return v π ,g whilst playing G is (weakly) higher than the expected return for M (without Generator ) i.e. v π ,g ( s, · ) ≥ v π ( s ) , ∀ s ∈ S , ∀ i ∈ N .
Theorem 2 shows that Generator ’s inﬂuence leads to an improvement in the system performance for the N agents.
Note that by Prop. 1, Theorem 2 compares the environment (extrinsic) rewards accrued by the agents so that the presence of Generator increases the total expected environment rewards.
We complete our analysis by extending Theorem 1 to capture (linear) function approximators which proves [IV] .
We ﬁrst deﬁne a projection Π by: ΠΛ :=
arg min ¯Λ ∈{ Φ r | r ∈ R p } (cid:13)(cid:13)
¯Λ − Λ (cid:13)(cid:13) for any function Λ .
Theorem 3 LIGS converges to the stable point of G , moreover, given a set of linearly independent basis functions Φ = { φ 1 , . . .
, φ p } with φ k ∈ L 2 , ∀ k .
LIGS converges to a limit point r (cid:63) ∈ R p which is the unique solution to Π F (Φ r (cid:63) )
=
Φ r (cid:63) where F Λ := ˆ R + γP max {M Λ , Λ } .
Moreover, r (cid:63) satisﬁes: (cid:107) Φ r (cid:63) − Q (cid:63) (cid:107) ≤ (1 − γ 2 ) − 1 / 2 (cid:107) Π Q (cid:63) − Q (cid:63) (cid:107) .
The theorem establishes the convergence of LIGS to a stable point (of G ) with the use of linear function approximators.
The second statement bounds the proximity of the convergence point by the smallest approximation error that can be achieved given the choice of basis functions.
E XPERIMENTS
We performed a series of experiments to test if LIGS: 1. learns to efﬁciently coordinate the agents’ joint exploration 2.
Optimises convergence points by inducing coordination.
Handles sparse
[Figure] [Figure]
6. C OOPERATIVE F ORAGING T ASKS
Experiment 1: Coordinated exploration.
We tested our ﬁrst claim that LIGS promotes coordinated exploration among agents.
To investigate this, we used a version of the level-based foraging environment (Papoudakis et al., 2020) as follows: there are n agents each with level a i .
Moreover, there are 3 apples with level K such that (cid:80) Ni
=1 a i = K .
The only way to collect the reward is if all agents collectively enact the collect action when they are beside an apple.
This is a challenging joint-exploration problem since to obtain the reward, the agents must collectively explore joint actions across the state space (rapidly) to discover that simultaneously executing collect near an apple produces rewards.
To increase the difﬁculty, we added a penalty for the agents failing to coordinate in collecting the apples.
For example, if only one agent uses the collect action near an apple, it gets a negative reward.
This results in a non-monotonic reward structure.
Fig.
1 shows the performance curves.
Fig, 1 shows LIGS demonstrates superior performance over the baselines.
Experiment 2: Optimal joint policies.
We next tested our second claim that LIGS can promote convergence to joint policies that achieve higher system rewards.
To do this, we constructed a challenging experiment in which the agents must avoid converging to suboptimal policies that deliver positive but low rewards.
In this experiment, the grid is divided horizontally in three sections; top, middle and bottom.
All grid locations in the top section give a small reward r/n to the agent visiting them where n is the number of tiles in the each section.
The middle section does not give any rewards.
The bottom section rewards the agents depending on their relative positions.
If one agent is at the top and the other at the bottom, the agent at the bottom receives a reward − r/n each time the other agent receives a reward.
If both agents are at the bottom, then one of the tiles in this section will give a
2.5 Steps 1e 0.
W i n R a t e 5m vs. 6m 0.
2.5 Steps 1e 0.
W i n R a t e 6h vs. 8z
Ours MAPPO
LIIR LICA
MAVEN 0.
2.5 Steps 1e 0.
W i n R a t e corridor reward R, r/ 2 < R < r to both agents.
The bottom section gives no reward otherwise.
The agents start in the middle section and as soon as they cross to one section they cannot return to the middle.
As is shown in Fig.
1, LIGS learns to acquire rewards rapidly in comparison to the baselines with
MAPPO requiring around 400k episodes to match the rewards produced by LIGS.
Experiment 3: Sparse rewards.
We tested our claim that LIGS can promote learning in MAS with sparse rewards.
We simulate a sparse reward setting using a competitive game between two teams of agents.
One team is controlled by LIGS while the other actions of the agents belonging to the other team are determined by a ﬁxed policy.
The goal is to collect the apple faster than the opposing team.
Collecting the apple results in a reward of 1, and rewards are 0 otherwise.
This is a challenging sparse reward since informative reward signals occur only apple when the apple is collected.
As is shown in Fig.
1 both LIGS and MAPPO perform well on the sparse rewards environment, whilst the other baselines are all unable to learn any behaviour on this environment.
We next investigated the workings of the LIGS framework.
We studied the locations where Generator added intrinsic rewards in Experiments 1 and 2.
As is shown in the heatmap visualisation in Fig.
2, for
Experiment 2, we observe that Generator learns to add intrinsic rewards that guide the agents towards the optimal reward (bottom left) and away from the suboptimal rewards at the top (where some other baselines converge).
This supports our claim that LIGS learns to guide the agents towards jointly optimal policies.
Moreover, as Fig.
2 shows, LIGS’s switching mechanism means that Generator only adds intrinsic rewards as the most useful locations for guiding the agents towards their target.
For Experiment 1, Fig. 2 shows that Generator learns to guide the agents towards the apple which delivers the high rewards.
Crucially Fig.
2 (Right) demonstrates a striking beneﬁt of the LIGS framework, namely it only activates the intrinsic rewards around the apple when both agents are at most 2 cells away from the apple.
Since the agents receive positive rewards only when they arrive at the apple simultaneously, this ensures the agents are encouraged to coordinate their arrival and receive the maximal positive rewards and avoids encouraging arrivals at times that lead to penalties.
6. L EARNING P ERFORMANCE IN S TAR C RAFT M ULTI -A GENT C HALLENGE
To study the performance of LIGS in highly complex environments, we compared its performance with the aforementioned baselines on three StarCraft Multi-Agent Challenge (SMAC) maps (Samvelyan et al., 2019) 5m vs. 6m (hard), 6h vs. 8z , and Corridor .
These maps vary in a range of MARL attributes such as number of units to control, environment reward density, unit action sets, and (partial)-observability.
In Fig.
3, we report our results showing ‘Win Rate’ vs ‘Steps’.
These curves are generated by computing the median win rate (vs the opponent) of the agent at regular intervals during learning.
We ran 3 of each algorithm (further setup details are in the Supp.
material Sec 13).
LIGS outperforms the baselines in all maps.
In 5m vs. 6m , the baselines do not approach the performance of LIGS.
In Corridor MAPPO requires over an extra million steps to match the performance of LIGS.
In 6h vs. 8z , LIGS slightly outperforms the baselines.
In summary, LIGS shows performance gains over all baselines in SMAC maps which encompass diverse MAS attributes.
C ONCLUSION
We introduced LIGS, a novel framework for generating intrinsic rewards which signiﬁcantly boosts performance of MARL algorithms.
Central to LIGS is a powerful adaptive learning mechanism that generates intrinsic rewards according to the task and the MARL learners’ joint behaviour.
Our experiments show LIGS induces superior performance in MARL algorithms in a range of tasks.
Appendix Table of Contents
8 Algorithm
Micromanagement II Maps 11 Flexibility of LIGS to Accommodate different Exploration Bonus Terms L 12 Further Experiment Demonstrating LIGS improved use of Exploration Bonuses.
13 Further Implementation Details 13.1 Hyperparameter Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14 Generator Termination Times 15 Notation & Assumptions 16 Proof of Technical Results
A LGORITHM
Input: Environment E
Initial agent policies π 0 = ( π 10 , . . .
π N 0 ) with parameters θ π 10 , . . .
θ π 1 N , Initial Generator switch policy g c 0 with parameters θ g c 0 , Initial Generator action policy g 0 with parameters θ g 0 ,
Randomly initialised ﬁxed neural network φ ( · , · ) , Neural networks h (ﬁxed) and ˆ h for
Augmented RND with parameter θ ˆ h , Buffer B , Number of rollouts N r , rollout length T , Number of mini-batch updates N u , Switch cost c , discount factor γ , learning rate α .
Output: Optimised agent policies π (cid:63) =
( π (cid:63), 1 , . . .
, π (cid:63),N ) 1 π = ( π 1 , . .
. , π N ) , g, g c ← π 0 , g 0 , g c2 for n = 1 , N r do // Collect rollouts for
1 , T do
Get environment s t from E
Sample a s t ))
Apply a t to environment E , get rewards r t =
1 t ,
, r next state s t +
Sample q t from g c ( s t ) //
Switching control if then
Sample θ ct from g ( s t )
Sample θ ct +1 from g ( s t +1 )
f it =
γφ ( s t +1 , θ ct +1 )
− φ ( s t , θ ct ) //
Calculate F ( s t , θ ct , s t +1 , θ ct +1 ) else θ ct , f it = 0 , 0 //
Dummy values
Append ( s t , a t , g t , θ ct , r t , f it , s t +1 ) to B for u = 1 , N u do
Sample data ( s t , a t , g t , θ ct , r t , f it , s t +1 ) from B if g t
= 1 then
Set reward to r st = r t + f it else
Set reward to r st = r t // Update Augmented RND
Loss RND =
|| h ( s t , a t ) − ˆ h ( s t , a t )
||
θ ˆ h ← θ ˆ h − α ∇ Loss RND //
Update Generator l t =
|| h ( s t , a t ) − ˆ h ( s t )
||
Compute L ( s t , a t ) c t = cg
t
Compute Loss g using ( s t , a t , g t , c t , r t , f it , l t , s t +1 ) using PPO loss // Section 4.
Compute Loss g c using ( s t , a t , g t , c t , r t , f it , l t , s t +1 ) using PPO loss // Section 4.
θ g ← θ g − α ∇ Loss g θ g c ← θ g c − α ∇ Loss g
c
Update agent j , for each j ∈ 1 , . .
N
Compute Loss π j using ( s t , a t , r j,st := r jt + f it , s t +1 ) using PPO loss // Section 4.
θ
← θ
− α ∇ Loss
A BLATION STUDY : P LUG & P LAY
In order to validate our claim that LIGS freely adopts RL learners, we tested the ability of LIGS to boost performance in a complex coordination task using independent Proximal policy optimization algorithm (IPPO)
(Schulman et al., 2017) as the base learner.
In this experiment, two agents are spawned at opposite sides of the grid.
The red agent is spawned in the left hand side and the blue agent is spawned in the right hand side of the grid in Fig.
The goal of the agents is to arrive at their corresponding goal states (indicated by the coloured square, where the colour corresponds to the agent whose goal state it is) at the other side of the grid.
Upon arriving at their goal state the agents receive their reward.
However, the task is made difﬁcult by the fact that only one agent can pass through the corridor at a time.
Therefore, in this setup, the only way for the agents to complete the task is for the agents to successfully coordinate, i.e. one agent is required to allow the other agent to pass through before attempting to traverse the corridor.
It is known that independent learners in general, struggle to solve such tasks since their ability to coordinate systems of RL learners is lacking (Yang et al., 2020).
This is demonstrated in Fig.
4 (right) which displays the performance curve of for IPPO which fails to score above 0 .
As claimed, when incorporated into the LIGS framework, the agents succeed in coordinating to solve the task.
This is indicated by the performance of IPPO + LIGS (blue).
[Figure]
R ESULTS ON A DDITIONAL S TAR C RAFT M ICROMANAGEMENT II M APS
Here we present results on two more maps: MMM2 and 3s5z vs 3s6z As can be seen in Fig.
5, LIGS outperforms the leading baseline MAPPO in all maps yielding a 50% improvement in win rate over
MAPPO in MMM
[Figure]
F LEXIBILITY OF LIGS TO A CCOMMODATE DIFFERENT E XPLORATION B ONUS T ERMS L
To demonstrate the robustness of our method to different choices of exploration bonus terms in
Generator ’s objective
[Figure]
we conducted an Ablation study on the L -term (c
[Figure]
f. Equation 3) where we replaced the RND L term with a basic count-based exploration bonus.
To exemplify the high degree of ﬂexibility
[Figure]
we replaced the RND with a simple exploration bonus term L ( s ) = 1 Count(s)
+1 for any given state s ∈ S where Count ( s ) refers to a simple count of the number of times the state s has been visited.
We conducted the Ablation study on all three Foraging environments presented in Sec. 6.1.
We note that despite the simplicity of the count-based measure, generally the performance of both versions of LIGS is comparable and in fact the count-based variant is superior to the RND version for the joint exploration environment
[Figure]
by method
F URTHER E XPERIMENT D EMONSTRATING LIGS IMPROVED USE OF E XPLORATION B ONUSES .
As we have shown above, LIGS can accommodate a variety of exploration bonuses and perform well.
Here, we did a experiment to further justify using LIGS against simpler exploration bonus methods.
We compared LIGS against and MAPPO with an RND intrinsic reward in the agents’ objectives (MAPPO+RND) and vanilla MAPPO.
Fig.
7 shows performance of these two methods on coordination environment shown in Fig.
We note that LIGS markedly outperforms both
MAPPO+RND and vanilla MAPPO.
Due to the added beneﬁt of switching controls and intrinsic reward selection performed by Generator , we observe that LIGS is able to signiﬁcantly augment the beneﬁts of applying RND directly to the agents’ objectives.
[Figure]
The additional machinery of switching-controls and intrinsic reward selection allows LIGS to make better use of exploration bonuses.
In this case, LIGS demonstrates signiﬁcant improvement over
MAPPO with RND intrinsic rewards.
F URTHER I MPLEMENTATION D ETAILS Details of Generator and F (
Object
Description f
Fixed feed forward NN that maps R d (cid:55)→ R m [512, ReLU , 512, ReLU , 512, m ] Θ
Discrete action set which is size of output of f , i.e., Θ is set of integers { 1 , ..., m } g
Fixed feed forward NN that maps R d (cid:55)→ R m [512, ReLU , 512, ReLU , 512, m ]
Intrinsic-reward function φ
[Equation]
d =Dimensionality of states; m ∈ N - tunable free parameter.
In all experiments we used the above form of F as follows: a state s t is input to the g network and the network outputs logits p t .
We softmax and sample from p t to obtain the action θ ct .
This action is one-hot encoded.
Then, the action θ ct is multiplied with f ( s t ) to compute the second term of F .
A similar process is used to compute the ﬁrst term.
In this way the policy of Generator chooses the intrinsic-reward.
13. H YPERPARAMETER S ETTINGS
In the table below we report all hyperparameters used in our experiments.
Hyperparameter values in square brackets indicate ranges of values that were used for performance tuning.
Clip Gradient Norm γ E 0.
λ 0.
Learning rate 1 x 10 −
Number of minibatches
Number of optimisation epochs
Number of parallel actors
Optimisation algorithm
Adam
Rollout length
Sticky action probability 0.
Use Generalized Advantage Estimation
True
Coefﬁcient of extrinsic reward [1, 5]
Coefﬁcient of intrinsic reward [1, 2, 5, 10, 20, 50]
Generator discount factor 0.
Probability of terminating option [
[Table]
Generator T ERMINATION T IMES
There are various possibilities for the termination times { τ 2 k } (recall that { τ 2 k +1 } are the times which the F is switched on using g c ).
One is for Generator to determine the sequence.
Another is to build a construction of { τ 2 k } that directly incorporates the information gain that a state visit provides: let w : Ω → { 0 , 1 } be a random variable with Pr( w = 1)
=
p and Pr( w = 0)
= 1 − p where p ∈ ]0 , 1] .
Then for any k = 1 , 2 , . . .
, and denote by ∆ L ( s τ k ) :
= L ( s τ k ) − L ( s τ k − 1 ) , then we can set:
[Equation]
To explain, since { τ 2 k } k ≥ 0 are the times at which F is switched off then if F is deactivated at exactly after j time steps then I ( s τ 2 k +1 + l )
=
I ( s τ 2 k +1 ) for any 0 ≤ l < j and I ( s τ 2 k
+1 + j )
=
I ( s τ 2 k +2 ) .
We now see that (4) terminates F when either the random variable w attains a 0 or when ∆ L ( s τ k + j ) ≤ 0 which occurs when the exploration bonus in the current state is lower than that of the previous state.
N OTATION & A SSUMPTIONS
We assume that S is deﬁned on a probability space (Ω , F , P ) and any s ∈ S is measurable with respect to the Borel σ -algebra associated with R p .
We denote the σ -algebra of events generated by { s t } t ≥by F t ⊂ F .
In what follows, we denote by ( V , (cid:107)(cid:107) )
any ﬁnite normed vector space and by H the set of all measurable functions.
Where it will not cause confusion (and with a minor abuse of notation) for a given function
h we use the shorthand h
( π i ,π − i ) ( s ) =
h ( s, π i , π − i ) ≡ E π i ,π − i
[ h ( s, a i , a − i )] .
The results of the paper are built under the following assumptions which are standard within RL and stochastic approximation methods:
Assumption 1 The stochastic process governing the system dynamics is ergodic, that is the process is stationary and every invariant random variable of { s t } t ≥ 0 is equal to a constant with probability 1 .
Assumption 2 The constituent functions of the agents’ objectives R , F and L are in L 2 .
Assumption 3 For any positive scalar c , there exists a scalar µ c such that for all s ∈ S and for any t ∈ N we have: E [1 + (cid:107) s t (cid:107) c | s
=
s ] ≤ µ c (1 + (cid:107) s (cid:107) c ) .
Assumption 4 There exists scalars C 1 and c 1 such that for any function J satisfying | J ( s ) | ≤
C 2 (1 + (cid:107) s (cid:107) c 2 ) for some scalars c 2 and C 2 we have that: (cid:80) ∞
t =0 | E [ J ( s t ) | s 0 = s ] − E
[ J ( s 0 )] | ≤
C 1 C 2 (1 + (cid:107) s t (cid:107) c
1 c 2 ) .
Assumption 5 There exists scalars c and C such that for any s ∈ S we have that: | J ( s, · ) | ≤
C (1 + (cid:107) s (cid:107) c ) for J ∈ { r i , F, L } .
We also make the following ﬁniteness assumption on set of switching control policies for Generator :
Assumption 6 For any policy g c , the total number of interventions is K < ∞ .
We lastly make the following assumption on L which can be made true by construction:
Assumption 7 Let n ( s ) be the state visitation count for a given state s ∈ S .
For any a ∈ A , the function L ( s, a ) =
for any n ( s ) ≥ M where 0 < M ≤ ∞ .
P ROOF OF T ECHNICAL R ESULTS
We begin the analysis with some preliminary lemmata and deﬁnitions which are useful for proving the main results.
Deﬁnition 1 A.1 An operator T : V → V is said to be a contraction w.r.t a norm (cid:107) · (cid:107) if there exists a constant c ∈ [0 , 1[ such that for any V 1 , V 2 ∈ V we have that:
[Equation]
Deﬁnition 2 A.2 An operator T : V → V is non-expansive if ∀ V 1 , V 2 ∈ V we have:
[Equation]
Lemma
For any f : V → R , g : V → R , we have that:
[Equation]
Proof 1 We restate the proof given in (Mguni, 2019):
[Equation]
[Equation] [Equation]
Deducting max a ∈V g ( a ) from both sides of (9) yields:
[Equation]
After reversing the roles of f and g and redoing steps (8) - (9), we deduce the desired result since the
RHS of (10) is unchanged.
Lemma
A.4 The probability transition kernel P is non-expansive, that is:
[Equation]
Proof
The result is well-known e.g. (Tsitsiklis & Van Roy, 1999).
We give a proof using the
Tonelli-Fubini theorem and the iterated law of expectations, we have that:
[Equation]
where we have used Jensen’s inequality to generate the inequality.
This completes the proof.
P ROOF OF P ROP .
Proof 3 (Proof of Prop. 1) To prove the proposition it sufﬁces to prove that the term (cid:80)
Tt =
γ t F ( θ t , θ t − 1 ) I ( t ) converges to 0 in the limit as T → ∞ .
As in classic potential-based reward shaping (Ng et al., 1999), central to this observation is the telescoping sum that emerges by construction of F .
First recall v π ,g ( s, I 0 ) , for any ( s, I 0 )
∈ S × { 0 , 1 } is given by:
[Equation]
Hence it sufﬁces to prove that E π ,
[Equation]
Recall there a number of time steps that elapse between τ k and τ k +1 , now ∞
[Equation]
where we have used the fact that by construction θ ct ≡
whenever t = τ 1 , τ 2 , . .
and by construction φ ( s, 0) ≡ 0 for any s .
With this we readily deduce that v π ,g ( s ) =
E π ,g [ (cid:80) ∞ t =
γ t R ( s t , a t )] which is a measure of environment rewards only from which statement (i) can be readily deduced.
For part (ii) we note ﬁrst that it is easy to see that v π ,g c ( s 0 , I 0 ) is bounded above, indeed using the key result in the proof of part (i) and the properties of c we have that
[Equation]
[Equation]
using the triangle inequality
[Equation]
the deﬁnition of R θ and the (upper-)boundedness of L and R (Assumption 5).
We now note that by the dominated convergence theorem we have that ∀ ( s 0 , I 0 )
∈ S × { 0 , 1 }
[Equation]
where K is the total number of switch activations or interventions and where again we have used the key result in the proof of (i) and Assumption 6 in the last step, after which we deduce (ii) since v π and v π ,g c differ by only a constant.
Note that by (ii) we heron may consider the quantity for the Generator expected return:
[Equation]
P ROOF OF T HEOREM
Proof 4 Theorem 1 is proved by ﬁrstly showing that when the agents jointly maximise the same objective there exists a ﬁxed point equilibrium of the game when all agents use Markov policies and
Generator uses switching control.
The proof then proceeds by showing that the MG G admits a dual representation as an MG in which all agents N × { c } jointly maximise the same objective which has
MPE that can be computed by solving an MDP.
Thereafter, we use both results to prove the existence of a ﬁxed point for the game as a limit point of a sequence generated by successively applying the
Bellman operator to a test function.
Therefore, the scheme of the proof is summarised with the following steps:
I)
Prove that the solution to Markov Team games (that is games in which all agents maximise an identical objective) in which one of the agents uses switching control is the limit point of a sequence of Bellman operators (acting on some test function).
II) Prove that for the MG G that is there exists a function B π ,g : S × { 0 , 1 } → R such that
[Equation]
III) Prove that the MG, G has a dual representation as a Markov Team Game which admits a representation as an MDP.
P ROOF OF P ART I
Our ﬁrst result proves that the operator T is a contraction operator.
First let us recall that the switching time τ k is deﬁned recursively τ k = inf { t > τ k − 1 | s t ∈ A, τ k ∈ F t } where
A = { s ∈
S , m ∈ M | g c ( m | s t ) > 0 } .
To this end, we show that the following bounds holds:
Lemma
The Bellman operator T is a contraction, that is the following bound holds:
[Equation]
Proof 5 Recall we deﬁne the Bellman operator T ψ of G acting on a function Λ : S × N → R by
T ψ Λ( s τ k , I ( τ k )) :
=
max
[Equation] [Equation]
In what follows and for the remainder of the script, we employ the following shorthands:
[Equation]
To prove that T is a contraction
[Equation]
we consider the three cases produced by (27)
[Equation]
that is to say we prove the following statements:
i)
[Equation]
≤ γ (cid:107) ψ − ψ (cid:48) (cid:107) ii) (cid:107)M π ,g ψ −
[Equation]
π ,
[Equation]
(cid:48) (cid:107)
[Equation]
(cid:107)
[Equation]
(cid:48) (cid:107) , (and hence M is a contraction).
iii) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:20)
[Equation] [Equation]
[Equation]
γ
[Equation]
ψ
[Equation]
where z t ≡ ( s t , I t ) ∈ S × { 0 , 1 } .
We begin by proving i)
[Equation]
Indeed, for any a ∈ A and ∀ z t ∈ S × { 0 , 1 } , ∀ θ t , θ t − 1 ∈ Θ , ∀ s
[Equation]
∈ S we have that
[Equation]
again using the fact that P is non-expansive and Lemma 1.
We now prove ii).
For any τ ∈ F , deﬁne by τ (cid:48)
=
inf { t > τ | s t ∈ A, τ ∈ F t } .
Now using the deﬁnition of M we have that for any s τ ∈ S
[Equation]
|
[Equation]
using the fact that P is non-expansive.
The result can then be deduced easily by applying max on both sides.
We now prove iii).
We split the proof of the statement into two cases:
Case 1:
[Equation]
We now observe the following:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where we have used the fact that for any scalars a, b, c we have that | max { a, b } − max { b, c }| ≤ | a − c | and the non-expansiveness of P .
[Table]
[Equation]
[Table] [Table]
[Equation]
again using the fact that P is non-expansive.
Hence we have succeeded in showing that for any Λ ∈ L 2 we have that
[Equation]
Gathering the results of the three cases gives the desired result.
P ROOF OF P ART II
To prove Part II
[Equation]
we prove the following result:
Proposition 3 For any π ∈ Π and for any Generator policy g , there exists a function B π ,g :
S × { 0 , 1 } → R such that
[Equation]
where in particular the function B is given by:
[Equation]
for any ( s 0 , I 0 ) ∈ S × { 0 , 1 }
[Equation]
Proof 6 Note that by the deduction of (ii) in Prop
[Equation]
we immediately observe that
[Equation]
We therefore immediately deduce that for any two Generator policies g and g (cid:48)
the following expression holds ∀ ( s 0 , I 0 )
∈ S × { 0 , 1 } :
[Equation]
Our aim now is to show that the following expression holds ∀ ( s 0 , I 0 ) ∈ S × { 0 , 1 } :
[Equation]
For the ﬁnite horizon case, the result is proven by induction on the number of time steps until the end of the game.
Unlike the inﬁnite horizon case, for the ﬁnite horizon case the value function and policy have an explicit time dependence.
We consider the case of the proposition at time T − 1 that is we evaluate the value functions at the penultimate time step.
In this case, we have that:
[Equation]
We now observe that for any π ∈ Π and for any g we have that B π ,
gT
( I T , s T ) =
E (cid:104)(cid:80)
[Equation] [Equation]
jτ
[Equation]
, moreover we have that for any π ∈ Π and for any g
E (cid:34) (cid:88) a T ∈
A
[Equation]
Hence we ﬁnd that
[Equation] [Equation]
Now
[Equation]
[Table]
[Equation]
46) using Assumption 6.
Hence we ﬁnd that
[Equation] [Equation]
Hence, we have succeeded in proving that the expression (30) holds for T − k when k = 1 .
Our next goal is to prove that the expression holds for any 0 < k ≤ T .
Note that for any T ≥ k > 0 , we can write B π ,gT − k as
[Equation]
Now we consider the case when we evaluate the expression (30) for any 0 < k ≤ T .
Our inductive hypothesis is the the expression holds for some 0 < k ≤ T , that is for any 0 < k ≤ T we have that:
[Equation]
It remains to show that the expression holds for k + 1 time steps prior to the end of the horizon.
The result can be obtained using the dynamic programming principle and the base case ( k = 1 ) result, indeed we have that
[Equation]
− ( k +1) ∼ d θ (cid:104) B π ,gT − ( k +1)
( I T − ( k +1) , s T − ( k +1) )
− B π (cid:48) ,g T − ( k +1)
( I T − ( k +1) , s T − ( k +1) ) (cid:105) =
E s T − ( k +1) ∼ d θ (cid:34) (cid:88) a T − ( k +1) ∈
A π ( a T − ( k +1) ; s T − ( k +1) )
  R ( s T − ( k +1) , a T − ( k +1) )
+ (cid:88) k ≥∞ (cid:88)
j = T −c ( I j , I j − 1 )
δ jτ k   + γ (cid:88) s T − k ∈ S (cid:88) a T − ( k +1)
∈
A π ( a T − ( k +1) ; s T − ( k +1) )
P ( s T − k ; a T − ( k +1) )
B π ,gT − k
( I T − k , s T − k ) − (cid:32) (cid:88) a (cid:48)
T − ( k +1)
∈
A π (cid:48)
( a (cid:48)
T − ( k +1) ; s T − ( k +1) )
  R ( s T − ( k +1) , a (cid:48)
T − ( k +1) )
+ (cid:88) k ≥∞ (cid:88)
j = T −c ( I j , I j − 1 )
δ jτ k   + γ (cid:88) s T − k ∈ S (cid:88)
a (cid:48)
T − ( k +1)
∈
A π (cid:48)
( a (cid:48)
T − ( k +1) ; s T − ( k +1) )
P ( s T − k ; a (cid:48)
[Equation]
(cid:35)(cid:33)
[Equation]
using the inductive hypothesis and where we have used the fact that
[Equation]
via similar reasoning as before and after which which we deduce the result in the ﬁnite case.
For the inﬁnite horizon case, we must prove that there exists a measurable function B : Π × S → R such that the following holds for any i ∈ N and ∀ π i , π (cid:48)
i ∈
Π i , ∀ π − i ∈
Π − i and ∀ s ∈ S :
[Equation] [Equation]
The result is proven by contradiction.
To this end, let us ﬁrstly assume ∃ c (cid:54) =
such that
[Equation]
Let us now deﬁne the following quantities for any s ∈ S and for each π i ∈ Π i and π − i ∈
Π − i and ∀ i ∈ N :
[Equation]
and
[Equation]
so that the quantity v π
[Equation]
( s ) measures the expected cumulative return until the point T (cid:48) < ∞ .
Hence
[Equation]
we deduce that
[Equation]
Next we observe that:
[Equation]
[Table]
Considering the last expectation and its coefﬁcient and denoting the product by κ , using the fact that by the Cauchy-Schwarz inequality we have (cid:107) AX − BY (cid:107) ≤ (cid:107) A (cid:107)(cid:107) X (cid:107) + (cid:107) B (cid:107)(cid:107) Y (cid:107) , moreover whenever A, B are non-expansive we have that (cid:107) AX − BY (cid:107) ≤ (cid:107) X (cid:107) + (cid:107) Y (cid:107) , hence we observe the following κ ≤ (cid:107) κ (cid:107) ≤ 2 γ T (cid:48) ( (cid:107) v i (cid:107) + (cid:107) B (cid:107) ) .
Since we can choose T (cid:48) freely and γ ∈ ]0 , 1[ , we can choose T (cid:48) to be sufﬁciently large so that γ T (cid:48) ( (cid:107) v i (cid:107) + (cid:107) B (cid:107) ) <
14 | c | .
This then implies that
[Equation]
which is a contradiction since we have proven that for any ﬁnite T (cid:48)
it is the case that
[Equation]
and hence we deduce the result in the inﬁnite horizon case.
P ROOF OF P ART III
To prove Part III , we ﬁrstly deﬁne precisely the notion of a stable point of the MG, G :
Deﬁnition
A policy proﬁle σ (cid:63) =
( g
[Equation]
, π
[Equation]
, π
[Equation]
∈ Π is a Markov perfect equilibrium (MPE) in
Markov strategies if the following condition holds for any i ∈ N × { 0 } :
[Equation]
The condition characterises strategic conﬁgurations which are stable points of the MG, G .
In particular, an MPE is achieved when at any state no agent can improve their expected cumulative rewards by unilaterally deviating from their current policy.
We denote by NE {G} the set of MPE strategies for the MG, G .
Next we prove that the set of maxima of the function B are the MPE of the MG G :
Proposition 4 The following implication holds:
[Equation]
where B is the function in Prop. 3.
Prop. 4 indicates that the game has an equivalent representation in which all agents maximise the same function and thus play a team game.
Proof
We do the proof by contradiction.
Let σ =
( π 1 , . . .
, π N , g ) ∈ arg sup
[Equation]
( s ) for any s ∈ S .
Let us now therefore assume that σ / ∈ NE {G} , hence there exists some other policy proﬁle ˜
σ = ( π 1 , . . .
˜ π
i , . . .
, π N , g ) which contains at least one proﬁtable deviation by one of the agents i ∈ N × { 0 , } .
For now let us consider the case in which the proﬁtable deviation is for a agent i ∈ N
so that π (cid:48)
i (cid:54) =
π i for i ∈ N i.e. v ( π
[Equation]
i
[Equation]
) (using the preservation of signs of integration).
Prop. 3 however implies that B ( π (cid:48)
i ,π − i ) ,g ( s ) − B ( π i ,π − i ) ,g ( s ) > 0 which is a contradiction since σ = ( π i , π − i , g ) is a maximum of B .
The proof can be straightforwardly adapted to cover the case in which the deviating agent is Generator after which we deduce the desired result.
The last result completes the proof of Theorem 1.
P ROOF OF P ROPOSITION
Proof 7 (Proof of Prop. 2)
The proof is given by establishing a contradiction.
Therefore suppose that M π ,
[Equation] [Equation]
s τ k , I ( τ k )) and suppose that the switching time τ (cid:48) 1 > τ 1 is an optimal switching time.
Construct the Generator g (cid:48) and ˜ g policy switching times by ( τ (cid:48) 0 , τ (cid:48) 1 , . . .
, ) and g (cid:48)policy by ( τ (cid:48) 0 , τ 1 , . . . )
respectively.
Deﬁne by l = inf { t > 0; M π ,g ψ ( s t , I 0 ) =
ψ ( s t , I 0 ) } and m = sup { t ; t < τ (cid:48) 1 } .
By construction we have that
[Equation]
We now use the following observation E (cid:104) R ( s
[Equation]
, a
[Equation]
1 ) + γ M
[Equation] [Equation]
π
[Equation]
≤
[Equation]
Using this we deduce that
[Equation]
where the ﬁrst inequality is true by assumption on M .
This is a contradiction since g (cid:48) is an optimal policy for Generator .
Using analogous reasoning
[Equation]
we deduce the same result for τ (cid:48)
k < τ k after which deduce the result.
Moreover, by invoking the same reasoning, we can conclude that it must be the case that ( τ 0 , τ 1 , . . .
, τ k − 1 , τ k , τ k +1 , . . .
, ) are the optimal switching times.
P ROOF OF T HEOREM
Proof 8 (Proof of Theorem 2)
The proof which is done by contradiction follows from the deﬁnition of v c .
Denote by v π ,g ≡
i value function an agent i ∈ N excluding Generator and its intrinsic-reward function.
Indeed, let ( ˆ π , ˆ g ) be the policy proﬁle induced by the Nash equilibrium policy proﬁle and assume that the intrinsic-reward F leads to a decrease in payoff for agent i .
Then by construction v π ,g ( s ) < v π ,g ≡ 0 ( s ) which is a contradiction since ( ˆ π , ˆ g ) is an MPE proﬁle.
P ROOF OF T HEOREM
To prove the theorem, we make use of the following result:
Theorem 4 (Theorem 1, pg 4 in (Jaakkola et al., 1994)) Let Ξ t ( s ) be a random process that takes values in R n and given by the following:
[Equation]
then Ξ t ( s ) converges to 0 with probability 1 under the following conditions:
i)
≤ α t ≤ 1 , (cid:80) t α t = ∞ and (cid:80) t
α t < ∞ ii) (cid:107) E
[ L t |F t ] (cid:107) ≤ γ (cid:107)
Ξ t (cid:107) , with γ < 1 ; iii)
Var [ L t |F t ] ≤ c (1 + (cid:107) Ξ t (cid:107) 2 ) for some c > 0 .
Proof 9 To prove the result, we show (i) - (iii) hold.
Condition (i) holds by choice of learning rate.
It therefore remains to prove (ii) - (iii).
We ﬁrst prove (ii).
For this, we consider our variant of the
Q-learning update rule:
[Equation] [Equation]
) (cid:21) .
After subtracting Q (cid:63) ( s t , I t , a t ) from both sides and some manipulation we obtain that:
[Equation]
t
[Equation]
where Ξ t ( s t , I t , a t ) :
=
Q t ( s t , I t , a t ) − Q (cid:63) ( s t , I t , a t ) .
Let us now deﬁne by
L
[Equation]
τ
[Equation]
max
[Equation]
Then Ξ
[Equation] [Equation]
We now observe that
E
[Equation]
Now, using the ﬁxed point property that implies Q (cid:63)
=
T φ Q (cid:63) , we ﬁnd that
[Equation] [Equation]
using the contraction property of T established in Lemma 3.
This proves (ii).
We now prove iii), that is
[Equation]
Now by (65) we have that
Var [
[Equation]
] =
Var
[Equation]
Var
[Equation]
2 ) , for some c > 0 where the last line follows due to the boundedness of Q (which follows from
Assumptions 2 and 4).
This concludes the proof of the Theorem.
P ROOF OF C ONVERGENCE WITH F UNCTION A PPROXIMATION
First let us recall the statement of the theorem:
Theorem 3 LIGS converges to a limit point r (cid:63) which is the unique solution to the equation:
[Equation]
where we recall that for any test function Λ ∈ V , the operator F is deﬁned by F Λ := Θ + γP max {M Λ , Λ } .
Moreover, r (cid:63) satisﬁes the following:
[Equation]
The theorem is proven using a set of results that we now establish.
To this end, we ﬁrst wish to prove the following bound:
Lemma 4 For any Q ∈ V we have that
[Equation]
so that the operator F is a contraction.
Proof 10 Recall, for any test function ψ , a projection operator Π acting Λ is deﬁned by the following
[Equation]
Now, we ﬁrst note that in the proof of Lemma 3, we deduced that for any Λ ∈ L 2 we have that
[Equation]
Λ (cid:48) (cid:107)
[Equation]
(c.f. Lemma 3).
Setting Λ = Q and ψ =
Θ , it can be straightforwardly deduced that for any Q, ˆ Q ∈ L 2 :
[Equation]
ˆ
[Equation]
≤ γ (cid:13)(cid:13)(cid:13) Q − ˆ Q (cid:13)(cid:13)(cid:13) .
Hence, using the contraction property of M , we readily deduce the following bound:
[Equation] [Equation]
We now observe that F is a contraction.
Indeed, since for any Q, Q (cid:48) ∈
L 2 we have that:
[Equation]
using (71) and again using the non-expansiveness of P .
We next show that the following two bounds hold:
Lemma
For any Q ∈ V we have that
i)
[Equation]
ii)
[Equation]
Proof
The ﬁrst result is straightforward since as Π is a projection it is non-expansive and hence:
[Equation]
using the contraction property of F .
This proves i)
[Equation]
For ii)
[Equation]
we note that by the orthogonality property of projections we have that (cid:104)
Φ r (cid:63)
[Equation]
Q (cid:63) , Φ r (cid:63)
[Equation]
(cid:63) (cid:105) , hence we observe that:
[Equation]
after which we readily deduce the desired result.
Lemma 6 Deﬁne the operator H by the following: HQ ( z ) =
[Equation]
z
[Equation]
, if
[Equation]
Q ( z )
[Equation]
Q ( z ) , otherwise , and ˜ F by: ˜ F Q := Θ + γPHQ .
For any Q, ¯ Q ∈ L 2 we have that
[Equation]
and hence ˜ F is a contraction mapping.
Proof 12 Using (71)
[Equation]
we now observe that
[Equation]
Lemma 7 Deﬁne by ˜ Q :=
Θ + γPv ˜ π where
[Equation]
then ˜ Q is a ﬁxed point of ˜ F ˜ Q , that is ˜ F ˜ Q = ˜ Q .
Proof 13 We begin by observing that
[Equation]
Hence,
[Equation]
which proves the result.
Lemma
The following bound holds:
[Equation]
Proof 14 By deﬁnitions of v ˆ π and v ˜ π
(c.f (73) ) and using Jensen’s inequality and the stationarity property we have that,
[Equation]
Now recall that ˜ Q :
=
Θ + γPv ˜ π and Q (cid:63) :=
Θ + γPv π (cid:63) , using these expressions in (76) we ﬁnd
that
[Equation]
Moreover, by the triangle inequality and using the fact that F (Φ r (cid:63) ) =
˜ F (Φ r (cid:63) ) and that F Q (cid:63) = Q (cid:63) and F ˜ Q =
˜ Q (c.f. (75) )
we have that
[Equation]
γ (cid:13)(cid:13)(cid:13) ˜
Q − Φ r (cid:63) (cid:13)(cid:13)(cid:13) + γ (cid:13)(cid:13)(cid:13) Q (cid:63) − ˜ Q (cid:13)(cid:13)(cid:13) , which gives the following bound:
[Equation]
from which, using Lemma 5, we deduce that (cid:13) (cid:13) (cid:13) ˜ Q − Q (cid:63) (cid:13) (cid:13) (cid:13) ≤ 2 (cid:104) (1 − γ ) (cid:112) (1 − γ 2 ) (cid:105) − 1 (cid:13) (cid:13) (cid:13)
˜ Q − Φ r (cid:63) (cid:13) (cid:13) (cid:13) , after which by (77) , we ﬁnally obtain
[Equation]
, as required.
Let us rewrite the update in the following way:
[Equation]
where the function Ξ : R 2 d × R p
→ R p is given by:
[Equation]
for any w ≡ ( z, z (cid:48) ) ∈ ( N × S ) 2 where z
=
( t, s ) ∈ N × S and z (cid:48)
=
( t, s (cid:48) )
∈ N × S and for any r ∈ R p .
Let us also deﬁne the function Ξ : R p
→ R p by the following:
[Equation]
Lemma
The following statements hold for all z ∈ { 0 , 1 } × S :
i) ( r − r (cid:63) )
Ξ k ( r ) < 0 , ∀ r
[Equation]
r (cid:63) , ii) Ξ k
( r (cid:63) ) = 0 .
Proof 15 To prove the statement, we ﬁrst note that each component of Ξ k ( r ) admits a representation as an inner product, indeed:
[Equation]
, using the iterated law of expectations and the deﬁnitions of P and F .
We now are in position to prove i).
Indeed, we now observe the following:
[Equation]
where in the last step we used the orthogonality of ( 1 − Π) .
We now recall that Π F Φ r (cid:63) =
Φ r (cid:63) since Φ r (cid:63) is a ﬁxed point of Π F .
Additionally, using Lemma 5 we observe that (cid:107) Π F Φ r − Φ r (cid:63) (cid:107) ≤ γ (cid:107) Φ r − Φ r (cid:63) (cid:107) .
With this we now ﬁnd that
[Equation]
which is negative since γ < 1 which completes the proof of part i).
The proof of part ii) is straightforward since we readily observe that
[Equation]
as required and from which we deduce the result.
To prove the theorem, we make use of a special case of the following result:
Theorem 5 (Th. 17, p. 239 in (Benveniste et al., 2012)) Consider a stochastic process r t : R × {∞} × Ω
→ R k which takes an initial value r 0 and evolves according to the following:
[Equation]
for some function s : R 2 d × R k
→ R k and where the following statements hold:
1. { s t | t = 0 , 1 , . . .
} is a stationary, ergodic Markov process taking values in R 2 d 2.
For any positive scalar q , there exists a scalar µ q such that E [1 + (cid:107) s t (cid:107) q | s ≡ s 0 ] ≤ µ q (1 + (cid:107) s (cid:107) q ) 3.
The step size sequence satisﬁes the Robbins-Monro conditions, that is (cid:80) ∞ t =
α t = ∞ and (cid:80) ∞ t =
α 2 t < ∞ 4.
There exists scalars c and q such that (cid:107) Ξ( w, r ) (cid:107) ≤ c (1 + (cid:107) w (cid:107) q ) (1 + (cid:107) r (cid:107) ) 5.
There exists scalars c and q such that (cid:80) ∞ t =
(cid:107) E
[Ξ( w t , r )
| z 0 ≡ z ] − E [Ξ( w 0 , r )] (cid:107) ≤ c (1 + (cid:107) w (cid:107) q ) (1 + (cid:107) r (cid:107) ) 6.
There exists a scalar c > 0 such that (cid:107) E
[Ξ( w 0 , r )] − E [Ξ( w 0 , ¯ r )] (cid:107) ≤ c (cid:107) r − ¯ r (cid:107) 7.
There exists scalars c > and q > such that (cid:80) ∞ t =0 (cid:107) E
[Ξ( w t , r ) | w 0 ≡ w ] − E
[Ξ( w 0 , ¯ r )] (cid:107) ≤ c (cid:107) r − ¯ r (cid:107) (1 + (cid:107) w (cid:107) q ) 8.
There exists some r (cid:63) ∈ R k such that Ξ ( r )( r − r (cid:63) ) < 0 for all r
[Equation]
=
[Equation]
and
[Equation]
= 0 .
Then r t converges to r (cid:63) almost surely.
In order to apply the Theorem 5, we show that conditions 1 - 7 are satisﬁed.
Proof 16 Conditions 1-2 are true by assumption while condition 3 can be made true by choice of the learning rates.
Therefore it remains to verify conditions 4-7 are met.
To prove 4, we observe that
[Equation]
Now using the deﬁnition of M , we readily observe that (cid:107)M Φ( z (cid:48)
[Equation]
+
[Equation]
+ γ (cid:107) Φ (cid:107) using the non-expansiveness of P .
Hence, we lastly deduce that
[Equation]
we then easily deduce the result using the boundedness of φ, Θ and ψ
[Equation]
Now we observe the following Lipschitz condition on Ξ :
[Equation]
φ ( z ) (cid:107) (cid:107) φ (cid:48) ( z ) r − φ ( z )¯ r (cid:107) ≤ γ (cid:107) φ ( z ) (cid:107) (cid:107) φ (cid:48)
( z (cid:48) )
r − φ (cid:48)
( z (cid:48) )
¯ r (cid:107) + (cid:107) φ ( z ) (cid:107) (cid:107) φ (cid:48)
( z ) r − φ (cid:48)
( z )¯ r (cid:107) ≤ (cid:107) φ ( z ) (cid:107) ( (cid:107) φ ( z ) (cid:107) + γ (cid:107) φ ( z ) (cid:107) (cid:107) φ (cid:48)
( z (cid:48) ) − φ (cid:48)
( z (cid:48) )
(cid:107) )
(cid:107) r − ¯ r (cid:107) ≤ c (cid:107) r − ¯ r (cid:107) , using Cauchy-Schwarz inequality and that for any scalars a, b, c we have that | max { a, b } − max { b, c }| ≤ | a − c | .
Using Assumptions 3 and 4, we therefore deduce that ∞ (cid:88) t =
[Equation]
Part 2 is assured by Lemma 5 while Part 4 is assured by Lemma 8 and lastly Part 8 is assured by
Lemma 9.
[Figure]