Surprise Minimizing Multi-Agent Learning with Energy-based Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Multi-Agent Reinforcement Learning (MARL) has demonstrated signiﬁcant success by virtue of collaboration across agents.
Recent work, on the other hand,introduces surprise which quantiﬁes the degree of change in an agent’s environment.
Surprise-based learning has received signiﬁcant attention in the case ofsingle-agent entropic settings but remains an open problem for fast-paced dynamicsin multi-agent scenarios.
A potential alternative to address surprise may be realizedthrough the lens of free-energy minimization.
We explore surprise minimization inmulti-agent learning by utilizing the free energy across all agents in a multi-agentsystem.
A temporal Energy-Based Model (EBM) represents an estimate of surprisewhich is minimized over the joint agent distribution.
Our formulation of the EBMis theoretically akin to the minimum conjugate entropy objective and highlightssuitable convergence towards minimum surprising states.
We further validate ourtheoretical claims in an empirical study of multi-agent tasks demanding collaboration in the presence of fast-paced dynamics.
Our implementation and agent videosare available at the anonymous Project Webpage .
Introduction
The rise of RL has led to an increasing interest in the study of multi-agent systems [34, 58], commonlyknown as Multi-Agent Reinforcement Learning (MARL).
In the case of partially observable settings,
MARL enables the learning of policies with centralised training and decentralised control [26].
Thishas proven to be useful for exploiting value-based methods which motivate collaboration across largenumber of agents.
But how do agents behave in the presence of sudden environmental changes?
Consider the problem of autonomous driving wherein a driver (agent) autonomously operates avehicle in real-time.
The driver learns to optimize the reward function by maintaining constantspeed and covering more distance in different trafﬁc conditions.
Whenever the vehicle approachesan obstacle, the driver acts to avoid it by utilizing the brake and directional steering commands.
However, due to the fast-paced dynamics of the environment, say fast-moving trafﬁc, the agent mayabruptly encounter an obstacle ( a person running across the street ) which may result in a collision.
Irrespective of the optimal action ( pushing of brakes ) executed by the agent, the vehicle may fail toevade the collision as a result of the abrupt temporal change.
The above arises as a consequence of surprise , which is deﬁned as a statistical measure of uncertainty.
Surprise minimization [3] is a recent phenomenon observed in the case of single-agent RL methodswhich deals with environments consisting of rapidly changing states.
In the case of model-based
RL [24], surprise minimization is used as an effective planning tool in the agent’s model [3].
Inthe case of model-free RL, surprise minimization is witnessed as an intrinsic motivation [1, 36] orgeneralization problem [9].
On the other hand, MARL does not account for surprise across agents as a result of which agents remain unaware of drastic changes in the environment [35].
Thus, surpriseminimization in the case of multi-agent settings requires attention from a critical standpoint.
A potential pathway to treat surprising states may be realized in light of free-energy minimization.
The free-energy principle depicts convergence to local niches and provides a general recipe forstability among agents.
Through this lens, we unify surprise with free-energy in the multi-agentsetting.
We construct a temporal EBM which represents an estimate of surprise agents may face inthe environment.
All agents jointly minimize this estimate utilizing temporal difference learning upontheir value functions and the EBM.
Our formulation of free-energy minimization is theoretically akinto minimizing the entropy in conjugate gradient space.
This insight provides a suitable convergenceresult towards minimum surprising states (or niches) of the agent state distributions.
In an empirical study of multi-agent tasks which present signiﬁcant collaboration bottlenecks and fast-paceddynamics, we validate our theoretical claims and motivate the practical usage of EBMs in MARL.
Related Work
Surprise Minimization:
Despite the recent success of value-based methods [39, 22] RL agents sufferfrom spurious state spaces and encounter sudden changes in trajectories.
Quantitatively, surprisehas been studied as a measure of deviation [3, 9] among states encountered by the agent duringits interaction with the environment.
While exploring [7, 56] the environment, agents tend to havehigher deviation among states which is gradually reduced by gaining a signiﬁcant understanding ofstate-action transitions.
In the case of model-based RL, agents can leverage spurious experiences [3]and plan effectively for future steps.
On the other hand, in the case of model-free RL, surprise resultsin sample-inefﬁcient learning [1].
This is primarily addressed by making use of rigorous explorationstrategies [52, 31].
High-dimensional exploration further requires extrinsic feature engineering [27]and meta models [16].
A suitable way to tackle high-dimensional dynamics is by utilizing surprise asa penalty on the reward [9].
This leads to improved generalization for single-agent interactions [45].
Our proposed approach is parallel to the aforesaid methods.
Energy-based Models: EBMs have been successfully implemented in single-agent RL methods[42, 19].
These typically make use of Boltzmann distributions to approximate policies [32].
Such aformulation results in the minimization of free energy within the agent.
While policy approximationdepicts promise in the case of unknown dynamics, inference methods [57] play a key role in optimizinggoal-oriented behavior.
A second type of usage of EBMs follows the maximization of entropy [65].
The maximum entropyframework [20] highlighted in Soft Q-Learning (SQL) [19] allows the agent to obey a policy whichmaximizes its reward and entropy concurrently.
Maximization of agent’s entropy results in diverse andadaptive behaviors [64] which may be difﬁcult to accomplish using standard exploration techniques[7, 56].
The maximum entropy framework is akin to approximate inference in the case of policygradient methods [49].
Such a connection between likelihood ratio gradient techniques and energybased formulations leads to diverse and robust policies [17].
Furthermore, their hierarchical extensions[18] preserve the lower levels of hierarchies.
In the case of MARL, EBMs have witnessed limitedapplicability as a result of the increasing number of agents and complexity within each agent [8].
While the framework is readily transferable to opponent-aware multi-agent systems [63], cooperativesettings consisting of coordination between agents require a ﬁrm formulation of energy.
Thisformulation must be scalable in the number of agents [15] and account for environments consistingof spurious states [62].
Our theoretical formulation is motivated by these methods in literature.
Preliminaries
Multi-Agent Learning: We review the cooperative MARL setup.
The problem is modeled as a
Decentralized Partially Observable Markov Decision Process (Dec-POMDP)
[43] deﬁned by thetuple ( S , A , r, N, P, Z, O, γ )
where the state space S and action space A are discrete, r : S
×
A →[ r min , r max ] presents the reward observed by agents a ∈ N where N is the set of all agents,
P : S ×S ×A → [0 , 1] presents the unknown transition model consisting of the transition probabilityto the next state s (cid:48)
∈ S given the current state s ∈ S and joint action u ∈ A (a combination ofeach agent’s action u a ∈
A
a ) at time step t and γ is the discount factor.
Our setting consists of theﬁnite-horizon discounted problem case where episodes terminate at timestep T with the terminal state being s T .
As a result, task returns remain bounded for each episode.
We consider a partiallyobservable setting in which each agent n draws individual observations z ∈ Z according to theobservation function O ( s, u ) : S ×
A → Z .
We consider a joint policy π θ ( u | s ) as a function ofmodel parameters θ .
Standard RL deﬁnes the agent’s objective to maximize the expected discountedreward E π θ [ (cid:80) Tt =
γ t r ( s t , u t )] as a function of the parameters θ .
The joint action-value function foragents is represented as Q ( u, s ; θ ) =
E π θ [ (cid:80) Tt =1 γ t r ( s, u ) | s = s t , u = u t ] which is the expectedsum of payoffs obtained in state s upon performing action u by following the policy π θ .
We denotethe optimal policy π θ ∗ ( shorthand π ∗ ) such that Q ( u, s ; θ ∗ ) ≥ Q ( u, s ; θ ) ∀ s ∈ S, u ∈ A .
In the caseof multiple agents, the joint optimal policy can be expressed as the Nash Equilibrium [40] of the
Stochastic Markov Game as π ∗ =
( π 1 , ∗ , π 2 , ∗ , ...
π N, ∗ ) such that Q ( u a , s ; θ ∗ ) ≥ Q ( u a , s ; θ ) ∀ s ∈
S, u ∈ A, a ∈ N .
Q-Learning is an off-policy, model-free algorithm suitable for continuous andepisodic tasks.
The algorithm uses semi-gradient descent to minimize the Temporal Difference (TD)error in Equation 1.
[Equation]
where y = r + γ max u (cid:48) ∈
A
[Equation] [Equation]
(cid:48)
[Equation]
s
[Equation]
; θ − ) is the TD target consisting of θ − as the target parameters and Rdenotes the replay buffer.
Energy-based Models: EBMs [29, 30]
have been successfully applied in the ﬁeld of machine learning [55] and probabilistic inference [37].
A typical EBM E formulates the equilibrium probabilities[47] P ( v, h ) =
exp( −E ( v,h ))
(cid:80) ˆ v, ˆ h
[exp( −E (ˆ v, ˆ h ))] via a Boltzmann distribution [32] where v and h are the valuesof the visible and hidden variables and ˆ v
and ˆ h are all the possible conﬁgurations of the visibleand hidden variables respectively.
The probability distribution over all the visible variables can beobtained by summing over all possible conﬁgurations of the hidden variables.
This is mathematicallyexpressed in Equation 2.
[Equation]
Here, E ( v, h ) is called the equilibrium free energy which is the minimum of the variational freeenergy and (cid:80) ˆ v, ˆ h exp ( −E (ˆ v, ˆ h )) is the partition function.
Energy-based Surprise Minimization
We begin by constructing surprise minimization as an energy-based problem in the temporal setting.
The motivation behind an energy-based formulation stems from rapidly changing states as an undesired niche among agents in the case of partially-observed settings.
To steer agents away from thisniche, we further construct a method which incorporates the theoretical aspect of the study.
4.1 The Surprise Minimization Objective
To make analysis tractable towards valid function spaces and surprising states, we take into accounttwo assumptions which form the central basis of surprise minimization among multiple agents.
Assumption 1.
(Completeness of value function space)
The space Π : S ×
A of all Q value functions Q ( s, u ) ∈ Π , ∀ s ∈ S , ∀ u ∈
A is a nonempty complete metric space.
Assumption 1 restricts the formulation of individual agent value functions Q a to the nonemptycomplete metric space.
A nonempty space conﬁrms the presence of candidate functions Q a upperbounded by the optimal function Q ∗ , i.e.- Q a ≤ Q ∗ , ∀ a ∈ N [5].
The completeness counterpart, onthe other hand, provisions a ﬁxed interior int Π for optimization [6].
Assumption 2.
(Constant surprise at Equilibrium)
In the limit of convergence lim π a → π ∗ to an optimal policy π ∗ , all agents a ∈ N incur a ﬁnite surprise ζ > 0 between consecutive states s and s (cid:48) until termination state s T .
Assumption 2 is directly based on the constant and continuous temporal aspect of surprise minimization [50, 12].
Corresponding to the lifetime of each agent a ∈ N , a desired minima bakes in theoptimal distribution of actions which correspond to minimum but ﬁnite instantaneous surprise.
We formulate the energy-based objective consisting of surprise as a function of states s , joint actions uand standard deviation σ of observations for each agent a .
In the case of high-dimensional state spaces(such as multiple opponents), σ informs agents of the abrupt statistical change that would take placeupon executing action u .
We formulate surprise as T V a surp ( s, u, σ ) which serves as an uncertaintyquantiﬁer Unc(s,a) of the state-action distribution.
Here V a surp ( s, u, σ ) denotes the surprise valuefunction which serves as a mapping from agent and environment dynamics to surprise.
Deﬁne anoperator presented in Equation 3 which sums surprising conﬁgurations across all agents.
[Equation]
136 Remark 1.
T V a surp ( s, u, σ ) intuitively provides a global estimate of surprise.
If all agents are equallylikely to face a surprising state, then T V a surp ( s, u, σ ) captures their individual contributions.
The formulation makes use of the soft-maximum operator [2].
The operator T V a surp ( s, u, σ ) is similarto prior energy formulations [19] where the energy across different actions is evaluated.
In our case,inference is carried out across all agents with actions as prior variables.
However, in the special caseof using an EBM as a Q -function, our approach suitably generalizes to the above methods (details in
Appendix B).
Our choice of T V a surp ( s, u, σ ) is based on its unique mathematical properties which result in betterconvergence.
Of these properties, the most useful result is that T forms a contraction on the surprisevalue function V a surp ( s, u, σ ) indicating a guaranteed minimization of surprise within agents.
Thisis formally stated in Theorem 1 while utilizing the completeness criterion of Assumption 1 whichprovides a tractable value function space.
All proofs are deferred to Appendix A.
Theorem 1.
Given a surprise value function V a surp ( s, u, σ ) ∀ a ∈ N , the energy operator
T V a surp ( s, u, σ )
= log (cid:80)
Na =1 exp ( V a surp ( s, u, σ )) forms a contraction on V a surp ( s, u, σ ) .
Theorem 1 provides a suitable guarantee of T V a surp ( s, u, σ ) converging to a ﬁxed point niche.
Thecontraction result is directly based on Banach’s ﬁxed point property and suggests the generalizationof convergence in any nonempty complete metric space ( X, d ) [5].
We now consider a weighted combination of Q ( s, u ) with T V a surp ( s, u, σ ) wherein we denote β as atemperature parameter,
[Equation]
155 Remark 2.
Equation 4 is an instance of value function regularization wherein the Q values aresubject to a joint penalty while observing surprising states.
Interestingly, upon considering the Legendre transform f ∗ ( x ) [6, 14] (convex conjugate functioncorresponding to the conjugate space X of a differentiable function f (z) ) of T V a surp ( s, u, σ ) , weobtain the following,
[Equation]
[Equation] [Equation]
[Equation]
162 Remark 3.
The Legendre Transform of T V a surp ( s, u, σ ) given by f ∗ ( x ) =
(cid:80) x x log( x ) when utilizedas value function regularization ˆ Q = Q − f ∗ ( x ) corresponds to the minimum entropy formulation inconjugate space E π θ (cid:104) (cid:80) T t =
γ t ( r ( s t , u t ) − λ H ( x )) (cid:105) for x = ∇ z f (z) ∈ X .
Based on the above insight, minimizing entropy to express ∇ z f ( z ) in conjugate space is akin tominimizing uncertainty among all agents in the value function space Π .
Intuitively, H ( x ) denotesthe uncertainty for each agent a ∈ N in the multi-agent population which is directly related to its ability of accurately interpreting the environment.
Minimizing H ( x ) leads to an increase in theexpressiveness of value function.
This in turn, induces an expressive state visitation distributionwhich steers the agent away from sudden changes in its environment.
Note that the setting doesnot minimize entropy in value function space which would stand contrary to the maximum entropyformulation [20] (see Appendix B).
Figure 1 presents an illustration of the intuition behind surprise minimization using the energy-based scheme.
Agentscollaborate in partially-observed worlds to attain a joint niche.
Interpreting the space of all surprising states as an energylandscape, MARL agents move from high energy states tolow energy states which consist of minimum surprise.
Duringtraining, agents train to ﬁnd policies which not only providerewarding actions, but also avoid risky states by minimizing
T V a surp ( s, u, σ ) .
Seeking these states leads to ﬁnding the minima on the energy landscape.
Thus, it is by virtue of regularizedvalue estimates ˆ Q that the minimization scheme informs
4.2 Surprise
Minimization with Function Approximation
We utilize the above insights as surprise-based regularizationin the TD learning setting.
Upon replacing Q ( u, s ; θ ) withˆ Q ( u, s, ; θ ) in the RL construction of Equation 1 one obtainsthe following,
[Equation]
where ˆ y is given by the following expression,
[Equation]
Collecting the log terms yields the following,
[Equation] [Equation]
Here, E is deﬁned as the surprise ratio .
The surprise value function V a
[Equation]
u
[Equation]
, σ
[Equation]
) is expressed asthe negative free energy and (cid:80) Na =1 exp ( V a surp ( s, u, σ )) as the partition function of a conventional
EBM described in Equation 2.
Alternatively, V a surp ( s, u, σ ) can be formulated as the negative freeenergy with (cid:80) Na =1 exp ( V a surp ( s (cid:48) , u (cid:48) , σ (cid:48) )) as the partition function.
The TD objective incorporatesthe minimization of surprise across all agents as minimizing the energy in rapidly changing states.
Remark 4.
The above formulation of βE can be realized as intrinsic motivation steering the agenttowards subgoals with reduced surprise.
The energy formulation E provides a tractable distribution over all surprising conﬁgurations in thestate space S .
This guarantees convergence to minimum surprise at optimal policy π ∗ and is formallyexpressed in Theorem 2 (see Appendix C for a detailed convergence analysis).
Theorem 2.
Upon agent’s convergence to an optimal policy π ∗ , total energy of π ∗ , expressed by E ∗ will reach a thermal equilibrium consisting of minimum surprise among consecutive states s and s (cid:48) .
Theorem 2 demonstrates an intuitive convergence result of agent populations collaborating to residein a mutual ecological niche [12].
The multi-agent population with minimum surprise exhibits theoptimal policy π ∗ which results in minimum energy corresponding to each surprising state in thestate distribution S .
Orthogonally, agents may continue to experience ﬁnite and constant surprise inthe long-horizon while acting optimally to visit non-surprising and rewarding states.
This presentssurprise minimization as a secondary surrogate objective in MARL.
4.3 Energy-based MIXer (EMIX)
[Figure]
Based on our theoretical analysis, we incorporate learning of surprise as global intrinsic motivationacross all agents in the multi-agent system.
A global estimate of surprise, following the energyoperator T V a surp ( s, u, σ ) , is beﬁtting from a computational perspective as well.
An individualestimate of surprise for each agent may be intractable to obtain due to the non-stationarity of theenvironment.
Instead, we seek to minimize surprise jointly across all agents using an expressive
Energy-based MIXer (EMIX) architecture which is compatible with any multi-agent RL algorithm.
Figure 2 illustrates our learning scheme.
Learning of surprise in the high-dimensional value function space is cumbersome with the numberof actions scaling linearly in the number of agents.
This imposes an inherent restriction to learnglobal surprise efﬁcaciously across all agents at a given timestep.
Towards this goal, EMIX encodesindividual value functions Q 1 , Q 2 , ...
Q n corresponding to each agent using local value encoders.
These encoders capture the local change in value functions arising over subsequent TD learningiterations [60].
A global state encoder maps environment states s 1 , s 2 , ...
s T to a low dimensionalrepresentation.
Further, a state deviation encoder encodes deviations across all states s 1 , s 2 , ...
s Twithin the given batch.
Akin to a model-based method [23], the state deviation encoder accounts foruncertainty in an agent’s state visitation distribution.
Note that the encoder does not construct anexplicit model of states, but only represents their variation in the agent’s environment.
This insight isessential to account for abrupt dynamics encountered by agents.
Representations obtained from stateand value function encoders are concatenated and compressed using a ﬁnal surprise encoder whichestimates a distribution of surprise values.
The distribution implicitly represents the density of stateswherein an agent may encounter most surprise.
A value estimate V a surp ( s, u, σ ) sampled from thesurprise distribution depicts the variational free energy conﬁguration upon application of T whichserves as global intrinsic motivation.
Practical training of EMIX proceeds with backpropagation [46]using gradient descent and the reparameterization trick [25] for sampling of V a surp ( s, u, σ ) .
4.4 Practical Implementation
Algorithm 1 presents the EMIX framework (in green ) combined with QMIX [44], an off-the-shelf
MARL algorithm.
The total Q -value Q θtot is computed by the mixer network with its inputs asthe Q -values of all the agents conditioned on s via the hypernetworks.
Similarly, the target mixersapproximate Q θ − i conditioned on s (cid:48) .
In order to evaluate surprise within agents, we compute thestandard deviations σ and σ (cid:48) across all observations z and z (cid:48) for each agent using s and s (cid:48) respectively.
The surprise value function, called the Surprise-Mixer, estimates surprise V a surp ( s, u, σ ) conditionedon s , u and σ .
The same computation is repeated using the Target-Surprise-Mixer for estimatingsurprise V a surp ( s (cid:48) , u (cid:48) , σ (cid:48) ) within next-states in the batch.
Application of the energy operator along thenon-singleton agent dimension for V a surp ( s, u, σ ) and V a surp ( s (cid:48) , u (cid:48) , σ (cid:48) ) yields the energy ratio E whichis used in Equation 8 to evaluate L ( θ ) .
We then use batch gradient descent to update parameters ofthe mixer θ .
Target parameters θ − i are updated every update − interval steps.
( Q 1 , Q 2 ..., Q N , s (cid:48) ), ∀ i = 1 , 2 .., m 10:
Calculate σ and σ (cid:48) using s and s (cid:48) 11:
V a surp ( s, u, σ ) ←− Surprise-Mixer( s, u, σ ) 12:
V a surp ( s (cid:48) , u (cid:48) , σ (cid:48) )
←− Target-Mixer( s (cid:48) , u (cid:48) , σ (cid:48) ) 13:
E ←− log (cid:18) (cid:80) Na =1 exp( V a surp ( s (cid:48) ,u (cid:48) ,σ (cid:48) ))
(cid:80)
Na =1 exp( V a surp ( s,u,σ ))
(cid:19) 14:
Calculate L ( θ ) using E in Equation15:
θ ←− θ − α ∇ θ L ( θ ) 16: end for 17: end if 18: if update-interval steps have passed then 19: θ − i ←− θ, ∀ i = 1 , 2 .., m 20: end if 21: end for
Experiments
Our experiments aim to evaluate the theoretical claims presented by EMIX along with its performanceto prior MARL methods.
Speciﬁcally, we aim to answer the following questions;
(1) How does the provision of an EBM for surprise minimization compare to current MARL methods?(2)
Does the algorithm validate the theoretical claims corresponding to its components?
5.1 Energy-based Surprise Minimization
We assess the validity of EMIX, when combined with QMIX, on multi-agent StarCraft II micromanagement scenarios [48] as these consist of a larger number of agents with different action spaces.
This in turn motivates a greater deal of coordination.
Additionally, micromanagement scenariosin StarCraft II consist of multiple opponents which introduce a greater degree of surprise withinconsecutive states.
We compare our method to prior methods namely; (1)
[Table]
[53], consisting of linear additive factorization of Q function; (3) Counterfactual Multi-Agent Policy
Gradients (COMA) [11], which consist of counterfactual actor-critic updates in a centralized critic;and (4) Independent Q Learning (IQL) [54], wherein each agent acts independent of other agents.
In order to compare our surprise minimization scheme against pre-existing mechanisms, we compare
EMIX additionally to a model-free implementation of SMiRL [3] in QMIX.
We use the generalizedversion of SMiRL as it demonstrates reduced variance across batches [9].
This implementation isdenoted as SMiRL-QMIX for comparisons.
Details related to the implementation of EMIX arepresented in Appendix D.
Table 5 presents the comparison of success rate percentages between EMIX and prior MARLalgorithms on 6 StarCraft II micromanagement scenarios.
Corresponding to each scenario, algorithmsdemonstrating higher success rate values in comparison to other methods have their entries highlightedin bold (see Appendix E.2 for a statistical analysis).
Out of the 6 scenarios considered, EMIX presentshigher success rates on 5 of these scenarios depicting the suitability of the proposed approach.
Incases of so_many_baneling and 5m_vs _6m having large number of opponents and a greater levelof surprise, EMIX aptly improves over prior methods.
When compared to QMIX, EMIX depictsimproved success rates on all of the 6 scenarios.
On comparing EMIX with SMiRL-QMIX, EMIXdemonstrates higher average success rates indicating surprise robust policies.
5.2 Ablation Study
We now present the ablation study for the various components of EMIX.
Our experiments aim todetermine the effectiveness of the energy-based surprise minimization method.
Additionally, wealso aim to evaluate the utility of dual approximators for surprise estimation in accordance with theprecept from RL literature [21, 13, 20].
EMIX Objective: To weigh the effectiveness ofenergy-based scheme, we ablate the energy operator
T and only utilize V a surp .
Since this implementationemploys dual approximators V a surp , ( i ) i ∈ { 1 , 2 } forstability, we call this implementation as TwinQMIX.
Thus, we compare between QMIX, TwinQMIX and
EMIX to assess the contributions of each of the proposed methods.
Figure 3 presents the comparison of average successrates for QMIX, TwinQMIX and EMIX on 3 differentscenarios.
In comparison to QMIX, TwinQMIX addsstability to the original objective by incorporating surprising estimates.
On comparing TwinQMIX to
EMIX we note that dual approximators play little role in improving convergence.
Thus, the energybased surprise minimization scheme is the main facet for signiﬁcant performance improvement.
This is demonstrated in the 5m_vs_6m scenario wherein the EMIX implementation improves theperformance of TwinQMIX in comparison to QMIX by utilizing a surprise-robust policy.
In the caseof so_many _baneling scenario which consists of a large number of opponents (27 banelings), EMIXtackles surprise effectively by preventing a signiﬁcant drop in performance which is observed in casesof QMIX and TwinQMIX.
Surprise Minimization with Temperature: The importance of β can be validated by assessing its usagein surprise minimization.
We observe the variation of
E as it is a collection of surprise-based sample estimates across the batch.
Additionally, E consists ofprior samples V a surp ( s, u, σ ) for V a surp ( s (cid:48) , u (cid:48) , σ (cid:48) ) which makes inference tractable.
Figure 4 presents the variation of Energy ratio E withthe temperature parameter β during learning.
We compare two stable variations of E at β = 0 .
001 and β = 0 .
The objective minimizes E over thecourse of learning and attains thermal equilibrium with minimum energy.
Intuitively, equilibriumcorresponds to convergence to optimal policy π ∗ which validates the claim in Theorem 2.
Withβ = 0 .
01 , EMIX presents improved convergence and surprise minimization for 5 out of the 6considered scenarios, hence validating the suitable choice of β .
The choice of β is further validatedin Figure 5 wherein β = 0 .
01 provides consistent stable improvements over other values.
Lowervalues of β , such as β = 0 .
001 , do little to minimize surprise or improve performance.
5.3 Predator-Prey Benchmark
We extend our comparison of EMIX on the Predator-Prey (particle world) tasks.
In addition to thedifﬁculty of task, we vary the number of opponents.
This helps quantify the variation in performanceagainst increasing level of surprise under ﬁxed dynamics.
Table 2 presents average returns.
Whileall agents present comparable performance on the easier tasks, EMIX improves over QMIX and
TwinQMIX on the more challenging punish and hard tasks.
In the case of punish , EMIX is the onlymethod to achieve greater than 20 returns.
Additional results can be found in Appendix E.4.
[Table]
Discussion
Conclusion:
In this paper, we presented an energy-based perspective towards surprise minimizationin multi-agent RL.
Towards this goal we introduce EMIX
[Table]
an energy-based intrinsic motivationframework for surprise minimization in MARL algorithms.
EMIX utilizes a temporal EBM toestimate and minimize surprise jointly across all agents.
Our theoretical claims on the formulation ofminimization of temporal energy with surprise are corroborated upon utilizing EMIX on a suite ofchallenging MARL tasks requiring signiﬁcant collaboration under fast-paced dynamics.
Future Work: While EMIX serves as a practical example of EBMs in cooperative MARL
[Table]
it presentsseveral new avenues for future work.
We shed light on 2 such aspects,
(1) Provision of an energy-based model naturally raises the question of how can we efﬁciently
samplefrom the surprise distribution?
Advances in sampling methods depict promise towards this aspect.(2)
Although suitable for lower dimensions, the scalability of EBMs towards high dimensionalaction spaces remains an open question.
We conjecture that the utility of density-based methods andgenerative models can address the scalability gap.
These directions are left for future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
A Proofs
Theorem 1.
Given a surprise value function V a surp ( s, u, σ ) ∀ a ∈ N , the energy operator
T V a surp ( s, u, σ )
= log (cid:80)
Na =1 exp ( V a surp ( s, u, σ )) forms a contraction on V a surp ( s, u, σ ) .
Proof.
We follow the process of [2].
Let us ﬁrst deﬁne a norm on surprise values
|| V 1 − V 2 || ≡max s,u,σ | V 1 ( s, u, σ )
− V 2 ( s, u, σ ) | .
Suppose (cid:15) =
||
V 1 − V 2 || ,log
N (cid:88) a = exp ( V 1 ( s, u, σ )) ≤ log
N (cid:88) a = exp ( V 2 ( s, u, σ ) + (cid:15) )
= log
N (cid:88) a = exp ( V 1 ( s, u, σ )) ≤ log exp ( (cid:15) )
N (cid:88) a = exp ( V 2 ( s, u, σ )) = log
N (cid:88) a = exp ( V 1 ( s, u, σ )) ≤ (cid:15) + log
N (cid:88) a = exp ( V 2 ( s, u, σ )) = log
N (cid:88) a = exp ( V 1 ( s, u, σ ))
− log
N (cid:88) a = exp ( V 2 ( s, u, σ )) ≤ || V 1 − V
||
Similarly, using (cid:15) with log (cid:80) Na =1 exp ( V 1 ( s, u, σ )) ,log
N (cid:88) a = exp ( V 1 ( s, u, σ ) + (cid:15) ) ≥ log
N (cid:88) a =
[Equation]
Results in Equation 9 and Equation 10 prove that the energy operation is a contraction.
Theorem 2.
Upon agent’s convergence to an optimal policy π ∗ , total energy of π ∗ , expressed by E ∗will reach a thermal equilibrium consisting of minimum surprise among consecutive states s and s (cid:48) .
Proof.
We begin by initializing a set of M policies { π 1 , π 2 ..., π M } having energy ratios{ E 1 , E 2 ..., E M } .
Consider a policy π 1 with surprise value function V 1 .
E 1 can then be expressed as
[Equation]
Invoking Assumption 2 for s and s (cid:48) , we can express V a 1 ( s (cid:48) , u (cid:48) , σ (cid:48) )
=
V a 1 ( s, u, σ ) + ζ 1 where ζ 1 is aconstant.
Using this expression in E 1 we get,
[Equation]
Similarly, E 2 = ζ 2 , E 3 = ζ 3 ..., E M = ζ M .
Thus, the energy residing in policy π is proportional to thesurprise between consecutive states s and s (cid:48) .
Clearly, an optimal policy π ∗ is the one with minimumsurprise.
Mathematically,
[Equation]
Thus, proving that the optimal policy consists of minimum surprise at thermal equilibrium.
B Relation to Maximum Entropy Framework
B.1 Similarities & Differences
We conceptually compare EMIX to the maximum entropy framework.
Similarities: Both methods utilize an auxilary objective as intrinsic motivation to tackle uncertainty.
While the maximum entropy formulation assigns low energy to uncertain actions, our method assignslow energy to uncertain encoded representations od states (as presented in Fig. 2).
Differences: Our method differs from maximum entropy in its optimization process and learningscheme.
The maximum entropy formulation aims to maximize entropy in the value function space soas to motivate exploration.
Our proposed scheme, on the other hand, aims to minimize surprise in thelow-dimensional representation space to obtain dynamics-aware robust policies.
B.2 Connection to Soft Q-Learning
The Soft Q-Learning objective with V θ − soft ( s (cid:48) ) and Q soft ( u, s ; θ ) as state and action value functionsrespectively is given by-
[Equation]
The gradient of this objective can be expressed as-
[Equation]
[Equation]
And the gradient of the EMIX objective is obtained as-
[Equation] [Equation]
[Table]
Comparing Equation 11 to Equation 12 we notice that Soft Q-Learning and EMIX are related toeach other as they utilize EBMs.
Soft Q-Learning makes use of a discounted energy functionwhich downweights the energy values over longer horizons.
Actions consisting of lower energyconﬁgurations are given preference by making use of Q soft ( u, s ; θ ) as the negative energy.
Onthe other hand, EMIX makes use of a constant energy function weighed by β which minimizessurprise-based energy between consecutive states.
Both the objectives can be thought of as energyminimizing models which search for an optimal energy conﬁguration.
Soft Q-Learning searches foran optimal conﬁguration in the action space whereas EMIX favours optimal behavior on spuriousstates.
In fact, EMIX can be realized as a special case of Soft Q-Learning if the mixer agent utilizesan energy-based policy and attains thermal equilibrium.
This leads us to express Theorem 3.
Theorem 3.
Given an energy-based policy π with its target function V ( s (cid:48) )
=
log (cid:80) u ∈
A exp Q ( u (cid:48) , s (cid:48) ; θ − ) , the surprise minimization objective L ( θ ) reduces to the Soft QLearning objective L ( θ soft ) in the special case surprise absent between consecutive states, 563 (cid:80)
Na =1 exp ( V a surp ( s (cid:48) , u (cid:48) , σ (cid:48) ))
= (cid:80) Na =1 exp ( V a surp ( s, u, σ )) .
Proof.
We know that the EMIX objective is given by-
[Equation]
Replacing the greedy policy term max u (cid:48)
Q
[Equation]
u
[Equation]
s
[Equation]
θ − ) with the energy-based value function V ( s (cid:48) )
=log (cid:80) u (cid:48)
∈
A exp Q ( u (cid:48) , s (cid:48) ; θ − ) , we get,
[Equation]
=
[Equation]
(cid:80)
[Equation]
(cid:33)
[Equation]
(cid:33)
[Equation]

[Equation]
14)= L ( θ ) =
[Equation]
u
[Equation]
∼ R
[Equation]
(cid:32)
[Equation]
At thermal equilibrium, (cid:80) Na =1 exp ( V a surp ( s, u, σ ))
= (cid:80) Na =1 exp ( V a surp ( s (cid:48) , u (cid:48) , σ (cid:48) )) ,=
L ( θ ) =
E s
[Equation]
u
[Equation]
s (cid:48) ∼ R (cid:34) (cid:32)
r
[Equation]
+
[Equation]
(cid:18)(cid:80)
[Equation]
(cid:80)
[Equation]
V a
[Equation] [Equation]
[Equation]
Equation 16 represents the Soft Q-Learning objective, hence proving the result.
C Convergence Analysis
We now analyze convergence of the surprise minimization scheme during policy optimization.
Ournotation denotes B V k − 1 = r + γV k − 1 as the Bellman operator which obeys monotonicity andcontraction.
Monotonicity: V 1 ≤ V 2 = ⇒ B V 1 ≤ B V 2 ; Contraction: (cid:107)B V 1 − B V 2 (cid:107) 2 ≤ γ (cid:107) V 1 − V 2 (cid:107)
We now denote ˆ V k
= r k + γ ˆ V k − 1 + β log (cid:80) Na =1 exp
( V a surp , ( k ) ( s, u, σ )) as the total value at step k .
As per the deﬁnition of B , this gives us
[Equation]
Consider
[Equation]
with V ∗ being the optimal value at convergence,
[Equation]
Where β log (cid:80) Na =1 exp( V a surp , ( k ) ) > 0 as per constant positive surprise ζ > 0 in 2.
We impose thisconstraint by adding ReLU nonlinearities in surprise encoder to obtain positive V a surp , ( k ) values. [Equation] [Equation] [Equation]
[Equation]
[Equation]
( V
[Equation]
k ) )
(cid:35)(cid:33) − V
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) (20)
Thus, for k iterations, we have,
[Equation]
V
[Equation]
( i )
[Equation]
(cid:35)(cid:33)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
( i ) )
(cid:35)(cid:33)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation] ( i ) )
(cid:35)(cid:33) [Equation] (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
We now absorb the sum of surprise values from time index i = 1 , .., k in a single variable V a tot .
Thus,using V a tot = (cid:80) ki =1 V a surp , ( i ) and utilizing the Triangle Inequality, we get,
[Equation]
We now bound the two terms separately.
Considering the ﬁrst term and following the results of valueiteration convergence [5],
[Equation]
[Equation]
[Equation]
wherein V µ denotes an approximation to V .
Utilizing the triangle inequality yields,
[Equation]
The two terms are bounded using the convergence result of [4].
[Equation]
Now, considering the second term in Equation 24 and denoting V ∗ tot = (cid:80) ki
=1 V ∗ surp , ( i ) as the sum ofoptimal surprise values,
[Equation]
using the triangle inequality,
[Equation]
Since T = log (cid:80) Na =1 exp( V a tot ) is a contraction following Theorem 1, for the ﬁrst term we have,
[Equation]
The second term in the above relation is bounded due to the completeness assumption,
[Equation]
The ﬁrst term, on the other hand, is simpliﬁed by applying
Jensen’s Inequality on the deﬁnitions of V a tot and V ∗ tot ,
[Equation]
[Equation]
Denoting (cid:13)(cid:13)(cid:13)(cid:13) V a surp , ( i ) −
V
[Equation]
i
[Equation]
= RMSE
[Equation]
V a surp
[Equation]
( i ) ) , we obtain the following result,
[Equation]
Finally, combining Equation 28 and Equation 32 in Equation 24, we obtain the desired convergencebound.
[Equation]
While the ﬁrst term in Equation 33 denotes the convergence of policy optimization, the secondterm indicates the bounded convergence of surprise to ecological niches with ﬁnite (yet nonzero)surprising elements.
The policy optimization process converges at a geometric rate O ( γ k ) towards itsstable ﬁxed points.
The surprise minimization process, on the other hand, demonstrates an annealingbehavior which depends on the temperature parameter β .
Furthermore, convergence to stable ﬁxedpoint V a tot is bounded in respect to each agents individual surprise values V a tot .
This insight indicatesthat different agents converge towards different locally optimal values of surprise.
Finally, thepresence of constant ζ corroborates prior claims [50, 12] that agents continue to experience surpriseirrespective of their convergence to minimum energy niches.
To further develop intuition for thisclaim, consider the special case wherein (cid:80)
ki =1 RMSE ( V a surp , ( i ) )
→ 0 , i.e.- surprise estimation errorfor all iterations goes to zero.
Irrespective of global convergence among all agents, a ﬁnite yet smallζ continues to contribute to the upper bound of (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
− V ∗ (cid:13)(cid:13)(cid:13)(cid:13) .
Role of β : We further discuss the role of β which is of balancing the terms at successive iterations.
While the ﬁrst term geometrically decays with O ( γ k ) rate, the second term approaches a ﬁniteconstant βζ as V atot → V ∗ tot .
Irrespective of our choice of β , the LHS (cid:107) V k − V ∗ (cid:107) 2 is upper boundedby a constant which validates the claims of minimum yet ﬁnite surprise values.
We do note that asmall β is still desirable to remove any approximation errors in order to push V k
→ V ∗ .
However,this comes at the cost of increased surprise if β is not selected appropriately.
D Implementation DetailsD.1 Model Speciﬁcations
Architecture:
This section highlights model architecture for the surprise value function.
At thelower level, the architecture consists of 3 independent networks called state_net , q_net and surp_net .
Each of these networks consist of a single layer of 256 units with ReLU non-linearity as activations.
Similar to the mixer-network, we use the ReLU non-linearity in order to provide monotonicityconstraints across agents.
Using a modular architecture in combination with independent networksleads to a richer extraction of joint latent transition space.
Outputs from each of the networks areconcatenated and are provided as input to the main_net consisting of 256 units with ReLU activations.
The main_net yields a single output as the surprise value V a surp ( s, u, σ ) which is reduced along theagent dimension by the energy operator.
Alternatively, deeper versions of networks can be used inorder to make the extracted embeddings increasingly expressive.
However, increasing the number oflayers does little in comparison to additional computational expense.
Computation of σ : The deviation σ corresponds to the standard deviation across each dimension ofthe state s .
Considering the state as a tensor of size B ×
A × M with B as the batch size, A as thenumber of agents and M as the observation dimension, we compute σ by calculating the standarddeviation across the M dimension.
This yields σ as a B × A × 1 dimensional array.
Computation of surprise estimates: V surp denotes the surprise value function which quantiﬁes theamount of surprise experienced by agents.
Analogous to a Q value function which provides estimatesof returns, V surp provides estimate of surprise.
Our framework learns V surp much like any othervalue function (using a neural network), but by additionally undergoing a log (cid:80) exp transformationto obey the ﬁxed point property.
This is achieved by realizing log-sum-exp as an energy operator
T = log (cid:80) exp which can be computed using standard computation libraries.
Since our code isimplemented in PyTorch, we implement this as T_V = torch.logsumexp(V_surp, dim=1) .
Global State Encoder:
The global state encoder serves as a mapping from the state space to a lowdimensional representation space S → Z .
The encoder takes in a sequence of states { s 1 , s 2 , ..., s T }as input and outputs a latent representation z state .
We use a standard pyramid MLP network consistingof 2 hidden layers of 256 units each with ReLU non-linearity.
Embeddings obtained from the encoderare concatenated with other latent embeddings before being passed to the ﬁnal surprise encoder.
Standard Deviation Encoder:
The standard deviation encoder serves as a mapping from standarddeviations across state dimensions to a low dimensional representation space.
Each standard deviationσ is computed across dimensions of the state s t .
These deviations are then packed in a sequence{ σ 1 , σ 2 , ..., σ T } and passed as inputs to the standard deviation encoder.
Intuitively, the encoder learnschanges across states in a batch of observations.
This is similar to a dynamics model predicting futurestates, except that we map these states to a low dimensional embedding.
We use a standard pyramid
MLP network consisting of 2 hidden layers of 256 units each with ReLU non-linearity.
Embeddingsobtained from the encoder are concatenated with other latent representations and used by the ﬁnalsurprise encoder to estimate the surprise distribution.
D.2 Hyperparameters
Table 3 presents hyperparameter values for EMIX.
A total of 2 target Q -functions were used as themodel is found to be robust to any greater values.
[Table] [Table]
[Table] [Table]
D.3 Selection & Tuning of β
One can manually tune β using a ﬁne-grained hyperparameter search.
We tune β between 0.001 and1 in intervals of 0.01 with best performance observed at β = 0 .
However, we ﬁnd two additionalmethods helpful for obtaining more accurate values.
These are described as followsArmijo’s Line Search: One can borrow from optimization theory and utilize Armijo’s line search[41] by setting a termination condition.
The method starts with a constant value of β which isiteratively incremented/decremented until a termination criterion (example- (cid:107)∇ L ( θ ) (cid:107) < (cid:15) with (cid:15) aconstant) is reached.
While line search is proven to converge towards globally optimal values,
its
O ( n 2 ) convergence may be computationally expensive that too in the MARL setting.
Thus, we turnto the more efﬁcient automatic tuning.
Automatic Tuning: We choose to automatically tune β following single-agent RL literature [20, 28].
This is achieved by treating β as a parameter and adaptively optimizing over it using Adam.
We treata surprise value of 0 as our target value.
The method works well in practice and provides β valuescloser to 0 .
01 (our manual selection).
E Additional Results
[Figure] [Figure]
We visualize and compare behaviors learned by surprise minimizing agents to the prior method of
QMIX.
Fig.
6 presents the comparison of EMIX and QMIX agent trajectories (in yellow arrows) onthe challenging so_many_baneling task.
The task consists of 27 baneling opponents which rapidlyattack the agent team on a bridge.
QMIX agents naively move to the central alley of the bridge andstart attacking enemies early on.
While QMIX agents naively maximize returns, EMIX agents learn adifferent strategy.
EMIX agents rearrange themselves ﬁrst at the corners of the bridge.
Note that thesecorners provide cover from enemy’s ﬁre.
Thus, EMIX agents learn to take cover before approachingthe enemy head-on.
This indicates that the surprise-robust policy is aware of the incoming fast-pacedassault.
As another example, Fig.
7 presents behaviors on the 2s_vs_1sc task wherein two agents mustcollaborate together to defeat a SpineCrawler enemy.
The enemy, having a long tentacle pointing tothe front, chooses to attack any one of the agents randomly in front of it.
Additionally, the tentaclehas a ﬁxed length and cannot extend beyond this range.
Random intermittent attacks indicate thatthe agents face a greater degree of surprise with no prior knowledge of the enemy’s movement.
Weobserve that QMIX agents take turns to attack the enemy by moving back and forth to minimizedamage.
EMIX agents, on the other hand, learn a different strategy.
One of the EMIX agents stands ata distance to attack th enemy while the other agent goes around to attack from behind.
This indicatesthat the policy is aware of enemy’s limited movement.
E.2 Statistical Signiﬁcance
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
We follow the recommendation of [33] and evaluate the statistical signiﬁcance of our results bycarrying out the Mann-Whitney U test [38].
All 5 seeds of an algorithm (on each task) are comparedto that of QMIX to yield the U statistic.
U here denotes the statistical signiﬁcance of performancewith higher values being desirable.
Table 4 presents the comparison of U statistic on the StartCraft II benchmark.
EMIX demonstratesconsistently high values of U across a diverse set of tasks when compared to SMiRL and prior MARLagents.
This highlights the consistent surprise-minimizing performance of EMIX across randomseeds.
E.3 StarCraft II Benchmark
[Table]
E.4 Predator-Prey Benchmark
[Figure]
We consider a simple toy task from the Predator-Prey benchmark todemonstrate the importance of surprise minimization.
We select predator_prey_easy due to its simplicity and convenient dynamics.
The taskconsists of 3 agents and 3 opponents.
We increase the number of opponentswhile keeping the task ﬁxed.
This way the dynamics of the MDP remainunchanged and the only changing factor is opponent behaviors.
Fig.
8 presents the variation of average returns for EMIX and QMIXover 5 random seeds.
While QMIX agents undergo a steady decrease inperformance, EMIX agents are found robust to this fast degradation.
Evenafter the addition of 20 opponents (against only 3 agents), EMIX is ableto retain positive returns.
The algorithm acquires a surprise robust-policyearly on during training to tackle fast-paced changes introduced by the largenumber of agents.
E.5 Note on Minimum Entropy Conjugate Objective
The minimum conjugate entropy objective denotes the dual problem to surprise minimization.
If we compute the Legendre Transform of our energy-based operator T V a surp ( s, u, σ ) =log (cid:80) Na =1 exp( V a surp ( s, u, σ ))
we obtain the entropy function H ( x ) where x is the gradient of the operator, x = T V a surp ( s, u, σ ) .
This insight indicates that minimizing the energy operator T V a surp ( s, u, σ )is same as minimizing entropy in the space of gradients.
Intuitively, our objective aims to minimizeuncertainty in the learning signal.
F Additional Related Work on Multi-Agent Value Factorization
We discuss recent MARL methods within the Centralised Training and Decentralised Controlparadigm [26] which improve value factorization.
The original work of QTRAN [51] improvesrepresentational capacity of factorization schemes by generalizing methods such as QMIX [44] and
VDN [53].
More recent advances combine techniques from dueling networks and temporal abstraction to learn MARL agent factorizations with sufﬁcient representations [59].
A notable work is thatof [60] which employs pretrained action representations to learn agent-speciﬁc roles.
Decomposingpolicy optimization into role selection and role execution stages allows larger number of MARLagents to collaborate well even in unseen scenarios.
Alternate works consider information theoreticobjectives to introduce diversity in optimization and representation of shared multi-agent parameters[10].
Lastly, [61] extend these ideas by decomposing value learning in multi-agent actor-criticmethods.
Within the off-policy setting, these agents highlight sufﬁcient representational capacity inboth discrete and continuous action spaces.