Inﬂuencing Towards Stable Multi-Agent Interactions
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Learning in multi-agent environments is difﬁcult due to the nonstationarity introduced by an opponent’s or partner’s changing behaviors.
Insteadof reactively adapting to the other agent’s (opponent or partner) behavior, we propose an algorithm to proactively inﬂuence the other agent’s strategy to stabilize –which can restrain the non-stationarity caused by the other agent.
We learn a lowdimensional latent representation of the other agent’s strategy and the dynamicsof how the latent strategy evolves with respect to our robot’s behavior.
With thislearned dynamics model, we can deﬁne an unsupervised stability reward to trainour robot to deliberately inﬂuence the other agent to stabilize towards a singlestrategy.
We demonstrate the effectiveness of stabilizing in improving efﬁciencyof maximizing the task reward in a variety of simulated environments, including autonomous driving, emergent communication, and robotic manipulation. Weshow qualitative results on our website.
Introduction
Reinforcement learning has shown impressive performance in recent years [1, 2], but much of thefocus has been on single-agent environments.
Many real-world scenarios, on the other hand, involvecoordinating with multiple agents whose behavior may change over time, making the task nonstationary.
For example, when playing volleyball, our teammate may play more aggressively overtime if they are hitting the ball well, so we may need to step aside and set the ball for them moreoften.
Many techniques in multi-agent reinforcement learning (MARL) have been proposed fordealing with such non-stationarity in the other agent’s strategies [3].
One set of approaches reactively adapts to the other agent’s change in strategy, for example bysimply forgetting past experiences and training on more recent data [4, 5, 6, 7, 8].
Other works try tolearn a model of the other agent and take into account how the robot’s actions affect the other agent’sstrategies over time [9, 10, 11, 12, 13].
With knowledge of the other agent’s strategy dynamics, wecan take actions to inﬂuence the other agent in a way that maximizes our long-term reward [10, 13].
Although inﬂuencing other agents to maximize reward is a useful approach in principle, it can bechallenging in practice.
Indeed, if we have access to a faithful model of the other agent’s strategydynamics, then we should choose actions that carefully nudge their high-level strategy to maximizeour long-term reward.
However, learning and leveraging such a model of the other agent’s strategydynamics can be impractical.
For one, even just estimating the other agent’s current strategy isalready difﬁcult with limited data, let alone learning the dynamics between strategies.
On top ofthat, the robot may have to ﬁgure out a best response for a large number of opponent strategies,making the full learning process extremely sample inefﬁcient.
Instead, we should take into account the difﬁculty of learning for our ego agent.
1 That is, we wouldstill like to inﬂuence the other agent in a useful way, but in a manner that minimizes the burden ofthe ego agent’s learning algorithm.
Our key insight is that we can stabilize the other agent’s strategy a multi-agent environment to reduce non-stationarity and facilitate learning a best response to the other agent.
Our approach is a speciﬁc form of inﬂuencing – learning how to stabilize – thataims to reduce the non-stationarity of the other agent, and in turn makes learning easier for the egoagent.
Intuitively, stabilizing the other agent’s strategy allows us to avoid learning about complexhigh-level strategy dynamics, and focus our efforts on maximizing reward against a ﬁxed strategy.
As an example of stabilizing the other agent, imagine again that we are playing beach volleyballwith a teammate.
When the ball comes perfectly in between our teammate and us, it is not obviouswhich of us should dig for the ball (see Fig.
1 (left)).
In principle, it is possible to carefully modelthe dynamics of how our teammate’s strategy changes based on our actions and precisely predictwho should get the next ball that falls in between us.
However, it can take many interactions beforewe learn a good model of our opponents, and constantly switching roles between digging for the balland staying back can make learning difﬁcult.
Instead, we can stabilize our teammate by making ourintentions clear: we actively take a step away from the ball so our teammate knows to dig for the ball,as in Fig.
1 (right).
This simpliﬁes the learning process by reducing the need to learn complicatedstrategy dynamics, and making the task more stationary.
In particular, our method combines the robot’s task reward with a stabilizing reward, where stabilityis measured with respect to the other agent’s strategies learned in an unsupervised way.
The taskreward pushes the robot to do well in the environment, while the stabilizing reward aims to makethe learning process easy and sample efﬁcient.
We demonstrate the strength of our approach in7 different tasks, including autonomous driving, an emergent communication task, and simulatedrobot environments.
Our main contributions in this paper are as follows:
• We propose augmenting our robot’s reward function with a stability reward in order
to
explicitly encourage the robot to inﬂuence the other agent towards a stable strategy.•
We demonstrate that the stability reward can be deﬁned as an unsupervised reward usinglearned representations of the other agent’s strategy.•
We perform experiments in simulated environments demonstrating that stabilizing the other
agent’s strategy improves our robot’s ability to coordinate and maximize the task reward.
Related Work
Multi-Agent Reinforcement Learning.
Learning in multi-agent environments can be extremelydifﬁcult when the other agents have non-stationary behavior [14, 15, 16].
Hernandez-Leal et al.[3] provide a taxonomy of existing methods that attempt to deal with this non-stationarity issue inmulti-agent environments.
Some works seek to forget about previous experiences with old opponentstrategies and improve adaptation by only training on more recent data [4, 5, 6, 7, 8, 17].
Further,centralized training has shown promise in multi-agent environments [18, 19].
Some works designexplicit communication channels in which agents can share their policy parameters or gradient updates [20, 21].
However, these methods require that there exists some centralized method in whichagents can communicate or share parameters.
We are mainly interested in the decentralized settingwhere we train an ego agent to interact with other agents that we do not have control over.
Opponent Modeling.
Instead of ignoring the opponent’s changing strategies, prior opponent modeling works design agents that estimate their opponent’s policies or parameter updates [22, 9, 23,11, 12, 24, 25] in a potentially recursive manner. Peer-rewarding has also emerged as a method ofencouraging cooperative behavior [26, 27, 16], but requires the assumption that an agent’s rewardfunction can be externally modiﬁed.
Several works recognize that the ego agent’s policy can directlyinduce change in the opponent’s policies [13, 28, 29, 30].
In [10], the ego agent learns not only arepresentation of the opponents’ strategies, but also learns a dynamics model for how the opponent’sstrategies change conditioned on the ego agent’s behavior.
Representation Learning of Latent Strategies.
Representation learning is critical to the tractability of opponent modeling.
It may be computationally inefﬁcient to learn all opponent intentions orexplicitly learn the parameters of the opponent’s policies.
Instead, we recognize that there is oftensome underlying structure in the opponent policy space, which can be captured by a low-dimensionallatent strategy.
Prior works learn latent context variables to help distinguish between separate tasks,which could correspond to different opponents [31, 32, 33, 34, 35].
Representation learning hasbeen shown to be useful in developing a dynamics model of the opponent’s policy in multi-agentenvironments [36, 10].
In our work, we show that it can be beneﬁcial to cluster latent strategy representations into a discrete latent space [37, 38] to identify shared structure among similar strategies.
Stability and Surprise in Reinforcement Learning.
To mitigate the difﬁculty of learning in a nonstationary environment, we seek to stabilize the opponent’s strategy – or minimize the surprise in theopponent’s strategy.
Recent works in reinforcement learning seek to do the opposite of designingmaximum entropy methods to improve exploration [39, 40, 41].
Stability in the opponent’s strategycan reduce the non-stationarity in the environment and improve efﬁciency of learning a best response[42].
Surprise minimizing reinforcement learning (SMiRL) trains an agent to seek predictable andrepeatable states to counteract the environment’s entropy [43].
SMiRL considers a long horizonnotion of stability in which the state should be of high likelihood with respect to the distributionof states encountered so far.
Our work is similar in principle to SMiRL; however, we consider apairwise notion of stability between consecutive latent strategies, as we recognize that an opponentthat unpredictably switches between a few strategies is highly non-stationary, yet each individualstrategy can be high likelihood with respect to the overall distribution of previously seen strategies.
Problem Statement
We consider two-player multi-agent settings, where there is an ego agent that we have control overand another agent that we do not have direct control over.
Since we only control the ego agent, wecan consider our problem formulation in terms of a single-agent Markov decision process where theopponent policy is absorbed as part of the environment transitions and rewards.
Non-stationarity inthe opponent policy then corresponds to non-stationarity in the environment transitions and rewards.
Hidden Strategies.
The agents repeatedly interact with each other across multiple interactions, witheach interaction lasting multiple timesteps.
We assume that the opponent policy z is ﬁxed withineach interaction but can change across interactions.
We consider environments with no hiddenstates, so that if the ego agent can observe the exact opponent strategy, then we can formulate theego agent’s task as a fully observable MDP.
However, since the ego agent does not directly observethe opponent’s strategy, we consider a Hidden Parameter MDP (HiP-MDP).
Hidden Parameter Markov Decision Process.
We formulate the interactions as a variant of a
HiP-MDP, which is a tuple M = h S , A , Z , T , T z , R , H,  i [44], which includes an additional T z 120 that governs the dynamics of the opponent’s strategies.
We let z j 2 Z be the opponent strategy 121 for the j -th interaction.
At each timestep t , the ego agent is in state s t 2 S .
The ego agent takesaction a t 2 A and the environment transitions to s t +1 with probability T ( s t +1 | s t , a t , z j ) .
The ego 123 agent receives the task reward
R task ( s t , a t , z j ) .
The interaction ends after H timesteps and  isthe discount factor.
Both the transition and reward function are unknown.
During interaction j , theego agent observes the trajectory ⌧ j = { ( s 1 , a 1 , R task ( s 1 , a 1 , z j )) , . . .
( s H , a H , R task ( s H , a H , z j )) } .
Notice that the ego agent does not observe the opponent’s strategy
z j in the trajectory, which makeslearning the value of an observation and action difﬁcult – the reward of an observation and actioncan change signiﬁcantly as the opponent’s strategy changes across interactions.
We consider the setting where the opponent’s change in strategy is governed by the trajectory of theprevious interaction.
At the end of interaction j , the opponent’s strategy transitions to z j +1 based on strategy dynamics T z ( z j +1 |
z j , ⌧ j ) .
Namely, the ego agent’s behavior during interaction j candirectly inﬂuence the opponent’s strategy for interaction j + 1 .
Next, we deﬁne a notion of stabilitybetween interactions j and j + 1 as follows:
Deﬁnition 1 Let z j , z
j +1 2 Z be two opponent strategies.
The strategies z j , z
j +1 are pairwise✏ -stable with respect to a metric space ( Z , d ) if and only if d ( z j +1 , z j ) < ✏ .
Stabilizing Opponent Strategies by Inﬂuencing Across Repeated Interactions.
Over repeated interactions, the ego agent explores HiP-MDP M over a sequence of opponent strategies ( z 1 , z 2 , . . . ) .
The ego agent’s objective is to maximize the cumulative discounted sum of rewards over all interactions, shown in Eq.
The ego agent’s ability to maximize the reward within each interaction isheavily dependent on their ability to estimate the opponent strategy z j for that interaction.
Withoutknowing the opponent’s strategy, the ego agent will have signiﬁcant difﬁculty taking an action thatis a best response to the opponent’s actions.
It can be prohibitive to learn a model of all the opponent’s strategies and form a general policy for the ego agent that can react to any opponent
strategy;however, if the ego agent can stabilize the opponent strategy to a ﬁxed point, i.e., 9 z k such that
T z ( z j +1 |
z j , ⌧ j ) >
if and only if d ( z j , z j +1 ) < ✏ , 8 j 2 { k . . .
N } (where N is the total numberof interactions), then the ego agent can more easily learn a best response to the opponent strategy ina stationary environment.
With z ﬁxed across interactions, the HiP-MDP is reduced to a fully observable MDP parameterizedby z .
At a high level, we are encouraging the agent to learn how to explore the HiP-MDP in a subsetof the state space, where the hidden opponent strategy is kept stable between interactions.
Thus, theego agent is effectively learning in a fully observable MDP, given that the ﬁxed – but unknown –opponent strategy z is not changing.
In terms of the exploration-exploitation tradeoff, stabilizing theopponent strategy would reduce the ego agent’s required exploration in order to ﬁnd a best responseto the opponent’s ﬁxed strategy.
We note that it is not always easy or possible to stabilize the opponent’s strategy.
Stabilizing requiresthat the dynamics of the opponent’s strategy are deﬁned such that the ego agent can form an interaction ⌧ j where T z ( z j +1 |
z j , ⌧ j ) >
if and only if z j and z j +1 are pairwise ✏ -stable.
In general, aslong as there is a ﬁxed point z k where the optimal value function satisﬁes V ⇤ ( s 1 , z k )  V
⇤ ( s 1 , z )for any other z 2 Z , then stabilizing should not hinder the task reward in the long run.
Stable Inﬂuencing of Latent Intent (SILI)
In this section, we present our approach of inﬂuencing the latent strategy of our opponent towardsa stable strategy.
At a high level, the ego agent learns an encoder that predicts the opponent’s nextlatent strategy given the previous trajectory.
The ego agent then conditions its policy for the nextinteraction on the predicted latent strategies of the opponent.
But, since we never observe the groundtruth latent strategies, we must learn the latent strategy representations in an unsupervised way.
Todo so, we follow the setup in [10], where we jointly learn a decoder that reconstructs the task rewardsand transitions for the next interaction using the predicted latent strategy.
Using this predicted latent strategy, we can measure the stability of the opponent’s latent strategy, and optimize for a weightedcombination of the task reward and a stability reward.
Learned Latent Strategies.
First, we describe the representation learning component of ourmethod shown in Fig.
2 (left).
Recall that we assume that the latent strategy dynamics follow
T z ( z j +1 |
z j , ⌧ j ) .
We can approximate these latent strategy dynamics by learning an encoder E that maps trajectories ⌧ j  1 to a latent representation z j of dimension k (chosen as a hyperparameter).
Here, we model the encoder as f ( z j | ⌧ j  1 ) instead of f ( z j | z j  1 , ⌧ j  1 ) , as in [10].
Thissimpliﬁcation, which often works well in practice, assumes that the previous trajectory ⌧ j alreadyhas enough information to reconstruct z j +1 .
Although we do not observe the ground truth latent strategies, we can still learn them in an unsupervised way, since we know that they should be predictive of the task reward and transitions.
Thus, we jointly train a decoder that predicts the task reward and transitions from the encoded latentrepresentation.
In particular, for interaction j and timestep t , the decoder D takes as input statesand actions s jt , a jt as well as the latent strategy embedding z j , and reconstructs the next state s jt +1and task reward r jt .
To train the encoder and decoder, we use the following maximum-likelihoodobjective: max  ,
[Equation]
Reinforcement Learning with Stable Latent Strategies.
As the opponent’s strategy changes,the ego agent’s policy should change in response.
Naturally, we should condition the ego agent’spolicy on the opponent’s latent strategies.
When considering the goal of keeping the opponent’sstrategy pairwise stable, it is important that the ego agent conditions their policy on at least the mostrecent two predicted latent strategies since the stability reward depends on the distance between theconsecutive latent strategies, as shown in Fig.
2 (right).
Concretely, the ego agent’s policy is deﬁnedas ⇡ ✓ ( a t | s t , z j , z j  1 ) , where z j = E  ( ⌧ j  1 ) .
We have deﬁned the environment reward to be R task .
We then deﬁne the stability reward to be thedistance between consecutive predicted opponent strategies, where d can be any generic distancemetric over the latent space Z as follows:
R stable =
d ( z j , z j  1 ) .
Here, we abuse notation and use z j to refer to the prediction of the opponent’s strategy since wenever observe the ground truth.
Finally, our new reward function is deﬁned as a weighted averageof these objectives, parameterized by stability weight  : R total =
(1   ) · R task +  · R stable .
With 197  = 1 , the ego agent’s entire objective is to stabilize the opponent strategy, and with  = 0 , the egoagent greedily maximizes the environment reward.
Optimizing Rewards Across Interactions.
Our ego agent seeks to inﬂuence the opponent towardsdesirable latent strategies.
With the encoder E  , the ego agent can deliberately take actions in trajectory ⌧ j in order to inﬂuence the opponent to a stable z j +1 .
To learn how to inﬂuence across repeated interactions, we train our ego agent’s policy ⇡ ✓ to maximize rewards across the repeated interactions
[Equation]
[Equation]
where ⇢ j ⇡ ✓ is the trajectory distribution induced by ⇡ ✓ under the transition function
T ( s t +1 | s t , a t , z j ) .
With the total reward including the stability reward, the ego agent learns totake interactions such that the opponent follows a stable latent strategy so the ego agent can easilylearn a best response to in turn maximize the task reward.
Modeling Latent Strategies.
We examine three methods to model and measure distances betweenthe changing latent strategies: continuous latent variables, discrete latent variables, and partial observations of the opponent’s strategy.
The ego agent never directly observes the opponent’s strategy.
1) Continuous Latent Variables.
In the most general case, the latent strategy space is allowed tobe continuous.
This enables the latent strategy space to represent an inﬁnite number of possible opponent strategies.
With the continuous latent variables Z , we use the Euclidean norm for the
stability reward: R stable
=  d ( z j z j  ) =
|
| z j  z j
| | 2) Discrete Latent Variables.
In this setting, we constrain the latent strategy space to be discrete categorical variables by using the Gumbel-Softmax distribution [37, 38].
Intuitively we cluster the
continuous latent variable representations to ﬁlter the noise of measuring distance in the latent space.
Further details of the discretization method can be found in Appendix A. With these categoricallatent variables Z , we use the discrete distance metric for the stability reward:
[Equation]
We found empirically that discrete latent variables improve the consistency of the stability rewardsignal compared to continuous representations, despite the decrease in expressivity and granularityprovided by the distance metric.
Intuitively, it is easier to identify whether or not a strategy issimilar enough to be clustered with another strategy than it is to exactly identify the distance betweenpredicted strategies in a latent space.3)
Partial Observations of Changing Strategy.
In this setting, we assume that the ego agent is ableto observe if their opponent’s strategy changes between timesteps, so no representation learning isnecessary to stabilize.
We shed the assumption that the latent strategies are ﬁxed within an interaction, so we denote the latent strategy at a timestep t as z t .
Thus, the stabilizing reward can be deﬁnedas R stable =
c t =
[ z t 6 =
z t  1 ] , which indicates whether the opponent strategy has changed.
This 230 corresponds to settings where it may be easy to estimate when an opponent’s strategy has changed,but it may be difﬁcult to estimate their exact strategy [12].
Experiments
We perform experiments in a range of environments across different latent strategy space settings:continuous latent variables, discrete latent variables, and partial strategy observations.
We compareour approach, Stable Inﬂuencing of Latent Intent (SILI) to various multi-agent learning baselineswith implementation details described in Appendix A:
Oracle.
Oracle is given the true opponent’s strategy as an observation.
Learning and Inﬂuencing Latent Intent (LILI) [10].
LILI combines soft actor-critic (SAC) [45]with a representation learning component to model the opponent’s strategy dynamics.
LILI does notexplicitly optimize for stabilizing the opponent’s strategy.
Surprise Minimizing RL (SMiRL) [43].
SMiRL optimizes for a different notion of stability, wherethe reward is proportional to the likelihood of a state given the history of previously visited states.
Stable.
We refer to Stable as the variant of SILI with  = 1 , only optimizing for stability.
SAC [45].
SAC does not model the opponent’s strategy and is the base RL algorithm for SILI.
5.1 Simulated Environments
The environments and trajectories of stabilizing behavior are shown in the left column of Fig.
Across all ﬁgures, the ego agent’s trajectory is drawn in blue, and grey corresponds to the opponent.
Circle Point Mass.
In this environment, the ego agent is trying to get as close to the opponent aspossible in a 2D plane, inspired by pursuit-evasion games [46].
The opponent moves between locations along the circumference of a circle.
The ego agent never observes the true location (strategy)of the opponent.
If the ego agent ends an interaction inside the circle, the opponent jumps counterclockwise to the next target location.
If the ego agent ends an interaction outside the circle, theopponent stays at the red target location for the next interaction.
We examine four variants: Circle(3 Goals), Circle (8 Goals), Circle (Continuous), Circle (Unequal).
In Circle (Continuous), there areinﬁnite possible opponent strategies, so we use continuous latent variables.
In Circle (Unequal), theego agent begins closer to two unstable opponent goals, but there is a stable goal farther away.
Driving.
A fast ego agent is attempting to pass a slow opponent driver [10].
There are 3 lanes anda road hazard upcoming in the center lane, so both the ego agent and opponent need to merge to anew lane.
If the ego agent merges to the left lane before the red line (giving the opponent enoughreaction time), then the opponent merges to the right lane during the next interaction, understandingthe convention of faster vehicles passing on the left.
Otherwise, the opponent will aggressively tryto cut off the ego agent by merging into the lane that the ego agent previously passed in.
Sawyer-Reach.
The opponent can choose between three goals on a table with their intent hiddenfrom the robot.
The ego agent is the Sawyer robot that is trying to move their end effector as close aspossible to the opponent’s chosen goal in 3D space without ever directly observing the opponent’sstrategy [47].
If the end-effector ends the interaction above a ﬁxed plane on the z-axis, the opponent’sstrategy stays ﬁxed.
Otherwise, the opponent’s strategy changes.
Semantically, we consider a robotserver trying to place food on a dish for a human and needing to move its arm away from the dishby the end of the interaction to avoid intimidating the human retrieving the food.
Detour Speaker-Listener.
In this environment, the agents must communicate to reach a goal, whichhas been a popular setting to explore emergent communication [18].
The speaker (opponent) doesnot move and observes the true goal of the listener (ego agent).
The ego agent cannot speak, but mustnavigate to the correct goal.
The speaker utters a message to refer to the goal, which the ego agentthen observes.
If the ego agent goes near the speaker (within some radius), the speaker follows thesame communication strategy during the next interaction.
Otherwise, the speaker chooses a randomnew strategy, or mapping of goal landmarks to communications.
Critically, in this environment, theopponent’s strategy is the communication strategy, not the true goal location.
[Figure] [Figure]
LILI
SMiRL
Stable
SAC
[Figure]
5.2 Results
Across all environments, without observations of the opponent’s strategy, SILI achieves a comparable task reward to Oracle, as seen in Fig.
Further analysis is available in the Appendix and athttps://sites.google.com/view/stable-marl/ with qualitative analysis of trajectories.
Continuous Latent Variables.
We perform an experiment with continuous latent variables in the
Circle (Continuous) environment (row 6, Fig. 3).
Both SILI and Oracle learn to stabilize in thisenvironment, although Oracle incurs various dips in stability reward at little to no cost to the taskreward because Oracle knows the true opponent’s strategy.
While these results are promising, weobserve in other environments that continuous latent variables and a smooth distance metric over thelatent space lead to more noisy rewards, which hinder the ego agent’s ability to learn effectively.
Discrete Latent Variables.
We further show our results in a variety of environments with discretelatent variables.
In Circle (3 Goals), SILI achieves the highest reward along with Oracle, but Oracledoes not require stabilizing the opponent’s strategy (row 1, Fig. 3).
In Circle (8 Goals), we demonstrate that with more opponent strategies, SILI still learns to stabilize and outperforms the baselines(row 2, Fig. 3), where we set the latent dimension hyperparameter to be an overestimate of the truenumber of opponent strategies.
In Circle (Unequal), SILI avoids trying to optimize for nearby, unstable opponent strategies, whereas the other baselines do not learn to stabilize since they greedilytry to track the nearby, unstable opponent strategies (row 3, Fig. 3).
In Driving, the task reward isbased on the number of collisions with the opponent, and our method is the only one that learns tosafely avoid all collisions (row 4, Fig. 3).
In Sawyer-Reach, SILI and Oracle achieve comparableperformance, but again, Oracle does not need to learn to stabilize (row 5, Fig. 3).
Partial Observations of Changing Strategy.
To test our method in a setting without representationlearning, the ego agent observes whether the opponent’s strategy has changed between timesteps.
We perform this experiment in the Detour Speaker-Listener environment, where the assumption thatthe opponent’s strategy is kept ﬁxed throughout an interaction is broken (row 7, Fig. 3).
SILI learns to stabilize and signiﬁcantly outperforms all other baselines in task reward, including
Oracle.
In this environment, although Oracle can observe the true opponent strategy, the Oracle stillneeds to learn the association of the opponent’s strategy and opponent’s communication with thetrue goal.
Further, the communication and strategy can change frequently between timesteps whilethe true goal stays constant, making learning difﬁcult.
The baseline Stable learns to stabilize butdoes not optimize for the task reward at all.
The rest of the baselines learn to move towards thecentroid of the goal locations since they cannot accurately decipher the opponent’s strategy.
Notice that around interaction 250 , SILI learns to stabilize, which requires moving in a detour awayfrom the true goals, but SILI maintains the stability and learns to maximize task reward well.
On thecontrary, LILI temporarily learns to stabilize at around interaction 1250 , but incurs a correspondingdecrease in task reward, which discourages LILI from maintaining the stable opponent strategy.
Discussion
Summary.
We propose a framework for learning in multi-agent environments that learns the dynamics of the opponent’s strategies in a latent space.
We then leverage this latent dynamics model todesign an unsupervised stability reward, which can be augmented with the task reward to explicitlyencourage the ego agent to learn how to stabilize and then more easily maximize the task reward.
Limitations and Future Work.
In this work, we deﬁne a notion of pairwise stability that considersconsecutive changes in the strategy.
However, it may be interesting to consider smoother, longerhorizon stability metrics, such as average change over a window of time.
We compare our method to
SMiRL, which optimizes for a probabilistic notion of state stability relative to states in an experiencereplay buffer, but show that SMiRL often does not learn to stabilize in a consistent manner.
Weplan to compare these different metrics of stability and their effects in future work.
In addition,our experiments were mainly done in simulation.
In the future, we plan to consider robot-robot andhuman-robot interactions, where we are required to learn much more complex dynamics that capturethe non-stationarity of humans who adapt their strategies over repeated interactions.
Stabilizing ahuman’s strategy can greatly improve the efﬁciency in which a robot learns the best way to performa task, but the exact method in which to stabilize human behavior should be further explored.