A M UTUAL I NFORMATION D UALITY A LGORITHM FOR M ULTI -A GENT S PECIALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The social behavior change in a heterogeneous population is an essential study of multi-agent learning.
The interactions between unique agents not only involves the optimization for single agent learning, agents’ behavioral changes also depend on the mutual similarity and dissimilarity between the agents.
Our study provides a theoretical derivation of the policies interactions under the formulation of joint policy optimization.
We discover that joint policy optimization in a heterogeneous population can affect the population behaviors through mutual information (MI) maximization.
In our study, we introduce a minimax formulation of MI (M&M) that optimizes the population behavior under MI minimization against the joint policy optimization.
The main findings of our paper show that MI minimization can minimize the behavioral similarity between agents and enable agents to develop individualized policy specialization.
Empirically M&M demonstrates a substantial gain in average population performance, diversity, and narrows the performance gap among the agents
I NTRODUCTION
From the success of multi-agent game plays [(OpenAI et al., 2019), (Vinyals et al., 2019)], the field of heterogeneous multi-agent learning is being actively studied in areas such as AI game design (Contributors, 2022), embedded IoTs (Toyama et al., 2021), and the research for future HumanAI interaction.
With the unique physique and specialized character attributes in a heterogeneous population, the ability to optimize policies for specific purposes is essential.
The difficulty as well as the goal of the heterogeneous population learning research is to search and define a general learning algorithm that optimizes an agent population such that the uniqueness of each agent can be fully utilized within the population.
To approach the problem, our research aims to understand the symmetric and asymmetric social behavior changes that occur during the population learning through mutual information formulation.
To study the learning and behavioral change, multi-agent learning has formed two branches of studies.
In the simplest form of multi-agent RL, individualized learning is performed to learn separate behavioral policies for each agent.
These prior works include Independent Q-Learning (IQL) (Tan, 1993), Policy Space Response Oracle (PSRO)
(Lanctot et al., 2017), and AlphaStar (Vinyals et al., 2019).
However, the empirical success of these prior approaches have excluded knowledge sharing.
Since the training is done independently, one agent’s learned experiences do not transfer to another.
The individualized behaviors result in high redundancy of re-exploration and low socialized behaviors among the agents.
On the contrary, joint policy optimization is proposed as a solution to the listed problems.
Joint policy optimization utilizes a single conditioned policy to learn the diverse skillset, and the character attributes of the population via distributed RL optimization.
Through population experiences sharing and joint policy optimization, a single conditioned policy network can learn a set of generalized policy skills that are transferable across the different agents.
Notable examples include OpenAIFive (OpenAI et al., 2019), HAPPO (Kuba et al., 2021) and NeuPL (Liu et al., 2022) that optimize the multi-agent behaviors under the expected population accumulated rewards.
Through knowledge sharing and joint policy optimization, population learning in interactive games has benefited from the increased learning efficiency, and learning generalization of agents’ social behaviors.
Our research focuses on the latter, where we analyze the cause of social behavior change.
By analyzing a heterogeneous population with pairwise interactions, we discover that the joint policy optimization of a two-player competitive game converges to MI maximization among the agents.
Unfortunately, MI maximization optimizes the commonality in a population.
We found that in a heterogeneous population where agents are individually unique, MI maximization benefits agents that have characteristics close to the population norm, but severely degrades the performance of unique agents with character attributes dissimilar to the average population.
To address the drawback of MI maximization, our research’s main contribution is a novel minimax MI (M&M) formulation of population learning that enables individual agents to learn specialization from the dual formation of MI maximization and minimization.
B ACKGROUND AND R ELATED W ORK
Multi-agent learning is a broad field of study that not only covers the necessary intelligence to achieve individual agent reward maximization, but also the social behavior change when agents interact with other agents.
Prior researches in population learning have studied the performance of the agent population in competitive and cooperative environments.
2. C OMPETITIVE B EHAVIORS L EARNING
In a competitive environment, the goal of the multi-agent research is to utilize competition to optimize the performance of an agent population.
One approach to perform the learning iteration is individualized learning.
Each agent improves its policy through learning a best-response (BR) against the prior iterations of agents.
The iterated elimination of dominated strategies optimizes a population of policies under Game Theory.
Prior studies such as (Jaderberg et al., 2018), PSRO,
PS-TRPO (Gupta et al., 2017), (Vinitsky et al., 2020) and Alphastar utilizes different variants of self-play (Heinrich et al., 2015) to learn competitive Nash Equilibrium behaviors for a population.
The different variants addresses the stability (TRPO constraint), robustness (adversarial population) and diversity (leagues of policies pretrained on human data) of individualized learning.
In contrast, a joint policy optimization framework proposes by (Foerster et al., 2016) and (Lowe et al., 2017) optimizes the population with Centralized Learning Decentralized Execution (CLDE).
The joint optimization enables common skill transfer across policies.
The commonality among the agents can be learned once and the learned behavior can be utilized by different agents of a population.
This form of joint policy optimization optimizes the population as an one-body problem with Mean Field Theory (Yang et al., 2018).
Under Mean Field Theory, the variations of individual agents can be averaged.
The modeling of the population behaviors are reduced from a many-body problem to an one-body problem.
Prior works include MADDPG, HAPPO (Kuba et al., 2021),
OpenAIFive (OpenAI et al., 2019) and NeuPL (Liu et al., 2022), where OpenAIFive and NeuPL have both further developed efficient graph solvers based on (Shoham & Leyton-Brown, 2008) to optimize the social graphs of the population.
The graph solver F optimizes the match pairing of the agents (x,y) with weighted edges Σ ( x,y ) .
The objective of F commonly optimize the policy learning to be robust against adversarial exploitation or agents with the most similar performance strengths.
2. C OOPERATIVE B EHAVIORS L EARNING
To develop social behaviors of cooperation, prior studies have proposed auxiliary rewards and regularization of mutual information as part of the objective function.
This includes OpenAIFive’s team spirit reward, (Chenghao et al., 2021)’s Q-value, (Cuervo & Alzate, 2020)’s
PPO, and (Mahajan et al., 2019)’s latent variable regularization of MI maximization.
While the above studies suggest that learning of cooperative social behaviors require auxiliary modifications with mutual information, (Dobbe et al., 2017) shows an interesting analysis on the distortion rate of the joint policy optimization versus the individualized learning with MI.
The study shows that even without auxiliary modification, there is a significant stability difference in distortion rate between the joint optimization and individualized learning.
This issue has shown to negatively impact cooperative learning.
2. A N EW P ERSPECTIVE O N E XISTING W ORK : F ROM P OLICY G RADIENT TO M UTUAL I NFORMATION
In this section we first show that policy gradient (Sutton et al., 1999) in multi-agent joint optimization can be reduced to MI maximization of an agent population.
We derive the learning objective of multi-agent learning as the following.
To model the learning dynamic of a two-players competitive zero-sum game with Player x and
Player y, we let Π θ ( a | s ) be a population joint policy.
Given a game state s , players act according to the policy Π θ ( a | s ) , and let a x and a y denote the actions taken by Player x and y. We define
Q ( s, a x , a y ) , V ( s ) , Adv ( s, a x , a y ) as the centralized learning of Q, value, and advantage functions respectively, where Adv ( s, a x , a y ) =
Q ( s, a x , a y ) − V
( s ) .
The objective of the policy optimization is to perform maximization of the expected return J ( θ ) from episodes of self-play.
The joint policy gradient of a single step optimization is then defined by:
[Equation] [Equation]
To analyze the behavioral change of the players, we derive Eq(3)
’s gradient to inspect the direction of policy change .
We take the derivative with the product rule and apply the log trick (Williams, 1992) to derive the expanded form of the joint gradient:
[Equation] [Equation]
Here we expand the Advantage function to Q ( s, a x , a y ) − V ( s ) .
With Π xθ ( a x | s ) and Π yθ ( a y | s ) specific to each side of x and y, we assume they are independent.
By applying the probability of independence and the logarithmic product rule (A.6), the joint gradient can be defined by:
[Equation]
The derivation of Eq(3) shows components of mutual information formulation.
By taking T steps of joint policy optimization, the integral of Eq(3) removes ∇ θ from the equation.
The convergence result of the joint policy optimization is a weighted variant of mutual information maximization shown below in Eq(4).
This derivation suggests that the long-term behavioral change between two competitive players will converge to mutual behaviors maximization .
[Equation]
In a heterogeneous population of agents { i, ii,... N } , we extend the two-player game (X,Y) to all pairings of unique agents, the expected joint gradient is then denoted as:
[Equation]
The key finding from Eq(3) to Eq(5) is that through the accumulated gradient update, the maximization of J ( θ ) results in MI maximization in the behavior of a heterogeneous population.
We find that the MI maximization formulation can impose a constraint on a heterogeneous population.
Particularly agents with character attributes dissimilar to the average population cannot realize their own competitive advantage when their behavioral policies are subject to MI maximization constraint with the average population.
An iterative BR variant of joint policy optimization is more difficult to derive the population-level behavior change during the joint policy optimization.
For the BR variant,
Eq(5) approximates only the population-level behavioral change as both sides of the policy response in a symmetric two-players game are near convergence.
i.e. the joint policies utilized by the two players are approximately the same.
M&M: M INI M AX M UTUAL I NFORMATION S PECIALIZATION
We introduce M&M as a MI minimax population learning that aims to address the drawback of joint policy optimization.
In particular, we propose a minimization of MI to optimize a disjoint set of policies against the MI maximized joint policy optimization.
This process enables individual agents to specialize behavioral policies relative to the average population.
We denote the Generalists as the joint policy that maximizes MI, and can be utilized on the general population.
We define Specialists as a disjoint set of policies that minimize MI against the Generalists.
Specialists’ behavioral policies deviate from the population norm through learning behavioral policies that align with agent’s innate attributes.
3. P ROBLEM F ORMULATION
We consider a non-Markov two-player zero-sum Game (X,Y) with a heterogeneous population of size N { i, ii... N } .
The game is defined by ( O, S, A, R ) where O is the observation space, S :
O × O is the joint observation of the two players as a fully observable game.
The discounted return for each player is defined as R t = (cid:80) ∞
τ = t γ τ r τ where γ is the discount factor γ ∈ [0 , 1) .
We denote the Generalists as { Π gθ ∗ } N g =
i , as an ϵ − Nash Equilibrium (NE) policy solution reached by the joint policy optimization after T iterations of population self-play optimization.
Here every player receives an expected payoff within ϵ of NE as a numerical relaxation of NE.
We define and initialize the Specialists as a set of disjoint policies from replication of the Generalists {
π ψ k 0 } Nk
= i ← { Π gθ ∗ } Ng
= i , where { π ψ
k 0 } Nk
= i denotes a set of N disjoint policies parameterized by ψ on the 0-th training iteration.
3. C ONDITIONAL M UTUAL I NFORMATION M INIMIZATION
For each set of games, we formulate M&M as a one-vs-all BR policy optimization of a Specialist against the population of the Generalists in a zero-sum two-players game.
Given a game state s , players act according to their own policy.
Let a k and a g denote the actions taken by a Specialist k and a Generalist g in a game.
We define Q ( s, a k , a g ) and V ( s ) as the Q and value functions individually for each Specialist.
The objective of the policy optimization is to maximize the expected return J ( ψ k ) while minimizing MI with the gradient.
In each optimization step we fix the parameter of the Generalists and treat the joint policy as a part of the fixed ϵ − NE solution.
We then define the effects of a g as a part of the environment and allow the disjoint policies to condition on the
MI maximization joint policy, and perform MI minimization.
Formally, we define the conditioned policy gradient for a Specialist k as:
[Equation]
By treating Π gθ ∗ ( a g | s ) as the average population’s fixed ϵ − NE response, a Specialist k ’s policy gradient optimization can be simplified to learn a BR against a stationary environment.
The derivative of the gradient w.r.t.
ψ k can be defined as:
[Equation]
By taking T steps of J ( ψ i ) maximization, the integral of Eq(7) maximizes the negative of a weighted conditional MI w.r.t.
the Generalists denote in Eq(8).
Alternatively, the optimization is equivalent to optimize π ψ k to minimizes its mutual policy response with respect to Π gθ ∗ by weighting the individual agent’s Q-value gain against the value-function estimation conditioned on the MI maximizing
Generalists population.
[Equation]
The overall expected social behavior change over time when averaging the disjoint policies’ gradients is shown in Eq(9).
w ( X k ; Y g ) denotes the weighting of Q ( s, a k , a g ) − V
( s |
Π gθ ∗ ) , and the full formulation shows a minimization effect of MI between every Specialist to the Generalists’ joint policy optimization.
Utilizing the dual properties of mutual information, M&M optimizes the Specialists population as individual agents that each specializes according to an agent’s innate attributes.
The benefit of specialization over joint policy optimization is that rather than relying on common skill transfers for policy improvement, a Specialist learns a unique policy that best aligns with its own characteristics.
This makes the population social interactions more diverse and enhances the performance of agents with out-of-distribution character attributes.
We show an algorithmic implementation of M&M policies optimization in (A.7)
3. S OCIAL G RAPH O PTIMIZATION
In our population learning of two-players competitive game, we have implemented a graph solver to both optimize and analyze the social interactions between vertices of agents’ policies.
Let F be defined as a graph solver, and each agent’s policy denotes a graph vertex.
The directional weighted edge between two vertices defines the interaction between two vertices as a two-players game.
The direction of the edge denotes the relationship between the pairs of vertices as dominance or dominated, and the weight of each edge denotes the probability of the competitive outcome for vertex k
as U k .
The initialization of all the edges are set to 0.5 as a 50% chance of winning or losing for each pair of vertices.
After each batch of finished games, the weights and directions of the edges involved are updated according to the game outcomes.
Based on the weights between the vertices, F updates the match distribution of vertex k by defining Σ k as the probability distribution of vertex k forms an interactive game to other vertices.
We define the optimization objective of F to prioritize the sampling of matchmaking based on the exploitability of vertex k relative to other policies.
Due to the probabilistic sampling of concurrent game interactions, the sampled adversary opponents form a set of mixed strategies we denote as σ .
E XPERIMENTS
Empirically we evaluate the specialization of agents on the Mobile Fighting Game: Naruto Mobile.
We compare and contrast the Generalists and Specialists population on advantage analysis, social graph connectedness, behavioral diversity, and agent performance.
We discuss the advantage and social graph for each agent to demonstrate how the social behavior change can be largely influenced by the methods of policies optimization. Additionally, our analysis on behavioral diversity and agent performance show that M&M optimizes agents’ policies for specialization.
By reducing MI, agent population can benefit from behavioral diversity and performance gain.
4. I NTRO TO N ARUTO M OBILE G AME M ECHANICS
Game mechanics : Naruto Mobile is a real-time 2D mobile Fighting Game of 1 vs 1 gameplay.
The game includes a pool of over 300 unique characters.
Each character is designed with a unique set of character attributes such as attack speed, range and movement variations.
Additionally, each character has skills of different cooldown, status effects and duration.
The large population of characters and real time competitive interactions provides a testbed for research on a heterogeneous population.
Before the start of a match, both sides of players start by selecting their characters, auxiliary skill scroll and a pet summon.
The key to winning a match is to reduce the opponent’s health point (HP) to zero, or having a higher HP at the end of a 60 second match timer.
4. A GENT S PECIALIZATION
We record the advantage of the Generalists’ and the Specialists’ population against their prior iteration of population policies.
The recorded advantage vectors of agent k comprise the expected advantage of agent k against its opponents over the duration of evaluation.
The recorded advantage vectors of an agent are projected on the unit circles to visualize the vectors’ magnitude and the vectors’ direction components.
The radial plots are measures in the population’s mean-deviation with the center of the circle representing zero advantage over the average population, and the radial edges represent the maximal deviation gain against a specific radial opponent.
From before to after agents’ specialization, Figure 1 shows Agent I. has an increased magnitude of agents’ advantage vectors.
This implies that under M&M specialization, the MI between Agent
I. and the general population is minimized.
Similarly, Agent K. is also able to minimize its MI to achieve its own specialization.
While both agents are able to achieve specialization, the direction components are different for the two agents.
Agent I. is specialized against prior iteration of itself and Agent K., and Agent K. is specialized against prior iteration of Agent I. and Agent O..
The unique directions of each agent’s specialization shows that M&M specialization not only minimizes
MI of the population, each agent is also optimized for its own agent-attributes.
Moreover, the Generalists and Specialists populations show different aggregated bounds.
With the aggregate of Generalists as the dotted line region and the Specialists population as the solid line region, the radial plot shows that the Generalists dotted bound is restricted close to the center.
With the center representing the average population, the closer the region to the center, the more generic the agent population behaves.
In the extreme case of conformity behavior convergence, MI maximization can reduce the region and eliminate the diversity in the population.
On the other hand,
Specialists show individual differences relative to the population norm.
The individual specialization creates a collective diversity for the population.
4. S OCIAL G RAPHS AND I NTERACTION S TABILITY
With NeuPL (Liu et al., 2022) and (Shoham & Leyton-Brown, 2008) graph solver, the different social graphs of the Generalists and Specialists are plotted on their corresponding radial charts.
In a social graph, the labeled agent is the center node of the graph and is connected with edges Σ( x, y ) outward to other agent nodes.
Σ( x, y ) are normalized with population mean-deviation.
The MI
Maximized Social Graph is shown on the left in blue, and M&M’s Social Graph on the right in red.
The social graphs display the relative frequency agents interact with each other and the deviation from the mean.
The larger an edge value is, the more frequent the agent-nodes interact.
MI Maximized Social Graph shown in Figure 2 illustrates that agents are connected with approximately equal connection strength in each edge.
This indicates a densely connected social graph.
This approximately translate to uniform distribution of { P X,Y ( x, y ) } Nx,y
= i with high stability in − H
( X, Y ) .
In contrast, M&M’s Social Graph shows few strongly connected edges, such as Agent
T. to Agent O., and sparse connections with other agents.
With sparsity in the social graphs, each agent is to receive a different distribution of experiences.
4. F ROM M ONOTONE B EHAVIOR TO D IVERSE I NTERACTIONS :
To analyze the agents’ behavior, we introduced several metrics to track the timing of agents’ skills.
ForcingMoves are metrics that measure agent’s attack initiation prior to the opponent’s attacks.
CounterMoves measure skills used to interrupt the opponent during their skill casting.
CloneSubstitute measures the frequency that an agent avoids or escapes from the opponent’s attack.
The metrics are displayed via mean-deviation frequency to account for the different skill cooldown among the agents.
We collect a batch of 100 games for each agent-type to look at how similar or different the learned strategies are.
Generalists and Specialists respond.
For the
Generalists, the mean-deviation shows similar change despite playing four uniquely different agents.
Except for the cooldown constraint, the actions of the four Generalists are largely the same response.
This shows the drawback of MI maximization optimization.
On the contrary, the Specialists are incentivized to formulate creative plays based on their individual agent-attributes.
M&M creates a wide variety of strategy timings and competitive behaviors.
In Figure 3, the stacked Generalists and Specialists action timing reveals the similarity and differences in different population’s strategy timing.
Specialists learn to time their skills differently when embodied in different agents.
Each agent adapts its behavior such that its current strategy is optimal relative to the general population.
In contrast, the Generalists population shows low variations in strategies.
With the similar action timing, the Generalists have developed social behaviors that conform with other agents.
While under MI maximization, the optimized generic behavior works across the different agents, but the diverse Specialists behaviors better demonstrate the essence of general intelligence for adaptive learning and creativeness.
4. P ERFORMANCE C OMPARISON :
The ϵ - Nash of the Generalists agents is iteratively optimized with 13 rounds of joint policy optimization via NeuPL’s graph solver (Heatmap shown in Appendix A.4).
Here we display the topperforming Generalists relative to the M&M optimized Specialists.
The displayed competitive performance are the weighted win rates of the 8 agents versus the ϵ - Nash of the Generalists population.
Aside from the performance gain from agent specialization, Figure 4 also shows an interesting observation that optimizing for population’s commonality (MI maximization) does not translate to a more balanced population.
With joint policy optimization, the population converges with high performance differences among the agents.
The standard deviation of MI maximized optimization is 9.95, while it is reduced to 5.99 after M&M’s specialization.
With approximately 30 % reduction in population performance deviation, it shows that equity of a population (performance gap) can be improved when individual agents specialize.
4. S AMPLE E FFICIENCY :
In our last experiment we look at the sample efficiency of the different optimizations.
The joint policy optimization of the Generalists are performed on a set of 16 agents under MI maximization for 9 rounds.
We then compare the relative performance gain to sample required with joint policy optimization and M&M for 2 additional rounds of optimization.
The experiment measures the sample complexity needed for the two methods to reach a similar level of relative performance prior to specialization.
In Figure 5, joint policy optimization demonstrates that it can use less learning steps per agent to optimize a population of agents.
In contrast, due to the diversified exploration M&M requires additional samples to explore the larger set of population interactions.
The main takeaway from this experiment is that it would be more learning efficient to first learn the common skills under joint policy optimization.
Sequentially, after Generalists’ convergence, M&M specialization can further benefit each agent with additional samples of training that diversify the learning experiences and enables specialization.
C ONCLUSION
In our paper, we have derived two formulations of MI from multi-agent policy gradient.
One brings a new perspective on the existing work of joint policy optimization, the other explores the potential of conditional MI minimization.
The two learning representations under the dual formulations of MI measure the long-term effects of a population’s social behavior change.
With the joint policy optimization, the MI maximization of gradients significantly increased the sample efficiency of population learning by learning a set of transferable skills across the agent policies.
Through maximizing MI as policy’s gradient, the joint policy optimization optimizes the heterogeneous population toward a set of mutually similar behaviors among the agents.
In contrast, our proposed M&M population learning aims to minimize the MI between the disjoint policies and the MI maximized joint policy agent population.
Our conditional MI minimization population learning enables each agent to define an unique policy that best aligns with the agent’s own character attributes.
Without the MI maximization constraint, the learned policy does not need to be transferable to other agents.
In a heterogeneous population the specialization of behavioral policies are especially beneficial to agents that have dissimilar character attributes relative to the average population.
As a result, M&M can optimize a diverse population of Specialists that better balance the performance inconsistency within a population.
We believe that through studying the gradient formulation of population policy gradient can help to bring better understanding on the social behavior change that occurs within a population.
Future Work: The current limitation of our method is on the additional compute resource for learning agent’s individualized specialization.
Due to the process of specialization being specific to each agent’s character attributes relative to the joint policy, sample efficiency improvement on joint and disjoint policies optimization can both benefit the training needed for social behavior study.
We thank the reviewers for suggestions and their valuable constructive feedback on our study.
MuJoCo Menagerie Contributors
users mickep
Prove log(xy)
= log x + log y. https
//math.stackexchange.com/q https://math.stackexchange.com/q/1115501 (
[Figure]
Basic Module - for game env stat & characters specific information.
Buff Module - buff and debuff info during gameplay.
Element Module - characters equipment of weapon IDs, armor, etc. 4.
Action Masking Module - gives the model the information on whether each action is currently available.
The separation of modules in Figure 6 allows us to have sparse connectivity in the model.
The compute load is reduced compared to fully connected layers.
This allowed us to inference our models locally on mobile devices.
MDP: The model receives from the Basic Module to identify the characters and equipped Summons and Scrolls.
Additionally, characters position, movement and skills’ buff & element information are received from Buff and Element Module.
The above are the input states s t to our model.
Our model predicts an action output a t to control the 2D movement of the agent, and the available attack and skills.
The reward can be customized, but in our standard mechanics it is based on the (weighting) of an agent’s own HP(10), opponent’s HP(10), the result of the battle(10), combo(5), and mana(5).
With the transition of the action, new state s t +1 are given to the model for the next iteration of MDP.
A.1. H YPERPARAMETERS AND H ARDWARE U SED Hyperparameters
• PPO: 0.
• n-step: 100 frames • Reward discount factor: 0.
• Learning rate: 1e-
Hardware Used
• CPUS: 5, • GPUS:• max Batch size:• Average Compute Time: –
≈ 180 Hrs (Generalist Training)
A. H ETEROGENEOUS A CTION S PACE R EPRESENTATION L EARNING :
With a large collection of distinct agents, the learning problem we face is their unique action sets and their relative similarity.
Inspired by word embedding (Mikolov et al., 2013), we approach the problem with a structure of information bottleneck to learn the representation of the action sets as a dense representation of a vector.
We define our information bottleneck via the structure of a neural net Π( a | S ) , where S is the learn-able state space that holds the contextual information of the involved agents ( x, y ) , action sets ( a x , a y ) and the joint observations.
The goal of the model structure is to compress the large contextual information S to learn a joint policy Π that captures the actions’ similarity in a specific interaction context.
A. I NDEPENDENT C AUSALITY B ETWEEN A GENTS ’ I NTERACTIONS :
VM3-AC (Kim et al., 2020) has pointed out the possible causality dependency of multi-agent interactions.
When player i decides its action a xt it may depend on player j ’s action a yt .
The obvious problem of reasoning is the circular causality of the two players, a xt depends on a yt and a yt depends on a xt .
To resolve this issue, (Kim et al., 2020) has introduced a random variable to disassociate the loop dependency via variational lower bound.
We define our agents’ causality graph without the Markov Assumption on agent’s action.
By removing the Markov Assumption , we can use opponent’s past actions a − π 0: t − 1 to predict opponent’s current move a − π t .
This breaks the loop and assume the casual graph of players’ actions are independent at time t .
This gives the agents temporal consistency of Forward Induction (Battigalli & Siniscalchi, 2002) of Game Theory for self-play.
The reached Nash Equilibrium under Forward Induction is self-consistent that the rational opponents in the past will continue to choose rationally in the future.
A. G ENERALISTS S ELF - PLAY A LGORITHM C ONSTRUCTION
Our neural population learning for joint policy optimization is performed under the multi-agent social graph of NeuPL(Liu et al., 2022).
The nodes are the different generations of agents, and are connected by the weighted edges of { Σ ( x,y ) } NxN .
NeuPL provides a population self-play framework that not only competes the current population
Π Nθ τ with combinations of distinct agents, but the weighted edge prioritization also extend to different generations of ϵ -Nash population Π Nθ 0: τ − 1 .
Each population is represented as a conditional joint policy that learns a set of BR strategies against all previous generations of multi-agents mixed-strategies.
The ablation evaluation shows joint policy optimization roughly diminished as the training approaches the 11th 13th iteration.
In particular, the 13th iteration only has a 57.6 % winrate against the 12th iteration of Generalist, which is close to the Nash equilibrium of 50 %.
In Figure 7, we evaluate all rounds of the Generalist policies against each other.
The heatmap shows monotonic convergences of the Generalist population.
At the 13th iteration, the Generalists population has converged as ε − Nash Equilibrium where ε ≈ 7 . 6% .
Further training may minimally increase the performance, but worsen the strategy diversity.
A. C ORRELATION OF D IVERSITY AND P ERFORMANCE
Since Agent T. is the only agent without performance gain after M&M specialization, we compare its Advantage vector change, performance chart with Agent H.’s.
From Figure 8, shows the positive correlation of strategy diversity and character performance still holds.
With Agent H., the radial plot shows a large deviation when the strategy is being converted from Generalist to Specialist.
This correlates with the notable increase in agent performance on the left chart of Figure 8.
On the other hand, Agent T. shows a more muted deviation within the
Generalist bound.
The lower strategy deviation is also reflected in Agent T.’s flat performance on the right chart of Figure 8.
The lack of specialization for Agent T. may be due to the Generalists’ converged strategy is already close to the optimal strategy for Agent T.
This may imply that Agent
T. has the most centered agent attributes in relation to the population.
As the Generalist policy converges towards the population mean, the converged strategy can become the optimal strategy for
Agent T.
A. I NTEGRATION :
Let Π xθ ( a x | s ) =
x , Π yθ ( a y | s )
=
y , and x, y > 0 then the change of variable approach (mickep , https://math.stackexchange.com/users/97236/mickep) (user65203):
Let log(x)
= (cid:90) x t =
∂t t (10) log ( xy ) =
(cid:90) xy t =
∂t t = (cid:90) x
∂t t + (cid:90) xy
t =
∂t t − (cid:90) x t = ∂t t
=
(cid:90) x t =
∂t t + (cid:90)
xy t = x
∂t t
Let u = t x, such that ∂t t
= ∂u u
= (cid:90) x t =
∂t t + (cid:90)
y u
= ∂u u (11)
This gives us log ( Pi ( xθ ( a x | s ) Pi ( yθ ( a y | s ))
=
log (Π ( xθ ( a x | s )Π ( yθ ( a y | s )) .
With the probability of independence assumption, Q ( a x , a y , s ) function has the joint probability of ∇ θ log (Π ( x,y )
θ ( a x , a y | s )) in Eq(3).
A. A LGORITHM P SEUDOCODE
In this section we breakdown M&M population learning into pseudocode, where the main components of the Specialists optimization are the neural population learning (NPL), ϵ − NE of Generalists policy ( Π nθ ∗ ), Specialists policies ( { π ψ k 0 } Nk = i ) and graph solver F .
Let NPL be the concurrent optimization of two-players game between two agents’ policies.
For every episode NPL simulates the match of ( π, σ, Π ), where π is matched against an agent member of Π and the population of Π collectively plays a mixed strategy σ .
We collect the policies interaction trajectories, T , into a replay buffer for policy gradient optimization.
Algorithm π, σ, replayBuffer ← { } // Initialize trajectory replay buffer for Episode ∈ 1 , ..., t do
Start game engine with policies ( π, Π ).
Store trajectories T from policies interactions ( π, σ, Π ) into replayBuffer .
replayBuffer ← replayBuffer (cid:83) T end for return replayBuffer
For a given heterogeneous population { i, ii, ...N } and an ϵ − NE Generalists population Π Nθ ∗ , M&M optimizes { π ψ k 0 } Nk
= i to minimize MI of the individual agent policy against the Generalists population Π Nθ ∗ .
Each one-vs-all matchmaking is sampled based on the priority given by the graph solver F .
After a batch of episodes of NPL, we optimize π ψ kτ with PPO optimization.
Additionally, we define
Eval() to compute the aggregate outcomes of NPL.
Algorithm
Input :
Population = { i,ii,...N } //
Heterogeneous population of N distinct agents { π ψ
k 0 } Nk
= i , // N disjoint policies
Π Nθ ∗ , //
ϵ − NE Generalists { Σ k :=
( π ψ k , { Π gθ ∗ ) } Ng = i } Nk
= i , //
Agent k’s social graph
F : R 1 × N → R 1 × N .
// Graph solver - NeuPL (Chen et al. 2022)
Parameter : Π Nθ ∗ , { π ψ k 0 } Nk = i .
Output : { π ψ kT } Nk
= i , Π N , { Σ k } Nk
=
i
Algorithm Starts : 1: for n ∈ Population do 2: π ψ n 0 ← Π nθ ∗ // initialize N disjoint policies 3: end for 4: while (true do) do 5: for k ∈ Population do 6: Π Σ k ← { Π g
( a g | s ) } N g
= i //
Specialist k’s social graph with the Generalists 7:
NPL ( π ψ kτ , σ k , Π Σ k ) //
One-vs-all population learning with π ψ kτ 8: π ψ kτ = PPO clip ( gradientStep ( π ψ kτ )) //
Optimize policy with PPO optimization 9:
U k ← Eval ( π ψ kτ , { Π g } Ng = i ) //
Eval() computes the aggregate values of 10: // vertex k’s game outcomes.
11: Σ k ← F ( U k ) //
Define k’s social graph 12: end for 13: Π N = Π N U { π ψ kτ ∗ } Nk
= i //
Adding Specialists to opponent pool 14: // Iteratively specialize Specialists { π ψ kτ } Nk = i .
15: end while 16: return {
π ψ k } Nk
= i , Π N , { Σ k } Nk
=
i
With Section 3.3’s defined social graph, we use U k to denote the probabilistic outcome distribution of all pairwise game matches with Specialist k .
We use F ( U k ) to update k ’s social graph weighted edges to prioritize sampling of adversarial opponents.
After all Specialists have converged, we add the population into the opponent pool set.
M&M iteratively optimizes the set of Specialized population until population performance convergence.
A. S OFTWARE AND L ICENSING
The models are implemented via Tensorflow, TensorflowLite (Abadi et al., 2015), IMPALA(Espeholt et al., 2018), and Horovod(Sergeev & Balso, 2018).
These softwares are all licensed under Apache License 2.0.