Towards Effective and Interpretable Human-AI Collaboration in MOBA Games
Anonymous Author(s)
AffiliationAddress email
Abstract
MOBA games, e.g., Dota2 and Honor of Kings , have been actively used as thetestbed for the recent AI research on games, and various AI systems have beendeveloped at the human level so far.
However, these AI systems merely focus onhow to compete with humans, less exploring how to collaborate with humans.
Tothis end, this paper makes the first attempt to investigate human-AI collaboration in
MOBA games.
In this paper, we propose to enable humans and agents to collaboratethrough explicit communications by designing an efficient and interpretable M etaC ommand C ommunication-based framework, dubbed MCC, for accomplishingeffective human-AI collaboration in MOBA games.
The MCC framework consistsof two pivotal modules: 1) an interpretable communication protocol, i.e., the
Meta-Command, to bridge the communication gap between humans and agents;2) a meta-command value estimation model, i.e., the Meta-Command Selector,to select a valuable meta-command for each agent to achieve effective human-AIcollaboration.
Experimental results in Honor of Kings demonstrate that MCCagents can collaborate reasonably well with human teammates and even generalizeto collaborate with different levels and numbers of human teammates.
Videos areavailable at https://sites.google.com/view/mcc-demo.
Introduction
Games, as the microcosm of real-world problems, have been widely used as testbeds to evaluatethe performance of Artificial Intelligence (AI) techniques for decades.
Recently, many researchersfocus on developing various human-level AI systems for complex games, such as board games like
Go [23, 24], First-Person Shooting (FPS) games like ViZDoom [12], Real-Time Strategy (RTS)games like StarCraft 2 [30] and Multi-player Online Battle Arena (MOBA) games like Dota 2 [18].
However, these AI systems focus merely on how to compete instead of collaborating with humans,leaving Human-AI Collaboration (HAC) in complex environments still to be investigated.
In this paper, we study the HAC problem in complex MOBA games, which is characterized by multiagent cooperation and competition mechanisms, long time horizons, enormous state-action spaces( 10 20000 ), and imperfect information [18, 22, 34].
HAC requires the agent to collaborate reasonablywith various human teammates.
One straightforward approach is to improve the generalization ofagents, that is, to collaborate with an enough diverse population of teammates during training.
Thereare some Population-Based Training (PBT) based algorithms and learning systems [1, 2, 8, 9, 27, 37]proposed to improve the generalization of agents in video games by constructing a diverse populationof agents in different ways.
However, this approach requires a vast amount of diverse data andmassive computing resources, posing a big computational obstacle for complex MOBA games.
Human team success in MOBA games requires not only subtle individual micro-operations but alsoexcellent communications and collaborations among teammates on macro-strategies, i.e., long-termintentions [6, 33]. Consequently, we focus on enabling humans and agents to collaborate through
[Figure]
Dota
Honor ofKings
Macro-Strategy Meta [Figure] Command
[Figure]
Explicit Message
[Figure]
A
B
C
D
E
[Figure]
Where
[Figure]
How long : Within 20 Seconds < L, E, T mc >
What
Where
How long (b) explicit communications and propose an efficient and interpretable Meta-Command Communicationbased human-AI collaboration framework, dubbed MCC, to solve the HAC problem in MOBAgames.
First, we design an interpretable communication protocol, i.e., the Meta-Command, as ageneral representation of macro-strategies to bridge the communication gap between agents andhumans.
Both macro-strategies sent by humans and messages outputted by agents can be convertedinto unified meta-commands (see Figure 1).
Second, following Gao et al.
[6], we construct ahierarchical model that includes the command encoding network (macro-strategy layer) and themeta-command conditioned action network (micro-action layer), used for agents to generate andexecute meta-commands, respectively.
Third, we propose a meta-command value estimation model,i.e., the Meta-Command Selector, to select the optimal meta-command for each agent to execute.
The training process of the MCC framework consists of three phases.
We first train the commandencoding network to learn the distribution of meta-commands sent from humans.
Afterward, wetrain the meta-command conditioned action network to ensure that the agent has the near-humancompletion rate for meta-commands.
Finally, we train the meta-command selector to ensure that theagent can select a valuable meta-command to achieve effective collaboration.
We train and evaluatethe agent in Honor of Kings
5v5 mode with a full hero pool (over 100 heroes).
Experimental resultsdemonstrate the effectiveness of the MCC framework.
In general, our contributions are as follows:
• To the best of our knowledge, we are the first to investigate the HAC problem in MOBA games.
We
propose an efficient and interpretable Meta-Command Communication-based framework dubbedMCC to achieve effective human-AI collaboration in MOBA games.
• We design an interpretable communication protocol to bridge the communication gap between humans and agents.
In addition, we propose a meta-command value estimation model to select a valuable meta-command for each agent to achieve effective human-AI collaboration.
• We introduce the training process of the MCC framework in a typical MOBA game Honor of Kingsand evaluate it in practical human-AI game tests.
Experimental results show that MCC agents can reasonably collaborate with different levels and numbers of human teammates.
Related Work2.1 MOBA Games AI Research
MOBA games, such as Dota 2 and Honor of Kings , have attracted much attention from AI researchersdue to their multi-agent cooperative and competitive mechanics, long time horizons, partial observation, and enormous state-action spaces [18, 34].
Recently, OpenAI et al.
[18] introduced an AI systemnamed OpenAI-Five that defeated professional players in Dota 2 5v5 mode under the condition oflimited heroes.
Ye et al.
[34, 35, 36] proposed another learning system named WuKong that cansurpass top e-sport players in Honor of Kings with a full hero pool.
Further, Wu [33] and Gao etal.
[6] proposed learning systems that enable the agent to learn human strategies to achieve policydiversity.
However, these AI systems can only defeat human players but cannot collaborate well dueto the communication gap between agents and humans, see Table 1.
In most real-world scenarios, theexcellent collaboration between humans and agents may make more sense than the competition.
2.2 Human-AI Collaboration
PBT is considered one way to solve the HAC problem.
Most PBT-based methods are devoted totraining an agent which can be compatible with unseen partners by maintaining a population of agentswith diverse behaviors in different ways [1, 2, 8, 9, 27, 37].
These methods have been validatedon objective and subjective metrics [26] in video games Overcooked and Capture the Flag andcard game Hanabi .
However, these tasks do not consider long-term macro-strategies planning andcollaborating, thus it is far simpler than MOBA games.
Besides, MOBA AI systems usually need tolearn billions of network parameters to cope with the enormous state-action spaces ( 10 20000 ) [34],which constitutes a prohibitive computational burden for learning.
As a more realistic topic of HAC, human-robot interaction in manufacturing is of great potential and also attracts much attention[11, 15, 21].
However, these studies are mainly limited to collaboration between a robot and a humanthrough one-way communication, i.e., humans give robots orders.
Therefore, there is still a largeroom to study RL with the participation of humans.
This work can be a stepping stone for broaderreal-world applications.
2.3 Multi-Agent Communication
Communication is often used in Multi-Agent Reinforcement Learning (MARL) to improve interagent collaboration.
Most communication-based MARL methods are mainly focused on exploringcommunication protocols between multiple agents with an end-to-end RL framework [4, 5, 7, 19,25, 28, 32].
Jiang and Lu [10] and Kim et al.
[13] proposed to model the value of multi-agentcommunication for effective collaboration.
Unfortunately, these methods all model communicationsin a latent space without considering human-AI interactions, making it less interpretable to humans.
Instead, we focus on enabling humans and agents to collaborate through explicit communications.
Human-AI Collaboration
We consider an interpretable communicative human-AI collaboration task, which can be extended from Partially Observable Markov Decision Process (POMDP) and formulated as a tuple< N, H, S , A N , A H , O , M , r, P, γ > , where N and H represent the numbers of agents and humans,respectively.
S is the space of global states.
A N = { A Ni } i =1 ,...,N and A H =
{ A Hi } i =1 ,...,H denotethe spaces of actions of N agents and H humans, respectively.
O =
{ O i } i =1 ,...,N + H denotesthe space of observations of N agents and H humans.
M represents the space of interpretablemessages, that is, the Meta-Commands in the MCC framework.
P : S
×
A N × A H → S andr : S ×
A N × A H → R denote the shared state transition probability function and reward functionof N agents, respectively.
Note that, r includes both individual reward and team reward.
γ ∈ [0 , 1)denotes the discount factor.
For each agent i in state s t ∈ S , it receives an observation o it ∈
O iand a selected message c it ∈ M , and then outputs an action a it =
π θ ( o it , c it ) ∈
A Ni and a newmessage m it +
=
π ϕ
( o it ) ∈ M , where π θ and π ϕ are action network and message encoding network, respectively.
A message selector c it = π ω ( o it , C t ) is introduced to receive a message set
C t = { m it } i =1 ,...,N + H ⊂ M from all agents and humans and select the optimal one to execute.
We divide the HAC problem in MOBA games into the Human-to-AI (H2A) and the AI-to-Human(A2H) scenarios.
The H2A Scenario: Humans send macro-strategies as messages to agent teammates,and agents combine them with their own messages to select the optimal one based on their ownmessage selector to execute, achieving effective collaboration with humans.
The A2H Scenario:
Agents send messages as macro-strategies to human teammates, and humans combine them withtheir own macro-strategies to select the optimal one based on their own value systems to execute,achieving effective collaboration with agents.
The goal of both tasks is that agents and humanscommunicate macro-strategies with pre-defined communication protocols, and then select valuablemacro-strategies for effective collaboration to win the game.
Meta-Command Communication-Based Framework
In this section, we present the proposed MCC framework in detail.
We first briefly describe three keystages of the MCC framework (see Section 4.1).
Then we introduce the two pivotal modules in the
MCC framework: 1) an interpretable communication protocol, i.e., the Meta-Command, as a generalrepresentation of macro-strategies to bridge the communication gap between agents and humans (see
Section 4.2); 2) a meta-command value estimation model, i.e., the Meta-Command Selector, to selecta valuable meta-command for each agent to achieve effective HAC in MOBA games(see Section 4.3).
Meta-Command Selector 𝜋 !
(II) Meta-Command Communication Stage 𝐶 𝑡 𝑜 𝑡 𝑐 𝑡
CommandConverter
Command Encoding Network 𝜋 " (I) Meta-Command ConversionStage 𝑜 𝑡 𝑚 𝑡 𝑜 !"
text signal 𝑚 !"
①②③ 𝑚 𝑡 𝑚
#$% 𝑚 #$& 𝑠 𝑡
Meta-Command Conditioned Action Network 𝜋 ’ (III) Human-AI Collaboration Stage 𝑎 𝑡 𝑐 𝑡 𝑜 𝑡 𝐶 𝑡 = {𝑚 !" ,𝑚 𝑡 } 𝑐 𝑡 𝑎 𝑡 𝑟 𝑡
III
III 𝑐 𝑡 𝑠 !
#$ 𝑎 !
#$ 𝑟 !
#$
III 𝑠 % 𝑎 % 𝑟 %
II
I 𝑅 !&’ … …
𝑠 %#$ 𝑎 %#$ 𝑟 %#$ 𝑅 %&’ … …
Step: t t+n
T
T+n
Environment ① ② ③ … … … 𝑐 𝑇 𝐶 𝑇 = {𝑚 %" ,𝑚 𝑇 } 𝑐 𝑇
4.1 Overview
The flow of the MCC framework can be divided into three stages: the meta-command conversion stage,the meta-command communication stage, and the human-AI collaboration stage, as plotted in Figure 2.
At the Meta-Command Conversion Stage , the MCC framework converts the macro-strategies sentby humans and the messages outputted by the command encoding network of agents into unifiedmeta-commands and then broadcasts them to all agents and humans.
At the Meta-Command
Communication Stage , the MCC framework uses the meta-command selector to estimate the valuesof all received meta-commands and select the optimal one for each agent to execute.
Note thathumans also select the optimal meta-command based on their value systems.
At the Human-AI
Collaboration Stage , the MCC framework adopts the meta-command conditioned action network topredict a sequence of actions for each agent to perform based on its selected meta-command.
Foreach game, humans and agents have to collaborate multiple times, that is, they need to perform theabove three stages multiple times to win the game.
4.2 Meta-Command
In MOBA games, we propose that a macro-strategy consists of three components: where to go, whatto do, and how long.
For example, a macro-strategy can be Come And Kill The Dragon , whichconsists of Come To The Dragon (where to go), Attack The Dragon (what to do), and Until The Dragon Is Killed (how long).
Thus, we propose a general representation of macro-strategies, i.e.,the Meta-Command, as an interpretable communication protocol to bridge the communication gapbetween agents and humans.
Meta-Command Definition.
We formulate the Meta-Command as a tuple <
L, E, T mc > , as shownin Figure 1(b), where L is the Location to go, E is the Event to do after reaching L , and T mc is the
Time Limit for executing the meta-command.
Among them, L is the key to the meta-command, whichcontains the intention of the macro-strategy.
E can be thought of as human micro-operation, which isimplemented through a pre-trained micro-action network π θ in the MCC framework.
T mc can be setto how long it normally takes a human to complete a macro-strategy in MOBA games, usually 20seconds corresponds to 80% completion rate for meta-commands, see Appendix A.12.1.
Meta-Command Conversion.
To realize interpretable human-AI communication, we convert theexplicit messages from humans and the implicit messages from agents into unified meta-commands.
To achieve the former, a hand-crafted command converter function f cc is used to generate L of metacommands by extracting the location from explicit messages, such as text and signals, sent by humans. To achieve the latter, we use a Command Encoding Network (CEN) π ϕ ( m | o ) to generate L of metacommands. The CEN is trained via supervised learning (SL) with the goal of learning the distributionof meta-commands sent from humans, as shown in Figure 3(a)(I).
The training dataset { < o, m > }is obtained by extracting the observation o and its corresponding meta-command m from expert data.
[Figure]
𝑜 𝑡 𝜙 𝜋 !
(𝑚 𝑡 |𝑜 " ) <𝑜 𝑡 ,𝑚 " >
Expert
[Figure]
SL
Meta-Command Selector (CS)
Meta-Command Conditioned Action Network (MCCAN) 𝜃 𝑜 𝑡 𝜙 𝑚 𝑡 𝑉 "#$" (𝑜 " ,𝑚 𝑡 )
𝜋 % (𝑎 𝑡 |𝑜 " ,𝑚 " )
𝑉 "&’" (𝑜 " ) 𝐺 " sample
RL 𝜙 𝑜 𝑡 𝑜 "() 𝑜 "(* 𝑚 𝑡 𝑚 "() 𝑚 "(* 𝜔 𝐶 𝑡 𝑄 𝑡 (𝑜 " ,𝐶 " ,𝑚 " ) 𝑄 "() (𝑜 " ,𝐶 " ,𝑚 "() ) 𝑄 "(* (𝑜 " ,𝐶 " ,𝑚 "(* ) 𝑉 ",- (𝑜 " ,𝐶 𝑡 ) 𝜃 𝐺 ",- RL 𝑐 𝑡 =softmax , 𝑄 𝑡 (𝑜 " ,𝐶 " ,𝑚)
𝑎 𝑡 𝐺 "&’" 𝐺 "#$"
[Figure]
pre [Figure] trained pre
[Figure]
MLPs
Shared CNNs State Embed Map Embed Map Embed MLP
[Figure]
Map Set Embed MLP
Fused Embed key [Figure] query query [Figure]
Q or [Figure] Observation [Figure]
[Figure] Command Q or 𝛑 [Figure]
(ℎ) [Figure] (ℎ, [Figure]
# ) 𝑚 . [Figure] After converting all messages into unified meta-commands, the MCC framework broadcasts them toall agents and humans.
Then, agents and humans receive an identical meta-command candidate set.
Meta-Command Execution.
After receiving a meta-command candidate set, agents can select one meta-command from it to execute.
We adopt a Meta-Command Conditioned Action Network (MCCAN)
π θ ( a | o, m ) for agents to perform actions based on the selected metacommand, as shown in Figure 3(a)(II).
The MCCAN is trained via goal-conditioned RL withthe goal of achieving a near-human completion rate for the meta-commands generated by thepre-trained CEN while ensuring that the win rate is not reduced.
We adopt an intrinsic rewardr intt ( s t , m t , s t +1 ) =
(cid:12)(cid:12) f
ce ( s t ) − m t (cid:12)(cid:12) − (cid:12)(cid:12)
f
ce ( s t +1 ) − m t (cid:12)(cid:12) to guide the process of executing the metacommand m t , where f ce is a hand-crafted command extraction function.
We train the MCCANwith the objective of maximizing the expectation over extrinsic and intrinsic discounted total rewards G t =
E s ∼ d πθ ,a ∼ π θ (cid:104)(cid:80) ∞ i =
γ i r t
+ i + α (cid:80) T mc j =0 γ j r intt + j (cid:105) , where α is a trade-off parameter andd π ( s ) =
lim t →∞ P (cid:0) s t
= s
| s 0 , π (cid:1) is the probability when following π for t steps from s 0 .
After training the CEN and MCCAN, we can achieve HAC by simply setting an agent to randomlyselect a meta-command derived from humans to execute.
However, such collaboration is nonintelligent and can even be a disaster for game victory because agents have no mechanism tomodel the values of meta-commands and cannot choose the optimal meta-command to execute.
While humans usually choose the optimal one based on their value systems for achieving effectivecollaboration to win the game.
Thus, we further propose a meta-command value estimation model toselect a valuable meta-command for each agent, as described in the following subsection.
4.3 Meta-Command Selector
In real-world MOBA games, the same macro-strategy often has different values for different humansin different situations.
For example, a macro-strategy can be Come And Kill The Dragon , as shown in
Figure 1(b).
It is more valuable for humans A and B to collaborate.
While another macro-strategy canbe Clean Up Top-Lane Minions , which is more valuable for human C rather than humans A and B.
Therefore, it is important to select the most valuable meta-command from the received meta-commandcandidate set C to achieve effective human-AI collaboration.
We propose a meta-command valueestimation model, i.e., the Meta-Command Selector (CS) π ω
( o, C ) , to estimate the values of allcurrent meta-commands and select the most valuable one for each agent to execute.
CS Optimization Objective.
Typically, the execution of a meta-command involves reaching location
L and doing event E , of which the latter is more important to the value of the meta-command.
For example, for the meta-command Come And Kill The Dragon , if Kill The Dragon event cannotbe done within T mc time steps, then it is pointless to Come To The Dragon .
Thus, the long-termreward R mc for executing a meta-command can be expressed as the total rewards within T mc timesteps by interacting with the environment: R mct = (cid:80) T L i =
r t
+ i + β (cid:80) T mc j = T L r t + j , where T L < T mcis the time for reaching L and β > 1 is a trade-off parameter.
Note that the reward function r includes both individual rewards and team rewards.
The optimization objective of CS is to selectthe optimal meta-command m ∗ t =
π ω
( o t , C t ) for each agent to maximize the expected discounted
[Equation]
-command execution return G
[Equation]
[-θ (cid:104)(cid:80) ∞ i =
 γ imc R mct + i · T mc (cid:105)-] , where o t ∈ O ,
C t is the meta-command candidate set in state s t , and γ mc ∈ [0 , 1) is the discount factor.
CS Training Process.
We construct a self-play training environment for CS where agents can sendmessages to each other.
Specifically, three tricks in Figure 3(a)(III) are adopted to increase thesample efficiency while ensuring efficient exploration.
First, each sent meta-command m is sampledwith the argmax rule from the results predicted by the pre-trained CEN.
Second, each agent sendsits meta-command with a probability p every T mc time steps.
Finally, each agent selects the finalmeta-command c sampled with the softmax rule from its CS output results and hands it over to thepre-trained MCCAN for execution.
We use the multi-head value mechanism [34] to model the valueof the meta-command execution, and the corresponding value loss can be formulated as:
[Equation]
212 where V kω ( S, C ) is the value of the k -th head.
For DQN-based methods [17, 29, 31], the Q loss is:
[Equation]
214 where w k is the weight of the k -th head and G mck is the Temporal Difference (TD) estimated value
[Equation]
kω ( S, C ) .
CS Model Structure.
We design a general network structure for CS towards MOBA games, asshown in Figure 3(b).
In MOBA games, the meta-commands corresponding to adjacent regionsusually have similar values.
Thus, we divide the meta-commands in the map into grids, a commonlocation description for MOBA games, and use the shared Convolutional Neural Network (CNN) toextract region-related information from the meta-commands to improve the generalization of CS toadjacent meta-commands.
Besides, we use the gating mechanism [16] to fuse the map embeddingof all received meta-commands and the state embedding of the observation information.
Finally, todirectly construct the relationship between the observation information and each meta-command, weintroduce a target attention module, where the query is the fused embedding h and the key is themap embedding m ′ of each meta-command.
The fused embedding h is used as the input into thesubsequent Q network Q ( h, m ′ ) and V network V ( h ) network of CS.
In this way, the Q networkcan also be easily converted to the policy network π
( m | h, m ′ ) .
Thus, the CS model structure can beeasily applied to most popular RL algorithms, such as PPO [20], DQN [17], etc.
Experiments
We evaluate the proposed MCC framework in Honor of Kings , one of the most popular MOBA gamesworldwide, which has been actively used as the testbed for recent game AI research [6, 33–36].
Weconduct all experiments in Honor of Kings 5v5 mode with a full hero pool (over 100 heroes), exceptablation studies with a 20 hero pool for exploring the influence of different model components moresufficiently and efficiently.
5.1 Experimental Setup
5.1.1 Training Setup
Due to the complexity of MOBA games and limited resources, instead of training jointly, we train the
CEN, MCCAN, and CS sequentially.
For all model training, the location L of meta-commands in themap is divided into 144 grids.
The time limit T mc for the meta-command execution is set to 20s.
CEN Training Settings.
We train the CEN via SL until it converges for 26 hours using 8 NVIDIA
P40 GPUs.
The batch size of each GPU is set to 512.
Adam[14] is adopted as the optimizer with aninitial learning rate of 0.0001.
{+MCCAN Training Settings. We train the MCCAN by finetuning a pre-trained micro-action network [34], the state-of-the-art (SOTA) model in Honor of Kings , which is conditioned on themeta-command sampled from the pre-trained CEN. The MCCAN is trained until it converges for 
+} VS
Agent-Only Environment
[Figure]
SimulatedHuman
Fixed Opponents
BroadcastChannel
VS
Train Environment
BroadcastChannel (a) … … … …
VS
Fixed Opponents
BroadcastChannel …
+
Human-AI Game Test (c)
VS
BroadcastChannel … …
Test I
Test II p … … … …
BroadcastChannel … …
CS …
[Figure]
hours using a physical computer cluster with 63,000 CPUs and [-
-]560 NVIDIA V100 GPUs.
The batchsize of each GPU is set to 256.
CS Training Settings.
We train the CS via self-play until it converges for 24 hours using a physicalcomputer cluster with 70,000 CPUs and 680 NVIDIA V100 GPUs.
The batch size of each GPU isset to 256.
The parameter β is set to 2.
Each agent sends a meta-command with a probability p of 0.8and an interval T mc of 20s, as shown in Figure 4(a).
5.1.2 Evaluating Setup
Our primary concern is whether the agents trained with the MCC framework, briefly called the MCCagents, can collaborate with humans well.
However, evaluating agents with humans is expensive,which is not conducive to model selection and iteration.
Therefore, we built two agent-only testingenvironments:
Test I and Test II, for the model selection and iteration process, as shown in Figure 4(b).
We also evaluate the MCC agents in practical human-AI game tests to examine the performance ofcollaborating with humans, as shown in Figure 4(c).
Compared Agents.
We
Agent-Only Environmental Settings.
Test I is the most complex environment where all agentteammates can send and receive meta-commands simultaneously with an interval of 20s.
Test I isused to evaluate the agents’ performance under extremely complex situations as well as in ablationstudies.
Test II is a simple environment to simulate practical game scenarios, where at most onehuman sends his macro-strategy at a time step.
Thus, in Test II, only one agent is randomly selectedto send its meta-command with an interval of 20s, and the other agents only receive meta-commands.
Human-AI Game Testing Settings.
We had different types of agents team up with different levelsand numbers of humans, including 15 strong humans (top1%) and 15 average humans (top30%),in m AI + n Human mode, where m + n = 5 .
For fair comparisons, each tester was not told the type of agent teammates.
To eliminate the effects of collaboration between agents, we prohibitagents from receiving meta-commands from their agent teammates, and the agent can only receivemeta-commands from humans.
In each game test, humans can send the converted meta-commandswhenever they think their macro-strategies are important.
To make the agent behave like humans(at most one human sends his macro-strategy at a time step), we restrict agents from sending theirmeta-commands.
We randomly choose a human teammate and use his observation and all agents’meta-commands as the CS input and select the final output of CS to send with an interval of 20s.
5.2 Results in Agent-Only Environment
5.2.1 AI Performance
The Kullback-Leibler (KL) divergence of the meta-command distribution between the CEN andhumans decreased from 4.96 to 0.44 as training converges.
The MCCAN is trained with the parameterα equal to 16.
The win rate of the trained agent against the SOTA agent [6, 34] is close to 50%.
The average completion rates of the trained agent and humans for meta-commands are 82% and80%, respectively.
Notably, we can train an agent with a higher completion rate by increasing α , butthis will significantly reduce the win rate because the meta-command executed is not necessarilyoptimal and may result in the death of agents.
We put the detailed experimental results of the CENand MCCAN in the Appendix A.10.1 and A.10.2 due to space limitations.
Figure 5(a) and (b) show the win rates of four types of agents who play against each other for 600matches in Test I and Test II, respectively.
We see that the MCC agent achieves the highest win rateagainst all the other agents in both testing environments, indicating that the CS can select a valuablemeta-command for each agent to collaborate, and such reasonable collaboration is conducive towinning the game.
The MC-Rand and MC-Rule agents are worse than the MC-Base agent, confirmingthat agents executing low-value meta-commands can hurt performance.
Notably, we find that the winrates of the MCC agent in Test I and Test II are close, suggesting that the MCC agent can generalizeto different numbers of meta-commands.
Figure 5(c) demonstrates the final Elo scores [3] of theseagents.
It clearly shows the effectiveness of CS in agent-only collaboration scenarios. [Figure]
5.2.2 Ablation Studies
[Figure]
We further investigatethe influence of differentcomponents, including
CNN feature extractionwith the gating mechanism(w/o
CNN-GM), targetattention module (w/o
TA), and PPO optimizationalgorithm
(MCC-PPO),on the performance of
CS.
We conduct ablationstudies in Test I with a20 hero pool.
In practicalgames, meta-commandswith adjacent regions often have similar intentions and values.
Thus the response rate of the agent toadjacent meta-commands should be as close as possible.
Besides, the higher the agent’s response rateto meta-commands, the more collaborative behaviors of the agent, thus we expect the response rateof CS as high as possible.
Generally, we expect the Response Rate (RR) of CS as high as possiblewhile ensuring that the Win Rate (WR) is not reduced.
Figure 6(a) demonstrates the WR of different CS ablation versions during the training process, and
Figure 6(b) shows the converged WR-RR results.
We see that after ablating the TA module, the WRand RR of CS are greatly reduced, indicating that the TA module can improve the accuracy of CSto meta-commands.
Besides, after ablating the CNN-GM module, the RR of CS is most affected,which is reduced by 20%.
It indicates that without the CNN-GM module, the value estimation of CSto adjacent meta-commands is not accurate enough, resulting in missing some actual high valuablemeta-commands.
We notice that the MCC and MCC-PPO in both metrics are close, confirming theversatility of the CS model structure.
[Table] [Table]
[Equation]
[Table]
5.3 Results in Human-AI Game Test
Due to space limitations, we only show the objective results in 4 AI + 1 Human mode.
Other modesresults and the subjective preference results of testers can be found in the Appendix A.10.3 and A.11.
Table 1 shows the WR of different human-AI teams who play against the MC-Base agent-only team.
We see that the MCC agent significantly outperforms other agents, regardless of whether they pairwith a strong or average human.
To explain why humans have a higher WR when paired with the MCC agents, we count the RR of agents to the meta-commands sent from human teammates (H2Ascenarios) and the RR of humans to the meta-commands sent from agent teammates (A2H scenarios),respectively, as shown in Table 2.
In H2A scenarios, the RRs of the MCC agents to average humansand strong humans are 61.17% and 73.05%, respectively, indicating that the MCC agents are morewilling to respond to valuable meta-commands sent from strong humans.
We also notice that the RRof the MCC agents to strong humans (73.05%) is very close to the RR of strong humans themselves(74.91%), suggesting that the CS is close to the value system of strong humans.
In A2H scenarios, the
RRs of average humans and strong humans to the MCC agents are 73.43% and 78.5%, respectively,which is significantly higher than that of MC-Rand agents (41.07% and 35.69%), indicating that themeta-commands sent from the MCC agents are more valuable and reasonable to humans.
Note thatthe RR of the MCC agents to the MC-Rand agents is 34.03%, which is close to that of strong humans(35.69%), once again confirming that the CS is close to the value system of strong humans.
We also visualize the comparison of CS and strong human value systems on a game scene with threemeta-commands existing, as shown in Figure 7.
We see that the CS selects the meta-command B forthe two heroes in the red dashed box to collaborate, selects the meta-command C for the two heroesin the purple dashed box to collaborate, and selects the meta-command A for the remaining heroto execute alone.
The CS selection results are consistent with the ranking results of strong humans,confirming the effectiveness of CS and the interpretability of the collaboration behavior between MCC agents and humans.
Conclusion
In this paper, we proposed an efficient and interpretable Meta-Command Communication-basedframework, dubbed MCC, to achieve effective human-AI collaboration in MOBA games.
To bridgethe communication gap between humans and agents, we designed an interpretable communicationprotocol, i.e., the Meta-Command, to convert the explicit messages from humans and the implicitmessages from agents into unified meta-commands.
To achieve effective collaboration, we constructeda meta-command value estimation model, i.e., the Meta-Command Selector, to select a valuablemeta-command for each agent to execute.
Finally, we introduced the training process of the MCCframework and conducted practical human-AI game tests in the typical MOBA game Honor of
Kings .
The experimental results show that the MCC agents can collaborate reasonably with humanteammates and even generalize to collaborate with different levels and numbers of human teammates.
We expect this work can be a foundation for future HAC research in complex environments.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See the abstract and the contributions in Section 1.
(b) Did you describe the limitations of your work?
[Yes] For margin reasons, we will discuss limitations here.
We have currently only verified the effectiveness of the MCC framework in MOBA games, and we will explore in more types of complex games, such as First-Person Shooting (FPS) and Massively Multiplayer Online (MMO) in the
future. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] The purpose of our method is only for academic research based on game environments, e.g., investigating the MOBA game-playing problems.
Like AlphaGo or AlphaStar,the potential negative societal impacts of our work will be limited to the development of gaming AI applications.
However, our research will contribute to the research community, the game industry, and the e-sports community.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The code and
the data are proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See the full experimental setup in Section 5.1.1 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] All results are reported over five random seeds (see Section 5.1.2).
For the convenience of presentation, the median is shown in AI Performance , while detailed error bars are presented in Ablation Studies .
Due to the high
cost of Human-AI Game Test , we only used the median model for testing.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the resources in Section
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] Honor of Kings is a released and recognized "testbed" for
MOBA-game-playing problems [36, 35, 34, 6, 33].
We contact the relevant author and
obtain authorization from the game provider.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Appendix.
All data in this paper is game related and has nothing to do with identity information.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable
[Yes] See Appendix.
In the Human- AI Game Test , all participants have more than three years of experience in
Honor of Kings and are familiar with all
theinformation in the game.
Before
the
test
we
also
inform the participants of the detailedtest instructions,
and
the participants voluntarily
choose whether to participate in
the
test.
We have detailed ethics descriptions in Appendix A.9. As stated in Section A.9.2,participants were given instructions before testing. As mentioned in point 5 (Line87-88) , participants’ game statistics will be only used for academic research, andparticipants can choose whether to participate or not.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes] We had performed a process similar to
IRB before the test was conducted in this paper.
The institution and all participantshave approved our research.
First, we analyze the risks of these experiments to theparticipants.
The risks mainly include the leakage of identity information and the timecost.
Then, a series of measures are implemented to prevent these risks in Appendix
A.9.2.
We make a risk statement for participants and sign an identity informationconfidentiality agreement.
We only use information related to the game state in ourresearch without identity information.
In addition, special equipment and accounts areprovided to the participants to prevent leakage of equipment and account informationduring the test.
The identity information of all participants is not disclosed to thepublic.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] In the Human-AI Game Test , participants
can get 5 dollars for each match and each match is about 10 to 20 minutes.