Towards Effective and Interpretable Human-AI Collaboration in MOBA Games
Anonymous Author(s)
Abstract
MOBA games, e.g., Dota2 and Honor of Kings , have been actively used as thetestbed for the recent AI research on games, and various AI systems have beendeveloped at the human level so far.
However, these AI systems merely focus onhow to compete with humans, less exploring how to collaborate with humans.
Tothis end, this paper makes the first attempt to investigate human-AI collaboration in
MOBA games.
In this paper, we propose to enable humans and agents to collaboratethrough explicit communications by designing an efficient and interpretable M etaC ommand C ommunication-based framework, dubbed MCC, for accomplishingeffective human-AI collaboration in MOBA games.
The MCC framework consistsof two pivotal modules: 1) an interpretable communication protocol, i.e., the
Meta-Command, to bridge the communication gap between humans and agents;2) a meta-command value estimation model, i.e., the Meta-Command Selector,to select a valuable meta-command for each agent to achieve effective human-AIcollaboration.
Experimental results in Honor of Kings demonstrate that MCCagents can collaborate reasonably well with human teammates and even generalizeto collaborate with different levels and numbers of human teammates.
Videos areavailable at https://sites.google.com/view/mcc-demo.
Introduction
Games, as the microcosm of real-world problems, have been widely used as testbeds to evaluatethe performance of Artificial Intelligence (AI) techniques for decades.
Recently, many researchersfocus on developing various human-level AI systems for complex games, such as board games like
Go [23, 24], First-Person Shooting (FPS) games like ViZDoom [12], Real-Time Strategy (RTS)games like StarCraft 2 [29] and Multi-player Online Battle Arena (MOBA) games like Dota 2 [18].
However, these AI systems focus merely on how to compete instead of collaborating with humans,leaving Human-AI Collaboration (HAC) in complex environments still to be investigated.
paper, we study the HAC problem in complex MOBA games, which is characterized by multiagent cooperation and competition mechanisms, long time horizons, enormous state-action spaces( 10 20000 ), and imperfect information [18, 22, 33].
HAC requires the agent to collaborate reasonablywith various human teammates.
One straightforward approach is to improve the generalization ofagents, that is, to collaborate with an enough diverse population of teammates during training.
Thereare some Population-Based Training (PBT) based algorithms and learning systems [1, 2, 8, 9, 26, 36]proposed to improve the generalization of agents in video games by constructing a diverse populationof agents in different ways.
However, this approach requires a vast amount of diverse data andmassive computing resources, posing a big computational obstacle for complex MOBA games.
Human team success in MOBA games requires not only subtle individual micro-operations but alsoexcellent communications and collaborations among teammates on macro-strategies, i.e., long-term
[Figure]
Dota
Honor ofKings
Macro
[Figure]
Explicit Message
[Figure]
A
[Figure]
C
D
E
[Figure]
How
[Figure]
: Within 20 Seconds < L, E, T mc >
What
Where
How long (b) intentions [6, 32]. Consequently, we focus on enabling humans and agents to collaborate throughexplicit communications and propose an efficient and interpretable Meta-Command Communicationbased human-AI collaboration framework, dubbed MCC, to solve the HAC problem in MOBAgames.
First, we design an interpretable communication protocol, i.e., the Meta-Command, as ageneral representation of macro-strategies to bridge the communication gap between agents andhumans.
Both macro-strategies sent by humans and messages outputted by agents can be convertedinto unified meta-commands (see Figure 1).
Second, following Gao et al.
[6], we construct ahierarchical model that includes the command encoding network (macro-strategy layer) and themeta-command conditioned action network (micro-action layer), used for agents to generate andexecute meta-commands, respectively.
Third, we propose a meta-command value estimation model,i.e., the Meta-Command Selector, to select the optimal meta-command for each agent to execute.
The training process of the MCC framework consists of three phases.
We first train the commandencoding network to learn the distribution of meta-commands sent from humans.
Afterward, wetrain the meta-command conditioned action network to ensure that the agent has the near-humancompletion rate for meta-commands.
Finally, we train the meta-command selector to ensure that theagent can select a valuable meta-command to achieve effective collaboration.
We train and evaluatethe agent in Honor of Kings
5v5 mode with a full hero pool (over 100 heroes).
Experimental resultsdemonstrate the effectiveness of the MCC framework.
In general, our contributions are as follows:
• To the best of our knowledge, we are the first to investigate the HAC problem in MOBA games.
We
propose an efficient and interpretable Meta-Command Communication-based framework dubbedMCC to achieve effective human-AI collaboration in MOBA games.
• We design an interpretable communication protocol to bridge the communication gap betweenhumans and agents.
In addition, we propose a meta-command value estimation model to select avaluable meta-command for each agent to achieve effective human-AI collaboration.•
We introduce the training process of the MCC framework in a typical MOBA game Honor of Kingsand evaluate it in practical human-AI game tests.
Experimental results show that MCC agents canreasonably collaborate with different levels and numbers of human teammates.
Related Work2.1 MOBA Games AI Research
MOBA games, such as Dota 2 and Honor of Kings , have attracted much attention from AI researchersdue to their multi-agent cooperative and competitive mechanics, long time horizons, partial observation, and enormous state-action spaces [18, 33].
Recently, OpenAI et al.
[18] introduced an AI systemnamed OpenAI-Five that defeated professional players in Dota 2 5v5 mode under the condition oflimited heroes.
Ye et al.
[33, 34, 35] proposed another learning system named WuKong that cansurpass top e-sport players in Honor of Kings with a full hero pool.
Further, Wu [32] and Gao etal.
[6] proposed learning systems that enable the agent to learn human strategies to achieve policy diversity.
However, these AI systems can only defeat human players but cannot collaborate well dueto the communication gap between agents and humans.
In most real-world scenarios, the excellentcollaboration between humans and agents may make more sense than the competition.
2.2 Human-AI Collaboration
PBT is considered one way to solve the HAC problem.
Most PBT-based methods are devoted totraining an agent which can be compatible with unseen partners by maintaining a population of agentswith diverse behaviors in different ways [1, 2, 8, 9, 26, 36].
These methods have been validated invideo games Overcooked and Capture the Flag and card game Hanabi .
However, these tasks do notconsider long-term macro-strategies planning and collaborating, thus far simpler than MOBA games.
Besides, MOBA AI systems usually need to learn billions of network parameters to cope with theenormous state-action spaces ( 10 20000 ) [33], which constitutes a prohibitive computational burdenfor learning.
As a more realistic topic of HAC, human-robot interaction in manufacturing is of greatpotential and also attracts much attention [11, 15, 21].
However, these studies are mainly limitedto collaboration between a robot and a human through one-way communication, i.e., humans giverobots orders.
Therefore, there is still a large room to study RL with the participation of humans.
This work can be a stepping stone for broader real-world applications.
2.3 Multi-Agent Communication
Communication is often used in Multi-Agent Reinforcement Learning (MARL) to improve interagent collaboration.
Most communication-based MARL methods are mainly focused on exploringcommunication protocols between multiple agents with an end-to-end RL framework [4, 5, 7, 19,25, 27, 31].
Jiang and Lu [10] and Kim et al.
[13] proposed to model the value of multi-agentcommunication for effective collaboration.
Unfortunately, these methods all model communicationsin a latent space without considering human-AI interactions, making it less interpretable to humans.
Instead, we focus on enabling humans and agents to collaborate through explicit communications.
Human-AI Collaboration
We consider an interpretable communicative human-AI collaboration task, which can be extended from Partially Observable Markov Decision Process (POMDP) and formulated as a tuple< N, H, S , A N , A H , O , M , r, P, γ > , where N and H represent the numbers of agents and humans,respectively.
S is the space of global states.
A N = { A Ni } i =1 ,...,N and A H =
{ A Hi } i =1 ,...,H denotethe spaces of actions of N agents and H humans, respectively.
O =
{ O i } i =1 ,...,N + H denotesthe space of observations of N agents and H humans.
M represents the space of interpretablemessages, that is, the Meta-Commands in the MCC framework.
P : S
×
A N × A H → S andr : S ×
A N × A H → R denote the shared state transition probability function and reward functionof N agents, respectively.
Note that, r includes both individual reward and team reward.
γ ∈ [0 , 1)denotes the discount factor.
For each agent i in state s t ∈ S , it receives an observation o it ∈
O iand a selected message c it ∈ M , and then outputs an action a it =
π θ ( o it , c it ) ∈
A Ni and a newmessage m it +
=
π ϕ
( o it ) ∈ M , where π θ and π ϕ are action network and message encoding network, respectively.
A message selector c it = π ω ( o it , C t ) is introduced to receive a message set
C t = { m it } i =1 ,...,N + H ⊂ M from all agents and humans and select the optimal one to execute.
The HAC problem in MOBA games can generally be divided into the Human-to-AI (H2A) and the AI-to-Human (A2H) scenarios.
The H2A Scenario: Humans send macro-strategies as messagesto agent teammates, and agents combine them with their own messages to select the optimal onebased on their own message selector to execute, achieving the collaboration with humans.
The A2H
Scenario: Agents send messages as macro-strategies to human teammates, and humans combinethem with their own macro-strategies to select the optimal one based on their own value systems toexecute, achieving the collaboration with agents.
The goal of both tasks is that agents and humanscommunicate macro-strategies with pre-defined communication protocols, and then select valuablemacro-strategies for effective collaboration to win the game.
Meta-Command Communication-Based Framework
In this section, we present the proposed MCC framework in detail.
We first briefly describe three keystages of the MCC framework (see Section 4.1).
Then we introduce the two pivotal modules in the
Meta-Command Selector 𝜋 !
(II) Meta-Command Communication Stage 𝐶 𝑡 𝑜 𝑡 𝑐 𝑡
CommandConverter
Command Encoding Network 𝜋 " (I) Meta-Command ConversionStage 𝑜 𝑡 𝑚 𝑡 𝑜 !"
text signal 𝑚 !"
①②③ 𝑚 𝑡 𝑚
#$% 𝑚 #$& 𝑠 𝑡
Meta-Command Conditioned Action Network 𝜋 ’ (III) Human-AI Collaboration Stage 𝑎 𝑡 𝑐 𝑡 𝑜 𝑡 𝐶 𝑡 = {𝑚 !" ,𝑚 𝑡 } 𝑐 𝑡 𝑎 𝑡 𝑟 𝑡
III
III 𝑐 𝑡 𝑠 !
#$ 𝑎 !
#$ 𝑟 !
#$
III 𝑠 % 𝑎 % 𝑟 %
II
I 𝑅 !&’ … …
𝑠 %#$ 𝑎 %#$ 𝑟 %#$ 𝑅 %&’ … …
Step: t t+n
T
T+n
Environment ① ② ③ … … … 𝑐 𝑇 𝐶 𝑇 = {𝑚 %" ,𝑚 𝑇 } 𝑐 𝑇
MCC framework: 1) an interpretable communication protocol, i.e., the Meta-Command, as a generalrepresentation of macro-strategies to bridge the communication gap between agents and humans (see
Section 4.2); 2) a meta-command value estimation model, i.e., the Meta-Command Selector, to selecta valuable meta-command for each agent to achieve effective HAC in MOBA games(see Section 4.3).
4.1 Overview
The flow of the MCC framework can be divided into three stages: the meta-command conversion stage,the meta-command communication stage, and the human-AI collaboration stage, as plotted in Figure 2.
At the Meta-Command Conversion Stage , the MCC framework converts the macro-strategies sentby humans and the messages outputted by the command encoding network of agents into unifiedmeta-commands and then broadcasts them to all agents and humans.
At the Meta-Command
Communication Stage , the MCC framework uses the meta-command selector to estimate the valuesof all received meta-commands and select the optimal one for each agent to execute.
Note thathumans also select the optimal meta-command based on their value systems.
At the Human-AI
Collaboration Stage , the MCC framework adopts the meta-command conditioned action network topredict a sequence of actions for each agent to perform based on its selected meta-command.
Foreach game, humans and agents have to collaborate multiple times, that is, they need to perform theabove three stages multiple times to win the game.
4.2 Meta-Command
In MOBA games, a macro-strategy comprises three components: where to go, what to do, and howlong.
For example, a macro-strategy can be Come And Kill The Dragon , which consists of Come To The Dragon (where to go), Attack The Dragon (what to do), and Until The Dragon Is Killed (howlong).
Thus, we propose a general representation of macro-strategies, i.e., the Meta-Command, as aninterpretable communication protocol to bridge the communication gap between agents and humans.
Meta-Command Definition.
We formulate the Meta-Command as a tuple <
L, E, T mc > , as shownin Figure 1(b), where L is the Location to go, E is the Event to do after reaching L , and T mc is the
Time Limit for executing the meta-command.
Among them, L is the key to the meta-command, whichcontains the intention of the macro-strategy.
E can be thought of as human micro-operation, which isimplemented through a pre-trained micro-action network π θ in the MCC framework.
T mc can be setto how long it normally takes a human to complete a macro-strategy in MOBA games, usually 300time steps (20 seconds) corresponds to 80% average completion rate for meta-commands.
Meta-Command Conversion.
To realize human-AI communication, we convert the explicit messages from humans and the implicit messages from agents into unified meta-commands.
To achievethe former, a hand-crafted command converter function f cc is used to generate L of meta-commands
[Figure]
𝑜 𝑡 𝜙 𝜋 !
(𝑚 𝑡 |𝑜 " ) <𝑜 𝑡 ,𝑚 " >
[Figure]
SL
Meta-Command Selector (CS)
Meta-Command Conditioned Action Network (MCCAN) 𝜃 𝑜 𝑡 𝜙 𝑚 𝑡 𝑉 "#$" (𝑜 " ,𝑚 𝑡 )
𝜋 % (𝑎 𝑡 |𝑜 " ,𝑚 " )
𝑉 "&’" (𝑜 " ) 𝐺 " sample
RL 𝜙 𝑜 𝑡 𝑜 "() 𝑜 "(* 𝑚 𝑡 𝑚 "() 𝑚 "(* 𝜔 𝐶 𝑡 𝑄 𝑡 (𝑜 " ,𝐶 " ,𝑚 " ) 𝑄 "() (𝑜 " ,𝐶 " ,𝑚 "() ) 𝑄 "(* (𝑜 " ,𝐶 " ,𝑚 "(* ) 𝑉 ",- (𝑜 " ,𝐶 𝑡 ) 𝜃 𝐺 ",- RL 𝑐 𝑡 =softmax , 𝑄 𝑡 (𝑜 " ,𝐶 " ,𝑚)
𝑎 𝑡 𝐺 "&’" 𝐺 "#$
[Figure]
CNNs
[Figure]
Embed
[Figure]
by extracting the location from explicit messages, such as text and signals, sent by humans.
Toachieve the latter, we use a Command Encoding Network (CEN)
π ϕ ( m | o ) to generate L of metacommands.
The CEN is trained via supervised learning (SL) with the goal of learning the distributionof meta-commands sent from humans, as shown in Figure 3(a)(I).
The training dataset { < o, m > }is obtained by extracting the observation o and its corresponding meta-command m from expert data.
After converting all messages into unified meta-commands, the MCC framework broadcasts them toall agents and humans.
Then, agents and humans receive an identical meta-command candidate set.
Meta-Command Execution.
After receiving a meta-command candidate set, agents can select one meta-command from it to execute.
We adopt a Meta-Command Conditioned Action Network (MCCAN)
π θ ( a | o, m ) for agents to perform actions based on the selected metacommand, as shown in Figure 3(a)(II).
The MCCAN is trained via goal-conditioned RL withthe goal of achieving a near-human completion rate for the meta-commands generated by thepre-trained CEN while ensuring that the win rate is not reduced.
We adopt an intrinsic rewardr intt ( s t , m t , s t +1 ) =
(cid:12)(cid:12) f
ce ( s t ) − m t (cid:12)(cid:12) − (cid:12)(cid:12)
f
ce ( s t +1 ) − m t (cid:12)(cid:12) to guide the process of executing the metacommand m t , where f ce is a hand-crafted command extraction function.
We train the MCCANwith the objective of maximizing the expectation over extrinsic and intrinsic discounted total rewards G t =
E s ∼ d πθ ,a ∼ π θ (cid:104)(cid:80) ∞ i =
γ i r t
+ i + α (cid:80) T mc j =0 γ j r intt + j (cid:105) , where α is a trade-off parameter andd π ( s ) =
lim t →∞ P (cid:0) s t
= s
| s 0 , π (cid:1) is the probability when following π for t steps from s 0 .
After training the CEN and MCCAN, we can achieve HAC by simply setting an agent to randomlyselect a meta-command derived from humans to execute.
However, such collaboration is nonintelligent and can even be a disaster for game victory because agents have no mechanism tomodel the values of meta-commands and cannot choose the optimal meta-command to execute.
While humans usually choose the optimal one based on their value systems for achieving effectivecollaboration to win the game.
Thus, we further propose a meta-command value estimation model toselect a valuable meta-command for each agent, as described in the following subsection.
4.3 Meta-Command Selector
In real-world MOBA games, the same macro-strategy often has different values for different humansin different situations.
For example, a macro-strategy can be Come And Kill The Dragon , as shown in
Figure 1(b).
It is more valuable for humans A and B to collaborate.
While another macro-strategy canbe Clean Up Top-Lane Minions , which is more valuable for human C rather than humans A and B.
Therefore, it is important to select the most valuable meta-command from the received meta-commandcandidate set C to achieve effective human-AI collaboration.
We propose a meta-command valueestimation model, i.e., the Meta-Command Selector (CS) π ω
( o, C ) , to estimate the values of allcurrent meta-commands and select the most valuable one for each agent to execute.
CS Optimization Objective.
Typically, the execution of a meta-command involves reaching location
L and doing event E , of which the latter is more important to the value of the meta-command.
For example, for the meta-command Come And Kill The Dragon , if Kill The Dragon event cannot be done within T mc time steps, then it is pointless to Come To The Dragon .
Thus, the long-termreward R mc for executing a meta-command can be expressed as the total rewards within T mc timesteps by interacting with the environment: R mct = (cid:80) T L i =
r t
+ i + β (cid:80) T mc j = T L r t + j , where T L < T mcis the time for reaching L and β > 1 is a trade-off parameter.
Note that the reward function rincludes both individual rewards and team rewards.
The optimization objective of CS is to selectthe optimal meta-command m ∗ t =
π ω
( o t , C t ) for each agent to maximize the expected discountedmeta-command execution return G mct = E
[Equation]
[Equation]
{+θ (cid:104)(cid:80) ∞ i =
 γ imc R mct + i · T mc (cid:105)+} , where o t ∈ O ,
C t is the meta-command candidate set in state s t , and γ mc ∈ [0 , 1) is the discount factor.
CS Training Process.
We construct a self-play training environment for CS where agents can sendmessages to each other.
Specifically, three tricks in Figure 3(a)(III) are adopted to increase thesample efficiency while ensuring efficient exploration.
First, each sent meta-command m is sampledwith the argmax rule from the results predicted by the pre-trained CEN.
Second, each agent sendsits meta-command with a probability p every T mc time steps.
Finally, each agent selects the finalmeta-command c sampled with the softmax rule from its CS output results and hands it over to thepre-trained MCCAN for execution.
We use the multi-head value mechanism [33] to model the valueof the meta-command execution, and the corresponding value loss can be formulated as:
[Equation]
210 where V kω ( S, C ) is the value of the k -th head.
For DQN-based methods [17, 28, 30], the Q loss is: [Equation] head k
[Equation]
212 where w k is the weight of the k -th head and G mck is the Temporal Difference (TD) estimated valueerror R mck + γ mc V kω
( S ′ , C ′ )
− V kω ( S, C ) .
CS Model Structure.
We design a general network structure for CS towards MOBA games, asshown in Figure 3(b).
In MOBA games, the meta-commands corresponding to adjacent regionsusually have similar values.
Thus, we divide the meta-commands in the map into grids, a commonlocation description for MOBA games, and use the shared Convolutional Neural Network (CNN) toextract region-related information from the meta-commands to improve the generalization of CS toadjacent meta-commands.
Besides, we use the gating mechanism [16] to fuse the map embeddingof all received meta-commands and the state embedding of the observation information.
Finally, todirectly construct the relationship between the observation information and each meta-command, weintroduce a target attention module, where the query is the fused embedding h and the key is themap embedding m ′ of each meta-command.
The fused embedding h is used as the input into thesubsequent Q network Q ( h, m ′ ) and V network V ( h ) network of CS.
In this way, the Q networkcan also be easily converted to the policy network π
( m | h, m ′ ) .
Thus, the CS model structure can beeasily applied to most popular RL algorithms, such as PPO [20], DQN [17], etc.
Experiments
We evaluate the proposed MCC framework in Honor of Kings , one of the most popular MOBA gamesworldwide, which has been actively used as the testbed for recent game AI research [6, 32–35].
Weconduct all experiments in Honor of Kings 5v5 mode with a full hero pool (over 100 heroes), exceptablation studies with a 20 hero pool for exploring the influence of different model components moresufficiently and efficiently.
5.1 Experimental Setup
5.1.1 Training Setup
Due to the complexity of MOBA games and limited resources, instead of training jointly, we train the
CEN, MCCAN, and CS sequentially.
For all model training, the location L of meta-commands in themap is divided into 144 grids.
The time limit T mc for the meta-command execution is set to 20s.
[-The MCCAN is trained until it converges for 
-]
VS
Agent-Only Environment
[Figure]
SimulatedHuman
Fixed Opponents
BroadcastChannel
VS
Train Environment
BroadcastChannel (a) … … … …
VS
Fixed Opponents
BroadcastChannel …
+
Human-AI Game Test (c)
VS
BroadcastChannel … …
Test I
Test II p … … … …
BroadcastChannel … …
CS …
[Figure]
CEN Training Settings.
We train the CEN via SL until it converges for 26 hours using 8 NVIDIA
P40 GPUs.
The batch size of each GPU is set to 512.
Adam[14] is adopted as the optimizer with aninitial learning rate of 0.0001.
MCCAN Training Settings.
We train the MCCAN by finetuning a pre-trained micro-action network [33] conditioned on the meta-command sampled from the pre-trained CEN.
The MCCAN istrained until it converges for 48 hours using a physical computer cluster with 63,000 CPUs and {+
+} NVIDIA V100 GPUs.
The batch size of each GPU is set to 256.
The parameter α is set to 16.
CS Training Settings.
We train the CS via self-play until it converges for 24 hours using a physicalcomputer cluster with 70,000 CPUs and 680 NVIDIA V100 GPUs.
The batch size of each GPU isset to 256.
The parameter β is set to 2.
Each agent sends a meta-command with a probability p of 0.8and an interval T mc of 20s, as shown in Figure 4(a).
5.1.2 Evaluating Setup
Our primary concern is whether the agents trained with the MCC framework, briefly called the MCCagents, can collaborate with humans well.
However, evaluating agents with humans is expensive,which is not conducive to model selection and iteration.
Therefore, we built two agent-only testingenvironments:
Test I and Test II, for the model selection and iteration process, as shown in Figure 4(b).
We also evaluate the MCC agents in practical human-AI game tests to examine the performance ofcollaborating with humans, as shown in Figure 4(c).
Compared Agents.
We compare the MCC agent with three different types of agents: the MC-Baseagent (agent only executes its own meta-command without communication), the MC-Rand agent(agent randomly selects a meta-command to execute), and the MC-Rule agent (agent selects thenearest meta-command to execute). For fair comparisons, we adopt the MC-Base agent as theopponent for all tests, and results are reported over five random seeds.
Agent-Only Environmental Settings.
Test I is the most complex environment where all agentteammates can send and receive meta-commands simultaneously with an interval of 20s.
Test I isused to evaluate the agents’ performance under extremely complex situations as well as in ablationstudies.
Test II is a simple environment to simulate practical game scenarios, where at most onehuman sends his macro-strategy at a time step.
Thus, in Test II, only one agent is randomly selectedto send its meta-command with an interval of 20s, and the other agents only receive meta-commands.
Human-AI Game Testing Settings.
We had different types of agents team up with different levelsand numbers of humans, including 15 strong humans (top1%) and 15 average humans (top30%), inm AI + n Human mode, where m + n = 5 .
For fair comparisons, each tester was not told the type ofagent teammates.
In each game test, humans can send the converted meta-commands whenever theythink their macro-strategies are important.
To make the agent behave like humans (at most one humansends his macro-strategy at a time step), we restrict agents from sending their meta-commands.
Werandomly choose a human teammate and use his observation and all agents’ meta-commands as the CS input and select the final output of CS to send with an interval of 20s.
5.2 Results in Agent-Only Environment
5.2.1 AI Performance
The Kullback-Leibler (KL) divergence of the meta-command distribution between the CEN andhumans decreased from 4.96 to 0.44 as training converges.
The average completion rates of agentsand humans for meta-commands are 82% and 80%, respectively.
Notably, we can train an agent witha higher completion rate by increasing α , but this will significantly reduce the win rate because themeta-command executed is not necessarily optimal and may result in the death of agents.
We put thedetailed experimental results of the CEN and MCCAN in the appendix due to space limitations.
Figure 5(a) and (b) show the win rates of four types of agents who play against each other for 600matches in Test I and Test II, respectively.
We see that the MCC agent achieves the highest win rateagainst all the other agents in both testing environments, indicating that the CS can select a valuablemeta-command for each agent to collaborate, and such reasonable collaboration is conducive towinning the game.
The MC-Rand and MC-Rule agents are worse than the MC-Base agent, confirmingthat agents executing low-value meta-commands can hurt performance.
Notably, we find that the winrates of the MCC agent in Test I and Test II are close, suggesting that the MCC agent can generalizeto different numbers of meta-commands.
Figure 5(c) demonstrates the final Elo scores [3] of theseagents.
It clearly shows the effectiveness of CS in agent-only collaboration scenarios.
5.2.2 Ablation Studies
[Figure]
We further investigatethe influence of differentcomponents, including
CNN feature extractionwith the gating mechanism(w/o
CNN-GM), targetattention module (w/o
TA), and PPO optimizationalgorithm
(MCC-PPO),on the performance of
CS.
We conduct ablationstudies in Test I with a20 hero pool.
In practicalgames, meta-commandswith adjacent regions often have similar intentions and values.
Thus the response rate of the agentto adjacent meta-commands should be as close as possible.
Besides, humans always want theirmeta-commands to be responded to as much as possible. Therefore, we design the response ratemetric to measure the generalization of CS.
Generally, we expect CS to have the highest possible Response Rate (RR) while ensuring that the Win Rate (WR) is not reduced.
Figure 6(a) demonstrates the WR of different CS ablation versions during the training process, and
Figure 6(b) shows the converged WR-RR results.
We see that after ablating the TA module, the WRand RR of CS are greatly reduced, indicating that the TA module can improve the accuracy of CSto meta-commands.
Besides, after ablating the CNN-GM module, the RR of CS is most affected,which is reduced by 20%.
It indicates that without the CNN-GM module, the value estimation of CSto adjacent meta-commands is not accurate enough, resulting in missing some actual high valuablemeta-commands.
We notice that the MCC and MCC-PPO in both metrics are close, confirming theversatility of the CS model structure.
[Table] [Table]
[Equation]
[Table]
5.3 Results in Human-AI Game Test
Due to space limitations, we only show the results in 4 AI + 1 Human mode.
More results canbe found in the appendix.
Table 1 shows the WR of different human-AI teams who play againstthe MC-Base agent-only team.
We see that the MCC agent significantly outperforms other agents,regardless of whether they pair with a strong or average human.
To explain why humans have ahigher WR when paired with the MCC agents, we count the RR of agents to the meta-commandssent from human teammates (H2A scenarios) and the RR of humans to the meta-commands sentfrom agent teammates (A2H scenarios), respectively, as shown in Table 2.
In H2A scenarios, the RRs of the MCC agents to average humans and strong humans are 61.17% and 73.05%, respectively,indicating that the MCC agents are more willing to respond to valuable meta-commands sent fromstrong humans.
We also notice that the RR of the MCC agents to strong humans (73.05%) is veryclose to the RR of strong humans themselves (74.91%), suggesting that the CS is close to the valuesystem of strong humans.
In A2H scenarios, the RRs of average humans and strong humans to the
MCC agents are 73.43% and 78.5%, respectively, which is significantly higher than that of MC-Randagents (41.07% and 35.69%), indicating that the meta-commands sent from the MCC agents are morevaluable and reasonable to humans.
Note that the RR of the MCC agents to the MC-Rand agents is34.03%, which is close to that of strong humans (35.69%), once again confirming that the CS is closeto the value system of strong humans.
We also visualize the comparison of CS and strong human value systems on a game scene with threemeta-commands existing, as shown in Figure 7.
We see that the CS selects the meta-command B forthe two heroes in the red dashed box to collaborate, selects the meta-command C for the two heroesin the purple dashed box to collaborate, and selects the meta-command A for the remaining heroto execute alone.
The CS’ value predictions of these three meta-commands are consistent with thestrong human ranking of the meta-command importance, confirming the effectiveness of the CS inhuman-AI collaboration scenarios.
Conclusion
In this paper, we proposed an efficient and interpretable Meta-Command Communication-basedframework, dubbed MCC, to achieve effective human-AI collaboration in MOBA games.
To bridgethe communication gap between humans and agents, we designed an interpretable communicationprotocol, i.e., the Meta-Command, to convert the explicit messages from humans and the implicitmessages from agents into unified meta-commands.
To achieve effective collaboration, we constructeda meta-command value estimation model, i.e., the Meta-Command Selector, to select a valuablemeta-command for each agent to execute.
Finally, we introduced the training process of the MCCframework and conducted practical human-AI game tests in the typical MOBA game Honor of
Kings .
The experimental results show that the MCC agents can collaborate reasonably with humanteammates and even generalize to collaborate with different levels and numbers of human teammates.
We expect this work can be a foundation for future HAC research in complex environments. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See the abstract and the contributions in Section 1.(b)
Did you describe the limitations of your work?
[Yes] For margin reasons, we willdiscuss limitations here.
We have currently only verified the effectiveness of the MCCframework in MOBA games, and we will explore in more types of complex games,such as First-Person Shooting (FPS) and Massively Multiplayer Online (MMO) in thefuture.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Thepurpose of our method is only for academic research based on game environments,e.g., investigating the MOBA game-playing problems.
Like AlphaGo or AlphaStar,the potential negative societal impacts of our work will be limited to the developmentof gaming AI applications.
However, our research will contribute to the researchcommunity, the game industry, and the e-sports community.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The code and
the data are proprietary.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See the full experimental setup in Section 5.1.1 and Appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] All results are reported over five random seeds (seeSection 5.1.2).
For the convenience of presentation, the median is shown in AI Performance , while detailed error bars are presented in Ablation Studies .
Due to the high
cost of Human-AI Game Test , we only used the median model for testing.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See the resources in Section 5.1.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a
work uses
[Table]
[Table]
(c) Did you include any new assets either in the supplemental
material
or as
a URL
[No]
d)
Did you discuss whether and how consent was obtained from people whose datayou’re using/curating?
[Yes] Honor of Kings is a released and recognized "testbed" for
MOBA-game-playing problems [35, 34, 33, 6, 32].
We contact the relevant author and
[Table]
obtain authorization from the game provider.
[Table]
e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[Yes] See Appendix.
All data in this paper is gamerelated and has nothing to do with identity information.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[Yes] See Appendix.
In the Human-AI Game Test , all participants havemore than three years of experience in Honor of Kings and are familiar with all the
[Table]
information in the game.
Before the test, we also inform the participants of the detailedtest instructions, and the participants voluntarily choose whether to participate in thetest.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[Yes] We had performed a process similar to
IRB before the test was conducted in this paper.
The institution and all participantshave approved our research.
First, we analyze the risks of these experiments to theparticipants.
The risks mainly include the leakage of identity information and thetime cost.
Then, a series of measures are implemented to prevent these risks.
Wemake a risk statement for participants and sign an identity information confidentialityagreement.
We only use information related to the game state in our research withoutidentity information.
In addition, special equipment and accounts are provided to theparticipants to prevent leakage of equipment and account information during the test.
The identity information of all participants is not disclosed to the public.
c)
Did you
include
the estimated hourly wage paid
to
participants and the total amount
[Table]
spent on
participant
compensation