Original: F EINT IN M ULTI -P LAYER G AMES
Revised: F EINT IN M ULTI -P LAYER G AMES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper introduces the first formalization, implementation and quantitative evaluation of Feint in Multi-Player Games.
Revised: This paper introduces the first formalization, implementation and quantitative evaluation of Feint in Multi-Player Games.

Original: Our work first formalizes Feint from the perspective of Multi-Player Games, in terms of the temporal, spatial and their collective impacts.
Revised: Our work first formalizes Feint from the perspective of Multi-Player Games, in terms of the temporal, spatial and their collective impacts.

Original: The formalization is built upon Non-transitive Active Markov
Revised: The formalization is built upon Non-transitive Active Markov

Original: Game Model , where Feint can have a considerable amount of impacts.
Revised: Game Model , where Feint can have a considerable amount of impacts.

Original: Then, our work considers practical implementation details of Feint in Multi-Player Games, under the state-of-the-art progress of multi-agent modeling to date (namely MultiAgent Reinforcement Learning).
Revised: Then, our work considers practical implementation details of Feint in Multi-Player Games, under the state-of-the-art progress of multi-agent modeling to date (namely MultiAgent Reinforcement Learning).

Original: Finally, our work quantitatively examines the effectiveness of our design and the results show that our design of Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption.
Revised: Finally, our work quantitatively examines the effectiveness of our design and the results show that our design of Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption.

Original: We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting
Revised: We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Game simulations, which only use Markov Game Model (Filar (1976)) or its variants (Wampler et al. (2010); Kim et al.
Revised: Game simulations, which only use Markov Game Model (Filar (1976)) or its variants (Wampler et al. (2010); Kim et al.

Original: (2022)), breed the needs for the diversity and the randomness to improve the game experiences.
Revised: (2022)), breed the needs for the diversity and the randomness to improve the game experiences.

Original: The trends of evolving more details into simulated games demand: ➊ the need for non-transitivity (i.e. there are no dominant gaming strategies), which allow players to dynamically change game strategies.
Revised: The trends of evolving more details into simulated games demand: ➊ the need for non-transitivity (i.e. there are no dominant gaming strategies), which allow players to dynamically change game strategies.

Original: In this way, the newly-incorporated strategies can maintain a high level of the diversity, which guarantee a high extent of unexploitability (Liu et al.
Revised: In this way, the newly-incorporated strategies can maintain a high level of the diversity, which guarantee a high extent of unexploitability (Liu et al.

Original: ; and ➋ the strict requirements on temporal impacts (and its implications on spatial and collective impacts), since modern game simulations are highly time-sensitive ( Nota & Thomas (2020)).
Revised: ; and ➋ the strict requirements on temporal impacts (and its implications on spatial and collective impacts), since modern game simulations are highly time-sensitive ( Nota & Thomas (2020)).

Original: Therefore, new optimizations on these game models are expected to be elegant and easy-to-implement, to preserve the original spirits of these games.
Revised: Therefore, new optimizations on these game models are expected to be elegant and easy-to-implement, to preserve the original spirits of these games.

Original: Our work first builds upon representative examples from the above two trends, by unifying two stateof-the-art progress of Multi-Player Games: ➊ we use Unified Behavioral and Response Diversity (described in Liu et al.
Revised: Our work first builds upon representative examples from the above two trends, by unifying two stateof-the-art progress of Multi-Player Games: ➊ we use Unified Behavioral and Response Diversity (described in Liu et al.

Original: (2021)), which exploits non-transitivity (i.e. no single dominant strategy in many complex games), to highlight the importance of the diversity in game policies.
Revised: (2021)), which exploits non-transitivity (i.e. no single dominant strategy in many complex games), to highlight the importance of the diversity in game policies.

Original: Moreover, we address the issue from their work, which fails to consider the intensity and future impacts from complex interactions among agents; and ➋ we incorporate Long-Term Behavior Learning (described in Kim et al.
Revised: Moreover, we address the issue from their work, which fails to consider the intensity and future impacts from complex interactions among agents; and ➋ we incorporate Long-Term Behavior Learning (described in Kim et al.

Original: (2022)), which proposes Active Markov Game Model to emphasize the convoluted future impacts from complex interactions among agents.
Revised: (2022)), which proposes Active Markov Game Model to emphasize the convoluted future impacts from complex interactions among agents.

Original: Based on the above two results, we unify them as a new model called Non-transitive Active Markov Game Model (NTAMGM), and use it throughout this work.
Revised: Based on the above two results, we unify them as a new model called Non-transitive Active Markov Game Model (NTAMGM), and use it throughout this work.

Original: This unification satisfies the need for a game model where (A) agents have intense and time-critical interactions; and (B) the design space of game policies is highly diverse.
Revised: This unification satisfies the need for a game model where (A) agents have intense and time-critical interactions; and (B) the design space of game policies is highly diverse.

Original: The definition of NTAMGM is described below.
Revised: The definition of NTAMGM is described below.

Original: • Non-transitive Active Markov Game Model : We define a K -agent Non-transitive Active Markov Game Model as a tuple ⟨ K, S, A, P, R, Θ , U ⟩ :
Revised: • Non-transitive Active Markov Game Model : We define a K -agent Non-transitive Active Markov Game Model as a tuple ⟨ K, S, A, P, R, Θ , U ⟩ :

Original: K = { 1 , ..., k } is the set of k agents; S is the state space;
Revised: K = { 1 , ..., k } is the set of k agents; S is the state space;

Original: A = { A i } Ki =1 is the set of action space for each agent, where there are no dominant actions; P performs state transitions of current state by agents’ actions: P : S
Revised: A = { A i } Ki =1 is the set of action space for each agent, where there are no dominant actions; P performs state transitions of current state by agents’ actions: P : S

Original: ×
Revised: ×

Original: A 1 × A 2 × ...
Revised: A 1 × A 2 × ...

Original: × A K
Revised: × A K

Original: → P ( S ) , where P ( S ) denotes the set of probability distribution over state space S ; R = { R i } Ki =1 is the set of reward functions for each agent; Θ = { Θ i } Ki =1 is the set of policy parameters for each agent; and
Revised: → P ( S ) , where P ( S ) denotes the set of probability distribution over state space S ; R = { R i } Ki =1 is the set of reward functions for each agent; Θ = { Θ i } Ki =1 is the set of policy parameters for each agent; and

Original: U =
Revised: U =

Original: { U i } K i =1 is the set of policy update functions for each agent.
Revised: { U i } K i =1 is the set of policy update functions for each agent.

Original: Based on the above assumption of Multi-Player Games, our goal is to incorporate Feint, a set of actions to mislead opponents, for strategic advantages in Multi-Player Games .
Revised: Based on the above assumption of Multi-Player Games, our goal is to incorporate Feint, a set of actions to mislead opponents, for strategic advantages in Multi-Player Games .

Original: Prior works simply incorporate Feint in the context of Two-Player Games (e.g. Wampler et al.
Revised: Prior works simply incorporate Feint in the context of Two-Player Games (e.g. Wampler et al.

Original: ; Won et al.
Revised: ; Won et al.

Original: (2021a)), and our works begins by addressing the limitations of the derived version (denoted as the basic formalization of Feint) from these works.
Revised: (2021a)), and our works begins by addressing the limitations of the derived version (denoted as the basic formalization of Feint) from these works.

Original: We find that: the basic formalization of Feint overlooks the complexity of potential impacts in Multi-Player Games, and therefore can not be generalized for Multi-Player Games.
Revised: We find that: the basic formalization of Feint overlooks the complexity of potential impacts in Multi-Player Games, and therefore can not be generalized for Multi-Player Games.

Original: To this end, we deliver the first comprehensive formalization of
Revised: To this end, we deliver the first comprehensive formalization of

Original: Feint, by separating the complex impacts into ➊ the temporal dimension; ➋ the spatial dimension; and ➌ the collective impacts from these two dimensions.
Revised: Feint, by separating the complex impacts into ➊ the temporal dimension; ➋ the spatial dimension; and ➌ the collective impacts from these two dimensions.

Original: We also show that how the above components of our formalization can be synergistically put together.
Revised: We also show that how the above components of our formalization can be synergistically put together.

Original: Based on the proposed formalization, we clear the implementation roadmap, under both Inference Learning and Reinforcement Learning models, to justify the applicability of our proposed formalization.
Revised: Based on the proposed formalization, we clear the implementation roadmap, under both Inference Learning and Reinforcement Learning models, to justify the applicability of our proposed formalization.

Original: To properly examine the benefits of our method, we first extensively build two complex scenarios, using Multi-Agent Deep Deterministic Policy Gradient (MADDPG Lowe et al.
Revised: To properly examine the benefits of our method, we first extensively build two complex scenarios, using Multi-Agent Deep Deterministic Policy Gradient (MADDPG Lowe et al.

Original: and
Revised: and

Original: Multi-Agent Actor-attention Critic (MAAC Iqbal & Sha (2019)), with six agents in total.
Revised: Multi-Agent Actor-attention Critic (MAAC Iqbal & Sha (2019)), with six agents in total.

Original: Then, we implement our formalization upon these two extensively-engineered scenarios.
Revised: Then, we implement our formalization upon these two extensively-engineered scenarios.

Original: Our quantitative evaluations show that our formalization and implementations have great potential in practice.
Revised: Our quantitative evaluations show that our formalization and implementations have great potential in practice.

Original: We first show that our work can make the game more interesting, via the following two metrics: for the
Revised: We first show that our work can make the game more interesting, via the following two metrics: for the

Original: Diversity Gain, our method can increase the exploitation of the search space by 1.98X, measured by the Exploitability metric; and for Gaming Reward Gain, our method can achieve 1.90X and 2.86X gains, when using MADDPG and MAAC respectively.
Revised: Diversity Gain, our method can increase the exploitation of the search space by 1.98X, measured by the Exploitability metric; and for Gaming Reward Gain, our method can achieve 1.90X and 2.86X gains, when using MADDPG and MAAC respectively.

Original: We then show that our method only incur negligible overheads, by using per-episode execution time as the metric: our method only introduces less than 5% more for the time consumption.
Revised: We then show that our method only incur negligible overheads, by using per-episode execution time as the metric: our method only introduces less than 5% more for the time consumption.

Original: We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting.
Revised: We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting.

Original:  B ACKGROUND AND M OTIVATION 2. E XISTING MARL MODELS
Revised:  B ACKGROUND AND M OTIVATION 2. E XISTING MARL MODELS

Original: Multi-Agent Reinforcement Learning (MARL) aims to learn optimal policies for agents in a multiagent environment, which consists of various agent-agent and agent-environment interactions.
Revised: Multi-Agent Reinforcement Learning (MARL) aims to learn optimal policies for agents in a multiagent environment, which consists of various agent-agent and agent-environment interactions.

Original: Many single-agent Reinforcement Learning methods (e.g. DDPG Lillicrap et al.
Revised: Many single-agent Reinforcement Learning methods (e.g. DDPG Lillicrap et al.

Original: SAC Haarnoja et al.
Revised: SAC Haarnoja et al.

Original: (2018), PPO Schulman et al.
Revised: (2018), PPO Schulman et al.

Original: (2017) and TD3 Fujimoto et al.
Revised: (2017) and TD3 Fujimoto et al.

Original: (2018)) can not be directly used in multi-agent scenarios, since the rapidly-changing multi-agent environment can cause highly unstable learning results (evidenced by Lowe et al.
Revised: (2018)) can not be directly used in multi-agent scenarios, since the rapidly-changing multi-agent environment can cause highly unstable learning results (evidenced by Lowe et al.

Original: Thus, recent efforts on MARL model designs aim to address such an issue.
Revised: Thus, recent efforts on MARL model designs aim to address such an issue.

Original: Foerster et al.
Revised: Foerster et al.

Original: (2018) proposes Counterfactual Multi-Agent (COMA) policy gradients, which uses centralised critic to estimate the Q-function and decentralised actors to optimize agents’ policies.
Revised: (2018) proposes Counterfactual Multi-Agent (COMA) policy gradients, which uses centralised critic to estimate the Q-function and decentralised actors to optimize agents’ policies.

Original: Lowe et al.
Revised: Lowe et al.

Original: (2017) proposes Multi-Agent Deep Deterministic Policy
Revised: (2017) proposes Multi-Agent Deep Deterministic Policy

Original: Gradient (MADDPG), which decreases the variance in policy gradient and instability of Q-function of DDPG in multi-agent scenarios.
Revised: Gradient (MADDPG), which decreases the variance in policy gradient and instability of Q-function of DDPG in multi-agent scenarios.

Original: Iqbal & Sha (2019) proposes Multi-Agent Actor-attention Critic (MAAC), which applies attention entropy mechanism to enable effective and scalable policy learning.
Revised: Iqbal & Sha (2019) proposes Multi-Agent Actor-attention Critic (MAAC), which applies attention entropy mechanism to enable effective and scalable policy learning.

Original: These models can have varied impacts within a diverse set of scenarios.
Revised: These models can have varied impacts within a diverse set of scenarios.

Original: 2. F EINT IN A N UTSHELL
Revised: 2. F EINT IN A N UTSHELL

Original: Feint is common for human players, as a set of active actions to obtain strategic advantages in real-world games.
Revised: Feint is common for human players, as a set of active actions to obtain strategic advantages in real-world games.

Original: Examples can include sports games such as boxing, basketball and car racing
Revised: Examples can include sports games such as boxing, basketball and car racing

Original: G¨uldenpenning et al.
Revised: G¨uldenpenning et al.

Original: (2017; 2018); Hyman (1989), and electronic games such as King of Fighters and Starcraft Team (2021); Critch & Churchill (2021).
Revised: (2017; 2018); Hyman (1989), and electronic games such as King of Fighters and Starcraft Team (2021); Critch & Churchill (2021).

Original: Though Feint is undoubtedly important in game simulations, there still lacks a comprehensive formalization of Feint for Non-People Characters (NPCs) in Multi-Player Games.
Revised: Though Feint is undoubtedly important in game simulations, there still lacks a comprehensive formalization of Feint for Non-Player Characters (NPCs) in Multi-Player Games.

Original: Only a limited amount of works tackle this issue.
Revised: Only a limited amount of works tackle this issue.

Original: Wampler et al.
Revised: Wampler et al.

Original: (2010) is an early example to incorporate Feint as a proof-of-concept, which focuses on constructing animations for nuanced game strategies for more unpredictability from NPCs.
Revised: (2010) is an early example to incorporate Feint as a proof-of-concept, which focuses on constructing animations for nuanced game strategies for more unpredictability from NPCs.

Original: More recently, Won et al.
Revised: More recently, Won et al.

Original: (2021a) uses a set of pre-defined Feint actions for the animation, which further serves under an optimized version of control strategy based on Online Reinforcement Learning (i.e. in animating combat scenes).
Revised: (2021a) uses a set of pre-defined Feint actions for the animation, which further serves under an optimized version of control strategy based on Online Reinforcement Learning (i.e. in animating combat scenes).

Original: However, these prior works (1) solely focus on Two-Player Games, which can not be effectively generalized to multi-player scenarios; and (2) lack an comprehensive exploration of potential implications from Feint actions in game strategies.
Revised: However, these prior works (1) solely focus on Two-Player Games, which can not be effectively generalized to multi-player scenarios; and (2) lack an comprehensive exploration of potential implications from Feint actions in game strategies.

Original: 2. N OVELTY OF O UR W ORK
Revised: 2. N OVELTY OF O UR W ORK

Original: The novelty of our work is three-folded.
Revised: The novelty of our work is three-folded.

Original: First, our work introduces the first formalization of Feint , which can be generalized to Multi-Player Games.
Revised: First, our work introduces the first formalization of Feint , which can be generalized to Multi-Player Games.

Original: Prior works solely focuses on Two-Player Games, which have the flexibility and scalability issue from the basic formalization.
Revised: Prior works solely focuses on Two-Player Games, which have the flexibility and scalability issue from the basic formalization.

Original: Second, our work provides effective implementations of Feint in Multi-Player Games, by exploiting our formalization appropriately on common parts of MARL models (i.e. the reward function).
Revised: Second, our work provides effective implementations of Feint in Multi-Player Games, by exploiting our formalization appropriately on common parts of MARL models (i.e. the reward function).

Original: Our formalization can be applied to existing MARL models, and is expected to be applicable in future MARL models.
Revised: Our formalization can be applied to existing MARL models, and is expected to be applicable in future MARL models.

Original: Third, our work identifies the unique characteristics of Feint , by differentiating Feint with other regular actions.
Revised: Third, our work identifies the unique characteristics of Feint , by differentiating Feint with other regular actions.

Original: Hence, our work is expected to be applicable in different scenarios, with only a limited amount of refinements.
Revised: Hence, our work is expected to be applicable in different scenarios, with only a limited amount of refinements.

Original:  F EINT F ORMALIZATION 3. T HE B ASIC F ORMALIZATION : D ERIVATION AND L IMITATIONS
Revised:  F EINT F ORMALIZATION 3. T HE B ASIC F ORMALIZATION : D ERIVATION AND L IMITATIONS

Original: We summarize two major limitations of existing works to justify that they cannot deliver a sufficient formalization of Feint in Multi-Player Games.
Revised: We summarize two major limitations of existing works to justify that they cannot deliver a sufficient formalization of Feint in Multi-Player Games.

Original: Since there are no prior formalization, we discuss relevant works and derive the key features to discuss them in detail.
Revised: Since there are no prior formalization, we discuss relevant works and derive the key features to discuss them in detail.

Original: ➊ The basic formalization on temporal impacts is insufficient for Multi-Player Games.
Revised: ➊ The basic formalization on temporal impacts is insufficient for Multi-Player Games.

Original: Multi-Player
Revised: Multi-Player

Original: Games require agents to account for future planning for decision-making, which is critical for deceptive actions like Feint Mnih et al.
Revised: Games require agents to account for future planning for decision-making, which is critical for deceptive actions like Feint Mnih et al.

Original: ; Naik et al.
Revised: ; Naik et al.

Original: ; Nota & Thomas (2020).
Revised: ; Nota & Thomas (2020).

Original: Several works simplify the temporal impacts of deceptive game strategies in different gaming scenarios.
Revised: Several works simplify the temporal impacts of deceptive game strategies in different gaming scenarios.

Original: Mnih et al.
Revised: Mnih et al.

Original: (2013) uses a discount factor γ to calculate the reward for following actions as (cid:80) ∞ t =

Revised: (2013) uses a discount factor γ to calculate the reward for following actions as (cid:80) ∞ t =

Original: γ t R i ( s t , a it , a − i t ) for agent i .
Revised: γ t R i ( s t , a it , a − i t ) for agent i .

Original: However, such a method suffers from the ”short-sight” issue
Revised: However, such a method suffers from the ”short-sight” issue

Original: Naik et al.
Revised: Naik et al.

Original: (2019), since they simply shrink smaller weights and deliver more future actions, which are not suitable for all gaming situations (discussed in Nota & Thomas (2020)).
Revised: (2019), since the weights for future actions’ rewards shrink exponentially with time, which are not suitable for all gaming situations (discussed in Nota & Thomas (2020)).

Original: More recently, Kim et al.
Revised: More recently, Kim et al.

Original: (2022) applies a long-term average reward, to equalize the rewards of all future actions as  T (cid:80) Tt =

Revised: (2022) applies a long-term average reward, to equalize the rewards of all future actions as 1 T (cid:80) Tt =

Original: R i ( s t , a it , a − i t )
Revised: R i ( s t , a it , a − i t )

Original: (i.e. for agent i ).
Revised: (i.e. for agent i ).

Original: However, such a method is restricted by the ”far-sight” issue, since there are no differentiation between near-future and far-future planning.
Revised: However, such a method is restricted by the ”far-sight” issue, since there are no differentiation between near-future and far-future planning.

Original: The mismatch between abstraction granularity heavily saddles with the design of Feint , because they use relatively static representations (e.g. static γ and T ).
Revised: The mismatch between abstraction granularity heavily saddles with the design of Feint , because they use relatively static representations (e.g. static γ and T ).

Original: Therefore, they cannot be aware of any potential changes of strategies in different phases of a game.
Revised: Therefore, they cannot be aware of any potential changes of strategies in different phases of a game.

Original: Hence, the temporal dimension is simplified for the basic Feint formalization.
Revised: Hence, the temporal dimension is simplified for the basic Feint formalization.

Original: ➋ The basic formalization cannot be effectively generalized to Multi-Player Game scenarios.
Revised: ➋ The basic formalization cannot be effectively generalized to Multi-Player Game scenarios.

Original: Prior works, which attempt to fuse Feint into complete game scenarios, only consider two-player scenarios Won et al.
Revised: Prior works, which attempt to fuse Feint into complete game scenarios, only consider two-player scenarios Won et al.

Original: (2021a)
Revised: (2021a)

Original: ; So et al. (2022).
Revised: ; So et al. (2022).

Original: However, in Multi-Player (more then two player)
Revised: However, in Multi-Player (more then two player)

Original: Games, gaming strategies (especially deceptive strategies) yield spatial impacts on other agents.
Revised: Games, gaming strategies (especially deceptive strategies) yield spatial impacts on other agents.

Original: Such impacts have been overlooked by all prior works.
Revised: Such impacts have been overlooked by all prior works.

Original: This is because an agent, who launches the Feint actions, can impact not only the target agent but also other agents in the scenario.
Revised: This is because an agent, who launches the Feint actions, can impact not only the target agent but also other agents in the scenario.

Original: Therefore, the influences of such an action needs to account for spatial impacts Liu et al.
Revised: Therefore, the influences of such an action needs to account for spatial impacts Liu et al.

Original: Moreover, with a new dimension accounted, the interactions between these two dimensions also raise a potential issue for their mutual collective impacts.
Revised: Moreover, with a new dimension accounted, the interactions between these two dimensions also raise a potential issue for their mutual collective impacts.

Original: 3. O UR F ORMALIZATION : G ENERALIZED FOR M ULTI -P LAYER G AMES
Revised: 3. O UR F ORMALIZATION : G ENERALIZED FOR M ULTI -P LAYER G AMES

Original: Therefore, to deliver an effective formalization of Feint in Multi-Player Games, it’s essential to consider the temporal, spatial and their collective impacts comprehensively.
Revised: Therefore, to deliver an effective formalization of Feint in Multi-Player Games, it’s essential to consider the temporal, spatial and their collective impacts comprehensively.

Original: We first discuss the
Revised: We first discuss the

Original: Temporal Dimension, then elaborate our considerations on Spatial Dimension, and finally summarize the design for the collective impacts from both temporal and spatial dimensions.
Revised: Temporal Dimension, then elaborate our considerations on Spatial Dimension, and finally summarize the design for the collective impacts from both temporal and spatial dimensions.

Original: 3.2. T EMPORAL D IMENSION : I NFLUENCE T IME
Revised: 3.2. T EMPORAL D IMENSION : I NFLUENCE T IME

Original: Different from prior works, our work consider the temporal dimension of Feint impacts by emulating them in a Dynamic Short-Long-Term manner.
Revised: Different from prior works, our work consider the temporal dimension of Feint impacts by emulating them in a Dynamic Short-Long-Term manner.

Original: The rationale behind such a design choice is that: the purpose of Feint is to obtain strategic advantages against the opponent in temporal dimension, aiming to benefit following attacks.
Revised: The rationale behind such a design choice is that: the purpose of Feint is to obtain strategic advantages against the opponent in temporal dimension, aiming to benefit following attacks.

Original: Hence, the Dynamic Short-Long-Term temporal impacts of
Revised: Hence, the Dynamic Short-Long-Term temporal impacts of

Original: Feint shall be (1) the actions that follow Feint actions (e.g. actual attacks) in a short-term period of time should have strong correlation to Feint ; (2) the actions in the long-term periods explicitly or implicitly depend on the effect of the Feint and its following actions; and (3) for different Feint actions in different gaming scenarios, the threshold that divides short-term and long-term should be dynamically adjusted to enable sufficient flexibility in strategy making.
Revised: Feint shall be (1) the actions that follow Feint actions (e.g. actual attacks) in a short-term period of time should have strong correlation to Feint ; (2) the actions in the long-term periods explicitly or implicitly depend on the effect of the Feint and its following actions; and (3) for different Feint actions in different gaming scenarios, the threshold that divides short-term and long-term should be dynamically adjusted to enable sufficient flexibility in strategy making.

Original: For Dynamic Short-Long-Term , we first set up a short-term planning threshold st to select the followup actions, which are decided by Feint policy π
Revised: For Dynamic Short-Long-Term , we first set up a short-term planning threshold st to select the followup actions, which are decided by Feint policy π

Original: ′ i at t 0 .
Revised: ′ i at t 0 .

Original: Note that actions { a it 0 +1 , ..., a it 0 + st } are strongly related to the Feint action a it 0 .
Revised: Note that actions { a it 0 +1 , ..., a it 0 + st } are strongly related to the Feint action a it 0 .

Original: For the actions bounded by the short-term threshold, a set of large weights α = { α t 0 , ..., α t 0 + st } are used to calculate the reward:
Revised: For the actions bounded by the short-term threshold, a set of large weights α = { α t 0 , ..., α t 0 + st } are used to calculate the reward:

Original: [Equation]
Revised: [Equation]

Original: since these actions are expected to deliver higher reward (i.e. the purpose of Feint is to obtain strategic advantages) via the Feint action.
Revised: since these actions are expected to deliver higher reward (i.e. the purpose of Feint is to obtain strategic advantages) via the Feint action.

Original: We then consider long-term planning after the short-term planning threshold st : we use a set of discount factor β = { β t 0 + st +1 , ..., β T } on the long-term average reward calculation (proposed by Kim et al.
Revised: We then consider long-term planning after the short-term planning threshold st : we use a set of discount factor β = { β t 0 + st +1 , ..., β T } on the long-term average reward calculation (proposed by Kim et al.

Original: to distinguish these reward from shortterm rewards:
Revised: to distinguish these reward from shortterm rewards:

Original: [Equation]
Revised: [Equation]

Original: where T denotes the end time of the game.
Revised: where T denotes the end time of the game.

Original: Finally, we put them together to formalize the Short-Long-Term reward calculation mechanism, when an agent i plans to perform a Feint action at time t 0 with a short-term planning threshold st and the end time of game T as:
Revised: Finally, we put them together to formalize the Short-Long-Term reward calculation mechanism, when an agent i plans to perform a Feint action at time t 0 with a short-term planning threshold st and the end time of game T as:

Original: [Equation]
Revised: [Equation]

Original: where λ short and λ long are weights for dynamically balancing the weight of short-term and longterm rewards for different gaming scenarios.
Revised: where λ short and λ long are weights for dynamically balancing the weight of short-term and longterm rewards for different gaming scenarios.

Revised: λ short and λ long are initially set as 0 .

Revised: 67 and 0 .

Revised: and are adjusted to achieve better performance with the iterations of training.

Original: 3.2. S PATIAL D IMENSION : I NFLUENCE R ANGE
Revised: 3.2. S PATIAL D IMENSION : I NFLUENCE R ANGE

Original: In a Multi-Player Game (i.e. usually more than two players), the strict one-to-one relationship between two agents is not realistic, since an agent can impact both the target agent and other agents.
Revised: In a Multi-Player Game (i.e. usually more than two players), the strict one-to-one relationship between two agents is not realistic, since an agent can impact both the target agent and other agents.

Original: Therefore, the influences to all other agents shall maintain different levels Liu et al.
Revised: Therefore, the influences to all other agents shall maintain different levels Liu et al.

Original: Therefore, our work includes the spatial dimension of Feint impacts by fusing spatial distributions.
Revised: Therefore, our work includes the spatial dimension of Feint impacts by fusing spatial distributions.

Original: The key idea of this design is to combine spatial distribution with the influence range during the game.
Revised: The key idea of this design is to combine spatial distribution with the influence range during the game.

Original: More specifically, we incorporate Behavioral Diversity from Liu et al.
Revised: More specifically, we incorporate Behavioral Diversity from Liu et al.

Original: to mathematically calculate and maximize the diversity gain of Feint actions in terms of the influence range.
Revised: to mathematically calculate and maximize the diversity gain of Feint actions in terms of the influence range.

Original: We formalize the influence range of an action policy on
Revised: We formalize the influence range of an action policy on

Original: K agent based on
Revised: K agent based on

Original: S ×
Revised: S ×

Original: A i × ...
Revised: A i × ...

Original: × A K , which follows a distribution of multi-to-one relationships
Revised: × A K , which follows a distribution of multi-to-one relationships

Original: T→ (
Revised: T

Original: [Equation]
Revised: [Equation]

Original: The influence distribution can have different factors in different gaming scenarios.
Revised: The influence distribution can have different factors in different gaming scenarios.

Original: We demonstrate a set of commonly used factors in boxing games Won et al.
Revised: We demonstrate a set of commonly used factors in boxing games Won et al.

Original: (2021b) where agent i plays against opponent − i :
Revised: (2021b) where agent i plays against opponent − i :

Original: V =
Revised: V =

Original: ( A ki , A j − i , positions ( i, − 1) , orientations ( i, − i ) , linear velocities ( i, − i ) , angular velocities ( i, − i )) , in which the factors represent the chosen action k of agent i , the chosen action j of opponent − i , the relative positions, the relative moving orientations, the linear velocities and angular velocities of agent i and opponent − i .
Revised: ( A ki , A j − i , positions ( i, − 1) , orientations ( i, − i ) , linear velocities ( i, − i ) , angular velocities ( i, − i )) , in which the factors represent the chosen action k of agent i , the chosen action j of opponent − i , the relative positions, the relative moving orientations, the linear velocities and angular velocities of agent i and opponent − i .

Original: When a Feint policy π ′ i is added, we aim to maximize the effective influence range under the influence distribution of Feint .
Revised: When a Feint policy π ′ i is added, we aim to maximize the effective influence range under the influence distribution of Feint .

Original: Assuming the row agent i maintains a policy pool P i
Revised: Assuming the row agent i maintains a policy pool P i

Original: = { π 1 i , π Mi } , such influence distribution can be fused into Behavior Diversity measurement of the effective influence range by maximizing the discrepancy between the old influence effectiveness of policy occupancy measure ρ π E ( T ) and the influence effectiveness when adding Feint policy of new policy occupancy
Revised: = { π 1 i , π Mi } , such influence distribution can be fused into Behavior Diversity measurement of the effective influence range by maximizing the discrepancy between the old influence effectiveness of policy occupancy measure ρ π E ( T ) and the influence effectiveness when adding Feint policy of new policy occupancy

Original: ρ π ′ i ,π E − i ( V ′ ) :
Revised: ρ π ′ i ,π E − i ( V ′ ) :

Original: [Equation]
Revised: [Equation]

Original: where the general f -divergence is use to measure the discrepancy of two distributions.
Revised: where the general f -divergence is use to measure the discrepancy of two distributions.

Original: 3.2. C OLLECTIVE I MPACTS : I NFLUENCE D EGREE
Revised: 3.2. C OLLECTIVE I MPACTS : I NFLUENCE D EGREE

Original: Solely relying on Temporal Dimension and Spatial Dimension overlooks the interactions between them, and these two dimensions are expected to have mutual influences for a realistic modeling Liu et al.
Revised: Solely relying on Temporal Dimension and Spatial Dimension overlooks the interactions between them, and these two dimensions are expected to have mutual influences for a realistic modeling Liu et al.

Original: Therefore, we consider the influence degree, so the collective impacts of these two dimensions can be aggregated in a proper manner.
Revised: Therefore, we consider the influence degree, so the collective impacts of these two dimensions can be aggregated in a proper manner.

Original: We formulate the collective impacts for a Feint policy π
Revised: We formulate the collective impacts for a Feint policy π

Original: ′ i in a Multi-Player Game that starts at t 0 and end at T as:
Revised: ′ i in a Multi-Player Game that starts at t 0 and end at T as:

Original: [Equation]
Revised: [Equation]

Original: where temporal impacts Rew temporal (Section 3.2.1) are aggregated on spatial domain and spatial impacts Rew spatial (Section 3.2.2) are aggregated on temporal domain.
Revised: where temporal impacts Rew temporal (Section 3.2.1) are aggregated on spatial domain and spatial impacts Rew spatial (Section 3.2.2) are aggregated on temporal domain.

Original: µ 1 and µ 2 denote the weights of aggregated temporal impacts and spatial impacts respectively, enabling flexible adaption to different gaming scenarios.
Revised: µ 1 and µ 2 denote the weights of aggregated temporal impacts and spatial impacts respectively, enabling flexible adaption to different gaming scenarios.

Revised: They are initially set as 0 .

Revised: and are adjusted to achieve better performance with the iterations of training.

Original: In addition to the collective impact of Feint itself in terms of temporal domain and spatial domain, our formalized Feint impacts can also result in response diversity of opponents, since different related opponents (spatial domain) at different time steps (temporal domain) can have diverse response.
Revised: In addition to the collective impact of Feint itself in terms of temporal domain and spatial domain, our formalized Feint impacts can also result in response diversity of opponents, since different related opponents (spatial domain) at different time steps (temporal domain) can have diverse response.

Original: Such diversity can be used as a reward factor that make the final reward calculation more comprehensive Nieves et al.
Revised: Such diversity can be used as a reward factor that make the final reward calculation more comprehensive Nieves et al.

Original: Liu et al.
Revised: Liu et al.

Original: Thus, to incorporate such diversity together with our final reward calculation model, we refer to Liu et al.
Revised: Thus, to incorporate such diversity together with our final reward calculation model, we refer to Liu et al.

Original: (2021) to characterize the diversity gain incurred by our collective impact formalization.
Revised: (2021) to characterize the diversity gain incurred by our collective impact formalization.

Original: When the impact Rew collective of Feint policy π M +1 in a M × N payoff matrix A P
Revised: When the impact Rew collective of Feint policy π M +1 in a M × N payoff matrix A P

Original: i × P
Revised: i × P

Original: i at when opponents choose policy π j − i is collectively calculated, the derived diversity gain can be measured as follows:
Revised: i at when opponents choose policy π j − i is collectively calculated, the derived diversity gain can be measured as follows:

Original: [Equation]
Revised: [Equation]

Original: where D ( a M +1 ||
Revised: where D ( a M +1 ||

Original: A P
Revised: A P

Original: i × P i )
Revised: i × P i )

Original: represents the diversity gain of the Feint action on current policy space.
Revised: represents the diversity gain of the Feint action on current policy space.

Original: We follow the method in Liu et al.
Revised: We follow the method in Liu et al.

Original: (2021) for the quantification of diversity gain.
Revised: (2021) for the quantification of diversity gain.

Original:  F EINT I MPLEMENTATIONS 4. I MPLEMENTING THE F ORMALIZATION AS A C OLLECTIVE R EWARD C ALCULATION
Revised:  F EINT I MPLEMENTATIONS 4. I MPLEMENTING THE F ORMALIZATION AS A C OLLECTIVE R EWARD C ALCULATION

Original: To provide a comprehensive reward calculation model for Feint in Multi-Player Games, we synthesis the above collective impacts and collective diversity gain into the overall Reward Calculation Model.
Revised: To provide a comprehensive reward calculation model for Feint in Multi-Player Games, we synthesize the above collective impacts and collective diversity gain into the overall Reward Calculation Model.

Original: The robustness of similar design idea is proved in Liu et al.
Revised: The robustness of similar design idea is proved in Liu et al.

Original: (2021) that the synthesised direct impacts and diversity gain can provide a more comprehensive reward calculation model for each player.
Revised: (2021) that the synthesised direct impacts and diversity gain can provide a more comprehensive reward calculation model for each player.

Original: Thus, we synthesis the collective impacts (Equation 5) and collective diversity gain (Equation 6) for a Feint policy π ′ i into the overall Collective Reward Calculation Model by applying weighted sum λ 1 of collective impact and λ 2 of collective diversity gain: 
Revised: Thus, we synthesize the collective impacts (Equation 5) and collective diversity gain (Equation 6) for a Feint policy π ′ i into the overall Collective Reward Calculation Model by applying weighted sum λ 1 of collective impact and λ 2 of collective diversity gain: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 4. F OR I NFERENCE L EARNING M ODELS
Revised: 4. F OR I NFERENCE L EARNING M ODELS

Original: Inference Learning module is used to predict whether an observed action is Feint or not and policy parameters of θ − i and policy dynamics U − i .
Revised: Inference Learning module is used to predict whether an observed action is Feint or not and policy parameters of θ − i and policy dynamics U − i .

Original: Model-based approaches use an explicit model to fit an agent with the learning strategies of other agents from the observation Kim et al.
Revised: Model-based approaches use an explicit model to fit an agent with the learning strategies of other agents from the observation Kim et al.

Original: but often suffer from the infinite recursion problem when an agent the model models the agent it self
Revised: but often suffer from the infinite recursion problem when an agent the model models the agent it self

Original: Tesauro (2003).
Revised: Tesauro (2003).

Original: Blei et al.
Revised: Blei et al.

Original: (2016) proposes a model-free approach using approximate variational inference and Kim et al.
Revised: (2016) proposes a model-free approach using approximate variational inference and Kim et al.

Original: (2022) optimizes a tractable evidence lower bound to infer accurate latent strategies of others.
Revised: (2022) optimizes a tractable evidence lower bound to infer accurate latent strategies of others.

Original: We add a random weight onto the ELBO to fit the randomness and uncertainty incurred by Feint actions.
Revised: We add a random weight onto the ELBO to fit the randomness and uncertainty incurred by Feint actions.

Original: Specifically, the random-weighted ELBO is defined together with an encoder p (ˆ z −
Revised: Specifically, the random-weighted ELBO is defined together with an encoder p (ˆ z −

Original: i k +1 |
Revised: i k +1 |

Original: τ i 0: t ; ϕ ienc ; γ ienc ) and a decoder p ( a − i t | s t , ˆ z − i t ; ϕ idec ; γ idec ) parameterized by a set of encoder and corresponding random weighted decision parameters { ϕ ienc ; γ ienc } and a set of decoder and corresponding random weighted decision parameters { ϕ idec ; γ idec } :
Revised: τ i 0: t ; ϕ ienc ; γ ienc ) and a decoder p ( a − i t | s [Equation] , ˆ z − i t ; ϕ idec ; γ idec ) parameterized by a set of encoder and corresponding random weighted decision parameters { ϕ ienc ; γ ienc } and a set of decoder and corresponding random weighted decision parameters { ϕ idec ; γ idec } :

Original: [Equation]
Revised: [Equation]

Original: where ˆ z − i t are latent strategies that represent inferred policy parameters of other agents θ − i t and τ i 0:
Revised: where ˆ z − i t are latent strategies that represent inferred policy parameters of other agents θ − i t and τ i 0:

Original: t = { s 0 , a i 0 , a − i 0 , r io , ..., s t } denotes i ’s trajectories up to timestep t .
Revised: t = { s 0 , a i 0 , a − i 0 , r io , ..., s t } denotes i ’s trajectories up to timestep t .

Original: The random weight parameters γ ienc and γ idec enable agents to randomly guess the probability of whether an action is a Feint action or not from the observations, since no inferred policy can properly select a Feint action.
Revised: The random weight parameters γ i enc and γ idec enable agents to randomly guess the probability of whether an action is a Feint action or not from the observations, since no inferred policy can properly select a Feint action.

Original: 4. F OR R EINFORCEMENT L EARNING M ODELS
Revised: 4. F OR R EINFORCEMENT L EARNING M ODELS

Original: Reinforcement Learning module updates the policy using various gradient ascending mechanisms
Revised: Reinforcement Learning module updates the policy using various gradient ascending mechanisms

Original: Lowe et al.
Revised: Lowe et al.

Original: (2017); Iqbal & Sha (2019)
Revised: (2017); Iqbal & Sha (2019)

Original: ; Yu et al.
Revised: ; Yu et al.

Original: Although these designs are different, one of the key components in policy updating is the reward function Q , which directly evaluates the reward of the policy and guides the policy updating process.
Revised: Although these designs are different, one of the key components in policy updating is the reward function Q , which directly evaluates the reward of the policy and guides the policy updating process.

Original: Our proposed reward calculation mechanism (Section 3) can thus be fused into the MARL model by replacing the Q functions.
Revised: Our proposed reward calculation mechanism (Section 3) can thus be fused into the MARL model by replacing the Q functions.

Original: Such replacement can directly provide temporal, spatial and their collective considerations in policy making and keep the main structures of current MARL models.
Revised: Such replacement can directly provide temporal, spatial and their collective considerations in policy making and keep the main structures of current MARL models.

Original: Thus, Feint actions can be effectively fused into policy learning process in current MARL models based on our reward calculation mechanism.
Revised: Thus, Feint actions can be effectively fused into policy learning process in current MARL models based on our reward calculation mechanism.

Original: We demonstrate the feasibility of fusion using two state-of-the-art MARL models, MADDPG Lowe et al.
Revised: We demonstrate the feasibility of fusion using two state-of-the-art MARL models, MADDPG Lowe et al.

Original: (2017) and MAAC Iqbal & Sha (2019) below.
Revised: (2017) and MAAC Iqbal & Sha (2019) below.

Original: In a K -agent game, agents { a 1 , ..., a k } maintain policies π
Revised: In a K -agent game, agents { a 1 , ..., a k } maintain policies π

Original: = { π 1 , ..., π k } parameterized by θ = { θ 1 , ..., θ k } , where the policies for other agents are inferred from the Inference Learning module 4.2.
Revised: = { π 1 , ..., π k } parameterized by θ = { θ 1 , ..., θ k } , where the policies for other agents are inferred from the Inference Learning module 4.2.

Original: Each agent i evaluates the expected reward Q πi ( s, a 1 , ..., a n ) at state s using reward term
Revised: Each agent i evaluates the expected reward Q πi ( s, a 1 , ..., a n ) at state s using reward term

Original: Rew i ( s ) in Equation 8.
Revised: Rew i ( s ) in Equation 8.

Original: In MADDPG Lowe et al.
Revised: In MADDPG Lowe et al.

Original: (2017) model,
Revised: (2017) model,

Original: centralized critic Q πi ( s, a 1 , ..., a n ) is used on decentralized execution to provide global information that can stabilize training.
Revised: centralized critic Q πi ( s, a 1 , ..., a n ) is used on decentralized execution to provide global information that can stabilize training.

Original: The centralized critic can be replaced by our reward calculation mechanism and the gradient descent of the policy learning is:
Revised: The centralized critic can be replaced by our reward calculation mechanism and the gradient descent of the policy learning is:

Original: [Equation]
Revised: [Equation]

Original: (10) where Rew i ( s ) is naturally fused into the gradient update function and provide influence range, influence degree and influence time length for adjusting learning outcomes.
Revised: (10) where Rew i ( s ) is naturally fused into the gradient update function and provide influence range, influence degree and influence time length for adjusting learning outcomes.

Original: In MAAC model Iqbal & Sha (2019), although the policy update mechanisms vary, the key part of policy updates is still the Q πi ( s, a 1 , ..., a n ) function, which can also be replaced by our reward calculation term.
Revised: In MAAC model Iqbal & Sha (2019), although the policy update mechanisms vary, the key part of policy updates is still the Q πi ( s, a 1 , ..., a n ) function, which can also be replaced by our reward calculation term.

Original: The slight difference is that MAAC uses a temperature parameter α to balance between the entropy and rewards.
Revised: The slight difference is that MAAC uses a temperature parameter α to balance between the entropy and rewards.

Original: Therefore, the gradient ascent for updating policy is:
Revised: Therefore, the gradient ascent for updating policy is:

Original: [Equation]
Revised: [Equation]

Original: 11) where Rew i ( s ) replace the original Q πi ( s, a 1 , ..., a n ) function and the multi-agent advantage function b ( o, a ı ) , since the expected return of multi-action interactions are already comprehensively calculated in Rew i ( s ) .
Revised: where Rew i ( s ) replace the original Q πi ( s, a 1 , ..., a n ) function and the multi-agent advantage function b ( o, a ı ) , since the expected return of multi-action interactions are already comprehensively calculated in Rew i ( s ) .

Original: Thus, our proposed reward calculation can be seamlessly fused into state-ofthe-art MARL models with simple replacement of the original Q -function while guaranteeing the feasibility.
Revised: Thus, our proposed reward calculation can be seamlessly fused into state-ofthe-art MARL models with simple replacement of the original Q -function while guaranteeing the feasibility.

Original:  E XPERIMENTAL M ETHODOLOGY
Revised:  E XPERIMENTAL M ETHODOLOGY

Original: Testbed Implementations.
Revised: Testbed Implementations.

Original: We implement two complex Multi-player Games to examine the effectivenss of our design.
Revised: We implement two complex Multi-player Games to examine the effectivenss of our design.

Original: We first re-implement and extend a strategic real-world game, AlphaS- tar Arulkumaran et al.
Revised: We first re-implement and extend a strategic real-world game, AlphaStar Arulkumaran et al.

Original: (2019), which is widely used as the experimental testbed in recent studies of Reinforcement Learning studies Risi & Preuss (2020);
Revised: (2019), which is widely used as the experimental testbed in recent studies of Reinforcement Learning studies Risi & Preuss (2020);

Original: Liu et al.
Revised: Liu et al.

Original: We make extra efforts to emulate a six-player game, where players are free to have convoluted interactions with each other.
Revised: We make extra efforts to emulate a six-player game, where players are free to have convoluted interactions with each other.

Original: And we implement Feint as dynamically generated policies, based on the 888 regular gaming policies.
Revised: And we implement Feint as dynamically generated policies, based on the 888 regular gaming policies.

Original: Then, we create a complex multi-player tagging game, based on Multi-Agent Particle Environment Mordatch & Abbeel (2017), an open-source environment from OpenAI.
Revised: Then, we create a complex multi-player tagging game, based on Multi-Agent Particle Environment Mordatch & Abbeel (2017), an open-source environment from OpenAI.

Original: We handcraft a tagging game scenario, where six agents can freely fight with each other with 30 nuanced and flexible actions.
Revised: We handcraft a tagging game scenario, where six agents can freely fight with each other with 30 nuanced and flexible actions.

Original: Such an implementation requires an extensive amount of efforts since current available codebases only have a limited set of actions, which are insufficient to demonstrate the impacts of
Revised: Such an implementation requires an extensive amount of efforts since current available codebases only have a limited set of actions, which are insufficient to demonstrate the impacts of

Original: Feint .
Revised: Feint .

Original: We follow the methodology from Wampler et al.
Revised: We follow the methodology from Wampler et al.

Original: ; Won et al.
Revised: ; Won et al.

Original: (2021a) to form Feint actions, based on 30 hand-crafted actions.
Revised: (2021a) to form Feint actions, based on 30 hand-crafted actions.

Original: Our tagging game resembles intense free fight scenarios in ancient Roman free fight scenarios Matz (2019), where interactions are intense and Feint is expected to be effective.
Revised: Our tagging game resembles intense free fight scenarios in ancient Roman free fight scenarios Matz (2019), where interactions are intense and Feint is expected to be effective.

Original: Experiment Procedure.
Revised: Experiment Procedure.

Original: We choose MADDPG Lowe et al.
Revised: We choose MADDPG Lowe et al.

Original: (2017) and MAAC Iqbal & Sha (2019) model in our experiments.
Revised: (2017) and MAAC Iqbal & Sha (2019) model in our experiments.

Original: We first train all six agents without Feint from our formalization on the state-of-the-art MARL models.
Revised: We first train all six agents without Feint from our formalization on the state-of-the-art MARL models.

Original: Then we randomly select 3 agents (always labeled as Agent 1, 2, and 3), who incorporate our formalization of Feint , and keep the other 3 agents regular.
Revised: Then we randomly select 3 agents (always labeled as Agent 1, 2, and 3), who incorporate our formalization of Feint , and keep the other 3 agents regular.

Original: All experiments are done with 4,000 training iterations on each model and 150 gaming iterations.
Revised: All experiments are done with 4,000 training iterations on each model and 150 gaming iterations.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We examine the effects of Feint using ➊ gaming rewards of training, ➋ diversity gain of policy space and ➌ overhead of computation load.
Revised: We examine the effects of Feint using ➊ gaming rewards of training, ➋ diversity gain of policy space and ➌ overhead of computation load.

Original: We first examine the learning outcomes (i.e. rewards) trained using both MADDPG and MAAC MARL model, by comparing the rewards of agents across all scenarios.
Revised: We first examine the learning outcomes (i.e. rewards) trained using both MADDPG and MAAC MARL model, by comparing the rewards of agents across all scenarios.

Original: We then examine the effects of Feint actions on how Feint can improve the diversity of gaming policies (Section 3).
Revised: We then examine the effects of Feint actions on how Feint can improve the diversity of gaming policies (Section 3).

Original: Finally, we perform overhead analysis, incurred by fusing Feint formalization in strategy learning.
Revised: Finally, we perform overhead analysis, incurred by fusing Feint formalization in strategy learning.

Original:  E XPERIMENTAL R ESULTS 6.G AMING R EWARD G AIN 
Revised:  E XPERIMENTAL R ESULTS 6.

Revised: [Figure]

Original: Figure 1 shows the rewards for each agent in two scenarios using MADDPG model.
Revised: Figure 1 shows the rewards for each agent in two scenarios using MADDPG model.

Original: We make three observations.
Revised: We make three observations.

Original: First, in ➊ in Figure 1, when no Feint is enabled, all agents’ rewards tend to progress to a similar level when after enough training iterations.
Revised: First, in ➊ in Figure 1, when no Feint is enabled, all agents’ rewards tend to progress to a similar level when after enough training iterations.

Original: However, in ➊ in Figure 1, when Feint actions are enabled on agent 1, 2, and 3, these agents gain significantly higher rewards than the agents who are not enabled to perform Feint actions (agent 4, 5, and 6).
Revised: However, in ➊ in Figure 1, when Feint actions are enabled on agent 1, 2, and 3, these agents gain significantly higher rewards than the agents who are not enabled to perform Feint actions (agent 4, 5, and 6).

Original: Second, when comparing ➊ with ➋ , the average rewards for agents who perform Feint actions (agent 1, 2, and 3 in ➋ ) is around 9.5, which is higher than the average rewards (around 5.0) for agents who do not perform Feint actions (all agents in ➊ ).
Revised: Second, when comparing ➊ with ➋ , the average rewards for agents who perform Feint actions (agent 1, 2, and 3 in ➋ ) is around 9.5, which is higher than the average rewards (around 5.0) for agents who do not perform Feint actions (all agents in ➊ ).

Original: These two observations demonstrate that our formalized Feint can provide effective improvement for agents’ rewards.
Revised: These two observations demonstrate that our formalized Feint can provide effective improvement for agents’ rewards.

Original: Third, the training results before 1000 iterations are not stable for both scenarios.
Revised: Third, the training results before 1000 iterations are not stable for both scenarios.

Original: This is mainly because the natural characteristics of MADDPG training process in which the training stables generally after 2000 iterations Lowe et al.
Revised: This is mainly because the natural characteristics of MADDPG training process in which the training stables generally after 2000 iterations Lowe et al.

Original: Figure 2 shows the rewards for each agent in two scenarios using MAAC model.
Revised: Figure 2 shows the rewards for each agent in two scenarios using MAAC model.

Original: When comparing to the results trained with MADDPG model, though the rewards differ in specific numbers, similar trends and observations are shown.
Revised: When comparing to the results trained with MADDPG model, though the rewards differ in specific numbers, similar trends and observations are shown.

Original: First, when no Feint are enabled, all agents’ rewards tend to progress to a similar level when after enough training training iterations ( ➊ in Figure 2), while when
Revised: First, when no Feint are enabled, all agents’ rewards tend to progress to a similar level when after enough training training iterations ( ➊ in Figure 2), while when

Original: Feint actions are enabled on agent 1, 2, and 3, these agents gain significantly higher rewards than the agents who are not enabled to perform Feint actions (agent 4, 5, and 6) ( ➋ in Figure 2).
Revised: Feint actions are enabled on agent 1, 2, and 3, these agents gain significantly higher rewards than the agents who are not enabled to perform Feint actions (agent 4, 5, and 6) ( ➋ in Figure 2).

Original: Second, when comparing ➊ with ➋ , the average rewards for agents who perform Feint actions (agent 1, 2, and 3 in ➋ ) is around 10.0, which is higher than the average rewards (around 3.5) for agents who do not perform Feint actions (all agents in ➊ ).
Revised: Second, when comparing ➊ with ➋ , the average rewards for agents who perform Feint actions (agent 1, 2, and 3 in ➋ ) is around 10.0, which is higher than the average rewards (around 3.5) for agents who do not perform Feint actions (all agents in ➊ ).

Original: And third, training in MAAC also suffers unstable results before the first 1000 iterations.
Revised: And third, training in MAAC also suffers unstable results before the first 1000 iterations.

Original: With the comparison of the results for MADDPG and MAAC model, we have an addition observation that our formalized reward calculation for Feint actions and the MARL fusion can be effectively adapted on current state-of-the-art MARL models, providing promising feasibility and scalability for extension studies.
Revised: With the comparison of the results for MADDPG and MAAC model, we have an addition observation that our formalized reward calculation for Feint actions and the MARL fusion can be effectively adapted on current state-of-the-art MARL models, providing promising feasibility and scalability for extension studies.

Original: 6. D IVERSITY G AIN
Revised: D IVERSITY G AIN

Original: To examine the impacts on the policy diversity in games, we perform a comparative study between
Revised: To examine the impacts on the policy diversity in games, we perform a comparative study between

Original: MARL training with and without Feint .
Revised: MARL training with and without Feint .

Original: Specifically, We use Exploitability and Population Efficacy (PE) to measure the diversity gain in the policy space.
Revised: Specifically, We use Exploitability and Population Efficacy (PE) to measure the diversity gain in the policy space.

Original: Exploitability Lanctot et al.
Revised: Exploitability Lanctot et al.

Original: (2017) measures the distance of a joint policy chosen by the multiple agents to the Nash equilibrium, indicating the gains of players compared to their best response.
Revised: (2017) measures the distance of a joint policy chosen by the multiple agents to the Nash equilibrium, indicating the gains of players compared to their best response.

Original: The mathematical expression of Exploitability is expressed as:
Revised: The mathematical expression of Exploitability is expressed as:

Original: [Equation]
Revised: [Equation]

Original: where π i stands for the policy of agent i and π − i stands for the joint policy of other agents.
Revised: where π i stands for the policy of agent i and π − i stands for the joint policy of other agents.

Original: Rew i denotes our formalized Reward Calculation Model (Section 4.1).
Revised: Rew i denotes our formalized Reward Calculation Model (Section 4.1).

Original: Thus, small Exploitability values show that the joint policy is close to Nash Equilibrium, showing higher diversity.
Revised: Thus, small Exploitability values show that the joint policy is close to Nash Equilibrium, showing higher diversity.

Original: In addition, we also use Population Efficacy (PE)
Revised: In addition, we also use Population Efficacy (PE)

Original: Liu et al.
Revised: Liu et al.

Original: (2021) to measure the diversity of the whole policy space.
Revised: (2021) to measure the diversity of the whole policy space.

Original: PE is a generalized opponent-free concept of Exploitability by looking for the optimal aggregation in the worst cases, which is expressed as:
Revised: PE is a generalized opponent-free concept of Exploitability by looking for the optimal aggregation in the worst cases, which is expressed as:

Original: [Equation]
Revised: [Equation]

Original: where π i stands for the policy of agent i and π − i stands for the joint policy of other agents.
Revised: where π i stands for the policy of agent i and π − i stands for the joint policy of other agents.

Original: α denotes an optimal aggregation where agents owning the population optimizes towards.
Revised: α denotes an optimal aggregation where agents owning the population optimizes towards.

Original: Rew i denotes our formalized Reward Calculation Model (Section 4.1) and opponents can search over the entire policy space.
Revised: Rew i denotes our formalized Reward Calculation Model (Section 4.1) and opponents can search over the entire policy space.

Original: PE gives a more generalized measurement of diversity gain from the whole policy space.
Revised: PE gives a more generalized measurement of diversity gain from the whole policy space.

Original: Figure 3 shows the experimental results for evaluating diversity gains.
Revised: Figure 3 shows the experimental results for evaluating diversity gains.

Original: From the figure, we obtain two observations.
Revised: From the figure, we obtain two observations.

Original: First, agents that can dynamically perform Feint actions (Agent 1, 2, and 3) achieve lower Exploitability (around 4 .
Revised: First, agents that can dynamically perform Feint actions (Agent 1, 2, and 3) achieve lower Exploitability (around 4 .

Original: 9 × 10 − 2 )
Revised: 9 × 10 − 2 )

Original: compared to agents who perform regular actions (around 9 .
Revised: compared to agents who perform regular actions (around 9 .

Original: 7 × 10 − 2 ) and have higher PE (lower negative PE - around 5 .
Revised: 7 × 10 − 2 ) and have higher PE (lower negative PE - around 5 .

Original: 3 × 10 − 2 ) than those who only perform regular actions (around 1 . 2 × 10 − 2 ).
Revised: 3 × 10 − 2 ) than those who only perform regular actions (around 1 . 2 × 10 − 2 ).

Original: This result shows that our formalized Feint can effectively increase the the diversity and effectiveness of policy space.
Revised: This result shows that our formalized Feint can effectively increase the the diversity and effectiveness of policy space.

Original: Second, agents with
Revised: Second, agents with

Original: Feint have slightly higher variations in both metrics.
Revised: Feint have slightly higher variations in both metrics.

Original: This is because Feint naturally incurs more randomness (e.g. succeed or not) in games, resulting in higher variations in metrics.
Revised: This is because Feint naturally incurs more randomness (e.g. succeed or not) in games, resulting in higher variations in metrics.

Original: 6. O VERHEAD A NALYSIS
Revised: 6. O VERHEAD A NALYSIS

Original: Figure 4 shows the results of our overhead analysis.
Revised: Figure 4 shows the results of our overhead analysis.

Original: We make two observations.
Revised: We make two observations.

Original: First, fusing Feint in MARL training do incur some overhead increment in terms of running time.
Revised: First, fusing Feint in MARL training do incur some overhead increment in terms of running time.

Original: This is because the formalization and fusion of Feint in MARL incur additional calculation load.
Revised: This is because the formalization and fusion of Feint in MARL incur additional calculation load.

Original: Secondly, in both MADDPG models and MAAC models, the increased overhead is generally lower than 5% , which still indicates that our proposed formalization of Feint actions can have enough feasibility and scalability on fusing with MARL models.
Revised: Secondly, in both MADDPG models and MAAC models, the increased overhead is generally lower than 5% , which still indicates that our proposed formalization of Feint actions can have enough feasibility and scalability on fusing with MARL models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present the first formalization, implementation and quantitative evaluations of Feint in MultiPlayer Games.
Revised: We present the first formalization, implementation and quantitative evaluations of Feint in MultiPlayer Games.

Original: Our work formalizes, implements and quantitatively examines Feint in Multi-Player
Revised: Our work formalizes, implements and quantitatively examines Feint in Multi-Player

Original: Games, on the temporal, spatial and their collective impacts.
Revised: Games, on the temporal, spatial and their collective impacts.

Original: The results show that our design of
Revised: The results show that our design of

Original: Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of the time consumption.
Revised: Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of the time consumption.

Original: We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting.
Revised: We conclude that our design of Feint is effective and practical, to make Multi-Player Games more interesting.

Original: Ourdesign is also expectedto be applicable for future models of Multi-Player Games.
Revised: Our expected