Self-Explaining Deviations for Coordination
Anonymous Author(s)
AffiliationAddress email
Abstract
Fully cooperative, partially observable multi-agent problems are ubiquitous in thereal world.
In this paper, we focus on a specific subclass of coordination problemsin which humans are able to discover self-explaining deviations (SEDs).
SEDs areactions that deviate from the common understanding of what reasonable behaviorwould be in normal circumstances.
They are taken with the intention of causinganother agent or other agents to realize, using theory of mind, that the circumstancemust be abnormal.
We first motivate SED with a real world example and formalizeits definition.
Next, we introduce a novel algorithm, improvement maximizingself-explaining deviations (IMPROVISED), to perform SEDs.
Lastly, we evaluate
IMPROVISED both in an illustrative toy setting and the popular benchmark setting
Hanabi, where it is the first method to produce so called finesse plays, which areregarded as one of the more iconic examples of human theory of mind
Introduction
Dispatcher: Oregon 911. Caller: I would like to order a pizza at... Dispatcher: You called 911 to order a pizza?
Caller: Uh, Yeah, apartment... Dispatcher: This is the wrong number to call for a pizza.
Caller: No no no... you’re not understanding me. Dispatcher: I’m getting you now.
Is the other guy still there?
Caller: Yep.
I need a large pizza.
Dispatcher: All right.
How about medical.
You need medical?
Caller: No. With pepperoni. Dispatcher: Turn your sirens off before you get there.
Caller ordered a pizza.
And agreed with everything I said that there’s domestic violence going on.
There has been a lot of progress in the deep multiagent collaboration domain ranging from better learning [34, 10] and planning [19, 29] algorithms designed to achieve strong performance in self-playto finding policies that collaborate well with otheragents include humans [11, 31].
However, these algorithms largely assume that the partners at test timewill follow a fixed policy.
Even in one of the mostpowerful search algorithms [19], the agent performing search assumes that other agents follow blueprintwhen planning forward, resulting in primarily unilateral deviations that improve the performance withoutrelying on other players also deviating in response.
Humans generally assume other human follow certainsocial norms (blueprint policies) when acting in thesociety, and interpret other humans’ behavior basedon such norms to infer hidden information about thepartially observable world.
For example, when adriver sees a car ahead suddenly stopping on a road,they may infer that there is some accident ahead orthe car is broken.
However, human also have theability to improvise when the conventional actions are restricted or there exist other actions that canbring superior outcomes in situations.
Taking the event in Figure 1 as an example, the caller deviatesfrom the conventional practice of stating the situation truthfully because it has negative consequences.
In this situation, the caller would perform this deviation because the deviation is only beneficial if the dispatcher figures out caller’s real intention and also deviates instead of simply ending the call.
This type of deviation is interesting because, firstly, it can be detected by the other person becauseit appears to be a mistake under common practice and, secondly, the expected deviation for theother person in response can be independently decided by both individuals based on the commonunderstanding of the world and the deviation chosen by the first person.
We refer to this type ofphenomena as self-explaining deviations (SEDs).
SEDs are actions that, under normal circumstances, would not make sense, given the agents’ commonunderstanding.
They are executed with the intention that teammates will use theory of mind to deducethat the situation is unusual in a particular and important way, and will adapt their behavior to accountfor this additional information.
While there is ample evidence that humans perform SEDs, such asthe finesse play in the card game Hanabi, we provide some evidence that neither existing planningnor learning algorithms can do so.
In this paper, we first formalize the problem setting and definition of SEDs.
Then we introduce anovel planning algorithm, IMPROVement maxImizing Self-Explaining Deviations (IMPROVISED),for performing SEDs.
We show that, under some assumptions, IMPROVISED performs the optimal
SED in terms of expected return maximization.
Next, we provide a motivating experiment, illustratingthat in a small toy problem designed to require SEDs to perform optimally, IMPROVISED is able tocompute an optimal joint policy, whereas other multi-agent algorithms are not.
Lastly, we presentexperiments on Hanabi, where we show that IMPROVISED is able to produce finesse plays, which isone of the most interesting techniques that human experts perform frequently.
Background
FOSG and Public POMDP.
For our notation, we use an adaption of factored observation stochasticgames ( FOSG ) [17].
W is the set of world states and w 0 is a designated initial state.
A =
A 1 ×· · ·×A N is the space of joint actions .
T is the transition function mapping W ×A → ∆( W ) .
R : W ×
A → R is the reward function .
O =
( O priv (1) , . . .
, O priv ( N ) , O pub ) is the observationfunction where O priv ( i ) :
W × A × W → O priv ( i ) specifies the private observation that player ireceives.
O pub : W ×
A × W → O pub specifies the public observation that all players receive.
O i
= O i ( w, a, w ′ )=(
O priv ( i ) ( w, a, w ′ ) , O pub ( w, a, w ′ )) is player
i ’s observation and a history is afinite sequence
h = ( w 0 , a 0 , . . . , w t ) .
The set of histories is denoted by H .
The information statefor player i at h = ( w 0 , a 0 , . . . , w t ) is s i ( h ) :
= ( O 0 i , a 0 i , . . .
, O ti ) .
The information state space forplayer i is S i :=
{ s i ( h ) | h ∈ H} .
The legal actions for player i at s i is denoted A i ( s i ) .
A jointpolicy is a tuple π = ( π 1 , . . .
, π N ) , where policy π i maps S i → ∆( A i ) .
The public state at h isthe sequence s pub ( h ) :
= s pub ( s i ( h )) :
= ( O 0 pub , . . .
, O t pub ) .
The information state set for player i ats ∈ S pub is S i ( s ) :
= { s i ∈ S i | s pub ( s i ) =
s } , where S pub is the space of public states.
Finally, thereach probability of h under π is P π ( h ) .
Rather than working with in a multi-agent setting directly, we invoke the public POMDPtransformation that maps cooperative multi-agent settings to equivalent single-agent POMDPs.
Given a common-payoff FOSG ⟨N , W , w 0 , A , T , R , O⟩ , we can construct an equivalent public
POMDP [21] ⟨ ˜ W , ˜ w 0 , ˜ A , ˜ T , ˜ R , ˜ O⟩ as follows:
The world states of the public POMDP ˜ W is theset { ( s 1 ( h ) , . . .
, s N ( h )) : h ∈ H} .
The initial world state of the public POMDP ˜ w 0 is the tuple( s 1 ( h 0 ) , . . .
, s N ( h 0 )) .
The actions of the public POMDP are called joint decision rules .
It is denotedby Γ and has N components.
The i th component of it Γ i is the decision rule for player i .
A decision rule Γ i maps s i to an element of A i ( s i ) for each s i ∈ S
i ( s pub ( h ))
; it instructs a playerin the common-payoff FOSG how to act as a function of its private information.
Given ˜ w ≡( s 1 , . . .
, s n ) and Γ , the transition distribution ˜ T ( ˜ w, Γ) is induced by T (( s 1 , . . .
, s n ) , a ) , where a ≡Γ(( s 1 , . . .
, s n ))
:=
(Γ 1 ( s 1 ) , . .
, Γ N ( s N )) 1 .
Given ˜ w ≡ ( s 1 , . . .
, s n ) and ˜ w ′ ≡ ( s ′ 1 , . .
, s ′ n ) , thereward and observation are given by ˜ R ( ˜ w, Γ , ˜ w ′ )
≡ R (( s 1 , . . . , s n ) , Γ(( s 1 , . . . , s n )) , ( s ′ 1 , . . .
, s ′ n ))and ˜ O ( ˜ w, Γ , ˜ w ′ )
≡ O pub (( s 1 , . . . , s n ) , Γ(( s 1 , . . . , s n )) , ( s ′ 1 , . . .
, s ′ n )) , respectively.
Every policy in the public POMDP corresponds to a joint policy in the underlying common-payoff
FOSG, which receives exactly the same expected return.
Therefore, it is sufficient to work with thepublic POMDP.
See [28, 29] for further discussion.
When the public POMDP is considered as a belief MDP, rather than as a POMDP, its belief states areof the form b t : ( s t 1 , . . .
, s tn )
(cid:55)→ P Γ 0 ,..., Γ t − 1 ( s t 1 , . . .
, s tN | s t pub ) .
In words, this is the joint distributionover private information states, conditioned on the historical policy (Γ 0 , . . .
, Γ t − 1 ) and the publicstate s t pub .
This public belief MDP is abbreviated as the PuB-MDP.
Self-Explaining Deviations
While the name self-explaining deviation (SED) is novel to this work, the idea behind SEDs is not.
Perhaps the most famous example of SEDs comes from the cooperative card game Hanabi [1] inthe form of a play called a “finesse”.
Readers unfamiliar with Hanabi and finesse may first jump
Section 5.2 and Section 5.3 for the detailed descriptions.
In a finesse play, the acting player, i.e., thefirst player who initiates the deviation as part of the finesse, intentionally communicates misleadinginformation to a receiving player, which would hurt the team’s score if the receiving player actedupon it using the prior convention.
The second player in the finesse, who acts after the first player butbefore the receiving player, realizes that the first player must have deviated from the prior conventionafter observing the seemingly disastrous move.
However, knowing that the first player is rational,the second player realizes that they can also deviate to reach a better outcome than the one that theoriginal convention would have led to.
After the second player plays its part of the joint deviation,the original information from the first player is no longer misleading and the subsequent players cancontinue to follow the prior conventions.
SEDs may take place in any cooperative situation with at least two players where common knowledgeblueprint policies that players follow under normal assumptions exist.
Intuitively, SEDs capturea form of joint deviations where one agent takes an action that at first appears to be a mistake orotherwise highly off-policy from another player’s perspective, i.e. zero or low probability under theblueprint.
On the presumption that the first agent chose that action intelligently and deliberately andthere is a unique off-policy action for the observing agent that could potentially result in an evenbetter-than-normal outcome for both, the observing agent may reason that the first agent “intends”them to take it—to take a leap of faith that the first agent has not erred, but rather knows both agentscan get that better outcome, even if the observing agent does not have the information themselves toprove that this outcome will result.
This work uses the term SEDs to describe this phenomenon ofcommunication via apparent mistakes in a general context.
We formalize SED in Section 3.1 andpropose IMPROVISED, a novel planning algorithm that performs the optimal SEDs under someassumptions, in Section 4.
3.1 Examining Self-Explaining Deviations
Having defined the problem setting, we are nowready to investigate SEDs.
To facilitate our investigation, let’s consider the trampoline-tigergame in Figure 2 as an example.
In the game,
Alice is standing on a balcony, while Bob standson the ground next to a lever.
Pulling the leverwill either deploy a trampoline below the balcony, or release a tiger.
From above, Alice canobserve which is the case, and then Alice decides whether to jump off the balcony or not.
Bob observes Alice’s choice but does not knowwhether the lever will deploy a trampoline ortiger, and decides to pull the lever or not.
Alicewants to get down from the balcony, but will diefrom the fall unless Bob pulls the lever and itreleases a trampoline.
If Bob pulls the lever andreleases a tiger, it will eat Bob.
Consider the joint policy ( ∗ (cid:55)→ N , ∗ (cid:55)→ N ) .
While not optimal, this joint policy is not unreasonable,as Bob won’t be eaten and Alice won’t fall to her death.
Indeed, independent of the probability ofa trampoline, this joint policy is a Nash equilibrium.
In addition, as we shall see in Section 5.1,state-of-the-art MARL algorithms converge to this solution most of the time.
However, there is a clear opportunity for a SED under this blueprint.
If there is a tiger, Alice should obviously never jump.
So if Bob observes Alice choosing to jump, and trusts that Alice is intelligent and would only havechosen to jump if it could lead to a better result, Bob should realize that there must be a trampoline.
Therefore Alice should trust that Bob will pull the lever if she jumps, as a result.
Under any planning algorithm that considers only unilateral deviations, Bob and Alice can’t escapethis local optimum.
Alice will never jump because she is under the belief that Bob will choose not topull the lever.
Therefore, finding SEDs in general requires considering multi-lateral deviations, i.e.simultaneous deviations by more than one player.
Since finding multi-lateral deviations in general games with partially observable state is a complexproblem, we restrict our focus to settings satisfying the following assumptions:
Assumption1.
We assume sequential and publicly observable actions.
SEDs may still be possible with partially-observable actions, but for simplicity we focus on when
Bob can directly observe Alice deviate from the blueprint, rather than having to deduce it from partialor incomplete observations.
Assumption2.
We consider only cases where Bob’s private information is not necessary for coordinating on the SED.
For simplicity, we always refer to the first player who initiates the joint deviation as the Alice and thesecond player who figures out Alice’s intention and plays their part of the joint deviation as Bob inthe following discussions beyond the scope of this toy game.
In other words, we only consider SEDswhere Bob needs only act as a function of the common-knowledge state and the Alice’s action, ratheras a function of private knowledge of their own.
Under these assumptions, we define the SED as follows:
Definition1.
Given a common knowledge blueprint (BP) policy π , common public belief of Alice and
Bob b , and information state of Alice s 1 , SED is any pair of joint deviations ( a ′ 1 , a ′ 2 ) for Alice and
Bob that satisfy all the following conditions:
• q π ( b, s 1 , a ′ 1 , a ′ 2 ) ≥ q π ( b, s 1 ) , i.e., it gives higher expected future return than BP;
• P ( a ′ 1 | π, b ) ≤ ϵ , i.e., a ′ 1 is highly unlikely under π under the common belief;
• a ′ 2 ∼ f ( b, a ′ 1 ) , i.e., a ′ 2 is a function of the public belief and Alice’s deviation action.
Here, q π ( b, s 1 , [ a 1 , a 2 ]) is Alice’s estimate of the expected future return after executing the optionala 1 , a 2 and following π afterwards.
ϵ is a hyper-parameter.
The function f is determined by thealgorithm designer under the constraint that f only takes b and a ′ 1 as input.
It is used both by Alice topredict Bob’s response and by Bob to decide the response independently.
IMPROVISED
In this section we derive IMPROVISED, an algorithm to perform the SEDs.
The core of the algorithmis to find a function f under the constraint of SED and an optimization procedure that Alice uses todecide whether to deviate given her information state s 1 and possible responses a 2 ∼ f from Bob.
4.1 Defining the Optimization Problem
With the observations in the prior sections, we are ready to write selecting a SED as a PuB-MDPoptimization problem:
[Equation]
where Γ 1 is the prescription for the acting player at current time step and Γ 2 is the prescription forthe acting player at the subsequent time step, ◦ denotes function composition, and, in accordance with Definition 1, Γ 2 ranges over the space of A 1 .
The series of equalities above begins with ageneric multi-lateral search in the PuB-MDP.
The LHS is ranging over Alice and Bob’s decision rulessuch that, if Alice uses Γ 1 and Bob uses Γ 2 in belief state b , and all the players play according to πthereafter, the expected return is maximized.
The first equality holds without applying any additionalassumptions, simply based on the fact that Alice is able to observe s 1 .
The second equality holds byapplying Assumptions 1 and 2, which imply both that Bob can observe Alice’s action and that it isthe only thing he needs to condition his decision upon (note that the change of the set over which Γ 2ranges is left implicit).
The final holds due to the design decisions that a) our algorithm only plays an SED for actions not supported in the BP and that b) players opt into the BP when there is no betterdeviation.
The inner maximization expresses the fact that Alice can observe s 1 and may opt intothe blueprint if the blueprint value exceeds that of the deviation.
Because Bob knows which actionsare supported by the blueprint, he can opt in exactly when Alice opts in, and play according to thedeviation otherwise.
4.2 An Easier Special Case
Unfortunately, expression (1) remains difficult to optimize, as the number of Γ 1 and Γ 2 is combinatorial.
To ameliorate, we can apply the simplifying constraint of only allowing Alice and Bob to deviatefor a single action pair regardless of Alice’s information state s 1 .
For simplicity, we also set ϵ = 0 .
Let D ( b ) be the set of plausible deviation actions for Alice, i.e., action that cannot be played underblueprint: D ( b ) :
= { a 1 [-
-]∈
A 1 |∀ s 1 ∈ b : π ( a 1 | b, s 1 ) = 0 } .
Let R be the set of all plausible actionsthat Bob can play in response in every possible state, i.e., R :=
(cid:84) s 2 ∈S 2[-S 
-] A 2 ( s 2 ) .
If R = ∅ , Alicewill skip searching for SED this turn and follow BP.
Now we can augment the expected value of adeviation with the allowed range of actions:
[Equation]
Then we can rewrite equation (1) using our newly defined q function as follows:
[Equation]
These values could be found in O ( |S 1 ||A 1 ||A 2 | ) time, given the two inner q functions.
Moreover,as all of these depend only on the public belief state, both players can compute them independently,assuming that there are no ties.
Once the deviation pair is computed, Alice decides whether to proceed with the deviation or notgiven her private information: Alice plays a ∗ 1 if ˆ q π ( b, s 1 , a ∗ 1 , a ∗ 2 ) >
q π ( b, s 1 ) , and the blueprint actionotherwise.
It is possible that ( a ∗ 1 , a ∗ 2 ) is not a valid deviation pair if no plausible deviations exist, e.g.,a ∗ 1 could be outside of D ( b ) .
However, in this case ˆ q π ( b, s 1 , a ∗ 1 , a ∗ 2 ) is −∞ , and so Alice will resortto playing blueprint.
Bob can detect the deviation as D ( b ) is a public knowledge and a ∗ 1 ∈ D ( b ) .
Therefore, depending on his observation, he can either play a ∗ 2 or respond as usual.
We can represent the value of the policy that allows a single deviation at b and plays blueprintafterwards as: q ∗ π ( b ) = max a 1 ,a 2 E s 1 ∼ b max(ˆ q π [ b, s 1 , a 1 , a 2 ) , q π ( b, s 1 )] .
It is easy to see that q ∗ π ≥q π , i.e., this algorithm has a weak policy improvement guarantee.
If we apply this procedure to the tiger-trampoline game with a blueprint policy that always no-ops:q ∗ π ( b ) = max a 1 ,a 2 E s 1 ∼ b max[ˆ q π ( b, s 1 , a 1 , a 2 ) , q π ( b, s 1 )]
= max a 1 ,a 2 [ P ( tiger ) max(ˆ q π ( b, tiger , a 1 , a 2 ) , 0) + P ( trampoline ) max(ˆ q π ( b, trampoline , a 1 , a 2 ) , 0)] = P ( trampoline ) .
As expected, we obtain a policy improvement with value equal to the probability of a trampoline.
We discover the SED where Alice jumps when a trampoline is present and Bob pulls the lever.
Weprovide a proof-of-principle implementation of IMPROVISED in the trampoline-tiger game that canbe run online at https://bit.ly/3KtMLT6 .
4.3 Coordination by Extending Conventions
In a coordination context, we cannot require Bob to perform arbitrary symmetry breaking, i.e., choosean action among several with the same expected value.
For example, consider a case in which a set of multiple action pairs tied for having the maximal value { ( a 1 = x, a 2 = y ) , ( a 1 = x, a 2 =z ) , ( a 1 = w, a 2 = z ) } .
Say that Alice picks a 1 = x .
This is problematic for Bob because he has noinformation about whether Alice is selecting x because she is in an information state in which ( x, y )is good or whether she is in an information state in which ( x, z ) is good (the states for which they aregood may be disjoint).
One way to resolve this issue is to force Alice to respect Bob’s inabilities to break ties by makingthe response function stochastic, e.g., a softmax over the expected values.
That is, given that Alicedeviates with a 1 , Alice assumes that Bob plays
[Equation]
with a temperature hyper-parameter to control the sharpness of the distribution.
Then Alice choosesto deviate using
[Equation]
Bob can either play according to the softmax or select the single maximum, as Alice has alreadypicked the deviation assuming that Bob cannot break symmetries.
4.4 Taking Alice’s Information State Into Account
Our discussion above has largely ignored the fact that Alice can perform different deviations givenher information state s 1 (i.e., private observation).
Nonetheless, it helps us find the responsefunction f
( b, a 1 ) = softmax a 2 [ E s 1 ∼ b max[ˆ q π ( b, s 1 , a 1 , a 2 ) , q π ( b, s 1 )] /t ] .
Given that Alice knows
Bob will response according to a 2 ∼ f ( b, a 1 ) , Alice is now free to optimize her action by taking herinformation state back into consideration.
By applying Jensen’s inequality, we observe that
[Equation]
Therefore, at information state s 1 , Alice opts out of the blueprint if the optimal deviation action ,a [Equation] = [Equation]
∼ f ( b,
[Equation]
[Equation]
ˆ q π ( b, s 1 , a 1 , a 2 ) , exceeds the expected return of the BP, q π ( b, s 1 ) .
This isthe final formulation of IMPROVISED E .
Please refer to the Appendix A for the detailed pseudocode.
We also define IMPROVISED P , where Bob plays according to the probability of improvement
[Equation]
(6) where I is the indicator function and Alice selects any action maximizing the expected improvement.
While IMPROVISED P does not maximize the expected return as IMPROVISED E does, it may betterreflect human SEDs since it finds deviation pairs that maximize the probability of improvement .
Experiments
[Figure]
We test the IMPROVISED in two different settings.
The first setting is the trampoline-tiger game explained before.
Secondly, we apply IMPROVISED tothree-player Hanabi, where we start from a blueprinttrained on human data.
5.1 Trampoline Tiger
As illustrated in Section 4.2, in the trampoline-tigergame IMPROVISED E and IMPROVISED P both recover the optimal SED, when starting from the no-op
BP in 100% of the runs.
Alice will decide to jumpwhenever there is a trampoline and Bob then pullsthe lever to open the door, leading to the optimalexpected return of 0.1.
For comparison, we also ran 20 unique seeds of MAPPO [34] and 24 different hyper-parametercombinations of QMIX [23] on the trampoline-tiger problem.
The results are shown in Figure 3.
[Table]
All runs converged extremely rapidly to a policy that avoids the highly negative payoffs, but only2/20 of the MAPPO runs and 0/24 of the QMIX runs discovered the optimal strategy within 500ktimesteps, leading to an average reward across runs of 0.01 for MAPPO and 0.0 for QMIX.
The codeand hyper-parameters are available in the supplementary material.
In our experiments, a variety of other standard multi-agent learning techniques could not solve thistoy problem at all, including value-decomposition network [32] and simplified action decoder [10].
5.2 Hanabi
Hanabi has been proposed as a benchmark challenge for multi-agent coordination, and has been thesubject of significant research [1, 12].
Briefly, in Hanabi, there is a a 50 card deck consisting of 5different suits (colors) and 5 rank of card within each suit, for a total of 25 unique combinations, withsome duplicates.
2 to 5 players cooperatively take turns playing cards, giving hints to other players,or discarding.
Players may see their partners’ hands, but not their own.
The team’s goal is to playexactly one card of each rank in each suit in increasing order, scoring 1 point per successful play.
Upon three plays of cards duplicated or out-of-order, the players lose and instead score 0.
Giving ahint spends one of 8 shared hint tokens, allowing one to name a color or rank in a partner’s hand andindicate all cards of that color or rank in that hand.
Discarding a card replenishes one hint token.
Seealso [1] for a more thorough description of the rules and basic strategy.
5.3 What is a Finesse?
As mentioned earlier in Section 3, a clear example of a SED is the finesse move in Hanabi.
Althoughdescribed in far more detail with an example in [1], we will recap it briefly here.
For convenience,we always refer to the current active player of a given turn as player 1 and the players who movesubsequently as player 2 , player 3 etc.
The standard, vanilla form of the finesse in Hanabi occurs when player 1 gives a hint to player 3(i.e.
the player who moves two steps later)
that under the conventions common between the playersimplies that player player 3 should play a particular card.
Unbeknownst to player 3 , that card, ifplayed, would be out-of-order and would fail disastrously.
Instead, player 2 has just drawn the cardthat is both playable and if played makes player 3 ’s card playable next.
Player 2 is expected to realizethat the only way that player 1 ’s hint could be good rather than a disaster is if player 2 themself hasnewly drawn this exact card, and to take a leap of faith and play that newly drawn card blindly, havingno explicit information about it.
The end result is that player 1 signals in only one action for bothother players to each successfully play a card, a large gain.
While other finesse-like patterns are possible, in our Hanabi experiments for simplicity we focus onlythose finesse moves that follow the pattern described above.
5.4 IMPROVISED in Hanabi
We design experiments to evaluate IMPROVISED in Hanabi from both qualitative and quantitativeperspectives.
In the qualitative evaluation, we study whether IMPROVISED can indeed performfinesses in manually selected situations where finesse could be completed (finesse-complete situations).
We are particularly interested in the finesse style SEDs as they are most natural and intuitive way to demonstrate IMPROVISED’s ability to perform SEDs.
Note that IMPROVISED discoversmany types of SEDs in Hanabi beyond the finesse style moves because it searches for any beneficialjoint deviations that when player 1 initiates a deviation, player 2 can independently figure out thecorrect deviation in response.
However, other types of SEDs may be hard for human to interpret andless intuitive to analyse and present.
In the quantitative evaluation, we show that IMPROVISED asa planning algorithm improves the expected return when applied to test finesse-able situations andentire games.
To implement IMPROVISED in Hanabi, we first need a belief function from which we can samplegame states given either public or private knowledge of the game to perform Monte Carlo rollouts.
Luckily, the belief over possible hands in Hanabi can be computed analytically [19].
The belief isfirst initialized to cover all possible hands and then incrementally updated by filtering out hands thatcontradict with public knowledge revealed through hints and that would have caused the players topick different moves at each time step.
Since we are mainly concerned with the finesse style SEDsin the experiments, we restrict the search action space
A 1 to contain only the hint moves that targetplayer 3 and A 2 to contain only the play moves.
Player 1 and player 2 compute their own copies ofthe response function f independently.
The detailed hyper-parameters and computational cost are inthe Appendix.
Finessable Situation Experiments To check IMPROVISED’s ability to perform finesse, we firstgenerate situations where seasoned human players may carry out finesse moves.
We use a blueprintpolicy π to generate selfplay games over a range of decks (game seeds) and look for situationswhere player 1 observes that player 2 ’s newest draw card is playable and has not been hinted at, andmeanwhile player 3 holds an un-hinted card that can be played after player 2 plays their newest card(finesse-able situations).
Then we manually override player 1 ’s and player 2 ’s moves in finesse-ablesituations to carry out the finesse and check if player 3 will play the designated card under theblueprint to make the finesse complete and worthwhile (finesse-complete situations).
We experimentwith 4 different blueprints and show their statistics of finesse-complete situations over 1000 gameseeds in Table 1a.
All the agents are trained following the settings in corresponding prior works.
Although finesse-able situations are pretty common in the games played by all four agents, the firstthree RL agents trained without human data rarely complete a manually enforced finesse move,making them undesirable for the subsequent experiment.
It also worth noting that none of the RLblueprints perform finesses by themselves while the behavior clone agent performs only 1 finesse outof the 1183 finesse-complete situations.
Finesse Execution Experiments We apply both IMPROVISED E and IMPROVISED P with thebehavior clone blueprint on 1000 randomly chosen finesse-complete situations to check how manyfinesse it performs.
The results are shown in Table 1b.
Due to the novelty of this problem setting,there has been no prior method to directly compare against.
For reference, we run SPARTA [19], astrong search algorithm for Dec-POMDP designed to find unilateral deviations that maximizes theexpect return, on the same situations with the same restrictions on the set of actions that the agentsmay deviate to.
From the table, we see that both IMPROVISED algorithms perform significantlymore finesses, indicating their effectiveness in finding SEDs on the fly.
In the situations where IMPROVISED does not perform finesses, it can either be that IMPROVISED finds no beneficialdeviations or it finds better, non-finesse SEDs.
It is also interesting to see that IMPROVISED P findsroughly the same amount of finesses than IMPROVISED E , contradicting the idea that humans mightuse the probability of improvement when reasoning about SEDs instead of relying on small differencebetween different (joint) q-values.
Full Game Experiments Then, we run IMPROVISED on the full game of Hanabi.
At any timestep, the active player will first check whether a deviation has been initiated by other players bychecking whether previous players have picked actions with low probability under the blueprint.
If adeviation has been initiated, they will play their corresponding role as either player 2 or player 3 .
Otherwise the active player will decide whether to deviate using the IMPROVISED method.
Overa fixed set of 100 game seeds, IMPROVISED E and IMPROVISED P performs 29 and 21 finessesrespectively while SPARTA only performs 3.
Although it is still much less frequent than whatexpert human players will do, IMPROVISED is nonetheless a meaningful step along this novel andchallenging direction.
Although our work is focused on understanding and replicating this human capability rather thanoptimizing for self-play score, we also report quantitative results showing a noticeable increase inscore using IMPROVISED compare to the plain blueprint, giving evidence that the deviations founddo have a meaningful impact on the game.
If we run IMPROVISED only at the 1000 finesse-completesituations from Table 1b and use blueprint for the rest of the timesteps, then IMPROVISED E and
IMPROVISED P achieve average scores of 18.08 ± 0.28 and 18.18 ± 0.27 respectively while theblueprint gets 17.18 ± 0.28 on those same games.
When we run IMPROVISED E on the full game,we improve the average score from 17.80 ± 0.85 to 23.54 ± 0.14.
Related Work
6.1 Public Belief Methods
Our method builds on prior work that models Dec-POMDPs as PuB-MDPs [21, 22, 7, 9, 29, 30].
More recently, this same approach has been extended to two-player zero-sum games [20, 18, 4, 14, 5]and two-team zero-sum games [15, 35].
PuB-MDPs enable theoretically sound planning algorithmswithout having to reason about the agents’ entire policies because the public belief state serves asa sufficient statistic for planning.
However, prior work on PuB-MDPs do not explicitly search for
SEDs.
6.2 Search in Dec-POMDPs
Search and planning algorithms have been a critical component to AI breakthroughs in complexmulti-agent environments, including backgammon [33], chess [6], go [25–27], and poker [20, 2, 3].
Closely related to our work are methods for policy improvement via planning in Dec-POMDPs, suchas SPARTA [19].
SPARTA conducts a one-step lookahead and chooses the action that maximizesexpected value assuming all players play according to a common-knowledge BP thereafter.
However,
SPARTA searches only for unilateral deviations from the BP rather than multilateral deviations,because searching over all of the latter would be intractable.
As discussed in Section 3.1, discovering
SEDs may require considering multi-lateral deviations from the BP.
IMPROVISED is able to discoverthese multi-lateral deviations by searching over a constrained set of multi-lateral deviations.
Otherprior work has also investigated planning algorithms for Dec-POMDPs [16].
However, most of thesetechniques have typically searched over public belief states [7, 8, 29].
This greatly increases thecomplexity of the search procedure.
Conclusions
The human ability to coordinate with others using minimal explicit agreement and extending conventions “on the fly” is one of the most intriguing reasoning abilities and far ahead of any of our learningmethods.
In this paper we formalize the definition of such behavior as self-explaining deviations .
We show why existing methods fail to learn these types of deviations and presented an optimalalgorithm named IMPROVISED that can both discover them and correspond correctly when they arebeing carried out by other agents at test time.
Crucially, IMPROVISED allows agents to improveperformance at test time by improving upon a common knowledge blueprint policy, even when theblueprint is a Nash equilibrium.
Limitation and Future Work
In this paper we first address SED with planning without taking intoaccount the extra complexity of learning and approximating due to its difficulty and novelty.
Theresulted algorithm is computationally expensive as it requires Monte Carlo rollouts to estimate manycrucial quantities at test time (more details in the Appendix).
However, it is possible to derive a moreefficient, learning based version of IMPROVISED since the joint Q-function q ( b, s 1 , a 1 , a 2 ) can belearned.
There are challenges to make these estimates accurate on the deviating states but our papernonetheless provides a valuable stepping stone along this direction. B Experimental Details for Tiger-Trampoline [Table]
In Section 5.1, we show the results of MAPPO and QMIX on the Tiger-Trampoline game.
For the
MAPPO we use the default parameters from the open sourced implementation 2 used for Hanabi,except with a hidden size of 128, reducing the episode length cap, and reducing the number of threadsby a factor of 2. For QMIX, we use the open sourced implementation 3 of the algorithm provided as part of the PyMARL framework [24]. We used the default agent and training configuration, except forthe four hyper-parameters listed in table 2. For those, we tried all combinations of the correspondingvalues, producing a total of 24 runs, each training for 500k steps, or 250k episodes.
C Experimental Details for Finesse in Hanabi
In the Hanabi experiments, we implement IMPROVISED as follows (better viewed together withthe pseudocode).
The belief b is the common public belief shared by player 1 and player 2 basedon common knowledge available to all players and their common private knowledge of player 3 ’shand.
We first draw M Player 2 hands s ′ 1 from b and compute blueprint actions a π = π
( b ( s ′ 1 ))and P π ( a ) .
We then consider joint actions
A 1 × A 2 = { ( a 1 , a 2 ) | P π ( a 1 ) ≤ 0 } for player 1 andplayer 2 .
Since our goal is to find finesse style joint deviations, we further restrict a 1 to be a hintmove to player 3 and a 2 to be a play move .
Given s ′ 1 , player 1 can further induce the privatebelief b ( s ′ 1 ) over their own hand.
For each of s ′ 1 , player 1 calculates Monte Carlo estimations ofq ( a 1 , a 2 , b, s ′ 1 , ) for ( a 1 ,
a 2 ) ∈
A 1 × A 2 and q π ( b, s ′ 1 ) with N samples drawn from b ( s ′ 1 ) .
So far wehave collected all the quantities required to compute the mapping f for IMPROVISED P and for
IMPROVISED E .
Finally, we draw another K samples from the true b ( s 1 ) where s 1 now is the realhand of player 2 to estimate δ = max a 1 E a 2 ∼ f ( b,a 1 ) q π ( b, s 1 , a 1 ) − q π ( b, s 1 ) .
Player 1 will deviate toargmax a 1 E a 2 ∼ a ∗ 2 ( a 1 ) q π ( b, s 1 , a 1 , a 2 ) if δ ≥ 0 . In the next turn, player 2 can carry out the samecomputation process to get P π ( a 1 ) and f ( b, a 1 ) to figure out whether player 1 has deviated and if sowhat is the correct response. Player 1 and player 2 do not share the random seed beforehand. In the experiments where we run IMPROVISED on finesse-complete situations only, we set M =1000 , N = 100 and K = 10000 / |A 1 | .
It takes roughly 2 hours in total for both player 1 and player2 to compute the deviations independently using 5 CPU cores and 1 GPU. In the experiments where we run IMPROVISED on the full game of Hanabi, we reduce M to 400 andshare the result of f ( b, a 1 ) between Player 1 and Player 2 instead of computing it twice independentlyas we empirically find that the statistic is stable enough against random seeds.
A full game then takesaround 10-12 hours using 20 CPU cores and 2 GPUs.
D Societal Impact
One of the most fundamental goals of multi-agent learning is to produce agents that collaborate wellwith human in real world.
Therefore, it is important for the artificial agents to not only understandhuman conventions, but also decode human’s intention when deviations occur so that agents canresponse correctly.
Such agents should also be capable to initiate such self-explaining deviationswhen beneficial to maximize their social values.
Therefore, this direction of work to understandand replicate human’s ability to perform SEDs using theory of mind is important and positive to thesociety.
We currently do not foresee any negative societal impact from this work.