Self-Explaining Deviations for Coordination
Abstract
Fully cooperative, partially observable multi-agent problems are ubiquitous in the real world.
In this paper, we focus on a specific subclass of coordination problems in which humans are able to discover self-explaining deviations (SEDs).
SEDs are actions that deviate from the common understanding of what reasonable behavior would be in normal circumstances.
They are taken with the intention of causing another agent or other agents to realize, using theory of mind, that the circumstance must be abnormal.
We motivate this idea with a real world example and formalize its definition.
Next, we introduce an algorithm for improvement maximizing SEDs (IMPROVISED).
Lastly, we evaluate IMPROVISED both in an illustrative toy setting and the popular benchmark setting Hanabi, where we show that it can produce so called finesse plays.
Introduction
Humans generally assume other humans follow certain social norms when acting in the society and interpret behavior using these norms to infer hidden information about the world.
For example, when a driver sees a car ahead suddenly stopping on a road, they may infer that there is some accident ahead or the car is broken.
However, humans also have the ability to improvise when the conventional actions are restricted or there exist other actions that can bring about superior outcomes.
Taking the event in Figure 1 as an example, the caller deviates from the conventional practice of stating the situation truthfully because it would have negative consequences.
This type of deviation involves two noteworthy aspects.
First, it can be detected by the dispatcher because it appears to be a mistake under common practice.
Second, the dispatcher’s response can be independently decided by both individuals based on the common understanding of the world and the deviation chosen by the caller.
We refer to this type of phenomena as self-explaining deviations (SEDs).
SEDs are actions that, under normal circumstances, would not make sense, given the agents’ common understanding.
They are executed with the intention that teammates will use theory of mind to deduce that the situation is unusual in a particular way, and will adapt their behavior to account for this additional information.
Dispatcher: Oregon 911.
Caller:
I would like to order a pizza at...
Dispatcher:
You called 911 to order a pizza?
Caller:
Uh, Yeah, apartment...
Dispatcher:
This is the wrong number to call for a pizza.
Caller:
No no
no... you’re not understanding me.
Dispatcher: I’m getting you now.
Is the other guy still there?
Caller: Yep.
I need a large pizza.
Dispatcher: All right.
How about medical.
You need medical?
Caller:
No.
With pepperoni.
Dispatcher: Turn your sirens off before you get there.
Caller ordered a pizza.
And agreed with everything I said that there’s domestic violence going on.
For our contribution, we first formalize the problem setting and definition of SEDs.
Then we introduce a novel planning algorithm, IMPROVement maxImizing Self-Explaining Deviations (IMPROVISED), for performing SEDs.
We show that, under some assumptions, IMPROVISED performs the optimal
SED in terms of expected return maximization.
Next, we provide a motivating experiment, illustrating that in a small toy problem designed to require SEDs to perform optimally, IMPROVISED is able to compute an optimal joint policy, whereas other multi-agent learning algorithms are not.
Lastly, we present experiments on the large scale benchmark Hanabi [1], where we show that IMPROVISED is able to produce finesse plays, which is one of the most interesting techniques that human experts perform frequently.
Background
FOSG and Public POMDP
For our notation, we use an adaption of factored observation stochastic games ( FOSG ) [7].
W is the set of world states and w 0 is a designated initial state.
A =
A 1 ×· · ·×A N is the space of joint actions .
T is the transition function mapping W ×A → ∆( W ) .
R : W ×
A → R is the reward function .
O =
( O priv (1) , . . .
, O priv ( N ) , O pub ) is the observation function where O priv ( i ) :
W × A × W → O priv ( i ) specifies the private observation that player i receives.
O pub : W ×
A × W → O pub specifies the public observation that all players receive.
O i
= O i ( w, a, w ′ )=(
O priv ( i ) ( w, a, w ′ ) , O pub ( w, a, w ′ )) is player
i ’s observation and a history is a finite sequence
h = ( w 0 , a 0 , . . . , w t ) .
The set of histories is denoted by H .
The information state for player i at h
=
( w 0 , a 0 , . . .
, w t ) is s i ( h ) :
= ( O 0 i , a 0 i , . . .
, O ti ) .
The information state space for player i is S i :
=
{ s i ( h ) | h ∈ H} .
The legal actions for player i at s i is denoted A i ( s i ) .
A joint policy is a tuple π
= ( π 1 , . .
. , π N ) , where policy π i maps S i → ∆( A i ) .
The public state at h is the sequence s pub ( h ) :
= s pub ( s i ( h )) :
= ( O 0 pub , . . .
, O t pub ) .
The information state set for player i at s ∈ S pub is S i ( s ) :
= { s i ∈ S i | s pub ( s i )
=
s } , where S pub is the space of public states.
Finally, the reach probability of h under π is P π ( h ) .
Rather than working with in a multi-agent setting directly, we invoke the public POMDP transformation that maps cooperative multi-agent settings to equivalent single-agent POMDPs.
Given a common-payoff FOSG ⟨N , W , w 0 , A , T , R , O⟩ , we can construct an equivalent public
POMDP [10] ⟨ ˜ W , ˜ w 0 , ˜ A , ˜ T , ˜ R , ˜ O⟩ as follows:
The world states of the public POMDP ˜ W is the set { ( s 1 ( h ) , . . .
, s N ( h )) : h ∈ H} .
The initial world state of the public POMDP ˜ w 0 is the tuple ( s 1 ( h 0 ) , . . .
, s N ( h 0 )) .
The actions of the public POMDP are called joint prescriptions .
It is denoted by Γ and has N components.
The i th component of it Γ
i is the prescription for player i .
A prescription Γ i maps s i to an element of A i
( s i ) for each s
i ∈ S
i ( s pub ( h ))
; it instructs a player in the common-payoff
FOSG how to act as a function of its private information.
Given ˜ w ≡ ( s 1 , . . .
, s n ) and Γ ,
the transition distribution ˜ T ( ˜ w, Γ) is induced by T (( s 1 , . . .
, s n ) , a ) , where a ≡ Γ(( s 1 , . . .
, s n ))
:=
(Γ 1 ( s 1 ) , . .
, Γ N ( s N )) 3 .
Given ˜ w ≡ ( s 1 , . . .
, s n ) and ˜ w ′ ≡ ( s ′ 1 , . .
, s ′ n ) , the reward and observa- tion are given by ˜ R ( ˜ w, Γ , ˜ w ′ )
≡ R (( s 1 , . . . , s n ) , Γ(( s 1 , . . . , s n )) , ( s ′ 1 , . . .
, s ′ n )) and ˜ O ( ˜ w, Γ , ˜ w ′ ) ≡
O pub (( s 1 , . . .
, s n ) , Γ(( s 1 , . .
, s n ))
, ( s ′ 1 , . . .
, s ′ n )) , respectively.
Every policy in the public POMDP corresponds to a joint policy in the underlying common-payoff
FOSG, which receives exactly the same expected return.
Therefore, it is sufficient to work with the public POMDP.
See [14, 15] for further discussion.
When the public POMDP is considered as a belief MDP, rather than as a POMDP, its belief states are of the form b t : ( s t 1 , . . .
, s tn )
(cid:55)→ P Γ 0 ,..., Γ t − 1 ( s t 1 , . . .
, s tN | s t pub ) .
In words, this is the joint distribution over private information states, conditioned on the historical policy (Γ 0 , . . .
, Γ t − 1 ) and the public state s t pub .
This public belief MDP is abbreviated as the PuB-MDP.
Self-Explaining Deviations
While the name self-explaining deviation (SED) is novel to this work, the idea behind SEDs is not.
One example of SEDs comes from the cooperative card game Hanabi [1] in the form of a play called a “finesse”.
Readers unfamiliar with Hanabi and finesse may first jump to Section 5.2 and Section 5.
for the detailed descriptions.
In a finesse play, the acting player, i.e., the first player who initiates the deviation as part of the finesse, intentionally communicates misleading information to a receiving player, which would hurt the team’s score if the receiving player acted upon it using the established convention.
The second player in the finesse, who acts after the first player but before the receiving player, realizes that the first player must have deviated from the established convention after observing the seemingly disastrous move.
Knowing that the first player is rational, the second player realizes that there must be a way that they can also deviate to reach a better outcome than the one that the original convention would have led to.
After the second player plays its part of the joint deviation, the original information from the first player is no longer misleading and the subsequent players can continue to follow the prior conventions.
SEDs may take place in any cooperative situation with at least two players where common knowledge blueprint policies that players follow under normal assumptions exist.
Intuitively, SEDs capture a form of joint deviations where one agent takes an action that at first appears to be a mistake or otherwise highly off-policy from another player’s perspective, i.e. zero or low probability under the blueprint.
On the presumption that the first agent chose that action intelligently and deliberately and there is an off-policy action for the observing agent that could potentially result in an even-better-thannormal outcome for both, the observing agent may reason that the first agent “intends” them to take it—to take a leap of faith that the first agent has not erred, but rather knows both agents can get that better outcome, even if the observing agent does not have the information themselves to prove that this outcome will result.
This work uses the term SEDs to describe this phenomenon of communication via apparent mistakes in a general context.
We formalize SED in Section 3.1 and propose IMPROVISED, a novel planning algorithm that performs the optimal SEDs under some assumptions, in Section 4.
3.1 Examining Self-Explaining Deviations
We are now ready to investigate SEDs.
To facilitate our investigation, let’s consider the trampoline-tiger game in Figure 2 as an example.
In this game, Alice is standing on a balcony, while Bob stands on the ground next to a lever.
Pulling the lever will either deploy a trampoline below the balcony, or release a tiger.
From above, Alice can observe which is the case, and then Alice decides whether to jump off the balcony (Y) or not (N).
Bob observes Alice’s choice but does not know whether the lever will deploy a trampoline or tiger, and decides to pull the lever (Y) or not (N).
Alice wants to get down from the balcony but will die from the fall unless Bob pulls the lever and it releases a trampoline.
If Bob pulls the lever and releases a tiger, it will eat Bob.
Consider the joint policy ( ∗ (cid:55)→ N , ∗ (cid:55)→ N ) .
While not optimal, this joint policy is not unreasonable, as Bob won’t be eaten and Alice won’t fall to her death.
Indeed, independent of the probability of a trampoline, this joint policy is a Nash equilibrium.
In addition, as we shall see in Section 5.1, a range of MARL algorithms converges to this solution.
However, there is a clear opportunity for a SED under this blueprint.
If there is a tiger, Alice should obviously never jump.
So if Bob observes Alice choosing to jump and trusts that Alice is intelligent and would only have chosen to jump if it could lead to a better result, Bob should realize that there must be a trampoline.
Therefore Alice should trust that Bob will pull the lever if she jumps, as a result.
Under any planning algorithm that considers only unilateral deviations, Bob and Alice can’t escape the original local optimum.
Alice will never jump because she is under the belief that Bob will choose not to pull the lever.
Therefore, finding SEDs in general requires considering multi-lateral deviations, i.e. simultaneous deviations by more than one player.
Since finding multi-lateral deviations in general games with partially observable state is a complex problem, we restrict our focus to settings satisfying the following assumptions:
Assumption 1.
We assume sequential and publicly observable actions.
SEDs may still be possible with partially-observable actions, but for simplicity we focus on when
Bob can directly observe Alice deviate from the blueprint, rather than having to deduce it from partial or incomplete observations.
Assumption 2.
We consider only cases where Bob’s private information is not necessary for coordinating on the SED.
For simplicity, we always refer to the first player who initiates the joint deviation as the Alice and the second player who figures out Alice’s intention and plays their part of the joint deviation as Bob in the following discussions beyond the scope of this toy game.
In other words, we only consider SEDs where Bob needs only act as a function of the common-knowledge state and the Alice’s action, rather as a function of private knowledge of their own.
Under these assumptions, we define the SED as follows:
Definition 1.
Given a common knowledge blueprint (BP) policy π , common public belief of Alice and Bob b , and information state of Alice s 1 , SED is any pair of joint deviations ( a ′ 1 , a ′ 2 ) for Alice and Bob that satisfy all the following conditions:
• q π ( b, s 1 , a ′ 1 , a ′ 2 ) ≥ q π ( b, s 1 ) , i.e., it gives higher expected future return than BP; • P ( a ′ 1 | π, b ) ≤ ϵ , i.e., a ′ 1 is highly unlikely under π under the common belief; • a ′ 2 ∼ f ( b, a ′ 1 ) , i.e., a ′ 2 is a function of the public belief and Alice’s deviation action.
Here, q π ( b, s 1 , [ a 1 , a 2 ]) is Alice’s estimate of the expected future return after executing the optional a 1 , a 2 and following π afterwards, and ϵ is a hyper-parameter.
The function f is determined by the algorithm designer under the constraint that f only takes b and a ′ 1 as input.
It is used both by Alice to predict Bob’s response and by Bob to decide the response independently.
IMPROVISED
In this section we derive IMPROVISED, an algorithm to perform the SEDs.
The core of the algorithm is to find a function f under the constraint of SED and an optimization procedure that Alice uses to decide whether to deviate given her information state s 1 and possible responses a 2 ∼ f from Bob.
4.1 Defining the Optimization Problem
With the observations in the prior sections, we are ready to write selecting a SED as a PuB-MDP optimization problem:
[Equation]
where Γ 1 is the prescription for the acting player at current time step and Γ 2 is the prescription for the acting player at the subsequent time step, ◦ denotes function composition, and, in accordance with Definition 1, Γ 2 ranges over the space of A 1 .
The series of equalities above begins with a generic multi-lateral search in the PuB-MDP.
The LHS is ranging over Alice and Bob’s prescriptions such that, if Alice uses Γ 1 and Bob uses Γ 2 in belief state b , and all the players play according to π thereafter, the expected return is maximized.
The first equality holds simply based on the fact that
Alice is able to observe s 1 and the fact Alice can always decide to opt into the blueprint (because
Bob knows which actions are supported by the blueprint, he can opt in exactly when Alice opts in, and play according to the deviation otherwise).
The second equality holds by applying Assumptions 1 and 2, which imply both that Bob can observe Alice’s action and that it is the only thing he needs to condition his decision upon (note that the change of the set over which Γ 2 ranges is left implicit).
4.2 An Easier Special Case
Unfortunately, expression (1) remains difficult to optimize, as the number of Γ 1 and Γ 2 is combinatorial.
To ameliorate, we can apply the simplifying constraint of only allowing Alice and Bob to deviate for a single action pair regardless of Alice’s information state s 1 .
For simplicity, we also set ϵ = 0 .
Let D ( b ) be the set of plausible deviation actions for Alice, i.e., action that cannot be played under blueprint: D ( b ) :
= { a 1 ∈
A 1 |∀ s 1 ∈ b : π ( a 1 | s 1 ) = 0 } .
Let R be the set of all plausible actions that Bob can play in response in every possible state, i.e., R :=
(cid:84) s 2 ∈S 2 A 2 ( s 2 ) .
If R = ∅ , Alice will skip searching for SED this turn and follow BP.
Now we can augment the expected value of a deviation with the allowed range of actions:
[Equation]
Then we can rewrite equation (1) using our newly defined q function as follows:
[Equation]
These values could be found in O ( |S 1 ||A 1 ||A 2 | ) time, given the two inner q functions.
Moreover, as all of these depend only on the public belief state, both players can compute them independently, assuming that there are no ties.
Once the deviation pair is computed, Alice decides whether to proceed with the deviation or not given her private information: Alice plays a ∗ 1 if ˆ q π ( b, s 1 , a ∗ 1 , a ∗ 2 ) >
q π ( b, s 1 ) and the blueprint action otherwise.
It is possible that ( a ∗ 1 , a ∗ 2 ) is not a valid deviation pair if no plausible deviations exist, e.g., a ∗ 1 could be outside of D ( b ) .
However, in this case ˆ q π ( b, s 1 , a ∗ 1 , a ∗ 2 ) is −∞ , and so Alice will resort to playing blueprint.
Bob can detect the deviation as D ( b ) is public knowledge and a ∗ 1 ∈ D ( b ) .
Therefore, depending on his observation, he can either play a ∗ 2 or respond as usual.
We can represent the value of the policy that allows a single deviation at b and plays blueprint afterwards as: q ∗ π ( b ) = max a 1 ,a 2 E s 1 ∼ b max(ˆ q π [ b, s 1 , a 1 , a 2 ) , q π ( b, s 1 )] .
It is easy to see that q ∗ π ≥ q π , i.e., this algorithm has a weak policy improvement guarantee.
If we apply this procedure to the tiger-trampoline game with a blueprint policy that always no-ops:
[Equation]
As expected, we obtain a policy improvement with value equal to the probability of a trampoline.
We discover the SED where Alice jumps when a trampoline is present and Bob pulls the lever.
We provide a proof-of-principle implementation of IMPROVISED in the trampoline-tiger game that can be run online at https://bit.ly/3KtMLT6 .
4.3 Coordination by Extending Conventions
In a coordination context, we cannot require Bob to perform arbitrary symmetry breaking, i.e., choose an action among several with the same expected value.
For example, consider a case in which a set of multiple action pairs tied for having the maximal value { ( a 1 = x, a 2 = y ) , ( a 1 = x, a 2 = z ) , ( a 1 = w, a 2 = z ) } .
Say that Alice picks a 1 = x .
This is problematic for Bob because he has no information about whether Alice is selecting x because she is in an information state in which ( x, y ) is good or whether she is in an information state in which ( x, z ) is good (the states for which they are good may be disjoint).
One way to resolve this issue is to force Alice to respect Bob’s inability to break ties by making the response function stochastic, e.g., a softmax over the expected values.
That is, given that Alice deviates with a 1 , Alice assumes that Bob plays
[Equation]
with a temperature hyper-parameter t to control the sharpness of the distribution.
Then Alice chooses to deviate using
[Equation]
Bob can either play according to the softmax or select the single maximum, as Alice has already picked the deviation assuming that Bob cannot break symmetries.
4.4 Taking Alice’s Information State Into Account
Our discussion above has largely ignored the fact that Alice can perform different deviations given her information state s 1 (i.e., private observation).
Define the response function f
( b, a 1 ) = softmax a 2 [ E s 1 ∼ b max[ˆ q π ( b, s 1 , a 1 , a 2 ) , q π ( b, s 1 )] /t ] .
Given that Alice knows Bob will respond according to a 2 ∼ f ( b, a 1 ) , Alice is now free to optimize her action by taking her information state back into consideration.
By applying Jensen’s inequality, we observe that
[Equation]
Therefore, at information state s 1 , Alice opts out of the blueprint if the optimal deviation action , a ∗ 1 = max a 1 E a 2 ∼ f ( b,a 1 ) ˆ q π ( b, s 1 , a 1 , a 2 ) , exceeds the expected return of the BP, q π ( b, s 1 ) .
This is the final formulation of IMPROVISED E .
Please refer to the Appendix A for the detailed pseudocode.
We also define IMPROVISED P , where Bob plays according to the probability of improvement
[Equation]
where I is the indicator function and Alice selects any action maximizing the expected improvement.
While IMPROVISED P does not maximize the expected return as IMPROVISED E does, it may better reflect human SEDs since it finds deviation pairs that maximize the probability of improvement .
Experiments
We test the IMPROVISED in two different settings.
The first setting is the trampoline-tiger game explained before.
Secondly, we apply IMPROVISED to three-player Hanabi, where we start from a blueprint trained on human data.
We provide the code for our Hanabi experiments at https://github.com/facebookresearch/off-belief-learning/blob/main/ pyhanabi/finesse.py .
5.1 Trampoline Tiger
As illustrated in Section 4.2, in the trampoline-tiger game IMPROVISED E and IMPROVISED P both recover the optimal SED, when starting from the no-op BP in 100% of the runs.
Alice will decide to jump whenever there is a trampoline and Bob then pulls the lever to open the door, leading to the optimal expected return of 0.1.
For comparison, we also ran 20 unique seeds of MAPPO [18] and 24 different hyper-parameter combinations of QMIX [12] on the trampoline-tiger problem.
The results are shown in Figure 3.
All runs converged rapidly to a policy that avoids the highly negative payoffs, but only 2/20 of the
[Figure]
MAPPO runs and 0/24 of the QMIX runs discovered the optimal strategy within 500k timesteps, leading to an average reward across runs of 0.01 for MAPPO and 0.0 for QMIX.
In our experiments, a variety of other standard multi-agent learning techniques could not solve this toy problem at all, including value-decomposition network [17] and simplified action decoder [4].
5.2 Hanabi
Hanabi is a benchmark challenge for multi-agent coordination.
Briefly, in Hanabi, there is a 50 card deck consisting of 5 different suits (colors) and 5 ranks of cards within each suit, for a total ofunique combinations, with some duplicates.
Two to five players cooperatively take turns playing cards, giving hints to other players, or discarding.
Players may see their partners’ hands, but not their own.
The team’s goal is to play exactly one card of each rank in each suit in increasing order, scoring 1 point per successful play.
Upon three plays of cards duplicated or out-of-order, the players lose and instead score 0.
Giving a hint spends one of 8 shared hint tokens, allowing one to name a color or rank in a partner’s hand and indicate all cards of that color or rank in that hand.
Discarding a card replenishes one hint token.
For a more thorough description of the rules and basic strategy see [1].
5.3 What is a Finesse?
As mentioned earlier in Section 3, a clear example of a SED is the finesse move in Hanabi.
We will recap it briefly here, for a more detailed explanation, see [1].
For convenience, we always refer to the acting player of a given turn as player 1 and the players who move subsequently as player 2 , player 3 , and so forth.
The standard, vanilla form of the finesse in Hanabi occurs when player 1 gives a hint to player(i.e.
the player who moves two steps later) that, under the conventions common between the players, implies that player 3 should play a particular card.
Unbeknownst to player 3 , that card, if played, would be out-of-order and would fail.
Instead, player 2 has just drawn the card that is both playable and if played makes player 3 ’s card playable next.
Player 2 is expected to realize that the only way that player 1 ’s hint could be good rather than a failure is if player 2 themself has newly drawn this exact card, and to take a leap of faith and play that newly drawn card blindly, having no explicit information about it.
The end result is that player 1 signals in only one action for both other players to each successfully play a card—a large gain.
While other finesse-like patterns are possible, in our Hanabi experiments for simplicity we focus only on those finesse moves that follow the pattern described above.
[Table]
5.4 IMPROVISED in Hanabi
We design experiments to evaluate IMPROVISED in Hanabi from both qualitative and quantitative perspectives.
In the qualitative evaluation, we study whether IMPROVISED can indeed perform finesses in manually selected situations where finesse could be completed (finesse-complete situations)
[Table]
We are particularly interested in the finesse style SEDs as they are the most natural and intuitive way to demonstrate IMPROVISED’s ability to perform SEDs.
Note that IMPROVISED discovers many types of SEDs in Hanabi beyond the finesse style moves because it searches for any beneficial joint deviations that when player 1 initiates a deviation, player 2 can independently figure out the correct deviation in response.
However, other types of SEDs may be hard for humans to interpret and less intuitive to analyze and present.
In the quantitative evaluation, we show that IMPROVISED as a planning algorithm improves the expected return when applied to test finesse-able situations and entire games.
To implement IMPROVISED in Hanabi
[Table]
we first need a belief function from which we can sample game states given either public or private knowledge of the game to perform Monte Carlo rollouts.
Luckily, the belief over possible hands in Hanabi can be computed analytically [8].
The belief is first initialized to cover all possible hands and then incrementally updated by filtering out hands that contradict with public knowledge revealed through hints and that would have caused the players to pick different moves at each time step.
Since we are mainly concerned with the finesse style SEDs in the experiments, we restrict the search action space
A 1 to contain only the hint moves that target playerand A 2 to contain only the play moves.
Player 1 and player 2 compute their own copies of the response function f independently.
The detailed hyper-parameters and computational cost are in Section C.
Finessable Situation Experiments To check IMPROVISED’s ability to perform finesse, we first generate situations where seasoned human players may carry out finesse moves.
We use a blueprint policy π to generate selfplay games over a range of decks (game seeds) and look for situations where player 1 observes that player 2 ’s newest draw card is playable and has not been hinted at and where player 3 holds an un-hinted card that can be played after player 2 plays their newest card (finesse-able situations).
Then we manually override player 1 ’s and player 2 ’s moves in finesse-able situations to carry out the finesse and check if player 3 will play the designated card under the blueprint to make the finesse complete and worthwhile (finesse-complete situations).
We experiment with 4 different blueprints and show their statistics of finesse-complete situations over 1000 game seeds in Table 1a.
All the agents are trained following the settings in corresponding prior works.
Although finesse-able situations are common in the games played by all four agents, the first three RL agents trained without human data rarely complete a manually enforced finesse move, making them undesirable for the subsequent experiment.
It is also worth noting that none of the RL blueprints perform finesses by themselves while the behavior clone agent performs only 1 finesse out of the 1183 finesse-complete situations.
Finesse Execution Experiments We apply both IMPROVISED E and IMPROVISED P with the behavior clone blueprint on 1000 randomly chosen finesse-complete situations to check how many finesse it performs.
The results are shown in Table 1b.
Due to the novelty of this problem setting, there has been no prior method to directly compare against.
For reference, we run SPARTA [8], a strong search algorithm for Dec-POMDP designed to find unilateral deviations that maximize the expected return, on the same situations with the same restrictions on the set of actions that the agents may deviate to.
From the table, we see that both IMPROVISED algorithms perform significantly more finesses, indicating their effectiveness in finding SEDs on the fly.
In the situations where
IMPROVISED does not perform finesses, it can either be that IMPROVISED finds no beneficial deviations or it finds better, non-finesse SEDs.
It is also interesting to see that IMPROVISED P finds roughly the same amount of finesses as IMPROVISED E .
Full Game Experiments Then, we run IMPROVISED on the full game of Hanabi.
At any time step, the active player will first check whether a deviation has been initiated by other players by checking whether previous players have picked actions with low probability under the blueprint.
If a deviation has been initiated, they will play their corresponding role as either player 2 or player 3 .
Otherwise the active player will decide whether to deviate using the IMPROVISED method.
Over a fixed set ofgame seeds, IMPROVISED E and IMPROVISED P perform 29 and 21 finesses respectively, while
SPARTA only performs 3.
Although it is still much less frequent than what expert human players will do, IMPROVISED is nonetheless a meaningful step along this novel and challenging direction.
Although our work is focused on understanding and replicating this human capability rather than optimizing for self-play score, we also report quantitative results showing a noticeable increase in score using IMPROVISED compared to the plain blueprint.
If we run IMPROVISED only at the 1000 finesse-complete situations from Table 1b and use blueprint for the rest of the timesteps, then IMPROVISED E and IMPROVISED P achieve average scores of 18.08 ± 0.28 and 18.18 ± 0.27 respectively while the blueprint gets 17.18 ± 0.28 on those same games.
When we run
IMPROVISED E on the full game, we improve the average score from 17.80 ± 0.85 to 23.54 ± 0.14.
Related Work
Public Belief Methods Our method builds on prior work that models Dec-POMDPs as PuBMDPs [10, 11, 2, 3, 15, 16].
PuB-MDPs enable theoretically sound planning algorithms without having to reason about the agents’ entire policies because the public belief state serves as a sufficient statistic for planning.
For example, SPARTA [8] conducts a one-step lookahead starting from the public belief state and chooses the action that maximizes expected value assuming all players play according to a common-knowledge BP thereafter.
However, SPARTA searches only for unilateral deviations from the BP rather than multilateral deviations, because searching over all of the latter would be intractable.
As discussed in Section 3.1, discovering SEDs may require considering multilateral deviations from the BP.
IMPROVISED is able to discover these multi-lateral deviations by searching over a constrained set of multi-lateral deviations.
Human-Like Coordination Outside of this work, another notable work in the direction of incorporating human-like behavior into AI agents is that of Ma et al.
Ma et al.
[9] show that certain architectures have better inductive biases for respecting the correspondence between action features and observational features.
Our work is complementary in the sense that we investigate a phenomenon that occurs in unusual situations, whereas Ma et al.
[9] investigate a phenomenon that occurs by default.
Conclusions
Coordinating with others using minimal explicit agreement and extending conventions “on the fly” is one of the most intriguing reasoning capabilities.
In this paper we formalize the definition of such behavior as self-explaining deviations .
We showed that existing methods tend not to perform these types of deviations and presented an algorithm called IMPROVISED that can both discover them and respond correctly when they are being carried out by other agents at test time.
Limitations and Future Work
We see two main limitations of IMPROVISED that would be worth addressing in future work.
The first is the fact that IMPROVISED requires exact knowledge of all teammates’ policies and assumes that other teammates are also running IMPROVISED.
In real coordination settings, it is both unrealistic to assume exact knowledge of the policies of externally specified teammates and unrealistic to assume that those teammates are implementing the same SED algorithm.
The second is the computational cost of IMPROVISED, as it requires Monte Carlo rollouts to estimate many crucial quantities at test time (more details in the appendix).
One concrete direction toward addressing this issue is to derive a learning based version of IMPROVISED by learning the
Q-function q ( b, s 1 , a 1 , a 2 ) .
B Experimental Details for Tiger-Trampoline
[Table]
In Section 5.1, we show the results of MAPPO and QMIX on the Tiger-Trampoline game.
For the
MAPPO we use the default parameters from the open sourced implementation used for Hanabi, except with a hidden size of 128, reducing the episode length cap, and reducing the number of threads by a factor of 2.
For QMIX, we use the open sourced implementation 5 of the algorithm provided as part of the PyMARL framework [13].
We used the default agent and training configuration, except for the four hyper-parameters listed in table 2.
For those, we tried all combinations of the corresponding values, producing a total of 24 runs, each training for 500k steps, or 250k episodes.
C Experimental Details for Finesse in Hanabi
In the Hanabi experiments, we implement IMPROVISED as follows (better viewed together with the pseudocode).
The belief b is the common public belief shared by player 1 and player 2 based on common knowledge available to all players and their common private knowledge of player 3 ’s hand.
We first draw M Player 2 hands s ′ 1 from b and compute blueprint actions a π = π
( b ( s ′ 1 )) and P π ( a ) .
We then consider joint actions
A 1 × A 2 = { ( a 1 , a 2 ) | P π ( a 1 ) ≤ 0 } for player 1 and player 2 .
Since our goal is to find finesse style joint deviations, we further restrict a 1 to be a hint move to player 3 and a 2 to be a play move .
Given s ′ 1 , player 1 can further induce the private belief b ( s ′ 1 ) over their own hand.
For each of s ′ 1 , player 1 calculates Monte Carlo estimations of q ( a 1 , a 2 , b, s ′ 1 , ) for ( a 1 ,
a 2 ) ∈
A 1 × A 2 and q π ( b, s ′ 1 ) with N samples drawn from b ( s ′ 1 ) .
So far we have collected all the quantities required to compute the mapping f for IMPROVISED P and for
IMPROVISED E .
Finally, we draw another K samples from the true b ( s 1 ) where s 1 now is the real hand of player 2 to estimate δ = max a 1 E a 2 ∼ f ( b,a 1 ) q π ( b, s 1 , a 1 ) − q π ( b, s 1 ) .
Player 1 will deviate to argmax
[Equation]
[Equation] [Equation]
if δ ≥ 0 .
In the next turn, player 2 can carry out the same computation process to get P π ( a 1 ) and f ( b, a 1 ) to figure out whether player 1 has deviated and if so what is the correct response.
Player 1 and player 2 do not share the random seed beforehand.
In the experiments where we run IMPROVISED on finesse-complete situations only, we set M = 1000 , N = 100 and K = 10000 / |A 1 | .
It takes roughly 2 hours in total for both player 1 and player 2 to compute the deviations independently using 5 CPU cores and 1 GPU.
In the experiments where we run IMPROVISED on the full game of Hanabi, we reduce M to 400 and share the result of f ( b, a 1 ) between Player 1 and Player 2 instead of computing it twice independently as we empirically find that the statistic is stable enough against random seeds.
A full game then takes around 10-12 hours using 20 CPU cores and 2 GPUs.
D Societal Impact
We do not anticipate any immediate negative impact from this work.