Social Interactions as Recursive MDPs
Anonymous Author(s)
AfﬁliationAddress email
Abstract: While machines and robots must interact with humans, providing themwith social skills has been a largely overlooked topic.
This is mostly a consequenceof the fact that tasks such as navigation, command following, and even gameplaying are well-deﬁned, while social reasoning still mostly remains a pre-theoreticproblem.
We demonstrate how social interactions can be effectively incorporatedinto MDPs (Markov decision processes) by reasoning recursively about the goalsof other agents.
In essence, our method extends the reward function to includea combination of physical goals (something agents want to accomplish in theconﬁguration space, a traditional MDP) and social goals (something agents wantto accomplish relative to the goals of other agents).
Our Social MDPs allowspecifying reward functions in terms of the estimated reward functions of otheragents, modeling interactions such as helping or hindering another agent (bymaximizing or minimizing the other agent’s reward) while balancing this with theactual physical goals of each agent.
Our formulation allows for an arbitrary functionof another agent’s estimated reward structure and physical goals, enabling morecomplex behaviors such as politely hindering another agent or aggressively helpingthem.
Extending Social MDPs in the same manner as I-POMDPs (Interactivepartially observed Markov decision processes) extension would enable interactionssuch as convincing another agent that something is true.
To what extent the Social MDPs presented here and their potential Social POMDPs variant account for allpossible social interactions is unknown, but having a precise mathematical model toguide questions about social interactions has both practical value (we demonstratehow to make zero-shot social inferences and one could imagine chatbots and robotsguided by Social MDPs) and theoretical value by bringing the tools of MDP thathave so successfully organized research around navigation to shed light on whatsocial interactions really are given their extreme importance to human well-beingand human civilization.
Introduction
Progress on modeling social interactions and giving machines social goals, such as being particularlynice to a user, is signiﬁcantly hampered by the lack of theoretical models which characterize whatsocial interactions are.
While microsociology has uncovered common structures in social interactions[1] a computational model is still elusive.
Until not too long ago, this was also the state of robotnavigation and sensing which was revolutionized by extending MDPs [2] to POMDPs [3].
Deﬁningthe problem clearly allowed us as a ﬁeld to understand what we can model and how to do so.
Until wetake this same step for social interactions, they will remain on shaky ground despite their importanceto virtually every interaction humans engage in.
We introduce an extension of MDPs, which we term Social MDPs.
In the process, we make severalassumptions.
First, that agents have both physical goals (e.g., bring the red key home) and social goals(e.g. , prevent John form getting his yellow key), and that their overall reward structure is an arbitrarycombination of the two, potentially accompanied by other terms.
Physical goals are precisely what MDPs can already express, a function of points in a conﬁguration space.
Social goals are a functionof the estimate of the reward structure of another agent.
For example, a reward that hinders anotheragent is a negative function of the estimated reward of that agent.
Complicating matters is the factthat social rewards like beliefs can be recursive: an agent may want to help another agent help them.
To model this, Social MDPs are recursive up to a bounded depth, much like interactive POMDPs [4], I-POMDPs.
Unlike I-POMDPs, Social MDPs are not recursive in terms of agent’s beliefs about the {+Solve 21 using: g 1 ,  12 (ground truth) e g 1 , 2 2 , e  1 , 2 21 (estimate agent 2) e 1 , 
 Solve e 1 , 1 2 using: e g 1 , 2 2 , e  1 , 2e g 2 , 1 1 , e  2 , 1 12 (estimate agent 1) e 2 , 
+}
{+Solve e 2 , 0 1 using: e g 2 , 
+}
[Figure] Require: l, s t , a tJ ,  ij , g i if l = 0 then solve MDP for agent i elsefor all e  i,l,tji , e g i,l,tj do compute P ( e  i,l,tji | s t  1 , a t  1 J ) P ( e g i,l,tj | s 1: t  1 ) e i,l  1 j ( s t , a tJ , e  i,l,tji , e g i,l,tj ) end for compute R li ( s t , a tJ ,  iJ , g i ) compute Q li ( s
[Equation]
[Equation]
[Equation]
[Figure]
We use the estimated social policy e i,l  1 j at previous time step to update the estimated physical and social goal as described in section 3.3.1.
At t = 0 , we assume P ( e g i,l,tj ) and P ( e  i,l,tjJ ) are from uniform distributions.
This algorithm is called at all recursion steps e i,l  1 j to estimate social policy for the other agent j 2 J .
The estimated goals and policies are used to compute the rewards and Q values for selecting the actions.
state of the world.
Instead, Social MDPs are recursive in terms of the rewards of the agents.
Thismakes Social MDPs and I-POMDPs orthogonal and complementary.
Social MDPs are speciﬁcallyformulated to not interfere with the standard extension from MDPs to POMDPs, making it possibleto include partial observability.
While we do not develop a joint Social I-POMDP here, this is areasonable extension which would cover far more of the space of social interactions, although onethat is computationally challenging.
Our contributions are: (1) formulating Social MDPs where an agent’s reward function is an arbitraryfunction of the recursive estimate of another agent’s reward and a physical goal, (2) an implementationwhere that function is a linear transformation, which captures many notions of helping and hindering,(3) demonstrating that the model performs zero-shot social reasoning in agreement with a humansubjects experiment, and (4) examples of the practical utility of recursive social reasoning,
In ananonymized online appendix 1 we fully enumerate all possible scenarios predicted by our model givenan environment simple enough to allow doing so, demonstrating that it captures a diverse set of socialbehaviors.
We also provide videos of the behavior of our model in all these scenarios.
Related Work
Modeling other agents In order to interact with other agents effectively, an agent must be ableto reason about the goals, preferences, and beliefs of other agents [5].
Theory-based models forsocial goal attribution [6, 7, 8, 9, 10], Bayesian inverse planning to infer an agent’s goal given theobservations of their behaviors [11, 12], and learning the reward functions of other agents [13] havebeen explored.
Prior research also tried to recognize social interactions such as waving and huggingin videos where people are involved in group activities [14, 15, 16].
These methods generally involvetwo separate stages [17]: a social perception stage and a coordination or collaboration stage whereagents interact.
In contrast, Social MDPs constantly reevaluate the goals of other agents enablingthem to adapt to changes in the plans of other agents.
Social MDPs also allow for enumerating socialsituations by formally deﬁning the space of what social interactions are, opening the doors to a moretheoretical approach to social interactions.
Game theory [18, 19] considers altruistic and spitefulbehavior through linear combinations of payoffs, similarly to what we consider here although such models are limited to level 1 reasoning and to scenarios where the goals are known a prior rather thanestimated on the ﬂy as they are in Social MDPs and in practical robotic applications.
Simulating social interactions Heider and Simmel [20] simulated agent trajectories to study socialperceptions through a set of animations involving the movements of geometrical ﬁgures.
Simulatingagent behaviors in physics engines had been explored to collect datasets in fully observable [21, 22]and partially observable environments [23, 17].
The collected datasets are used to study socialperception or build machine learning models that can recognize agents’ goals.
These frameworksassume that each agent has either a physical goal or a social goal.
Here we consider a more generalscenario where agents have both a physical goal and a social goal and the overall behavior is a blendof the two.
This is a more practical setting as when one decides to help another agent one does notabandon every other concern and physical care.
Our formulation allows for an arbitrary function ofanother agent’s reward recognizing for complex social interactions. We explore only a subset of thefull power of Social MDPs here by considering only linear functions.
Learning to interact with other agents Interactive POMDPs [24, 25, 26] (I-POMDPs) are extensions of POMDPs that recursively model the beliefs of other agents.
Social MDP and I-POMDPs areorthogonal.
Social MDPs allow agents to reason recursively about other agents’ reward functionswhile I-POMDPs allow agents to reason recursively about other agent’s beliefs about the state of theworld.
The two could in principle be combined, but while Social MDPs require solving a modestnumber of additional nested MDPs, I-POMDPs require signiﬁcantly nested inference, and whenthe two are combined the problem quickly becomes intractable. Xie et al.
[27] propose a differenttype of approach that does not require nested inference: learning a low-dimensional representationof another agent’s strategy.
This approach allows an agent to avoid another agent or to manipulateanother agent into some mutually-beneﬁcial behavior.
Social MDPs, on the other hand, allow buildingthe strategy of another agent directly into the reward function of an agent, enabling behaviors suchas helping or hindering regardless of what the other agent is trying to achieve.
Moreover, Social
MDPs are zero-shot, while this prior approach is not.
From the point of view of generalization andsample-efﬁcient robotics, a zero-shot approach is preferable; in addition, it opens new doors for amore theoretical understanding of social interactions.
We could combine Social MDPs with this priorwork to build in latent representations of strategies into reward functions [28, 29] creating moreefﬁcient approximations of Social MDPs.
Social MDPs
Social MDPs are recursive MDPs (Markov decision process) with nested estimates of other agent’sgoals. They are inspired by hierarchical models of games [30] and nested MDP that reason aboutthe beliefs of other agents [31, 32, 33].
Figure 1 shows an example of recursively estimating theother agent’s goals and policy in a two-agent scenario.
Like other nested models, e.g I-POMDPs, Social MDPs have the notion of a level.
A level
Social MDP is simply an MDP: agents reasonabout the map state.
A level [-
-]1 Social MDP enables each agent to reason about the physical goals ofother agents (those other agents are treated like level 0 agents).
A level 2 Social MDP enables eachagent to reason about the level 1 social goals of other agents.
To perform this nested inference, agentsmust have access to another agents’ physical and social goals.
These goals are estimated by solving Social MDPs recursively at every level.
A level 0 agent can take physical actions, but cannot reason socially.
A level 1 agent can take actionsrelative to another agent’s physical goals; such actions include helping, hindering, stealing, etc.
Alevel 2 agent can take actions relative to another agent’s social and physical goals; such actionsinclude avoiding an attempt to be hindered, recognizing that help is needed, joining in to help together.
Levels deeper than 2 continue to describe meaningful interactions although we do not consider themhere.
It is unclear what level of recursion is required before agents exceed the social reasoningcapacities of humans.
3.1 Assumptions
As we are in an MDP setting, all physical states are fully observable to all agents.
Note howeverthat the goals are not available to other agents.
In other words, agents can observe one another atany distance and regardless of any obstacles but they cannot know what another agent wants (eitherphysically or socially) by reading that agent’s mind.
Agents must infer the physical and social goals,if any, of other agents from their actions.
Agents reason completely independently of one another.
In this paper, we consider only pairwise Social MDPs; indeed, all our experiments involve settingswhere only two agents are present.
N-way Social MDPs that consider social interactions betweenmultiple agents at the same time are extensions of pairwise MDP and scale linearly at every level.
Computationally however, n-way Social MDPs, as with any such model, scale exponentially in n .
As with many multiagent MDP settings, the experiments performed here assume that agents areoptimal planners.
This is not an inherent limitation of Social MDPs, but relaxing this assumptionmakes estimating the other agents’ goals and rewards far more difﬁcult.
3.2 Formal deﬁnition of Social MDPs
A Social MDP for agent i with respect to all agents J consists of an arity (here we formulate thepairwise case) and a maximum level, l , and is deﬁned as:
[Equation]
where S is a set of states in the environment where s 2 S ;
A =
A J is the set of joint moves of all 138 agents in J .
a i is an action for agent i ; T is the probability distribution of going from state s 2 S tonext state s 0 2 S given actions of all agents in J : T ( s 0 | s, a J ) ;  iJ is agent i ’s social goal towardevery other agent in J .
For convenience,  iJ is {+a shorthand for S j 2 J,j 
 =+} i  ij ; g i is agent
i ’s physicalgoal; R li is the l -th level reward function for agent i based on its estimate of other agents’ rewards;and  is a discount factor,  2 (0 , 1) .
Reward Each agent has its own physical goal, e.g., going to a landmark, as well a social goal, e.g.,helping or hindering other agents.
What enables Social MDPs to go beyond regular MDPs is therecursive nature of the reward function which can be written in terms of the estimated rewards ofother agents.
The immediate reward of an agent i at level l is computed as follows:
[Equation]
where r ( · ) is the static reward given the agent’s own physical goal g i , e R l  1 ,i j ( · ) is the estimatedreward for agent j from agent i ’s point of view assuming agent j is a level l  1 agent, c ( · ) is the costfor taking an action.
For negative levels, the reward is deﬁned to be zero. ij is the social goal, it transforms the reward of another agent j into a goal that is part of the rewardof the target agent i .
In this paper, we instantiate the model with a linear transformation, so  ijis simply a reweighting of the estimated reward of the other agent.
If it is a negative value, thetarget agent will attempt to minimize the reward of another agent, i.e. hindering.
A positive valuecorresponds to helping.
Social goals can be eliminated entirely by setting this weight to zero.
In order to estimate another agent’s reward function, one needs to estimate that agent’s physical andsocial goals.
We use e
[Equation]
e g i,lj to denote the estimated social and physical goals.
The superscripti, l indicates agent i at level l is making the estimations.
We describe how to estimate the social andphysical goals in section 3.3.1.
3.3 Planning for Social MDPs
Analogous to MDPs, the Q function of Social MDPs is the sum of immediate reward and the expectedvalue in the future.
[Equation] Since agent i is interacting with other agents j 2 J , it needs to estimate what actions other agents arelikely to take in order to compute its state-action value.
Social MDPs take the expectation over theestimated goals and actions of agent j to compute V li ( s 0 ,  iJ , g i ) :
[Equation]
estimatesocialgoal
[Equation]
When solving agent i ’s MDP at level l , the estimated social and physical goals are further used toupdate the other agent j ’s social policy to the actions agent j may take.
We denote the estimatedsocial policy for agent j at reasoning level l  1 as
[Equation]
g
[Equation]
Figure 2summarizes the steps to compute the state-action values and select optimal actions for any level l attime step t .
We ﬁrst update the probability of the estimated goals of other agents using the observedstate and the estimated policy from the previous time step. The updated probability of goals are usedto update the policy of other agents and compute the reward and Q function of the target agent.
Therecursion happens at estimating the social policies of other agent at a lower level.
3.3.1 Updating social and physical goals of other agents
An agent’s estimate of another agent’s social and physical goals at time step t and level l canbe updated based on the actions performed by the agents.
At time step t = 0 , we use uniformdistributions for social and physical goals.
The social goal, estimated at time step t , is updated after actions taken by all agents at the previous 179 time step.
This update is similar to the belief update in the POMDP framework but based on theestimated social policy of the other agent j :
[Equation]
The physical goal g j of agent j is estimated by i as follows, similar to Shu et al.
[21] but marginalizedover the estimated social goal as the agent is estimating the social goal at the same time.
[Equation]
[Equation]
The l -level social policy e i,lj of the agent j is predicted by i using the Q-function at level l -1:
[Equation]
This is a softmax policy where we use a temperature parameter ⌧ to control how much the agent jfollows the greedy actions.
As shown in eq.
4, in order to use agent j ’s Q function at level l -1, itrequires to compute agent i ’s Q function at level l -2, and so on. This involves solving Social MDPsrecursively at levels 0 , 1 , · · · , l -1.
3.4 Time complexity
The time complexity of solving a Social MDP at level 0 is the same as that of solving an MDP.
Atlevel 1, an MDP must be solved for every agent independently in order to compute the likely physicalgoals of every other agent.
Assume that the number of models considered for each pair of agents ateach level is bounded by a number M (based on the number of social and physical goals to consider).
Solving a Social MDP at level l requires solving O ( M ( A  1) 2 l ) MDPs, where A is the numberof agents.
Social MDPs form a tree with branching factor A  1 as every agent must compute thepairwise social goal of every other agent until level 0 where the tree bottoms out.
There are manypotential speedups that can alleviate this runtime to allow for efﬁcient inference even in the face ofmany agents.
For example, a distance horizon could be used where far away agents could simplybe considered non-interacting.
Similar to Netanyahu et al.
[23], it is also possible to speed up thealgorithm by amortized inference over goals and relations by training a neural net to recognize goalsand relations as initial guesses and reﬁne them through probabilistic inference.
Results
We apply the Social MDP framework to a multi-agent grid world inspired by previous studies onsocial perceptions [12, 6, 34].
The 10 ⇥ 10 world consists of two agents, a yellow robot and red robot,two physical landmarks, a ﬂower and tree, and two objects, a yellow watering can and red wateringcan.
The yellow agent has a low cost for moving the yellow watering can, while it has a high cost formoving the red watering can.
Robots can have a physical goal of moving the watering can to a target [Figure] plant.
Robots can have a social goal of helping or hindering to different degrees.
In the grid world,agents can move in four direction (left, right, up, down) or choose not to move.98 different experiment scenarios 2 are systematically created in this grid world.
Each scenario hasagents as having either the same physical goal or different physical goals and one of 7 differentscaling factors on each of their social goals (-2, -1, -0.5, 0, 0.5, 1, 2) ( 2 ⇤ 7 ⇤ 7 = 98 scenarios).
2indicates that the social goal is weighted much more than the physical goal, and an agent wants tomaximize the other agent’s goal.
Similarly for -2, except that an agent wants to minimize anotheragent’s goal.
At 1, agents weigh their own social goals equally with their own physical goals.
At0.5, they put twice as much weight on their physical goals as their social goals.
These agents are lesslikely to help, particularly if helping will cost more time or energy than carrying out the physicalgoal.
Finally, either agent can have the factor set to zero; these agents only have a physical goal andno social goal.
The fact that we can enumerate all possible social interactions, as predicted by ourmodel, in a given scenario is an important feature missing from alternative representations.
All 98experiment scenarios correspond to reasonable interactions between agents.
The degree to which thisis true in more complex environments and the degree to which systematically unfolding the model in
Interactions scenarios
[Figure] [Table] model. more complex environments always results in what humans would describe as social interactions isan important topic for future work.
Each agent’s reward for reaching its physical goal is based on that agent’s geodesic distance from the goal after taking an action [12].
This physical reward function is parameterized by ⇢ and  that determines the scale and shape of the physical reward: r i ( s, a, g i ) = max ( ⇢ (1  distance( s, a, g i ) /  ) , 0) .
We set the cost, c , of an action a , to 1 for grid moves and 0.1to staying in place while ⇢ and  were set to 1.25 and 5, respectively.
The discount factor,  , was setto 0 . 99 .
Two of the scenarios selected for our experiment are shown in ﬁg. 3.
We used the Social MDP toselect actions for the yellow agent which has a physical goal, watering the ﬂower or tree, whileinteracting with the red agent.
The red agent had a physical goal, watering the tree, and a varyingsocial goal.
At every time step, the yellow agent estimates the physical and social goal of the redagent, depending on the Social MDP level.
At deeper levels, the agents behave more optimally andmore socially.
To quantitatively establish the quality of the social inferences made by the Social MDPs, we comparehuman judgements of 12 subjects against those of two baseline models: inverse planning [12] and arecent cue-based model [21].
Humans and models had to estimate the physical and social goals ofagents in these environments when the agents were acting both as level one agents (unaware that theother agents are also social) and as level two agents (who could account for the fact that the otheragents are social).
In ﬁg. 4 we show the raw judgements of humans and of our models, along witha best linear ﬁt.
The performance of all models against human judgements, was measured throughcorrelation coefﬁcient at 95% conﬁdence level, for social goal estimation (r = 0.89 for the Social
MDP vs. r = 0.81 for the Inverse Planning model vs. r = 0.23 for the Cue-based model) and physical goal estimation (r = 0.78 for the Social MDP vs. r = 0.72 for the Inverse Planning model vs. r = 0.08for the Cue-based model).
Our model performs considerably better than other models.
This is evenmore evident in the deep dive shown in ﬁg. 5.
For level one agents, agents that are social but thatassume that other agents are not social, all models agreed with human judgements.
Yet, for leveltwo agents, agents that are social and can assume that other agents are also social, our models arefar better aligned with human judgements.
Prior work could capture a far smaller space of socialinteractions than the Social MDPs we craft here.
Conclusion
Social MDPs are a ﬁrst step toward a theory of social interactions that ﬁts within the establishedframeworks we have in robotics.
They can perform zero-shot social recognition and planning fordiverse situations.
The fact that MDPs can be extended in a natural way that is also computationallytractable to account for many social interactions by nesting inference and allowing models totake arbitrary functions of the estimated rewards of other agents has not been noted before.
Ourexperiments clearly show that Social MDPs are superior to prior models and account for more socialinteractions.
In this work, we have only begun to explore what Social MDPs can represent.
The environment weconsider is very simple, yet, at the same time, more than enough to differentiate Social MDPs fromother models.
So far, we have unrolled Social MDPs only two levels; what exists at deeper levels isstill unclear.
It is likely that humans do not perform deeply-nested recursive reasoning to carry outsocial interactions, although, what the cutoff is, and if Social MDPs are close enough to a human’smental model to allow for measuring that cutoff is unknown.
We would like to see in the future that any MDP-based system can be augmented to be social bya straightforward extension with Social MDPs.
Much like virtually any approach can be easilyaugmented to partially-observed environments using POMDPs.
Social MDPs and POMDPs arecompatible, exploring their combinations and the implications of partial observability for socialinteractions remains as future work.