Interaction Modeling with Multiplex Attention
Anonymous Author(s)
AffiliationAddress email
Abstract
Modeling multi-agent systems requires understanding how agents interact.
Suchsystems are often difficult to model because they can involve a variety of typesof interactions that layer together to drive rich social behavioral dynamics.
Herewe introduce a method for accurately modeling multi-agent systems.
We present
Interaction Modeling with Multiplex Attention (IMMA), a forward predictionmodel that uses a multiplex latent graph to represent multiple independent types ofinteractions and attention to account for relations of different strengths.
We alsointroduce Progressive Layer Training, a training strategy for this architecture.
Weshow that our approach outperforms state-of-the-art models in trajectory forecastingand relation inference, spanning three multi-agent scenarios: social navigation,cooperative task achievement, and team sports.
We further demonstrate that ourapproach can improve zero-shot generalization and allows us to probe how differentinteractions impact agent behavior
Introduction
Modeling multi-agent systems is important for a wide variety of applications, including self-drivingcars, crowd navigation, and human-machine collaboration.
The dynamics of multi-agent systems canbe challenging to model as they are usually governed by various independent types of interactions.
Consider modeling crowd navigation at a social gathering, where agents’ behaviors might be governedby at least two types of interactions: target destinations (e.g. a person trying to meet up with a friend)and collision avoidance (e.g. navigating a busy sidewalk).
In social scenarios like this, humans oftenappear to navigate and make predictions based on estimated high-level intentions and relations amongthe other people [16, 45, 46].
Motivated by this observation, we aim to build a forecasting model formulti-agent systems that infers high-level abstractions in the form of graphs, entirely through the taskof predicting agent trajectories.
Leading approaches in modeling multi-agent systems, such as Neural Relational Inference (NRI) [22],
Relational Forward Model (RFM) [8], and their extensions [12, 27], use graph neural networks(GNNs) to infer edge types for every pair of entities in the interacting systems.
However, thisinductive bias does not explicitly handle the multiple layers of interactions present in social multiagent systems and, as shown empirically in Section 4, has led to at least two shortcomings: reducedperformance on long-term predictions and decreased interpretability.
Our approach, Interaction
Modeling with Multiplex Attention (IMMA), overcomes these issues by using a multiplex graph 1latent structure to model multiple interaction types, with attention graph layers that can capture thestrength of relations.
Furthermore, we propose a training approach for IMMA that we call Progressive Layer Training(PLT).
Learning in high-dimensional space often leads to highly-entangled representations that areuninterpretable [44], but adding explicit disentanglement objectives often leads to decreased modelperformance [10, 18].
Inspired by the concept of “starting easy” and curriculum learning [5], we trainour model to learn progressively, one latent layer at a time.
In addition to improving performance,training IMMA with PLT also improves interpretability—offering the potential to answer questionssuch as: "Who causes an agent to behave this way?"
Using our method, we conduct experiments in a range of social multi-agent environments inspired byreal-world scenarios: social navigation, cooperative task achievement, and team sports.
We evaluatemodels on trajectory prediction, we analyze the relations inferred by our model, and we explore theimportant role these inferred relations play in making accurate predictions.
Additionally, we showthat IMMA is effective at zero-shot generalization and conditional generation.
To summarize,
• We propose Interaction Modeling with Multiplex Attention (IMMA), a model that uses a
new latent space structure which we call Multiplex Attentional Graph.
It models relations
and utilizes these relations to make forecasting predictions.
• We propose a training technique that is enabled by the multiplexed latent structure:
Progres-
sive Layer Training (PLT).
PLT enables our model to learn types of interactions one at a
time and furthers the performance and interpretability of IMMA.
• Empirically, we show that our method (IMMA w/ PLT) outperforms the state-of-the-art
models on three different social multi-agent environments.
To understand
the interpretabilityof our approach, we evaluate our method on relational inference and conduct a conditionalgeneration experiment.
We also compare various ablated and modified versions of ourmethod, including incorporation of a well-established approach for learning disentangledrepresentations, to understand the contribution of each components.
Furthermore, weshowcase the superior sample efficiency and generalization ability of our model.
Related Work
Many approaches have been proposed to model multi-agent interactions, especially as a crucial steptoward achieving better performance in tasks such as model-based reinforcement learning [37, 48],trajectory prediction [34], and motion planning [40].
In particular, social dynamics modeling has beenstudied in domains such as human crowds [34], team sports [12, 22, 27], and traffic participants [35]where the interactions between entities lead to highly complicated motion or behavior patterns.
Manyearlier works, such as Social Forces [17], are deterministic regression models that model humansas objects affected by forces.
The rise of deep learning ushered in various neural network based models such as Social-LSTM [1]—an LSTM model with a social pooling layer, social-GAN [13]—a
GAN combined with sequence-to-sequence model, Social-Attention [42]—a recurrent neural networkthat makes use of attention mechanism, and others [11, 25].
Researchers also proposed to modelinteractions with networks that were inspired by a Theory of Mind framework [32, 33].
Our work isdifferent as we explicitly infer an interpretable discrete structure and perform inference over it.
In recent years, graph representation learning has been investigated for relational reasoning in multiagent systems, where nodes represent interacting entities and edges represent their relations [4].
Mostexisting graph-based methods infer interactions implicitly [3, 7, 19, 26, 29, 35, 36], meaning that littleto no interpretable abstractions can be extracted from the models.
Other works take pre-defined graphtopology based on heuristics as input [1, 23].
NRI [22], DNRI [12], RFM [39], and EvolveGraph[27] takes a step forward to reason about relations by inferring the topology of the underlyinginteraction graphs.
NRI [22] takes the form of a variational auto-encoder while RFM[39], havingnear-identical architecture as NRI, simply uses a forward prediction framework.
All these methodsmodel interactions as latent interaction graphs and leverage them to make future predictions.
Ourwork falls into this line of research with significant improvements to performance in terms of bothrelational inference and trajectory prediction because of (1) the novel multiplexed latent structure, (2)the incorporation of an attention mechanism, and (3) a progressive training strategy.
Methods
In this section, we define our problem and introduce our proposed model (IMMA) and trainingstrategy (PLT).
Our input consists of trajectories of N agents.
We denote by x ti the feature vector ofagent i at time t , e.g. location.
We denote by x t = { x t 1 , ..., x tN } the set of features of all N agents attime t and by X T 1 : T 2 = { x T 1 : T 2 i , i = 1 , ..., N } the set of features covering time steps from T 1 to T 2 .
The task is to take trajectory observations as input and learn to predict successive trajectories of allagents.
We aim to estimate p ( X T h
+1:
T h + T f | X 1: T h ) , where T h is the historical horizon and T f is theforecasting horizon.
Note that even though agents can have relations and goals that vary in differentinstances, these must be inferred entirely through the supervision signals of the trajectory predictiontask (with no supervision of the relations themselves).
3.1 Interaction Modeling with Multiplex Attention
We propose Interaction Modeling with Multiplex Attention (IMMA) - a model that consists of anencoder and a decoder trained jointly.
The encoder takes a sequence of observations (i.e. trajectories)as input and outputs a latent interaction graph.
Subsequently, the decoder predicts future trajectoriesby performing message passing among agents (nodes) with the inferred latent interaction graph.
IMMA assumes that the dynamics of multi-agent systems can be abstracted to a latent graph structure(i.e.
a vector of categorical latent variables)
z = { z ij } , where z ij represents relations between agentsi and j .
We optimize the variational lower bound, as in the conditional variational autoencoder(CVAE)
[Equation]
The encoder (approximated posterior) q ϕ
( z | X 1: T h + T f ) is a neural network with parameters ϕ , thedecoder (likelihood)
p θ ( X T h
+1:
T h + T f | X 1: T h , z ) is a neural network with parameters θ , and p ( z )denotes the prior 2 .
In previous works [22, 27, 39, 12], the latent graph z is inferred by performing edge-type classification.
That is, z represents the likelihoods of edges in the latent graph belonging to a set of classes.
However,in social dynamical systems multiple types of independent interactions can coexist, with differentstrengths of relations within an interaction type.
In this case, representing the interactions with asingle graph with many edge types requires many more edge types to accommodate all possiblecombinations of interactions.
More concretely, consider a navigational environment consisting of
Furthermore, since z is learned to encode likelihood of an edge belonging to a particular class, it maystruggle to capture relative strengths of interactions between agents.
In the following, we describe how IMMA (depicted in Fig. 2) tackles these limitations, as weintroduce the encoder and decoder in more detail.
Encoder with Multiplex Attentional Latent
Graphs The goal of the encoder is to infer a latentinteraction graph.
To achieve this, we use a GNN on a fully-connected directed graph withoutself-loops, where each vertex represents an agent.
Let x
j denote the feature vector of agent j acrossall input frames ( x 1 j , ..., x T h j ) .
The GNN embeds agent trajectories (e.g. x j ) into edge embeddings,which are then passed through a MLP to derive unnormalized edge weights (e.g. e ( i,j ) ).
e ( i,j ) is avector with dimensionality K , a hyperparameter that determines the number of layers in the latentgraph.
Details of the GNN can be found in the Appendix.
Previous approaches attempt to classify edges into one out of the K possible edge types.
They derivez ij by applying a softmax on e ( i,j ) such that (cid:80) k
= K k
=1 z kij
= 1 with z k the k -th layer of the latent graph.
We propose a novel way to construct the latent space using a multiplex attentional graph.
Instead ofperforming edge-type classification, we perform multi-class prediction on edge embeddings, yieldingmultiplexed layers, and introduce an attention mechanism between agents within each of these layers.
That is,
[Equation]
where N j denotes the neighbors of
agent j
(we assume a fully connected graph; thus, N j includesall other agents except j itself).
Note that z ij is different than z ji .
With Eq. 2, layers of the latentgraph are separable by construction.
By separable, we mean that individual layers of the latent graph(e.g. z 1 and z 2 ) can be inferred by separate encoders that does not share any parameters.
Intuitively,a multiplex latent space allows us to model independent interactions naturally, and the attentionmechanism captures strengths of relations.
Decoder The task of the decoder is to predict the dynamics of the system using the latent interactiongraph z and the past dynamics.
That is, we want to model p θ ( X T h
+1:
T h + T f | X 1: T h , z ) .
One commonissue when training such a decoder is that it might learn to ignore the latent variables while achieving only a marginally worse prediction loss.
To avoid this problem, our decoder first learns agent-centricrepresentations ˜ h tj = ˜ f emb ( x j ) (i.e. representations are embedded individually for each agent),and then uses these as node embeddings to perform message passing.
The decoder thus has to relyon the inferred latent graph to exchange information between agents.
After learning agent-centricrepresentations, we use GraphGRU [39] and a readout function to decode future trajectories:
[Equation]
The details of a GraphGRU block are elaborated in the Appendix.
Unlike a typical Graph Attention
Network [41] where the graph at every message passing layer is inferred, the same (multiplex) graphis used across all decoding steps.
We can re-evaluate the encoder for every time step to obtain adynamically changing latent graph, however we leave this to future work.
3.2 Progressive Layer Training
The role of the encoder is to infer a set of graph bases ( z 1 , z 2 ... )
that reflect the underlying dynamicsof the environment.
Since the messages are first passed within layers before summing up at the end, itwould be useful for the latent graph layers to represent different types of interactions between agents.
However, it is difficult for the model to learn all the interactions, let alone have them be disentangled.
Disentanglement is typically achieved by imposing additional loss, yet these approaches tend tocome with the trade-off of decreased reconstruction quality [10, 18, 24].
Motivated by curriculumlearning [5] and progressive learning [28] (or more broadly the concept of “starting easy”), wepropose to overcome this trade-off with Progressive Layer Training (PLT) — a strategy of learninga set of good graph bases by learning the high-level and most consequential interactions first andthen progressively growing the network to model lower-level and more intricate interactions.
Wefirst train a IMMA with a single-layered latent graph z 1 , then add a second layer z 2 by traininga second encoder while freezing the weights of the first.
This process can be repeated until nofurther improvement is observed.
The fact that we can choose to have a separable latent space anddo not have to share any parameters between these encoders is a result of our latent space beingmultiplexed.
Directly adding new components to a trained network can introduce a sudden shock tothe gradient.
To stabilize training, we adopt “fade-in” [20] to smoothly blend the new and existingnetwork components.
Refer to the Appendix for more details.
Experiments
We design our experiments to answer the following questions:
Q1 :
Does our approach (IMMA w/ PLT) consistently outperform prior methods across a diverse
set of social multi-agent environments and benchmarks?
Q2 :
Does the use of multiplex attentional latent graph give more interpretable abstractions?
Q3 :
How much does each component contribute to the final performance?
Q4 :
Is our approach sample efficient?
Can it generalize well to novel environments?
To answer Q1 , we test our approach in three multi-agent environments that each exhibited multipletypes of social interaction: our simulated Social Navigation Environment, PHASE [30], and the NBAdataset (used in [22, 27, 49, 47, 15]).
We develop the first benchmark ourselves, while the lattertwo are established benchmarks.
For each environment, the model receives multi-agent trajectoriesof length 24 as observation and predicts 10 future time steps.
For Social Navigation we use 100kmulti-agent trajectories (in total for training, validation, and testing), for NBA dataset we use 300kmulti-agent trajectories, and for PHASE we use 836 multi-agent trajectories.
We also perform dataaugmentation, as described in the Appendix.
To answer Q2 , we evaluate the inferred interactiongraphs for the Social Navigation Environment and PHASE, since we have the ground truth interactiongraph.
Note that we are not supervising on any explicit relations of agents.
We further conductqualitative experiments to understand the abstractions provided by the latent graph.
To answer Q3 ,we conduct an ablation study.
For Q4 , we perform a sample efficiency experiment and an experimentof zero-shot generalization.
Experiments for Q3 and Q4 are conducted on the Social Navigation
Environment as we have full control over it.
4.1 Experiment Setup
Here we describe the environments and baselines.
More details are in the Appendix.
Social Navigation Environment The Social Navigation Environment simulates a socially interactive crowd where agents seek to meet up with other agents while simultaneously navigating toavoid collisions within the crowd (as schematized in Fig. 1).
This environment allows fine-grainedcontrol by simulating lower-level interaction dynamics using Optimal Reciprocal Collision Avoidance(ORCA, [6]) like many other crowd navigation works [8, 9].
With this environment, we can easilyvary environmental parameters to generate diverse trajectories for our generalization experiment.
Weexperiment primarily with an environment consisting of five agents.
For each simulated episode,a social interaction graph of the agents is randomly generated: each agent is assigned to be thefollower of another agent.
Agents are initialized to positions along the edge of a circle.
Dynamics issimulated using ORCA by assigning the position of each agent’s leader (at each time step) as thetarget destination of that agent.
These environment settings yield reasonably complex trajectories thatwere clearly influenced by the social interaction graph and collision avoidance, as seen in exampletrajectories of Fig. 3.
[Table]
PHASE dataset PHASE [30] is a simulator for physically-grounded abstract social events, inwhich social recognition and social prediction algorithms can be benchmarked.
In PHASE, twoanimated agents move in a continuous 2D space and interact with each other and other objects (2balls).
Their actions are generated using a physics engine and a hierarchical planner [30].
We choosethe “collaboration” task for our experiment, in which two agents collaborate to move one of the ballsto a pre-specified landmark location.
Diverse trials are generated by randomizing the agents’ andballs’ starting locations as well as orientations, target ball, and target landmark.
Note that this datasetis substantially smaller than the two other datasets.
dataset The National Basketball Association (NBA) dataset consists of position trajectoriesfor ten interacting basketball players and the ball and uses real-world player tracking data.
The taskis to predict 10 future time steps (4s) based on a history of 24 time steps (10s).
Team information isan additional input feature.
The unit reported is meters.
Baselines We consider the following baselines and ablations
MLP : a large MLP network that takes the concatenated features of all agents across all timestepsas input and output trajectory predictions of all agents at the same time.
• GAT_LSTM : a network with Graph Attention layers [41] followed by LSTMs.
• NRI [22]: a VAE model with recurrent GNN modules.
• EvolveGraph
[19]:a model that expands upon the framework of NRI and introduces a doublestage training pipeline to account for an evolving interaction graph. • RFM [39]: a recurrent GNN model of similar architecture to NRI, but with a supervised loss instead of a variational lower bound loss.
• RFM (skip 1) : a RFM model with the first edge type is “hard-coded” as “non-edge” (i
e. nomessages are passed along edges of the first layer).
• fNRI [43]: a NRI model that uses a multiplex latent graph structure (i
e., sigmoid activation).
• IMMA (SG) :
IMMA with a single layer of attentional latent graph.• IMMA (MG) :
IMMA with multiple layers of attentional latent graphs (a multiplex graph)trained simultaneously, without PLT.
We are aware that there are other relevant baselines such as Social-LSTM [1], Social-GAN [13], and
Trajectron++ [35] etc.
We choose to compare with EvolveGraph, because it consistently outperformsthese models across datasets [27].
4.2 Trajectory Prediction
To answer Q1 , we evaluate all models on their prediction performance of agent position on futuretrajectories with two metrics: average displacement error (ADE) and final displacement error (FDE).
Both of these are metrics widely used in trajectory prediction literature: ADE is defined as the averagedeviated distance of all entities within the prediction horizon, and FDE is defined as the deviateddistance of all entities at the last predicted time step.
The results on the Social Navigation Environment, PHASE, and the NBA dataset are shown in table1.
We found that using multiplex attentional latent graph with progressive layer training (MG w/ PLT)yields the best performance on all three datasets, outperforming previous state-of-the-art models.
Onthe Social Navigation Environment and the NBA dataset, IMMA without progressive layer training
[Table]
(MG) still performs better than all baselines, showing the effectiveness of using multiplex attentionalgraph as latent structure.
We visualize the prediction results of our method and the best performingbaseline (RFM) in Fig.
3 on the Social Navigation Environment.
Visualizations of PHASE and the
NBA dataset are shown in Fig.
4 (more can be found in the Appendix).
4.3 Relational Analysis and Interpretability
To answer Q2 , we conduct experiments to understand what abstractions can be extracted from theinferred multiplex attentional graph.
We first compare the performance of our approach to the sameset of baselines used in the previous experiment on relational inference.
Intuitively, this is to evaluatehow well models can answer the question “What causes an agent to behave this way?”
Refer tothe Appendix for details on how we calculate the graph accuracy.
We conduct this experiment inthe Social Navigation Environment and PHASE, since we have access to the ground-truth socialinteraction graph for these two environments.
IMMA more accurately estimates the underlying social interaction graph (Table 2).
The results canalso explain the benefits of training IMMA with PLT: by leveraging the graph accuracy predictionthat arises from using a single graph layer, but with the expressive power of a full multiplex graph.
To further understand how IMMA uses the inferred social graph, we assess how the trajectorypredictions are altered if we manually manipulate the latent graph.
More formally, we ask our decoderto generate new trajectories conditioned on the new latent graph p θ ( X T h
+1:
T h + T f | X 1: T h , z new ) .
In
Fig.
5, we see that with IMMA, changing the leader of an agent clearly alters the predicted trajectoryto target that new leader while keeping the predictions for other agents intact, whereas the generatedtrajectories of the baseline consist of unrealistic turns (red agent 0) and the predicted trajectories ofother agents deteriorates at the same time. Using the trajectories shown in Fig.
5, we ran a humanstudy and our model’s prediction is judged as more reasonable than the RFM model’s in 76.67% ofthe trials.
Find more details of the human study in Appendix section C.
4.4 Ablation Study
To answer Q3 , we experiment with three ablated versions of our model.
We ensure that all ablated versions have around the same amount of total parameters and have two layers of latent graphs.
[Table]
Additionally, we measure the information dependency between the two layers of latent graph with Normalized Mutual Information (NMI) score (details in the Appendix).
Results are shown in Table 3.
Thefirst column in the table shows the performance of a model where the latent graph structure are edgetypes between all pairs of agents.
It has an NMI score of 1.0 (complete dependency) because the second layer of latent graph is simply one minus the first layer of latent graph.
We observe that the biggestgain arose from using a multiplex attention graph and Progressive Layer Training furthers the performance.
We also see that both components contribute to decreased information dependency between
[Table]
two latent graph layers without explicitly encouraging disentanglement.
Although imposing an additional loss todisentangle between two latent graphlayers does yield the smallest NMIscore, it comes with the trade-off ofdecreased forecasting accuracy as observed in other works [10, 18, 24].
4.5 Sample Efficiency and Generalization
To answer Q4 , we investigatesample efficiency and observethat our method consistently outperforms the strongest baseline(RFM) on smaller sample complexity regimes (Fig. 6).
We testzero-shot generalization performance in response to three environment modifications: agentsmoving twice as fast (2x speed),crowding the agents into half as much space (2x smaller environment), and doubling the number ofagents (2x more agents).
As shown in Table 4, MG and MG+PLT versions of IMMA outperformthe baselines in all metrics.
Additionally, as shown in Appendix Table 5, the change in performanceon the generalization scenarios relative to the original environment was similar to or better than thebaselines in all metrics
Conclusion
Multi-agent dynamics can often be complex as a result of many layers of underlying social interactions.
Agents can be influenced by multiple independent types of relationships with each agent, leading toproperties (such as intentionality or cooperation) that are often absent in simpler physical systems.
We developed a method that uses multiplex attentional graphs as latent representations to model thedynamics that can arise from such multi-layered multi-agent interactions.
Addressing key aspects ofsocial interaction, our method consistently outperforms other state-of-the-art methods in modelingthe dynamics of social multi-agent systems.
We foresee minimal direct negative societal impact —please see Appendix for further discussion.
Future work will extend this approach to active learningsettings and investigate using our method to guide model-based reinforcement learning agents.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will release
all code and data after we refactor and tidy-up our code, no later than the publication date.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4 and the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See the Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
We will release the data upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We only used public benchmark datasets and synthetic datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]