Y OU MOSTLY WALK ALONE : ANALYZING FEATURE ATTRIBUTION IN TRAJECTORY PREDICTION
A BSTRACT
Predicting the future trajectory of a moving agent can be easy when the past trajectory continues smoothly but is challenging when complex interactions with other agents are involved
Recent deep learning approaches for trajectory prediction show promising performance and partially attribute this to successful reasoning about agent-agent interactions.
However, it remains unclear which features such black-box models actually learn to use for making predictions.
This paper pro - poses a procedure that quantiﬁes the contributions of different cues to model performance based on a variant of Shapley values.
Applying this procedure to stateof-the-art trajectory prediction methods on standard benchmark datasets shows that they are, in fact, unable to reason about interactions.
Instead, the past trajectory of the target is the only feature used for predicting its future.
For a task with richer social interaction patterns, on the other hand, the tested models do pick up such interactions to a certain extent, as quantiﬁed by our feature attribution method.
We discuss the limits of the proposed method and its links to causality
I NTRODUCTION
Predicting the future trajectory of a moving agent is a signiﬁcant problem relevant to domains such as autonomous driving (Weisswange et al., 2021; Xu et al., 2014), robot navigation (Chen et al., 2018), or surveillance systems (Morris & Trivedi, 2008).
Accurate trajectory prediction requires successful integration of multiple sources of information with varying signal-to-noise ratios: whereas the target agent’s past trajectory is quite informative most of time, interactions with other agents are typically sparse and short in duration, but can be crucial when they occur.
Graph neural networks (N. Kipf & Welling, 2017) and transformers (Vaswani et al., 2017) have led to recent progress w.r.t.
average prediction error (Mohamed et al., 2020; Yu et al., 2020; Salzmann et al., 2020; Mangalam et al., 2020), but the predicted distributions over future trajectories remain far from perfect, particularly in unusual cases.
Moreover, the black-box nature of state-of-the-art models (typically consisting of various recurrent, convolutional, graph, and attention layers) makes it increasingly difﬁcult to understand what information a model actually learns to use to make its predictions.
To identify shortcomings of existing approaches and directions for further progress, in the present work, we propose a tool for explainable trajectory prediction .
In more detail, we use Shapley values (Shapley, 1953; Lundberg & Lee, 2017) to develop a feature attribution method tailored speciﬁcally to models for multi-modal trajectory prediction.
This allows us to quantify the contribution of each input variable to the model’s performance (as measured, e.g., by the negative log likelihood) both locally (for a particular agent and time point) and globally (across an entire trajectory, scene, or dataset).
Moreover, we propose an aggregation scheme to summarize the contributions of all neighboring agents into a single social interaction score that captures how well a model is able to use information from interactions between agents.
1 for
Overall our analysis shows that established trajectory prediction datasets are suboptimal for benchmarking the learning of interactions among agents but existing approaches do have the capability to learn such interactions on more appropriate datasets.
We highlight the following contributions:
• We address, for the ﬁrst time, feature attribution for trajectory prediction to gain insights about the actual cues contemporary methods use to make predictions.
We achieve this by designing a variant of Shapley values that is applicable to a large set of trajectory prediction models (§ 3).
• With this method, we quantify feature attributions both locally (per scenario; § 3.1) and globally (over the whole dataset; § 3.2) and study the robustness of given models (§ 3.3).
• Our study uncovers that, on popular benchmarks, existing models do not use interaction features.
However, when those features have a strong causal inﬂuence on the target’s future trajectory, these models start to reason about them (§ 4.4).
B ACKGROUND & R ELATED W ORK
Our primary goal is to better understand the main challenges faced by existing trajectory prediction models.
First, we formalize the problem setting (§ 2.1) and review prior approaches (§ 2.2), as well as explainability tools (§ 2.3), which we will use to accomplish this goal in § 3.
2. T HE T RAJECTORY P REDICTION P ROBLEM S ETTING
Let { X 1 t , ..., X n t } , t ∈ Z denote a set of time series corresponding to the trajectories of n agents that potentially interact with each other; see Fig.
2 (a) for an illustration.
All X it are assumed to take values in R d , that is, the state of agent i at time t is a vector x it ∈ R d encoding, e.g., its 2D position, velocity, and acceleration.
We refer to an observed temporal evolution x i 1: T =
( x i 1 , ..., x iT ) of a single agent as a trajectory of length T and to the collection of trajectories x 1: n 1: T =
( x 11: T , ..., x n 1: T ) of all n agents as a scene .
Assume that we have access to a training dataset consisting of M such scenes.
The trajectory prediction task then consists of predicting the future trajectory ( X it +1 , ...,
X it +∆ t ) of a given target agent i and at a given time t up to a time horizon ∆ t given the observed past x 1: n t − h : t of both the target agent itself and of all the other agents, where h is the length of the history that is taken into account.
Formally, we thus aim at learning the distributions P ( X it +1:
t +∆ t | X 1: n t − h : t ) for any t ∈ Z and i ∈ N := { 1 , ..., n } with given ∆ t and h .
Since any given agent (e.g., a particular person) typically only appears in a single scene in the training dataset, solving the trajectory prediction task requires generalizing across scenes and agent [Figure] identities.
In other words, it is necessary to learn how an agent’s future trajectory, in general , depends on its own past and on the past behavior of other neighboring agents.
2. E XISTING A PPROACHES FOR T RAJECTORY P REDICTION
Fig.
2 (b)–(d) shows a general framework that uniﬁes most existing approaches for trajectory prediction.
It consists of three main modules:
(b) a history encoder ψ that learns an embedding F i =
ψ ( X it − h : t ) of the history of each agent; (c) an interaction encoder ξ that incorporates information from neighboring agents by learning an embedding for the edge between two agents E ij = ξ ( X it − h : t , X jt − h : t ) ; and (d) a decoder ϑ that combines both history and interaction features to generate the predicted future trajectory ˆ x it +1:
t +∆ t :=
ϑ ( F i , ¯ E i ) of the target agent, where ¯ E i is the result of aggregating all edge embeddings (see below).
Existing approaches mostly differ in the choices of these modules.
To handle the temporal evolution of the agent state, LSTMs (Hochreiter & Schmidhuber, 1997) are widely used for ψ and ϑ , i.e., to encode the history and decode the future trajectory, respectively (Alahi et al., 2016; Zhang et al., 2019).
Since the future is highly uncertain, stochastic decoders are typically used to sample multiple trajectories, ˆ x it +1:
t +∆ t ∼ ϑ ( . ) , e.g., using GANs (Gupta et al., 2018) or conditional VAEs (Lee et al., 2017; Salzmann et al., 2020).
Alternatively, Makansi et al.
(2019) proposed a multi-head network that directly predicts the parameters of a mixture model over the future trajectories.
To handle interactions between moving agents (Fig. 2c), existing works model a scene as a graph in which nodes correspond to the state embeddings F i of the n agents and edges are speciﬁed by an adjacency matrix A with A ij = 1 iff.
agents i and j are considered neighbors (based, e.g., on their relative distance).
Given this formulation, local social pooling layers have been used to encode and aggregate the relevant information from nearby agents within a speciﬁc radius (Alahi et al., 2016;
Gupta et al., 2018).
More recently, Mangalam et al.
(2020) proposed a non-local social pooling layer, which uses attention and is more robust to false neighbor identiﬁcation, while Salzmann et al.
(2020) modeled the scene as a directed graph to represent a more general set of scenes and interaction types, e.g., asymmetric inﬂuence.
In a different line of work, Mohamed et al.
(2020) proposed undirected spatio-temporal graph convolutional neural networks to encode social interactions, and Yu et al.
(2020) additionally incorporated transformers based on self-attentions to learn better embeddings.
In general, the edge encodings { E ij } j for agent i are then aggregated over the set of its neighbors (i.e., those j with A ij = 1 ) to yield the interaction features ¯ E i , e.g., by averaging as follows:
[Equation]
Although some of the above works show promising results on common benchmarks, it is unclear what information these methods actually use to predict the future.
In the present work, we focus mainly on this aspect and propose an evaluation procedure that quantiﬁes the contribution of different features to the predicted trajectory, both locally and globally.
2. E XPLAINABILITY AND F EATURE A TTRIBUTION
An important step toward interpretability in deep learning are feature attribution methods which aim at quantifying the extent to which a given input feature is responsible for the behavior (e.g., prediction, uncertainty, or performance) of a model.
Among the leading approaches in this ﬁeld is a concept from cooperative game theory termed Shapley values (Shapley, 1953), which fairly distributes the payout of a game among a set of n players.
Within machine learning, Shapley values can be used for feature attribution by mapping an input x = ( x 1 , ..., x n ) to a game in which players are the individual features x i and the payout is the model behavior f : X 1 × ...
× X n → R on that example (Lundberg & Lee, 2017).
Formally, one deﬁnes a set function ν : 2 N → R with
N := { 1 , ..., n } whose output ν ( S ) for a subset S ⊂ N corresponds to running the model on a modiﬁed version of the input x for which features not in S are “dropped” or replaced (see below for details).
The contribution of x i , as quantiﬁed by its Shapley value φ ( x i ) , is then given by the difference ν ( S ∪ { i } ) − ν ( S ) between including and not including i , averaged over all subsets S : φ ( x i ) =
[Equation]
There are different variants of Shapley values based mainly on the following two design choices:
(i)
What model behavior (prediction, uncertainty, performance) is to be attributed?
(choice of f ) (ii)
What is the reference baseline, that is, how are features dropped?
(choice of ν )
A common choice for (i) is the output (i.e., prediction) of the model.
Alternatively, Janzing et al.
(2020a) deﬁned f as the uncertainty associated with the model prediction, thus quantifying to what extent each feature value contributes to or reduces uncertainty.
In the present work, we will focus on attributing model performance.
As for (ii) the set function ν , Lundberg & Lee (2017) originally proposed to replace the dropped features x N \ S with samples from the conditional data distribution given the non-dropped features: ν ( S )
=
E [ f ( x S , X N \ S ) | X S = x S ] ; they then approximate this with the marginal distribution, ν ( S ) =
E [ f ( x S , X N \ S )] , using the simplifying assumption of feature independence.
Janzing et al.
(2020b) argued from a causal perspective that the marginal distribution is, in fact, the right distribution to sample from since the process of dropping features naturally corresponds to an interventional distribution, rather than an observational (conditional) one.
In an alternative termed baseline Shapley values, the dropped features are replaced with those of a predeﬁned baseline x (cid:48) :
ν ( S ) =
f ( x S , x (cid:48)
N \ S ) , see Sundararajan & Najmi (2020) for details.
E XPLAINABLE T RAJECTORY P REDICTION
In the present work, we are interested in better understanding what information is used by trajectory prediction models to perform well, that is, quantifying the contribution of each input feature to the performance of a given model (rather than to its actual output).
Therefore, we deﬁne the behavior f to be attributed (i.e., choice (i) in § 2.3) as the error of the output prediction f :=
L (ˆ x it +1:
t +∆ t , x it +1: t +∆ t ) , where L denotes any loss function (see § 4.3 for common choices).
3. W HICH S HAPLEY V ALUE V ARIANT : H OW TO D ROP F EATURES ?
Next, we need to decide how to deﬁne the set function ν (choice (ii) in § 2.3), i.e., how to drop features.
Since this choice can be highly context-dependent, we need to ask: what is the most reasonable treatment for dropped features in the context of trajectory prediction ?
Intuitively, dropping a subset of agents should have the same effect as if they were not part of the scene to begin with: ideally, we would like to consider the behavior difference between agents being present or not.
As discussed in § 2.3, Shapley values are commonly computed by replacing the dropped features with random samples from the dataset or with a speciﬁc baseline value x (cid:48) .
For the latter, choosing the baseline value x (cid:48) is often non-trivial: for instance, Dabkowski & Gal (2017) use the feature-wise mean as a baseline, while Ancona et al.
(2019) set the baseline to zero.
However, for either choice the dropped features still contribute some signal (e.g., causing gray-ish or black pixels, respectively, in the context of images) that may affect the feature attribution scores.
The same caveat applies to using random samples which—in addition to the computational overhead—also makes this variant unattractive for our needs.
For trajectory prediction, we thus opt for a modiﬁed version of baseline Shapley values where we choose the baseline to be a static, non-interacting agent.
This means that the contribution of the past trajectory of a target agent is quantiﬁed relative to a static non-moving trajectory, and the contributions of neighboring agents are quantiﬁed relative to the same scene after removing those agents.
Owing to the ﬂexible formulation of the interaction encoder as a graph (see § 2.2 and Fig. 2c), dropping a neighboring agent from the scene is equivalent to cutting the edge between the target agent and that neighbor, and thus easy to implement.
Formally, we achieve this by modifying the adjacency matrix of the graph used in (1) as follows:
[Equation]
This formulation applies to most existing frameworks using directed (Salzmann et al., 2020) or undirected graphs (Alahi et al., 2016; Gupta et al., 2018;
Mangalam et al., 2020; Mohamed et al., 2020).
3. C ONTRIBUTION A GGREGATION : THE S OCIAL I NTERACTION S CORE
With the above formulation, quantifying the contribution of features (i.e, the past trajectory of the agent-of-interest and the interaction with other agents) in a speciﬁc scenario is possible.
On the other hand, aggregating these Shapley values over multiple scenarios is important to draw conclusions about the behavior of the model on a speciﬁc dataset.
Formally, given the Shapley values of a feature for a speciﬁc scenario φ ( x i ) , we aim at quantifying the contribution of the feature over the whole dataset φ i .
The common methodology is to average the obtained Shapley values for every feature over multiple scenarios.
This is problematic when features vary over scenarios.
For instance, the neighbors of the agent-of-interest may change over time and are different over scenes, thus aggregating their contribution over the whole dataset yields the wrong statistics.
To address this problem, we split the features into two types: permanent features P and temporary features T .
The past trajectory of the agent-of-interest is an example of the former type, while the neighboring agents are examples of the latter.
For every permanent feature, we use the common average aggregation to estimate the overall contribution of that feature φ i =
E [ φ ( x i )] .
For the temporary features, we ﬁrst aggregate them locally (within the same scenario) and then globally.
For the global aggregation, we use the average operator.
Since interactions with neighbors are usually sparse (i.e., only few neighbors have true causal inﬂuence on the target), we choose the max operator for the local aggregation.
In other words, we quantify the contribution of the most inﬂuential neighboring agent, which we call the social interaction score :
[Equation]
A larger non-negative score indicates that at least one of the neighbors has a signiﬁcant positive contribution to the predicted future while a zero score indicates that none of the neighbors are used.
3. R OBUSTNESS A NALYSIS
Beside the contribution analysis of existing features (such as neighboring agents) via our formulation of the Shapley values, we propose to use the same methodology to study the robustness of pretrained models when random agents are added to the scene.
Thanks to the structure of the interaction encoder (modeled as a graph), adding a random neighbor is equivalent to adding a new incoming edge in Fig.
2 (c).
The random agent can be drawn from the same scene at a different time or from another scene.
Random agents should not contribute to the future of the agent-of-interest,thus their Shapley values given a robust model should be close to zero; see the dummy property of
Shapley values (Sundararajan & Najmi, 2020).
In other words, this analysis tests the model ability in distinguishing between real and random neighbors. [Figure]
E XPERIMENTS
Given the proposed method, we analyze the role of interaction both locally and globally for different state-of-the-art models in trajectory prediction.
4. M ODELS
Our analysis is based on three recent state-of-the-art models: Social-STGCNN (Mohamed et al., 2020), Trajectron++ (Salzmann et al., 2020) and PECNet (Mangalam et al., 2020).
We refer to § 2.
for details about the implementation choices of the key modules (history/interaction encoders and decoder).
For PECNet, we compare the three provided pretrained models (Mangalam et al., 2020).
Alternatively, we propose a new variant of the Trajectron++ (named Trajectron++Edge) in which we make both the history and interaction encoders stronger.
For the history encoder (LSTM), we make it one layer deeper.
To handle a variable number of neighboring agents, they propose to aggregate all neighbors from the same type (e.g., all pedestrians) into one node and learn a single edge to the target agent; see Fig.
3 (a).
Alternatively, we propose to learn separate edges to all neighboring agents and to add ”dummy” agents to the scene until we reach a predeﬁned maximum number; see Fig.
3 (b).
Notably, giving the network the freedom to learn separate edges between any two agents is more powerful.
For instance, two neighboring pedestrians could have different effects on the future of the target agent.
Using our evaluation procedure, we can compare the two variants by quantifying the contribution of both features (history and interaction).
4. D ATASETS
ETH-UCY is one of the most common benchmarks for trajectory prediction.
The dataset is a combination of the ETH (Pellegrini et al., 2009) and UCY (Leal-Taix ´ e et al., 2014) datasets withdifferent scenes and 1 , 536 pedestrians.
We use the standard 5 -fold cross validation for our analysis.
Stanford Drone Dataset (SDD) (Robicquet et al., 2016) is a large dataset with 20 different scenes covering multiple areas at Stanford university.
It consists of over 11 k agents of different types (e.g, pedestrians, bikers, skaters, cars, etc.).
We follow the standard train/test split used in previous works (Mangalam et al., 2020; Gupta et al., 2018).
nuScenes (Caesar et al., 2020) is one of the largest autonomous driving datasets with more than 1 , 000 scenes collected from Boston and Singapore, where each is 20 seconds long.
The dataset also has semantic maps with 11 different layers such as drivable areas and pedestrian crossings.
We also use the standard training/testing splits (Salzmann et al., 2020).
SportVU (Yue et al., 2014) is a tracking dataset for games recorded from multiple seasons of the
NBA.
Every scene has two teams of 5 players and the ball.
We pre-process the dataset to remove short scenes and randomly select 7,000 scenes for training and 100 scenes for testing.
4. E VALUATION M ETRICS
min-ADE is the minimum over all predicted trajectories of the average displacement error (L distance) to the ground truth.
Similarly, min-FDE is the minimum ﬁnal displacement error, i.e., the minimum L2 distance over all predicted trajectories to the ground truth end point at time t + ∆ t . [Figure]
The previous two metrics are biased since they report only the minimum error over many predictions and are commonly used for testing multi-modality of the prediction.
To assess all predicted trajectories for those methods predicting a multimodal probabilistic distribution of the future, we report also the NLL of the ground truth given the predicted distribution.
For the Shapley values, we choose the NLL of the model as the output of the set function ν for both variants of Trajectron++ and Social-STGCNN.
On the other hand, PECNet is a non-probabilistic approach and can only output multiple trajectories by decoding multiple samples drawn from the latent space.
Therefore, we choose the min-ADE of the model when evaluating PECNet models.
4. R ESULTS
Analysis on common benchmarks.
As discussed in § 3, we report the contribution for the past trajectory (since it is a permanent feature) and the neighboring agent with the maximum contribution (our social interaction score).
Additionally, we also compare to the contribution of a random agent added to the scene (robustness).
From the results shown in Fig.
4 on ETH-UCY (a), nuScenes (b) and SDD (c-d), we observe the following: (1) the past trajectory has the largest contribution on the predicted future, (2) the contribution of the neighbors (our social interaction score ) is insigniﬁcant and close to zero, and (3) the contribution of a random neighbor is almost identical to existing neighbors.
Notably, for those datasets with heterogeneous agents (e.g, pedestrians, vehicles, bikers, etc.), we report the average results over different types of agents.
From the last two observations, we conclude that recent methods are unable to exploit information from neighboring agents to predict the future on these common benchmarks.
Analysis on the SportVU dataset.
We conduct the same analysis on a dataset with rich patterns of interactions and show the results in Fig.
4 (e, f).
Although the past trajectory still has the largest contribution, neighboring agents start to play an important role by contributing to the predicted future.
Moreover, there is a signiﬁcant difference between the maximum contribution of neighbors (our social interaction score ) and the contribution of a random neighbor.
Our analysis is not only important to explain and test models but also to compare them.
As can be seen in Fig.
4 (e), the contributions of both the past trajectory and the neighbors of our Trajectron++Edge variant are signiﬁcantly larger than the original model.
Referring to § 4.1, both the history and interaction encoders are stronger in our variant which explains the increase in their contributions.
On the other hand, both variants are robust to random neighbors added to the game.
In Fig.
4 (f), we compare PECNet and both variants of Trajectron++ on the SportVU dataset where
[Table]
we observe that our Trajectron++Edge has the largest social interaction score while PECNet is largely based on the past.
Similarly to our social interaction score, Tab. 1 shows the relative difference in the average error when dropping all neighbors.
Here we also observe that social interaction is only important on the
SportVU dataset.
Instead of dropping all neighbors during inference, we drop them during training and report the difference in Appendix C.
Local analysis.
In Fig.
5, we show the results of our Trajectron++Edge on two interesting scenarios.
Here, every player (and the ball) is represented as a circle and colored according to the team.
We plot the past trajectories of all players (solid lines), the ball (yellow) and the future trajectory of the target player (dashed line).
We overlay the predicted distribution of the model (blue heatmap) on top of the scene.
Additionally, we show the Shapley values of the scenario (right) for the past (red), neighboring players and the ball.
For the ﬁrst scenario (a), we clearly see that the past is the main feature contributing to the predicted future where all other players have insigniﬁcant contributions.
On the other hand, for the second scenario where interaction with other players is critical to make the correct prediction (b), we can identify that the three main players (counter attacking) and the ball contribute positively to the predicted future, while the past has relatively small inﬂuence on the prediction.
Interestingly, agent 6 moves to the right (opposite of the true future of the target) and thus has a negative contribution on the prediction of the target agent.
Overall, our method is able to identify and visualize which feature has a large inﬂuence on the target agent future for speciﬁc local events.
D ISCUSSION
Links to Granger causality.
Although the relevance of features for predicting the target should not be confused with their causal impact , the two are tightly linked.
According to the seminal work of Granger (1969), a time series X t causally inﬂuences a target Y t whenever the past of X helps better predict Y t from its own past and that of all features other than X .
This reasoning is justiﬁed—and follows from the causal Markov condition and causal faithfulness, see (Peters et al., 2017, Thm.
10.3)—whenever the following three assumptions are satisﬁed: (i) every feature inﬂuences others only with a nonzero time lag (i.e., no instantaneous inﬂuence); (ii) there are no latent (unobserved) common causes of the observed features (causal sufﬁciency); and (iii) the prediction is optimal in the sense that it employs the full statistical information from the past of the features under consideration.
Although these assumptions are hard to satisfy exactly in practice, Granger causality has been widely applied since decades due to its simplicity.
Since our attribution analysis is based on comparing model performance (as opposed to predictions ) with and without a certain feature, it is loosely related to Granger causality.
The main differences are that we (a) perform a post-hoc analysis on a single model, rather than training separate ones with and without the feature of interest; and (b) average over subsets of the remaining features, instead of always including them.
[Figure]
Interpretation and limitations.
While assumptions (i) and (ii) above describe the general scenarios for which our attribution method may allow a causal interpretation, assumption (iii) points to an important limitation.
Since a model simply may not have learned to pick up and make use of certain inﬂuences, we cannot conclude from small Shapley values that no causal inﬂuence exists .
If, however, we know a priori that a causal inﬂuence does exist and the corresponding Shapley value is (close to) zero, then this points to a failure of the model.
Similarly, if there is a causal inﬂuence and the model can pick this up, we will see it in the Shapley value.
For the trajectory prediction task, where only past information is used for prediction, positive Shapley values thus suggest the existence of a causal inﬂuence, subject to (i) and (ii).
Exclusion vs randomization of features.
As opposed to detecting causal inﬂuence, we note that quantifying its strength via the degree of predictive improvement is not always justiﬁed, contrary to what is often believed.
Janzing et al.
(2013, Sec.
3.3 and Example 7) argued, based on a modiﬁcation of a paradox described by Ay & Polani (2008), that assessing causal strength by excluding the feature for prediction is ﬂawed and showed that it should be randomized instead.
However, our experiments in Appendix B show only a minor difference between exclusion and randomization in our case.
C ONCLUSION
We addressed feature attribution for trajectory prediction by analyzing to what extent the available cues are actually used by existing methods to improve predictive performance.
To this end, we proposed a variant of Shapley values for quantifying feature attribution, both locally and globally, and for studying the robustness of given models.
Subject to the assumptions and caveats discussed in § 5, our attribution method can be interpreted as a computationally efﬁcient way of (approximately) quantifying causal inﬂuence in the context of trajectory prediction, both for indicating the strength of causal inﬂuence of a dataset and for benchmarking how well a particular model uses such inﬂuence for its prediction.
Using our method, we reveal that existing methods, contrary to their claims, do not rely on interaction features when trained on popular data sets, but that such information is used on other data sets such as SportVU where interactions play a larger role, see
Appendix D for further discussion of our ﬁnding.
A CKNOWLEDGMENTS
We would like to thank Lenon Minorics and Atalanti Mastakouri for the useful discussions and feedback.
This work was supported by the German Federal Ministry of Education and Research (BMBF): T¨ubingen AI Center, FKZ: 01IS18039B and by the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645.
We also thank Amazon Web Services for providing enough resources to conduct the experiments
A O THER F EATURE A TTRIBUTION M ETHODS
Other feature attribution methods can generally be categorized into two streams: perturbation-based and gradient-based approaches.
The former perturb some parts of the input and use the change in the output as a measure of feature relevance (Zeiler & Fergus, 2014).
Such methods are sensitive to the applied perturbation and violate the sensitivity axiom (Sundararajan et al., 2017).
Gradient-based approaches estimate the feature attributions by investigating the local gradients of the model.
To address the saturation problem of gradients, DeepLIFT (Shrikumar et al., 2017) and LRP (Binder et al., 2016) approximate gradients with discrete differences, which results in violating the implementation invariance axiom (Sundararajan et al., 2017).
Sundararajan et al.
(2017) proposed the
Integrated Gradients (IG) to estimate the feature attribution relevant to a baseline by integrating over the gradient along the path between the given input and the baseline, which satisfy all required axioms and can be seen as a variant of the Shapley values (Sundararajan & Najmi, 2020).
B E XCLUSION VS R ANDOMIZATION OF F EATURES
In Fig.
6, we compare the obtained Shapley values using our method and the ones obtained by running the marginal Shapley values.
For the latter, we replace the feature by a random feature drawn from the marginal distribution.
In other words, instead of removing a neighboring agent, we replace it by a random agent from another game and average over multiple replacements.
Clearly, we draw the same conclusions (as in § 4.4) using the marginal Shapley values while our method is more computationally efﬁcient since we do not need to marginalize over many samples.
C R ETRAINING W ITHOUT I NTERACTION
To further support our ﬁndings, we conduct an additional experiment where we train one of the
SOTA models from scratch on a dataset without interaction by removing neighbors to assess if models need interaction or not.
We selected the Traj++ model, retrained it on the ETH-UCY dataset, and compared the results to the same model trained with interactions.
Tab.
2 shows that both models (with and without interaction) yield the same error, which is consistent with our ﬁndings.
D F URTHER D ISCUSSION
Here we provide two complementary explanations for our ﬁnding that existing models behave differently between the popular benchmarks (e.g., ETH-UCY) and the SportVU dataset: (i) a large difference in the frequency of interactions, and (ii) a large difference in the nature of interactions. As
[Table]
for (i), the concurrent work (Makansi et al., 2021)highlights the long-tailed distribution of common pedestrian benchmarks (e.g., ETH-UCY)
meaning that most scenarios can be predicted well using a simple linear Kalman ﬁlter (so-called easy scenarios) while more challenging scenarios (where interaction is needed) are rare; this is less pronounced on the sports dataset where the proportion of hard scenes (requiring interaction modelling) is larger, see Fig.
As for (ii), in existing benchmark settings, people tend to walk toward their destinations mostly avoiding interactions , except for the purpose of avoiding obstacles and collisions, whereas interaction is an integral part of the activity in a sports scenario such as playing a game of basketball.
Concurrently to our work, Chen et al.
(2021)highlight the importance of the bias between training and testing environments and introduced a counterfactual analysis in which they intervene on the causal graph by cutting the edge between the predicted trajectory and the environment.
To do so, they utilize several choices for the counterfactual trajectory such as the mean trajectory or a random trajectory.
Our method, on the other hand, remove totally the inﬂuence of a feature (neighboring trajectory) by removing its edges in the interaction graph.
E I NTERACTION G RAPH L EARNING
An interesting and related line of works is to explicitly learn an interaction graph as part of the trajectory prediction task.
Kipf et al.
(2018) introduce the NRI framework, a variational auto-encoder, where the learned latent code represents the underlying interaction graph of the system.
Several recent works build upon the NRI framework by allowing the interaction graph to change over time (i.e., dynamic relations) (Li et al., 2020; Graber & Schwing, 2020).
L¨owe et al.
(2020) leverage the shared knowledge across different samples and scenes and learn the causal relations between time series (e.g., moving agents).
We adapted the latter work into the state-of-the-art framework for tra- [Table] jectory prediction (Traj++ (Salzmann et al., 2020)) by replacing the edge encoder of the framework with a learned interaction graph (referred as Traj++NRI) and train the model on the ETH-UCY/univ scene.
Tab.
3 shows an analysis of the Traj++ model with its original edge encoder and the learned interaction graph from L¨owe et al.
(2020) for different prior graphs.
We observe that learning an interaction graph did not lead to a better model and dropping the edges during inference yielded similar/better performance.
This supports our hypothesis that on common benchmarks, social interaction features are not learned.
F S CENE -A GENT I NTERACTION
Although our work targets mainly the analysis between agents, extending this analysis to the interaction between dynamic agents and the scene (e.g., semantic map) is valuable.
To this end, one option would be to learn a compact set of high-level map features (e.g., via a CNN encoder) and to add this as an additional single node to the interaction graph.
Our approach could then be applied to quantify also the contribution of map information to model performance.