Original: A GENT P RIORITIZATION WITH I NTERPRETABLE R ELATION FOR T RAJECTORY P REDICTION
Revised: A GENT P RIORITIZATION WITH I NTERPRETABLE R ELATION FOR T RAJECTORY P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we present a novel multi-agent trajectory prediction model, which discovers interpretable relations among agents and prioritize agent’s motion.
Revised: In this paper, we present a novel multi-agent trajectory prediction model, which discovers interpretable relations among agents and prioritize agent’s motion.

Original: Different from existing approaches, our interpretable design is inspired by the fundamental navigation and motion functions of agent movements, which represent ’where’ and ’how’ the agents move in the scenes.
Revised: Different from existing approaches, our interpretable design is inspired by the fundamental navigation and motion functions of agent movements, which represent ’where’ and ’how’ the agents move in the scenes.

Original: Specifically, it generates the relation matrix, where each element indicates the motion impact from one to another.
Revised: Speciﬁcally, it generates the relation matrix, where each element indicates the motion impact from one to another.

Original: In addition, in highly interactive scenarios, one agent may implicitly gain higher priority to move, while the motion of other agents may be impacted by the prioritized agents with higher priority (e.g., a vehicle stopping or reducing its speed due to crossing pedestrians).
Revised: In addition, in highly interactive scenarios, one agent may implicitly gain higher priority to move, while the motion of other agents may be impacted by the prioritized agents with higher priority (e.g., a vehicle stopping or reducing its speed due to crossing pedestrians).

Original: Based on this intuition, we design a novel motion prioritization module to learn the agent motion priorities based on the inferred relation matrix.
Revised: Based on this intuition, we design a novel motion prioritization module to learn the agent motion priorities based on the inferred relation matrix.

Original: Then, a decoder is proposed to sequentially predict and iteratively update the future trajectories of each agent based on their priority orders and the learned relation structures.
Revised: Then, a decoder is proposed to sequentially predict and iteratively update the future trajectories of each agent based on their priority orders and the learned relation structures.

Original: We first demonstrate the effectiveness of our prediction model on simulated Charged Particles (Kipf et al., 2018) dataset.
Revised: We ﬁrst demonstrate the effectiveness of our prediction model on simulated Charged Particles (Kipf et al., 2018) dataset.

Original: Next, extensive evaluations are performed on commonly-used datasets for robot navigation, human-robot interactions, and autonomous agents: real-world NBA basketball (Yue et al., 2014) and INTERACTION (Zhan et al., 2019).
Revised: Next, extensive evaluations are performed on commonly-used datasets for robot navigation, human-robot interactions, and autonomous agents: real-world NBA basketball (Yue et al., 2014) and INTERACTION (Zhan et al., 2019).

Original: Finally, we show that the proposed model outperforms other state-of-the-art relation based methods, and is capable to infer interpretable, meaningful relations among agents.
Revised: Finally, we show that the proposed model outperforms other state-of-the-art relation based methods, and is capable to infer interpretable, meaningful relations among agents.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Revised: Past

Revised: Trajectory

Original: [Figure]
Revised: [Figure]

Original: paradigm of relation
Revised: Encoding Future

Original: [Figure]
Revised: [Figure]

Original:  agent motions, and make in-order prediction based on their priorities.
Revised: relations, prioritize agent motions, and make in-order prediction based on their priorities.

Original: Multi-agent trajectory prediction is an essential component in a wide range of applications from robot navigation to autonomous intelligent systems.
Revised: Multi-agent trajectory prediction is an essential component in a wide range of applications from robot navigation to autonomous intelligent systems.

Original: While navigating in crowded scenes, autonomous agents (i.e., robots and vehicles) not only themselves interact, but also should have ability to observe others’ interactions and anticipate where other agents will move in near future.
Revised: While navigating in crowded scenes, autonomous agents (i.e., robots and vehicles) not only themselves interact, but also should have ability to observe others’ interactions and anticipate where other agents will move in near future.

Original: This ability is crucial for autonomous agents to avoid collisions and plan meaningful machinehuman/machine-machine interactions.
Revised: This ability is crucial for autonomous agents to avoid collisions and plan meaningful machinehuman/machine-machine interactions.

Original: Designing a robust and accurate trajectory prediction model has attracted much of recent research efforts.
Revised: Designing a robust and accurate trajectory prediction model has attracted much of recent research efforts.

Original: In fact, meaningful reasoning about interactions among agents provides valuable cues to improve the trajectory prediction accuracy, especially in highly interactive scenarios.
Revised: In fact, meaningful reasoning about interactions among agents provides valuable cues to improve the trajectory prediction accuracy, especially in highly interactive scenarios.

Original: However, how to learn/discover meaningful relations among agents from historical motion data to improve prediction accuracy remains a challenging task in recent research.
Revised: However, how to learn/discover meaningful relations among agents from historical motion data to improve prediction accuracy remains a challenging task in recent research.

Original: There is a large body of recent research focused on modeling interaction among agents for future trajectory prediction.
Revised: There is a large body of recent research focused on modeling interaction among agents for future trajectory prediction.

Original: Some notable research in this field learn interaction features using advanced deep learning techniques such as graph neural networks (Scarselli et al., 2008), social pooling mechanism (Alahi et al., 2016; Gupta et al., 2018), or attention networks (Kamra et al., 2020).
Revised: Some notable research in this ﬁeld learn interaction features using advanced deep learning techniques such as graph neural networks (Scarselli et al., 2008), social pooling mechanism (Alahi et al., 2016; Gupta et al., 2018), or attention networks (Kamra et al., 2020).

Original: These works follow a common paradigm, as shown in
Revised: These works follow a common paradigm, as shown in

Original: Figure 1a, to infer the relations among agents using their historical motions.
Revised: Figure 1a, to infer the relations among agents using their historical motions.

Original: The main limitation of these approaches is that they lack a mechanism to learn the motion importance of each agent in the scene.
Revised: The main limitation of these approaches is that they lack a mechanism to learn the motion importance of each agent in the scene.

Original: In realistically interactive scenarios, it is often that the agent movements implicitly imply that an agent gains higher priority to decide where and when to move, their movements will impact the others in the scene.
Revised: In realistically interactive scenarios, it is often that the agent movements implicitly imply that an agent gains higher priority to decide where and when to move, their movements will impact the others in the scene.

Original: For example, in driving scenarios, vehicles yield to crossing pedestrians (higher prioritized agents).
Revised: For example, in driving scenarios, vehicles yield to crossing pedestrians (higher prioritized agents).

Original: In the basket ball game, the other players’ movements are likely to be conditioned (i.e., impacted ) by the ball-controlling offensive player.
Revised: In the basket ball game, the other players’ movements are likely to be conditioned (i.e., impacted ) by the ball-controlling offensive player.

Original: To handle the aforementioned limitations, we propose a new approach, shown in Figure 1b, to prioritize each agent motion based on their interpretable relations.
Revised: To handle the aforementioned limitations, we propose a new approach, shown in Figure 1b, to prioritize each agent motion based on their interpretable relations.

Original: Our model first learns the interaction among agents at each time-step.
Revised: Our model ﬁrst learns the interaction among agents at each time-step.

Original: Inspired by the relation learning method (Fujii et al., 2021) for animal movements, we design a inter-agent encoder that consists of two sub-encoders, each of which represents innate movement and navigation capacities of agents.
Revised: Inspired by the relation learning method (Fujii et al., 2021) for animal movements, we design a inter-agent encoder that consists of two sub-encoders, each of which represents innate movement and navigation capacities of agents.

Original: While the navigation encoder captures the agent relations based on the moving directions (i.e, where to move), the motion encoder infers the relation based on the motion capacity (i.e., how to move).
Revised: While the navigation encoder captures the agent relations based on the moving directions (i.e, where to move), the motion encoder infers the relation based on the motion capacity (i.e., how to move).

Original: Next, the prioritization module quantifies the importance score (i.e., priority) of each agent by measuring their motion impacts on other agents.
Revised: Next, the prioritization module quantiﬁes the importance score (i.e., priority) of each agent by measuring their motion impacts on other agents.

Original: Based on the orders of priorities, sequential predictions are made to allow the predicted future trajectories of agents with higher priorities to impact on the lower prioritized ones within their relation structures.
Revised: Based on the orders of priorities, sequential predictions are made to allow the predicted future trajectories of agents with higher priorities to impact on the lower prioritized ones within their relation structures.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: • We propose a novel prediction pipeline with motion prioritization module to prioritize the importance of each agent based on its motion impacts on other agents within their interpretable relation structures.
Revised: • We propose a novel prediction pipeline with motion prioritization module to prioritize the importance of each agent based on its motion impacts on other agents within their interpretable relation structures.

Original: • We design an interpretable interaction encoder to capture the agent relations from both navigational and motion perspectives.
Revised: • We design an interpretable interaction encoder to capture the agent relations from both navigational and motion perspectives.

Original: The relationships among agents are learned interpretably at each observed time step to produce meaningful relation structures for prioritization task.
Revised: The relationships among agents are learned interpretably at each observed time step to produce meaningful relation structures for prioritization task.

Original: • We evaluate our prediction model on several highly interactive datasets: Charged Particles, NBA, and INTERACTION.
Revised: • We evaluate our prediction model on several highly interactive datasets: Charged Particles, NBA, and INTERACTION.

Original: We show that the proposed model is able to learn meaningful interaction features and outperforms state-of-the-art models on these datasets.
Revised: We show that the proposed model is able to learn meaningful interaction features and outperforms state-of-the-art models on these datasets.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Multi-Agent Trajectory Prediction Multi-agent trajectory prediction is an actively researched problem due to its broad applications in robot planning (Schmerling et al., 2018), traffic prediction (Liao et al., 2018), sport video analysis (Felsen et al., 2017).
Revised: Multi-Agent Trajectory Prediction Multi-agent trajectory prediction is an actively researched problem due to its broad applications in robot planning (Schmerling et al., 2018), trafﬁc prediction (Liao et al., 2018), sport video analysis (Felsen et al., 2017).

Original: Recent research have focused on modeling the relations among agents seeking to improve trajectory prediction.
Revised: Recent research have focused on modeling the relations among agents seeking to improve trajectory prediction.

Original: In general, existing approaches employ common structures such as graph neural network, social-GAN (Mohamed et al., 2020), attention network, transformer, etc. to learn agent interactions from their motions.
Revised: In general, existing approaches employ common structures such as graph neural network, social-GAN (Mohamed et al., 2020), attention network, transformer, etc. to learn agent interactions from their motions.

Original: Notably,
Revised: Notably,

Original: Kosaraju et al.
Revised: Kosaraju et al.

Original: (Kosaraju et al., 2019) on graph attention network (Veliˇckovi´c et al., 2017) to decide how much information to share between agents.
Revised: (Kosaraju et al., 2019) on graph attention network (Veliˇckovi´c et al., 2017) to decide how much information to share between agents.

Original: Kamra et al.
Revised: Kamra et al.

Original: (Kamra et al., 2020) developed a dedicated attention mechanism for trajectory prediction from the inductive bias of motion and intents.
Revised: (Kamra et al., 2020) developed a dedicated attention mechanism for trajectory prediction from the inductive bias of motion and intents.

Original: Kipf et al.
Revised: Kipf et al.

Original: (Kipf et al., 2018) proposed neural relational inference (NRI) proposed by taking the form of a variational auto-encoder;.
Revised: (Kipf et al., 2018) proposed neural relational inference (NRI) proposed by taking the form of a variational auto-encoder;.

Original: Jiachen et al.
Revised: Jiachen et al.

Original: (Li et al., 2020) improved NRI by using graphs that evolve over time.
Revised: (Li et al., 2020) improved NRI by using graphs that evolve over time.

Original: Although these approaches can capture some interactions among agent, they lack mechanism to reason the agent priorities, important feature cues to improve trajectory results.
Revised: Although these approaches can capture some interactions among agent, they lack mechanism to reason the agent priorities, important feature cues to improve trajectory results.

Original:  Another closely related research theme focuses on discovering the Grangercausal (GC) relationship among agents.
Revised: Relation Discovery Another closely related research theme focuses on discovering the Grangercausal (GC) relationship among agents.

Original: SENN (Alvarez Melis & Jaakkola, 2018) is the first selfexplanatory network, a class of intrinsically interpretable models, which explains the contributions concepts (i.e., raw inputs) to predictions.
Revised: SENN (Alvarez Melis & Jaakkola, 2018) is the ﬁrst selfexplanatory network, a class of intrinsically interpretable models, which explains the contributions concepts (i.e., raw inputs) to predictions.

Original: SENN was applied to infer GC relationship via generalized vector autoregression model (GVAR) (Marcinkeviˇcs & Vogt, 2021), which captures the GC relationship via coefficient matrices.
Revised: SENN was applied to infer GC relationship via generalized vector autoregression model (GVAR) (Marcinkeviˇcs & Vogt, 2021), which captures the GC relationship via coefﬁcient matrices.

Original: These models have shown promising performances to learn
Revised: These models have shown promising performances to learn

Original: MotionEncoder
Revised: MotionEncoder

Original: Inter-agent
Revised: Inter-agent

Original: Encoder
Revised: Encoder

Original: Motion Prioritization
Revised: Motion Prioritization

Original: Relation Inference 1 2 1  3 > >
Revised: Relation Inference 1 2 1  3 > >

Original: Past trajectory
Revised: Past trajectory

Original: Future trajectory ℎ ! 𝑧 ",!
Revised: Future trajectory ℎ ! "

Original: Motion Decoder 𝑧 $,!
Revised: Motion Decoder " $,!

Original: #𝑌 !
Revised: #$ !

Original: Φ % agent interaction in simulated environments, where the GC relationships are known and static.
Revised: Φ % agent interaction in simulated environments, where the GC relationships are known and static.

Original: Recently, ABM model (Fujii et al., 2021) was proposed to further extend GVAR model to capture animal interactions such as approaching or repulsing.
Revised: Recently, ABM model (Fujii et al., 2021) was proposed to further extend GVAR model to capture animal interactions such as approaching or repulsing.

Original: ABM relies on scientific knowledge and assumes that if an agent goes straight then there is no interactions.
Revised: ABM relies on scientiﬁc knowledge and assumes that if an agent goes straight then there is no interactions.

Original: For relation discovery in trajectory prediction, (Makansi et al., 2021) analyzed feature attributes to provide insights on the impacts of relation feature and discussed its links to causality inference.
Revised: For relation discovery in trajectory prediction, (Makansi et al., 2021) analyzed feature attributes to provide insights on the impacts of relation feature and discussed its links to causality inference.

Original: Recently, M2I (Sun et al., 2022) learn to predict, but limits to pairs of relations, and only applicable for traffic scenes.
Revised: Recently, M2I (Sun et al., 2022) learn to predict, but limits to pairs of relations, and only applicable for trafﬁc scenes.

Original: Different from these works, we build a new pipeline and extend relation modeling schemes to discover agent relations in real-world interacting scenarios.
Revised: Different from these works, we build a new pipeline and extend relation modeling schemes to discover agent relations in real-world interacting scenarios.

Original: Based on the learned relation structures, agent motions are prioritized for the purpose of future trajectory prediction.
Revised: Based on the learned relation structures, agent motions are prioritized for the purpose of future trajectory prediction.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this work, we tackle the problem of multi-agent trajectory prediction.
Revised: In this work, we tackle the problem of multi-agent trajectory prediction.

Original: Given the past trajectories
Revised: Given the past trajectories

Original: X =
Revised: X =

Original: [ X 1 , X 2 ..., X N ∈ R T o × d ] ∈ R N × T o × d of N agents with d -dimensional spatial locations in the past T o time steps, we aim to predict their trajectories Y =
Revised: [ X 1 , X 2 ..., X N 2 R T o ⇥ d ] 2 R N ⇥ T o ⇥ d of N agents with d -dimensional spatial locations in the past T o time steps, we aim to predict their trajectories Y =

Original: [ Y 1 , Y 2 , ..., Y N ∈ R T p × d ] ∈ R N × T p ×
Revised: [ Y 1 , Y 2 , ..., Y N 2 R T p ⇥ d ] 2 R N ⇥ T p ⇥

Original: d in the next T p future time steps.
Revised: d in the next T p future time steps.

Original: The overview of our prediction model is shown in Figure 2.
Revised: The overview of our prediction model is shown in Figure 2.

Original: Our model consists of three main modules: (1) Motion Encoder: encodes historical motion of each agents.
Revised: Our model consists of three main modules: (1) Motion Encoder: encodes historical motion of each agents.

Original: (2) Inter-agent Encoder: encodes interactions among agents; (3) Decoder: infer relations, prioritize motion, sequentially predict future trajectories of agents.
Revised: (2) Inter-agent Encoder: encodes interactions among agents; (3) Decoder: infer relations, prioritize motion, sequentially predict future trajectories of agents.

Original: Next, we present the details of each module.
Revised: Next, we present the details of each module.

Original: 3. M OTION E NCODER
Revised: 3. M OTION E NCODER

Original: The motion encoder p ( h | X ) captures the individual movement of agents by learning the hidden motion feature h ∈ R N × d h conditioned on X .
Revised: The motion encoder p ( h | X ) captures the individual movement of agents by learning the hidden motion feature h 2 R N ⇥ d h conditioned on X .

Original: We encode the hidden state of each agent i as:
Revised: We encode the hidden state of each agent i as:

Original: [Equation]
Revised: [Equation]

Original: where h ti is the hidden motion state of target agent i at current time step t , o ti is the output feature of
Revised: where h ti is the hidden motion state of target agent i at current time step t , o ti is the output feature of

Original: GRU, C is the size of embedded feature e i .
Revised: GRU, C is the size of embedded feature e i .

Original: 3. M OTION D ECODER
Revised: 3. M OTION D ECODER

Original: The motion decoder q ( ˆ Y | h ) decodes the future trajectories of each agent i from the learned hidden motion feature h ti .
Revised: The motion decoder q ( ˆ Y | h ) decodes the future trajectories of each agent i from the learned hidden motion feature h ti .

Original: We formulate this decoding process as:
Revised: We formulate this decoding process as:

Original: [Equation]
Revised: [Equation]

Original:  where fc is a fully connected layer.
Revised: (5) where fc is a fully connected layer.

Original: To cope with multi-modal nature of future trajectories, we predict a bivariate Gaussian distribution N ( µ t i , σ t +1 i , p t i ) in each future time step t ∈ { t 0 + 1 , ..., t [-
-] + T p } , where µ t +1 i =
Revised: To cope with multi-modal nature of future trajectories, we predict a bivariate Gaussian distribution N ( µ t i ,  t +1 i , p t i ) in each future time step t 2 { t 0 + 1 , ..., t [-
-]0 + T p } , where µ t +1 i =

Original: ( µ x , µ y ) t +1 i , σ t +1 i =
Revised: ( µ x , µ y ) t +1 i ,  t +1 i =

Original: ( σ x , σ y ) t +1 i , p ti are the mean, standard deviation, and correlation coefficient.
Revised: (  x ,  y ) t +1 i , p ti are the mean, standard deviation, and correlation coefﬁcient.

Original: We then randomly select K samples from the distribution for the final multiple trajectories prediction.
Revised: We then randomly select K samples from the distribution for the ﬁnal multiple trajectories prediction.

Original: Due to complex interactions among agents, relying on historical motions is not adequate for accurate predictions.
Revised: Due to complex interactions among agents, relying on historical motions is not adequate for accurate predictions.

Original: Thus, in the next stages, we extend the decoder to incorporate relation features from inter-agent encoder and from the motion prioritization module.
Revised: Thus, in the next stages, we extend the decoder to incorporate relation features from inter-agent encoder and from the motion prioritization module.

Original: The decoder is formulated as q ( ˆ Y | h, z a , z m ) , where z a and z m are relation features from inter-agent encoder and motion prioritization, which will be presented next.
Revised: The decoder is formulated as q ( ˆ Y | h, z a , z m ) , where z a and z m are relation features from inter-agent encoder and motion prioritization, which will be presented next.

Original: 3. I NTER - AGENT E NCODER
Revised: 3. I NTER - AGENT E NCODER

Original: The encoder p a ( z a , Φ | X ) , depicted in Figure 3, learns interpretable relation features z a given the past locations of all agents.
Revised: The encoder p a ( z a ,  | X ) , depicted in Figure 3, learns interpretable relation features z a given the past locations of all agents.

Original: It also produces relation matrices (i.e., coefficient matrices) Φ , consisting of relation matrix Φ t ∈ R N × N at each observed time step t ∈ { t 0 − T o + 1 , ..., t 0 } .
Revised: It also produces relation matrices (i.e., coefﬁcient matrices)  , consisting of relation matrix  t 2 R N ⇥ N at each observed time step t 2 { t 0  T o + 1 , ..., t 0 } .

Original: Φ ti,j is an asymmetric square matrix, which elements indicate the motion impact of an agent on another.
Revised:  ti,j is an asymmetric square matrix, which elements indicate the motion impact of an agent on another.

Original: Thus, the value of Φ ti,j is high if there is strong motion impact from agent i to j .
Revised: Thus, the value of  ti,j is high if there is strong motion impact from agent i to j .

Original: In other words, agent i contains rich information that improves future prediction of agent j .
Revised: In other words, agent i contains rich information that improves future prediction of agent j .

Original: Conversely, Φ ti,j = 0 indicates no relation i → j .
Revised: Conversely,  ti,j = 0 indicates no relation i ! j .

Original: The relation matrices are learned during the training process via inducing sparsity loss, introduced in Section 3.4.[Figure] [Equation] [Figure]
Revised: The relation matrices are learned during the training process via inducing sparsity loss, introduced in Section 3.4.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Equation] [Figure] To learn interpretable and meaningful relation matrix, we design an disentangled inter-agent encoder, which consists of an interactionbased navigation encoder F n (Φ n | X ) and an interaction-based movement encoder
Revised: To learn interpretable and meaningful relation matrix, we design an disentangled inter-agent encoder, which consists of an interactionbased navigation encoder F n (  n | X ) and an interaction-based movement encoder

Original: F m (Φ m | X ) .
Revised: F m (  m | X ) .

Original: Our design is inspired by that autonomous agents innately possess common navigation and movement functions, which enable them an ability to plan the future movements.
Revised: Our design is inspired by that autonomous agents innately possess common navigation and movement functions, which enable them an ability to plan the future movements.

Original: While the navigation encoder captures the relation based on the future directions of agents, the movement encoder infers the relation based on the strength of their movements.
Revised: While the navigation encoder captures the relation based on the future directions of agents, the movement encoder infers the relation based on the strength of their movements.

Original: Specifically, we implement our inter-agent encoder following the concepts of self-explanatory neural networks
Revised: Speciﬁcally, we implement our inter-agent encoder following the concepts of self-explanatory neural networks

Original: SENN(Alvarez Melis & Jaakkola, 2018), and its extensions to to discover causal relationships (GVAR (Marcinkeviˇcs & Vogt, 2021), ABM (Fujii et al., 2021)).
Revised: SENN(Alvarez Melis & Jaakkola, 2018), and its extensions to to discover causal relationships (GVAR (Marcinkeviˇcs & Vogt, 2021), ABM (Fujii et al., 2021)).

Original: These networks consists of a link function g ( · ) , basis conceptsψ ( x ) , and explainable function θ ( x ) to each concept to the predictions.
Revised: These networks consists of a link function g ( · ) , basis concepts ( x ) , and explainable function ✓ ( x ) to each concept to the predictions.

Original: The general form is:
Revised: The general form is:

Original: [Equation]
Revised: [Equation]

Original: In our work, we consider the conceptsψ ( x ) being motion feature
Revised: In our work, we consider the concepts ( x ) being motion feature

Original: h
Revised: h

Original: i , the relation matrix Φ t is the result of explainable function θ ( x ) , and g ( · ) is the sum.
Revised: i , the relation matrix  t is the result of explainable function ✓ ( x ) , and g ( · ) is the sum.

Original: Equation 6 can be written as:
Revised: Equation 6 can be written as:

Original: [Equation]
Revised: [Equation]

Original: Φ t can be decomposed to Φ tm and Φ tn ; that is Φ t =
Revised:  t can be decomposed to  tm and  tn ; that is  t =

Original: Φ tm ⊙ Φ tn , where ⊙ is element-wise multiplication, ϵ t represent independent noise.
Revised:  tm   tn , where  is element-wise multiplication, ✏ t represent independent noise.

Original: Each matrix represents the agent relationships from motion and navigational perspectives.
Revised: Each matrix represents the agent relationships from motion and navigational perspectives.

Original: Formally, Φ tm and Φ tm are learned from motion and navigation functions as: Φ t m = F t m ( X t ) ∈ R Nd i × ( N − 1)
Revised: Formally,  tm and  tm are learned from motion and navigation functions as:  tm = F tm ( X t ) 2 R Nd i ⇥ ( N  1)

Original: d i and Φ t n = F t n ( u t n ) ∈ R Nd i × ( N − 1) d i , where u tn =
Revised: d i and  tn = F tn ( u tn ) 2 R Nd i ⇥ ( N  1) d i , where u tn =

Original: [ X t , r t ] ∈ R d u is the input feature, concatenating the observed location X t of all agents and its relative locations to others r t =
Revised: [ X t , r t ] 2 R d u is the input feature, concatenating the observed location X t of all agents and its relative locations to others r t =

Original: [∆ X i,j ( j ̸ = i ) ] .
Revised: [  X i,j ( j 6 = i ) ] .

Original: F m and F n are neural networks.
Revised: F m and F n are neural networks.

Original: Equation 7 makes our relation modeling to capture agent relations in each observed time step, which is different from
Revised: Equation 7 makes our relation modeling to capture agent relations in each observed time step, which is different from

Original: NRI (Kipf et al., 2018) and dNRI (Graber & Schwing, 2020).
Revised: NRI (Kipf et al., 2018) and dNRI (Graber & Schwing, 2020).

Original: We note that ABM (Fujii et al., 2021)) also shares a similar structure, but with limitations that hinder the ability to extend SENN to work in human interactive scenarios.
Revised: We note that ABM (Fujii et al., 2021)) also shares a similar structure, but with limitations that hinder the ability to extend SENN to work in human interactive scenarios.

Original: The key differences are highlighted below:
Revised: The key differences are highlighted below:

Original: (1) ABM model designs a specific navigation function F n to capture the relationship of few animal species, where there relationship occurs within a close distance (i.e., local scope).
Revised: (1) ABM model designs a speciﬁc navigation function F n to capture the relationship of few animal species, where there relationship occurs within a close distance (i.e., local scope).

Original: Thus, this model does not generalize well to other real-world applications, where the relation could be occurred in further distance.
Revised: Thus, this model does not generalize well to other real-world applications, where the relation could be occurred in further distance.

Original: In this work, we extend the navigation functions to captures relations in both local and global scopes as: F n ( u n ) = ς α F ln ( u ln )+(1 − ς α ) F gn ( u gn ) , where F ln and F gn are local and global navigation functions, which capture the relations within local areas and entire scene, respectively.
Revised: In this work, we extend the navigation functions to captures relations in both local and global scopes as: F n ( u n ) = & ↵ F ln ( u ln )+(1  & ↵ ) F gn ( u gn ) , where F ln and F gn are local and global navigation functions, which capture the relations within local areas and entire scene, respectively.

Original: The contribution of each function is weighted by ς α , a sigmoid function with learnable parameter α ; ς α ∈ [0 , 1] .
Revised: The contribution of each function is weighted by & ↵ , a sigmoid function with learnable parameter ↵ ; & ↵ 2 [0 , 1] .

Original: We present the implementation and impacts of these functions in the supplementary material.
Revised: We present the implementation and impacts of these functions in the supplementary material.

Original: (2) While previous models (Fujii et al., 2021; Marcinkeviˇcs & Vogt, 2021) infer the causal relationships without focusing on long-term trajectory prediction task.
Revised: (2) While previous models (Fujii et al., 2021; Marcinkeviˇcs & Vogt, 2021) infer the causal relationships without focusing on long-term trajectory prediction task.

Original: In contrast, our work focus on learning interpretable relation, which is meaningful for motion prioritization in the later stage and the entire pipeline is trained to improve the multi-step future prediction.
Revised: In contrast, our work focus on learning interpretable relation, which is meaningful for motion prioritization in the later stage and the entire pipeline is trained to improve the multi-step future prediction.

Original: 3. M OTION P RIORITIZATION
Revised: 3. M OTION P RIORITIZATION

Original: The motion prioritization module p m ( z m , r | Φ , ˆ Y ) learns to prioritize the impact of the agents based on the learned relation matrix from Inter-agent Encoder.
Revised: The motion prioritization module p m ( z m , r |  , ˆ Y ) learns to prioritize the impact of the agents based on the learned relation matrix from Inter-agent Encoder.

Original: It also produces the future-trajectories conditioned relational features
Revised: It also produces the future-trajectories conditioned relational features

Original: z m for each agent.
Revised: z m for each agent.

Original: We present the details of each module below:
Revised: We present the details of each module below:

Original: Future-Conditioned Relation Inference Given the learned relational matrix Φ t at each observed time step t ∈ { t 0 − T o + 1 , ..., t 0 } , we quantify the relation matrix among agents S ∈ R N × N as:
Revised: Future-Conditioned Relation Inference Given the learned relational matrix  t at each observed time step t 2 { t 0  T o + 1 , ..., t 0 } , we quantify the relation matrix among agents S 2 R N ⇥ N as:

Original: [Equation]
Revised: [Equation]

Original: where Φ =
Revised: where  =

Original: [Φ t 0 − T o +1 , ..., Φ t 0 ] , consisting the relation matrix at each observed time step, and Φ t =
Revised: [  t 0  T o +1 , ...,  t 0 ] , consisting the relation matrix at each observed time step, and  t =

Original: Φ tn ⊙ Φ tm .
Revised:  tn   tm .

Original: Due to the dynamic movements of agents, the recent relation matrix is more relevant to predict future time steps compared to those in further past.
Revised: Due to the dynamic movements of agents, the recent relation matrix is more relevant to predict future time steps compared to those in further past.

Original: To achieve this, we design a temporal kernel κ ( t, t 0 ) = ς a r (1 / | t 0 − t | − d th ) with ς a r is a sigmoid function with gain a r .
Revised: To achieve this, we design a temporal kernel  ( t, t 0 ) = & a r (1 / | t 0  t |  d th ) with & a r is a sigmoid function with gain a r .

Original: The kernel emphasizes the relation feature as it closer to the current time step, and weaken those that is away.
Revised: The kernel emphasizes the relation feature as it closer to the current time step, and weaken those that is away.

Original: Next, we construct the future-conditioned relation feature z m,i of target agent i as:
Revised: Next, we construct the future-conditioned relation feature z m,i of target agent i as:

Original: [Equation]
Revised: [Equation]

Original: Motion Prioritization
Revised: Motion Prioritization

Original: The intuition is that an agent gains higher priority when its motion is less affected by other’s motion and more impacts on others.
Revised: The intuition is that an agent gains higher priority when its motion is less affected by other’s motion and more impacts on others.

Original: In fact, the impact score can be measured from the relation matrix.
Revised: In fact, the impact score can be measured from the relation matrix.

Original: We associate each agent with a priority score m i 
Revised: We associate each agent with a priority score m i =

Original: = (cid:80) j
Revised: P [Equation]

Original: = N j =0 ,j ̸
Revised: N j [Equation] 0 ,j 6 =

Original: = i S i,j .
Revised: i S i,j .

Original: Based on the priority score, we propose to predict future trajectories in order of their priorities.
Revised: Based on the priority score, we propose to predict future trajectories in order of their priorities.

Original: This is to allow the predicted trajectories of higher prioritized agents have impacts on the ones with lower priorities.
Revised: This is to allow the predicted trajectories of higher prioritized agents have impacts on the ones with lower priorities.

Original: Considering motion prioritization, we can extend the motion decoder as q ( ˆ Y | h, z a , z m ) .
Revised: Considering motion prioritization, we can extend the motion decoder as q ( ˆ Y | h, z a , z m ) .

Original: In fact, the predicted trajectories of each agent can be updated (i.e., refined) multiple times to encourage the self-corrected among agent’s future trajectories and to fully utilize the future-conditioned relation features.
Revised: In fact, the predicted trajectories of each agent can be updated (i.e., reﬁned) multiple times to encourage the self-corrected among agent’s future trajectories and to fully utilize the future-conditioned relation features.

Original: To implement this idea, we iteratively decodes the future trajectories multiple times defined by N s ∈ R .
Revised: To implement this idea, we iteratively decodes the future trajectories multiple times deﬁned by N s 2 R .

Original: Intuitively, if N s = 0 , then there is no updates on predictions.
Revised: Intuitively, if N s = 0 , then there is no updates on predictions.

Original: In other words, the motion prioritization does not have effects.
Revised: In other words, the motion prioritization does not have effects.

Original: Otherwise, there will be N s loop over the prioritized list O = { i, j, ..., N } , where m i ≥ m j .
Revised: Otherwise, there will be N s loop over the prioritized list O = { i, j, ..., N } , where m i  m j .

Original: The details of decoding procedure and the impacts of N s on prediction accuracy are presented in our supplementary material.
Revised: The details of decoding procedure and the impacts of N s on prediction accuracy are presented in our supplementary material.

Original: Loss Function We jointly train the proposed models by using the following losses:
Revised: Loss Function We jointly train the proposed models by using the following losses:

Original: [Equation]
Revised: [Equation]

Original: 14) where L pred is the min-over-k mean-square-error (MSE) prediction loss, which encourages diversities among K predictions sampled from the predicted Gaussian distribution.
Revised: 14) where L pred is the min-over-k mean-square-error (MSE) prediction loss, which encourages diversities among K predictions sampled from the predicted Gaussian distribution.

Original: L sparsity (Φ t ) is sparsity-inducing penalty term used to learn the relation matrix.
Revised: L sparsity (  t ) is sparsity-inducing penalty term used to learn the relation matrix.

Original: In our implementation, we use the elastic-net-style penalty term (Nicholson et al., 2017)
Revised: In our implementation, we use the elastic-net-style penalty term (Nicholson et al., 2017)

Original: (Equation 14) with α = 0 . 5 , and || · || F is the Frobenius norm.
Revised: (Equation 14) with ↵ = 0 . 5 , and || · || F is the Frobenius norm.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We evaluate our model on simulated Charged Particles (Kipf et al., 2018) dataset and two complex interaction datasets, which commonly favors the applications of robot navigation and autonomous system: NBA (Yue et al., 2014) and INTERACTION (Zhan et al., 2019).
Revised: We evaluate our model on simulated Charged Particles (Kipf et al., 2018) dataset and two complex interaction datasets, which commonly favors the applications of robot navigation and autonomous system: NBA (Yue et al., 2014) and INTERACTION (Zhan et al., 2019).

Original: Charged Particles Charged particles (Kipf et al., 2018) is a simulated deterministic system, which is controlled by simple physics rules.
Revised: Charged Particles Charged particles (Kipf et al., 2018) is a simulated deterministic system, which is controlled by simple physics rules.

Original: In each scene, there are 5 charged particles.
Revised: In each scene, there are 5 charged particles.

Original: Each particle has either a positive or negative charge with equal probability.
Revised: Each particle has either a positive or negative charge with equal probability.

Original: Particles with the same charge repel each other, and vise versa.
Revised: Particles with the same charge repel each other, and vise versa.

Original: We set T = 100 and T o = 80 for each scene.
Revised: We set T = 100 and T o = 80 for each scene.

Original: We generate 50K scenes for training, and 10K each for validation and test respectively.
Revised: We generate 50K scenes for training, and 10K each for validation and test respectively.

Original: NBA Dataset
Revised: NBA Dataset

Original: This dataset contains tracking data from the 2012-2013 NBA season, and is provided in (Li et al., 2021).
Revised: This dataset contains tracking data from the 2012-2013 NBA season, and is provided in (Li et al., 2021).

Original: The dataset consist of trajectories of a ball and 10 players from both teams (i.e., 5 players each team).
Revised: The dataset consist of trajectories of a ball and 10 players from both teams (i.e., 5 players each team).

Original: We preprocess the data such that each scene has 50 frames and spans approximately 8 seconds of play, and the first 40 frames are historic, i.e. T = 50 and T o = 40.
Revised: We preprocess the data such that each scene has 50 frames and spans approximately 8 seconds of play, and the ﬁrst 40 frames are historic, i.e. T = 50 and T o = 40.

Original: INTERACTION
Revised: INTERACTION

Original: It consists of different realistic and interactive driving scenarios in roundabout, un-signalized intersection, signalized intersection, merging and lane changing.
Revised: It consists of different realistic and interactive driving scenarios in roundabout, un-signalized intersection, signalized intersection, merging and lane changing.

Original: In total, the dataset is collected from 11 locations using drones or fixed cameras.
Revised: In total, the dataset is collected from 11 locations using drones or ﬁxed cameras.

Original: We follow the same train/validation/test splits as proposed by (Zhan et al., 2019).
Revised: We follow the same train/validation/test splits as proposed by (Zhan et al., 2019).

Original: We set T o = 30 frames (3 seconds), T p = 10 frames ( second).
Revised: We set T o = 30 frames (3 seconds), T p = 10 frames ( second).

Original: The evaluations are made in comparison with state-of-the-art models: Social-GAN (Gupta et al., 2018), Fuzzy Query Attention (FQA) (Kamra et al., 2020), Dynamic Neural Relational Inference (dNRI) (Graber & Schwing, 2020), and GRIN (Li et al., 2021).
Revised: The evaluations are made in comparison with state-of-the-art models: Social-GAN (Gupta et al., 2018), Fuzzy Query Attention (FQA) (Kamra et al., 2020), Dynamic Neural Relational Inference (dNRI) (Graber & Schwing, 2020), and GRIN (Li et al., 2021).

Original: We evaluate our model using the commonly-used metrics: Average Displacement Error (ADE) and Final Displacement Error (FDE), which are the L 2 distance between the ground truth and predicted trajectories, and the L 2 distance between the ground truth final destination and predicted final destination.
Revised: We evaluate our model using the commonly-used metrics: Average Displacement Error (ADE) and Final Displacement Error (FDE), which are the L 2 distance between the ground truth and predicted trajectories, and the L 2 distance between the ground truth ﬁnal destination and predicted ﬁnal destination.

Original: Following GRIN (Li et al., 2021), we report the best-of-100 displacement error of each trajectory/destination.
Revised: Following GRIN (Li et al., 2021), we report the best-of-100 displacement error of each trajectory/destination.

Original: We present the details of comparison models and implementations in the supplementary material.
Revised: We present the details of comparison models and implementations in the supplementary material.

Original: 4. Q UANTITATIVE R ESULTS
Revised: 4. Q UANTITATIVE R ESULTS

Original: We provide quantitative comparisons with related methods on Charged Particles and NBA datasets in Table 1.
Revised: We provide quantitative comparisons with related methods on Charged Particles and NBA datasets in Table 1.

Original: We follow the same experiment setups used on GRIN (Li et al., 2021).
Revised: We follow the same experiment setups used on GRIN (Li et al., 2021).

Original: Our approach outperforms relation modeling methods (NRI (Kipf et al., 2018), dNRI (Graber & Schwing, 2020)) in ADE/FDE, which demonstrates the efficacy of discovering agent relations in each observed time step.
Revised: Our approach outperforms relation modeling methods (NRI (Kipf et al., 2018), dNRI (Graber & Schwing, 2020)) in ADE/FDE, which demonstrates the efﬁcacy of discovering agent relations in each observed time step.

Original: Additionally, our model also outperforms state-of-the-art model (Kamra et al., 2020) and interpretable model (Li et al., 2021) on both datasets by large margins.
Revised: Additionally, our model also outperforms state-of-the-art model (Kamra et al., 2020) and interpretable model (Li et al., 2021) on both datasets by large margins.

Original: It concludes that the proposed prioritization module plays a pivot role in improving prediction accuracy.
Revised: It concludes that the proposed prioritization module plays a pivot role in improving prediction accuracy.

Original: We perform the ablation study (see Table 2) to investigate the impact of each model component.
Revised: We perform the ablation study (see Table 2) to investigate the impact of each model component.

Original: The brief description of each module is as follows:
Revised: The brief description of each module is as follows:

Original: E using only motion encoder/decoder, F m is the motion encoder; F n
Revised: E using only motion encoder/decoder, F m is the motion encoder; F n

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: a) Ground

Original: truth    4     (b) Prediction    4    0.

Original: (c) Relation matrix    4    0.0 0.

Original: [Figure]
Revised: [Figure]

Original: Ground truth is the navigation encoder, Re is future-conditioned relation inference, Pri is motion prioritization module.
Revised: is the navigation encoder, Re is future-conditioned relation inference, Pri is motion prioritization module.

Original: We can see that the relation-based navigation encoder(i.e.
Revised: We can see that the relation-based navigation encoder(i.e.

Original: , E + F n ) and relation-based motion encoder (i.e., E + F m ) show their positive impacts when reducing the prediction errors on Charge Particles compared to the variant only individual motion encoder E .
Revised: , E + F n ) and relation-based motion encoder (i.e., E + F m ) show their positive impacts when reducing the prediction errors on Charge Particles compared to the variant only individual motion encoder E .

Original: However, it is interesting to observe that these variants worsen the results on NBA sport dataset.
Revised: However, it is interesting to observe that these variants worsen the results on NBA sport dataset.

Original: It is reasonable because the NBA consist of mostly realistic and complex interactions among players.
Revised: It is reasonable because the NBA consist of mostly realistic and complex interactions among players.

Original: The players make strategy movements based on the future movements of other higher prioritized players.
Revised: The players make strategy movements based on the future movements of other higher prioritized players.

Original: This explains why adding relational inference (i.e., module Re ) to condition one player’s movements on other’s future trajectories helps reduce the prediction errors.
Revised: This explains why adding relational inference (i.e., module Re ) to condition one player’s movements on other’s future trajectories helps reduce the prediction errors.

Original: Interestingly, our full model with Pri (last row) shows best results overall.
Revised: Interestingly, our full model with Pri (last row) shows best results overall.

Original: This indicates that each player has different motion impact that affects other players’ movements in NBA sport games.
Revised: This indicates that each player has different motion impact that affects other players’ movements in NBA sport games.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: We show the qualitative results consisting of the predicted trajectory and learned relation matrix.
Revised: We show the qualitative results consisting of the predicted trajectory and learned relation matrix.

Original: In Figure 4 the charged particles preserve very interesting swirling interactions, such as among two particles 2 and 4 in the first row, and two groups of swirling particles (1,4), and (0,3) in second row.
Revised: In Figure 4 the charged particles preserve very interesting swirling interactions, such as among two particles 2 and 4 in the ﬁrst row, and two groups of swirling particles (1,4), and (0,3) in second row.

Original: The learned relation matrices show that these interactions can be captured in the corresponding squares (1,3), (3,1), (4, 2), (2,4) for the first scenario, and (3,0), (0,3) (1,4), (4,1) for the second scenario.
Revised: The learned relation matrices show that these interactions can be captured in the corresponding squares (1,3), (3,1), (4, 2), (2,4) for the ﬁrst scenario, and (3,0), (0,3) (1,4), (4,1) for the second scenario.

Original: We note that the stronger relation among agents, the more yellowish (i.e., S ij → 1 ) in the corresponding square in the relation matrix, while blue ( S ij → 0 ) indicates there is no relation.
Revised: We note that the stronger relation among agents, the more yellowish (i.e., S ij ! 1 ) in the corresponding square in the relation matrix, while blue ( S ij ! 0 ) indicates there is no relation.

Original: In NBA dataset, we observe several interesting interaction scenarios, shown in Figure 5.
Revised: In NBA dataset, we observe several interesting interaction scenarios, shown in Figure 5.

Original: In the first row, there are several notable interaction pairs (2,7), (0, 5), (4,9) captured in our relation matrix.
Revised: In the ﬁrst row, there are several notable interaction pairs (2,7), (0, 5), (4,9) captured in our relation matrix.

Original: It can be observed that in this scenario player 7 is following along with player 2, moving from one side to another side of the court.
Revised: It can be observed that in this scenario player 7 is following along with player 2, moving from one side to another side of the court.

Original: At the same time, players 5, 9 play defense against players 0 and 4, respectively.
Revised: At the same time, players 5, 9 play defense against players 0 and 4, respectively.

Original: In second row, our model learns there are group of motion among players (0, 6, 8, 1, 7, 3), where player 0 has dominant motion impact on other players.
Revised: In second row, our model learns there are group of motion among players (0, 6, 8, 1, 7, 3), where player 0 has dominant motion impact on other players.

Original: In the last row, there are two
Revised: In the last row, there are two

Original: ADE
Revised: ADE

Original: FDE
Revised: FDE

Original: Models 1/20/
Revised: Models 1/20/

Original: FDE 1/20/
Revised: FDE 1/20/

Original: GRIN (Li et al., 2021) 1.80/1.27/0.
Revised: GRIN (Li et al., 2021) 1.80/1.27/0.

Original: Ours 1.46 / 1.16 / 0.
Revised: Ours 1.46 / 1.16 / 0.

Original: main groups (1,7) and (2,9, 4 ,5 6,), especially player 9 plays large attention to player 2 as playeris defending against player 2, who is trying to score.
Revised: main groups (1,7) and (2,9, 4 ,5 6,), especially player 9 plays large attention to player 2 as playeris defending against player 2, who is trying to score.

Original: [Figure]
Revised: [Figure]

Original: 4. R ESULTS ON INTERACTION DATASET
Revised: 4. R ESULTS ON INTERACTION DATASET

Original: In this section, we present our evaluation on INTERACTION dataset in comparisons with GRIN (Li et al., 2021) on different number of prediction samples: 1, 20, 100, as reported in Table 3.
Revised: In this section, we present our evaluation on INTERACTION dataset in comparisons with GRIN (Li et al., 2021) on different number of prediction samples: 1, 20, 100, as reported in Table 3.

Original: Both models are trained with 100 epochs.
Revised: Both models are trained with 100 epochs.

Original: Overall, our model outperforms GRIN model in both ADE/FDE in different number of predicted trajectories.
Revised: Overall, our model outperforms GRIN model in both ADE/FDE in different number of predicted trajectories.

Original: Finally, we provide visualization on different driving scenarios such as intersection, roundabout and merging in INTERACTION dataset, shown in Figure 6.
Revised: Finally, we provide visualization on different driving scenarios such as intersection, roundabout and merging in INTERACTION dataset, shown in Figure 6.

Original: Generally, interaction types such as following, or yielding are very common in driving scenarios.
Revised: Generally, interaction types such as following, or yielding are very common in driving scenarios.

Original: The results show that our model is able to capture these types of interactions in several scenarios.
Revised: The results show that our model is able to capture these types of interactions in several scenarios.

Original: As we can see that in first row (intersection scenarios), agent 2 is waiting for agent 0 and 1 to pass before it can turn left.
Revised: As we can see that in ﬁrst row (intersection scenarios), agent 2 is waiting for agent 0 and 1 to pass before it can turn left.

Original: Meanwhile, agent 1 makes a slight interaction (i.e., yield) to agent 3.
Revised: Meanwhile, agent 1 makes a slight interaction (i.e., yield) to agent 3.

Original: In the roundabout scenarios (second row), the most notable interaction is between 2 and 4.
Revised: In the roundabout scenarios (second row), the most notable interaction is between 2 and 4.

Original: In this scenario, agent 2 is waiting for agent 4 to cross, before it can go into the roundabout section.
Revised: In this scenario, agent 2 is waiting for agent 4 to cross, before it can go into the roundabout section.

Original: In the last row, there is interesting merging scenarios from 4 to 3.
Revised: In the last row, there is interesting merging scenarios from 4 to 3.

Original: and agent 3 following 2 when there merge from
Revised: and agent 3 following 2 when there merge from

Original: [Figure]
Revised: [Figure]

Original: 3 to 2 lanes.
Revised: 3 to 2 lanes.

Original: In addition, the priority map for each scenario indicates the agent’s importance, which provides meaning cues for improving trajectory prediction.
Revised: In addition, the priority map for each scenario indicates the agent’s importance, which provides meaning cues for improving trajectory prediction.

Original:  L IMITATIONS
Revised:  L IMITATIONS

Original: Although we have demonstrated that the proposed model has successfully captured several interaction behavior in the given datasets, discovering causalities in realistic scenarios such as in autonomous driving is still challenging.
Revised: Although we have demonstrated that the proposed model has successfully captured several interaction behavior in the given datasets, discovering causalities in realistic scenarios such as in autonomous driving is still challenging.

Original: As a possible solution, exploring scientific knowledge about traffics and the priorities of different agent types could be incorporated to improve the model accuracy.
Revised: As a possible solution, exploring scientiﬁc knowledge about trafﬁcs and the priorities of different agent types could be incorporated to improve the model accuracy.

Original: Secondly, our model discovers relations and prioritization from bird-eye views, which provides a global-view relation agents.
Revised: Secondly, our model discovers relations and prioritization from bird-eye views, which provides a global-view relation agents.

Original: However, robots or ego-vehicles also observe the relations with surrounding agent from the front camera views.
Revised: However, robots or ego-vehicles also observe the relations with surrounding agent from the front camera views.

Original: Thus, the future work of considering fusing relations from different views can be explored to strengthen the relation features.
Revised: Thus, the future work of considering fusing relations from different views can be explored to strengthen the relation features.

Original: Lastly, discovering relation and trajectory prediction (if the identity of subject is known, which is not the case in our work) can raise issues regarding privacy sensitivity.
Revised: Lastly, discovering relation and trajectory prediction (if the identity of subject is known, which is not the case in our work) can raise issues regarding privacy sensitivity.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce a novel multi-agent trajectory prediction.
Revised: In this paper, we introduce a novel multi-agent trajectory prediction.

Original: Our model discovers interpretable interaction feature from historical trajectories of all agents.
Revised: Our model discovers interpretable interaction feature from historical trajectories of all agents.

Original: We also developed motion prioritization module, which prioritize those agents which have higher impacts on other future motions.
Revised: We also developed motion prioritization module, which prioritize those agents which have higher impacts on other future motions.

Original: Based on the prioritized scores, the decoder makes sequential prediction of each agents with iterative predictions.
Revised: Based on the prioritized scores, the decoder makes sequential prediction of each agents with iterative predictions.

Original: In broader impacts, our prediction could be used to infer relations in other scenarios such as human motions, or inferring robot-human interactions and predictions.
Revised: In broader impacts, our prediction could be used to infer relations in other scenarios such as human motions, or inferring robot-human interactions and predictions.

Original:  D ETAILS OF I NTER -A GENT E NCODER
Revised:  D ETAILS OF I NTER -A GENT E NCODER

Original: We introduced in our main paper that the Inter-Agent encoder consists of movement function F m and navigation function F n , which capture agent relations from motion and navigation perspectives, inspired by animal movements Nathan et al.
Revised: We introduced in our main paper that the Inter-Agent encoder consists of movement function F m and navigation function F n , which capture agent relations from motion and navigation perspectives, inspired by animal movements Nathan et al.

Original: ; Fujii et al.
Revised: ; Fujii et al.

Original: Specifically, each function produces a relation matrix ( Φ m and Φ n ), where each element of Φ m (produced by F m ) represents the amplitude of interaction, and each element of Φ n (produced by Φ n ) represents the sign of interactions (e.g., positive : two agents approach, negative: two agents repulse).
Revised: Speciﬁcally, each function produces a relation matrix (  m and  n ), where each element of  m (produced by F m ) represents the amplitude of interaction, and each element of  n (produced by  n ) represents the sign of interactions (e.g., positive : two agents approach, negative: two agents repulse).

Original: To further extend the model to generalize well to real-world robot applications, where the relation could be occurred in further distance.
Revised: To further extend the model to generalize well to real-world robot applications, where the relation could be occurred in further distance.

Original: We implement our navigation function to capture interaction from local and global scopes, which capture the relations within local areas and entire scene, respective.
Revised: We implement our navigation function to capture interaction from local and global scopes, which capture the relations within local areas and entire scene, respective.

Original: Thus, F n can be decomposed into local function F ln and global function F gn .
Revised: Thus, F n can be decomposed into local function F ln and global function F gn .

Original: The local function F ln capture interaction with emphasies given a pre-defined thershold d ignore as follows:
Revised: The local function F ln capture interaction with emphasies given a pre-deﬁned thershold d ignore as follows:

Original: [Equation]
Revised: [Equation]

Original: (1) while the global function capture the interaction in entire scene (without threshold).
Revised: while the global function capture the interaction in entire scene (without threshold).

Original: [Equation]
Revised: [Equation]

Original: where ς a d and ς a v are sigmoid functions with gain a d and a v , respectively; d ignore is a threshold for defining the local area.
Revised: where & a d and & a v are sigmoid functions with gain a d and a v , respectively; d ignore is a threshold for deﬁning the local area.

Original: Thus, if the distance of two agents are greater than d ignore , then there is no interaction.
Revised: Thus, if the distance of two agents are greater than d ignore , then there is no interaction.

Original: r i,j = r j − r i is the relative distance of agent j to agent i with r i
Revised: r i,j = r j  r i is the relative distance of agent j to agent i with r i

Original: =
Revised: =

Original: ( x i , y i ) T .
Revised: ( x i , y i ) T .

Original: v i,j is the velocity of agent
Revised: v i,j is the velocity of agent

Original: i (i.e., v i ) in the direction of r ij , thus ς a v ( v i,j − 12 ) × 2 sign represents of effects of j on i .
Revised: i (i.e., v i ) in the direction of r ij , thus & a v ( v i,j  12 ) ⇥ 2 sign represents of effects of j on i .

Original: Then, the navigation function is formulated as:
Revised: Then, the navigation function is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where ς α is a sigmoid function with learnable parameter ς α ∈ [0 , 1] .
Revised: where & ↵ is a sigmoid function with learnable parameter & ↵ 2 [0 , 1] .

Original: In addition to our ablation studies presented in (Section 4, Table 2 in main paper), we validate the impact of the proposed navigation functions on trajectory prediction.
Revised: In addition to our ablation studies presented in (Section 4, Table 2 in main paper), we validate the impact of the proposed navigation functions on trajectory prediction.

Original: In this experiment, we train/test the different variants of Inter-Agent encoder on Charged Particles dataset.
Revised: In this experiment, we train/test the different variants of Inter-Agent encoder on Charged Particles dataset.

Original: As shown in
Revised: As shown in

Original: Table 1, we can observe that local navigation function F ln and F gn both improves the prediction accuracy.
Revised: Table 1, we can observe that local navigation function F ln and F gn both improves the prediction accuracy.

Original: This indicates the important roles of these functions in our model.
Revised: This indicates the important roles of these functions in our model.

Original:  D ETAILS OF D ECODING A LGORITHM
Revised:  D ETAILS OF D ECODING A LGORITHM

Original: We present the details of trajectory decoding process with motion prioritization in Algorithm 1.
Revised: We present the details of trajectory decoding process with motion prioritization in Algorithm 1.

Original: We note that the predicted trajectories of each agent can be updated (i.e., refined) multiple times to
Revised: We note that the predicted trajectories of each agent can be updated (i.e., reﬁned) multiple times to

Original: [Table]
Revised: [Table]

Original: encourage the correction among agent’s future trajectories and to fully utilize the future-conditioned relation features.
Revised: encourage the correction among agent’s future trajectories and to fully utilize the future-conditioned relation features.

Original: To implement this idea, we iteratively decodes the future trajectories multiple times defined by N s .
Revised: To implement this idea, we iteratively decodes the future trajectories multiple times deﬁned by N s .

Original: For example, if the prioritized list of agent is O = { 3 , 2 , 1 , 4 , 5 } , with 3 is the agent with highest priority, 5 is the lowest one.
Revised: For example, if the prioritized list of agent is O = { 3 , 2 , 1 , 4 , 5 } , with 3 is the agent with highest priority, 5 is the lowest one.

Original: One iteration (i.e., N s = 1 ) corresponds to a sequential prediction in order of 3 → 2 → 1 → 4 → 5 .
Revised: One iteration (i.e., N s = 1 ) corresponds to a sequential prediction in order of 3 ! 2 ! 1 ! 4 ! 5 .

Original: This allows the predicted trajectories of agent with higher priority influences on trajectories of agent with lower priority.
Revised: This allows the predicted trajectories of agent with higher priority inﬂuences on trajectories of agent with lower priority.

Original: We study the impact of different number of iterations to update the future trajectories, shown in Figure 1.
Revised: We study the impact of different number of iterations to update the future trajectories, shown in Figure 1.

Original: It is interesting to observe that only one step of refinement is enough to achieve the best prediction on Charged
Revised: It is interesting to observe that only one step of reﬁnement is enough to achieve the best prediction on Charged

Original: Particles.
Revised: Particles.

Original: On NBA dataset, although we achieve best results at 5 iterations, the results also seem to be saturated in the first iteration.
Revised: On NBA dataset, although we achieve best results at 5 iterations, the results also seem to be saturated in the ﬁrst iteration.

Original: This concludes that relational features has been fully shared from the higher-prioritized agent to the lower one in list of prioritized agent in the first iteration
Revised: This concludes that relational features has been fully shared from the higher-prioritized agent to the lower one in list of prioritized agent in the iteration.

Original: Input: List of relational matrices Φ =
Revised: Input: List of relational matrices  =

Original: [Φ t 0 − T o +1 , . . 
Revised: [  t 0  T o +1 , . . .

Original: . , Φ t 0 ] in T o historical time-steps 
Revised: ,  t 0 ] in T o historical time-steps Output

Original: Output: Predicted trajectories Y =
Revised: : Predicted trajectories Y =

Original: [ Y 1 , Y 2 , ..., Y N ∈ R T p × d ] of all N agents in the next T p future time steps.
Revised: [ Y 1 , Y 2 , ..., Y N 2 R T p ⇥ d ] of all N agents in the next T p future time steps.

Original: [Table]
Revised: 1 Calculate relational weights using Equation 9.

Original: [Table]
Revised: Estimate the priority score (i.e., impactness) for each agent.

Original: [Table] [Table]
Revised: m i = [Equation] j m i  m j do

Original: [Equation]
Revised: [Equation]

Original:  I MPLEMENTATION D ETAILS
Revised:  I MPLEMENTATION D ETAILS

Original: We implement our model using PyTorch Pytorch (2018), which has BSD-style license.
Revised: We implement our model using PyTorch Pytorch (2018), which has BSD-style license.

Original: We train the model with the Adam Zhang (2018) optimizer with the learning rate of 0.0001 and 100 epochs.
Revised: We train the model with the Adam Zhang (2018) optimizer with the learning rate of 0.0001 and 100 epochs.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: batch size is 128.
Revised: batch size is 128.

Original: For all datasets, we implement the movement encoders with 2 MLP layers, each has hidden size of 50.
Revised: For all datasets, we implement the movement encoders with 2 MLP layers, each has hidden size of 50.

Original: The hidden size of GRU in both encoders and decoders is 48.
Revised: The hidden size of GRU in both encoders and decoders is 48.

Original: In the training loss, we set α = 0 .
Revised: In the training loss, we set ↵ = 0 .

Original: 5 , and λ = 1 .
Revised: 5 , and  = 1 .

Original:  A DDITIONAL Q UALITATIVE R ESULTS
Revised:  A DDITIONAL Q UALITATIVE R ESULTS

Original: In this section, we present additional qualitative results on three datasets: Charged Particles, NBA,
Revised: In this section, we present additional qualitative results on three datasets: Charged Particles, NBA,

Original: INTERACTION.
Revised: INTERACTION.

Original: Results on Charged Particles Figure 2 shows the trajectory prediction and the learned relation matrix at different time steps.
Revised: Results on Charged Particles Figure 2 shows the trajectory prediction and the learned relation matrix at different time steps.

Original: We observe that at different time steps, particles involves in different interactions.
Revised: We observe that at different time steps, particles involves in different interactions.

Original: For example, at t=10 and t=60, there are mainly swirling interactions between particles (2, 4) and (1, 3).
Revised: For example, at t=10 and t=60, there are mainly swirling interactions between particles (2, 4) and (1, 3).

Original: At t=120, particle 3 reaches toward particle 0, and then goes further away at t=240.
Revised: At t=120, particle 3 reaches toward particle 0, and then goes further away at t=240.

Original: The learned relation matrix show that these subtle movements and interactions can be captured.
Revised: The learned relation matrix show that these subtle movements and interactions can be captured.

Original: Results on Charged Particles Next, we show predictions and priority scores of players in basketball games (Figure 3).
Revised: Results on NBA Next, we show predictions and priority scores of players in basketball games (Figure 3).

Original: NBA dataset consists of highly complex interactions among players.
Revised: NBA dataset consists of highly complex interactions among players.

Original: In the first scenario, player 0 and 4 in offensive team (red) have higher priority score (i.e., higher impact) as the player 0 is passing the ball to player 4; thus, their movements highly influence others’ motions.
Revised: In the ﬁrst scenario, player 0 and 4 in offensive team (red) have higher priority score (i.e., higher impact) as the playeris passing the ball to player 4; thus, their movements highly inﬂuence others’ motions.

Original: In the second scenario, player 0 holding the ball but does not have large impact; this is because the motions of the ball and player 0 is not changing much.
Revised: In the second scenario, player 0 holding the ball but does not have large impact; this is because the motions of the ball and player 0 is not changing much.

Original: Other players may suspect that the player 0 could pass the ball to other offensive players.
Revised: Other players may suspect that the player 0 could pass the ball to other offensive players.

Original: Thus, the movements of other players in the red team are more important to observe.
Revised: Thus, the movements of other players in the red team are more important to observe.

Original: Interestingly, player 2 has the most critical move and thus it generates the largest impact on other players.
Revised: Interestingly, player 2 has the most critical move and thus it generates the largest impact on other players.

Original: Results on INTERACTION
Revised: Results on INTERACTION

Original: We further visualize the trajectory prediction, priority map, and relation matrix for two interactive scenarios: merging and roundabout in Figure 4.
Revised: We further visualize the trajectory prediction, priority map, and relation matrix for two interactive scenarios: merging and roundabout in Figure 4.

Original: We can see that our model can capture intriguing relation matrix and learns the motion importance of agent effectively.
Revised: We can see that our model can capture intriguing relation matrix and learns the motion importance of agent effectively.

Original: In the merging scenarios (top row), agent 2 gains highest importance as it is going to merge and have high relations with others agents in front (agent 1), and behind (agent 3).
Revised: In the merging scenarios (top row), agent 2 gains highest importance as it is going to merge and have high relations with others agents in front (agent 1), and behind (agent 3).

Original: In the roundabout scenario (bottom row), we observe two groups: (1,3) and (0,2,4) on different sides of the roundabout.
Revised: In the roundabout scenario (bottom row), we observe two groups: (1,3) and (0,2,4) on different sides of the roundabout.

Original: Interestingly, we can see that the relation between agent 0 (pedestrian) and agent 2 (vehicle) can be
Revised: Interestingly, we can see that the relation between agent 0 (pedestrian) and agent 2 (vehicle) can be

Original: [-Ground truth 0 10 20 30 40 50        
-]
Revised: [-Ground truth 0 10 20 30 40 50        
-]

Original: [-Prediction 0 10 20 30 40 50       
 Priority map 0 10 20 30 40 50      1   6 
 Ground truth 0 10 20 30 40 50    501   6 
 Prediction 0 10 20 30 40 50      1   6 
-]
Revised: [-Prediction 0 10 20 30 40 50       
 Priority map 0 10 20 30 40 50      1   6 
 Ground truth 0 10 20 30 40 50    501   6 
 Prediction 0 10 20 30 40 50      1   6 
-][Figure]

Original: Priority Map

Original: [Figure]
Revised: [Figure]

Original: captured even though they still far-way.
Revised: captured even though they still far-way.

Original: The agent 2 gains most motion priority within the group (0,2,4) because it has relationship with both agent 4 and 0.
Revised: The agent 2 gains most motion priority within the group (0,2,4) because it has relationship with both agent 4 and 0.