Original: Forecasting Human Trajectory from Scene History
Revised: Forecasting Human Trajectory from Scene History

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Predicting the future trajectory of a person remains a challenging problem, dueto randomness and subjectivity.
Revised: Predicting the future trajectory of a person remains a challenging problem, dueto randomness and subjectivity.

Original: However, the moving patterns of human in aconstrained scenario typically conform to a limited number of regularities to acertain extent, because of the scenario restrictions ( e .
Revised: However, the moving patterns of human in aconstrained scenario typically conform to a limited number of regularities to acertain extent, because of the scenario restrictions ( e .

Original: g ., floor plan, roads andobstacles) and person-person or person-object interactivity.
Revised: g ., floor plan, roads andobstacles) and person-person or person-object interactivity.

Original: Thus, an individualperson in this scenario should follow one of the regularities as well.
Revised: Thus, an individualperson in this scenario should follow one of the regularities as well.

Original: In otherwords, a person’s subsequent trajectory has likely been traveled by others.
Revised: In otherwords, a person’s subsequent trajectory has likely been traveled by others.

Original: Basedon this hypothesis, we propose to forecast a person’s future trajectory by learningfrom the implicit scene regularities.
Revised: Basedon this hypothesis, we propose to forecast a person’s future trajectory by learningfrom the implicit scene regularities.

Original: We call the regularities, inherently derivedfrom the past dynamics of the people and the environment in the scene, scenehistory .
Revised: We call the regularities, inherently derivedfrom the past dynamics of the people and the environment in the scene, scenehistory .

Original: We categorize scene history information into two types: historical grouptrajectories and individual-surroundings interaction.
Revised: We categorize scene history information into two types: historical grouptrajectories and individual-surroundings interaction.

Original: To exploit these informationfor trajectory prediction, we propose a novel framework Scene History Excavating
Revised: To exploit these informationfor trajectory prediction, we propose a novel framework Scene History Excavating

Original: Network (SHENet), where the scene history is leveraged in a simple yet effectiveapproach.
Revised: Network (SHENet), where the scene history is leveraged in a simple yet effectiveapproach.

Original: In particular, we design two components, the group trajectory bankmodule to extract representative group trajectories as the candidate for future path,and the cross-modal interaction module to model the interaction between individualpast trajectory and its surroundings for trajectory refinement, respectively.
Revised: In particular, we design two components, the group trajectory bankmodule to extract representative group trajectories as the candidate for future path,and the cross-modal interaction module to model the interaction between individualpast trajectory and its surroundings for trajectory refinement, respectively.

Original: Inaddition, to mitigate the uncertainty in the evaluation, caused by the aforementionedrandomness and subjectivity, we propose to include smoothness into evaluationmetrics.
Revised: Inaddition, to mitigate the uncertainty in the evaluation, caused by the aforementionedrandomness and subjectivity, we propose to include smoothness into evaluationmetrics.

Original: We conduct extensive evaluations to validate the efficacy of proposedframework on ETH, UCY, as well as a new, challenging benchmark dataset PAV,demonstrating superior performance compared to state-of-the-art methods.
Revised: We conduct extensive evaluations to validate the efficacy of proposedframework on ETH, UCY, as well as a new, challenging benchmark dataset PAV,demonstrating superior performance compared to state-of-the-art methods.

Original:  Introduction
Revised:  Introduction

Original: Human trajectory prediction (HTP) aims to predict a target person’s future path from a video clip[5, 9, 19, 20].
Revised: Human trajectory prediction (HTP) aims to predict a target person’s future path from a video clip[5, 9, 19, 20].

Original: This is critical for intelligent transportation, as it enables vehicle to perceive the statusof pedestrian in advance so it can avoid potential collision.
Revised: This is critical for intelligent transportation, as it enables vehicle to perceive the statusof pedestrian in advance so it can avoid potential collision.

Original: Surveillance systems with HTP capabilitycan assist security officers to predict the possible escape path of suspects.
Revised: Surveillance systems with HTP capabilitycan assist security officers to predict the possible escape path of suspects.

Original: Although much work hasbeen done in recent years [1, 9, 12, 14, 18, 33], very few of them are reliable and generalizable enoughto be applied in real-world scenarios, primarily due to two challenges of the task: randomness andsubjectivity of human movement.
Revised: Although much work hasbeen done in recent years [1, 9, 12, 14, 18, 33], very few of them are reliable and generalizable enoughto be applied in real-world scenarios, primarily due to two challenges of the task: randomness andsubjectivity of human movement.

Original: However, most real world scenarios are constrained.
Revised: However, most real world scenarios are constrained.

Original: As shown in
Revised: As shown in

Original: Figure 1, given previously captured video in this scene, the target person’s (red box) future trajectorybecomes more predictable as the moving pattern of human typically complies with several underlyingregularities in this scenario, which the target person would follow.
Revised: Figure 1, given previously captured video in this scene, the target person’s (red box) future trajectorybecomes more predictable as the moving pattern of human typically complies with several underlyingregularities in this scenario, which the target person would follow.

Original: Therefore, to predict the trajectory,first we need to understand these regularities.
Revised: Therefore, to predict the trajectory,first we need to understand these regularities.

Original: We argue that these regularities are implicitly encodedin historical human trajectories (Figure 1 left), individual past trajectory, the surroundings, and theinteraction between them (Figure 1 right), which we refer to as scene history .
Revised: We argue that these regularities are implicitly encodedin historical human trajectories (Figure 1 left), individual past trajectory, the surroundings, and theinteraction between them (Figure 1 right), which we refer to as scene history .

Original: We separate the historical information into two categories: historical group trajectories (HGT) and individual-surroundings interaction (ISI).
Revised: We separate the historical information into two categories: historical group trajectories (HGT) and individual-surroundings interaction (ISI).

Original: HGT refers to the group representatives of all historicaltrajectories in a scene.
Revised: HGT refers to the group representatives of all historicaltrajectories in a scene.

Original: The reason of resorting to HGT is that given a new target person in the scenario,his/her path is more likely to share more similarities with one of the group trajectories than anyindividual instance of the historical trajectories, due to the aforementioned randomness, subjectivity,and regularities.
Revised: The reason of resorting to HGT is that given a new target person in the scenario,his/her path is more likely to share more similarities with one of the group trajectories than anyindividual instance of the historical trajectories, due to the aforementioned randomness, subjectivity,and regularities.

Original: Nevertheless, the group trajectories are less correlated to the individual’s past statusand the corresponding surroundings, influencing the person’s future trajectory as well.
Revised: Nevertheless, the group trajectories are less correlated to the individual’s past statusand the corresponding surroundings, influencing the person’s future trajectory as well.

Original: ISI is requiredto more comprehensively leverage the history information by extracting contextual information.
Revised: ISI is requiredto more comprehensively leverage the history information by extracting contextual information.

Original: Afew existing methods take into account similarity between individual past trajectory and historicaltrajectories.
Revised: Afew existing methods take into account similarity between individual past trajectory and historicaltrajectories.

Original: Most attempts [16, 19, 26, 33] explore only individual-surroundings interaction, wheretremendous efforts are spent on modeling the individual trajectory, the semantic information ofenvironment information, and relationship between them.
Revised: Most attempts [16, 19, 26, 33] explore only individual-surroundings interaction, wheretremendous efforts are spent on modeling the individual trajectory, the semantic information ofenvironment information, and relationship between them.

Original: Although MANTRA [20] models thesimilarity using encoders trained in a reconstruction manner, it performs the similarity computationon instance level instead of group level, making it sensitive to the capability of the trained encoder.
Revised: Although MANTRA [20] models thesimilarity using encoders trained in a reconstruction manner, it performs the similarity computationon instance level instead of group level, making it sensitive to the capability of the trained encoder.

Original: Based on the above analysis, we propose a simple yet effective framework, Scene History Excavating
Revised: Based on the above analysis, we propose a simple yet effective framework, Scene History Excavating

Original: Network (SHENet), to leverage HGT and ISI jointly for HTP.
Revised: Network (SHENet), to leverage HGT and ISI jointly for HTP.

Original: In particular, the framework consists oftwo major components: 1) a group trajectory bank (GTB) module, and 2) a cross-modal interaction(CMI) module.
Revised: In particular, the framework consists oftwo major components: 1) a group trajectory bank (GTB) module, and 2) a cross-modal interaction(CMI) module.

Original: GTB constructs the representative group trajectories from all historical individualtrajectories and provides a candidate path for future trajectory prediction.
Revised: GTB constructs the representative group trajectories from all historical individualtrajectories and provides a candidate path for future trajectory prediction.

Original: CMI encodes the observedindividual trajectory and the surroundings separately, and models their interaction with a cross-modaltransformer for refinement of the searched candidate trajectory.
Revised: CMI encodes the observedindividual trajectory and the surroundings separately, and models their interaction with a cross-modaltransformer for refinement of the searched candidate trajectory.

Original: In addition, in order to alleviate the evaluation uncertainty from two mentioned characteristics ( i .
Revised: In addition, in order to alleviate the evaluation uncertainty from two mentioned characteristics ( i .

Original: e .,randomness and subjectivity), we introduce curve smoothing (CS) into current metrics average andfinal displacement error ( i .
Revised: e .,randomness and subjectivity), we introduce curve smoothing (CS) into current metrics average andfinal displacement error ( i .

Original: e ., ADE and FDE), resulting in two new metrics CS-ADE and CS-FDE.
Revised: e ., ADE and FDE), resulting in two new metrics CS-ADE and CS-FDE.

Original: Moreover, to facilitate the development of research in HTP, we collect a new challenging dataset withdiverse movement patterns, named PAV.
Revised: Moreover, to facilitate the development of research in HTP, we collect a new challenging dataset withdiverse movement patterns, named PAV.

Original: The dataset is obtained by choosing the videos with fixedcamera view and complex human movement from MOT15 dataset [11].
Revised: The dataset is obtained by choosing the videos with fixedcamera view and complex human movement from MOT15 dataset [11].

Original: The contributions of the work can be summarized as follows: 1) We introduce group history tosearch individual trajectory for HTP.
Revised: The contributions of the work can be summarized as follows: 1) We introduce group history tosearch individual trajectory for HTP.

Original: We propose a simple and effective framework, SHENet, tojointly utilize two types of scene history ( i .
Revised: We propose a simple and effective framework, SHENet, tojointly utilize two types of scene history ( i .

Original: e ., historical group trajectories and individual-surroundingsinteraction) for HTP.
Revised: e ., historical group trajectories and individual-surroundingsinteraction) for HTP.

Original: We construct a new challenging dataset, PAV, and two new metrics consideringrandomness and subjectivity in human moving patterns are proposed to better benchmark HTPperformance.
Revised: We construct a new challenging dataset, PAV, and two new metrics consideringrandomness and subjectivity in human moving patterns are proposed to better benchmark HTPperformance.

Original: We conduct comprehensive experiments on ETH, UCY, and PAV to demonstrate thesuperior performance of SHENet and the efficacy of each component.
Revised: We conduct comprehensive experiments on ETH, UCY, and PAV to demonstrate thesuperior performance of SHENet and the efficacy of each component.

Original:  Related Work
Revised:  Related Work

Original: Unimodal Methods.
Revised: Unimodal Methods.

Original: Unimodal methods rely on learning the regularity of individual motion frompast trajectories in order to predict future trajectories.
Revised: Unimodal methods rely on learning the regularity of individual motion frompast trajectories in order to predict future trajectories.

Original: For example, Social LSTM [1] focuses onmodeling the interaction between individual trajectories through a social pooling module.
Revised: For example, Social LSTM [1] focuses onmodeling the interaction between individual trajectories through a social pooling module.

Original: STGAT [8]uses an attention module to learn spatial interactions and assign reasonable importance to neighbors.
Revised: STGAT [8]uses an attention module to learn spatial interactions and assign reasonable importance to neighbors.

Original: PIE [25] uses a temporal attention module to calculate the importance of observed trajectories ateach time step.
Revised: PIE [25] uses a temporal attention module to calculate the importance of observed trajectories ateach time step.

Original: Giuliari et al.
Revised: Giuliari et al.

Original: [6] use the Transformer networks [3, 29] to predict the trajectoriesof individual people, which weight all observed points in the trajectories according to an attentionmechanism.
Revised: [6] use the Transformer networks [3, 29] to predict the trajectoriesof individual people, which weight all observed points in the trajectories according to an attentionmechanism.

Original: These methods consider the scene history in two aspects: the individual’s past trajectoryand the interaction between the individual’s past trajectory and its neighboring trajectories.
Revised: These methods consider the scene history in two aspects: the individual’s past trajectoryand the interaction between the individual’s past trajectory and its neighboring trajectories.

Original: Multimodal Methods.
Revised: Multimodal Methods.

Original: Additionally, multimodal methods examine the influence of environmentinformation on HTP.
Revised: Additionally, multimodal methods examine the influence of environmentinformation on HTP.

Original: SS-LSTM [33] proposes a scene interaction module to capture global informationof the scene.
Revised: SS-LSTM [33] proposes a scene interaction module to capture global informationof the scene.

Original: Trajectron++ [26] uses a graph structure to model the trajectories and interacts withenvironmental information and other individuals.
Revised: Trajectron++ [26] uses a graph structure to model the trajectories and interacts withenvironmental information and other individuals.

Original: Liang et al.
Revised: Liang et al.

Original: [16] integrate the interactive modulesinto a person interaction module (person-person, person-object, and person-scene), and add a personbehavior module (person appearance) to encode rich visual information.
Revised: [16] integrate the interactive modulesinto a person interaction module (person-person, person-object, and person-scene), and add a personbehavior module (person appearance) to encode rich visual information.

Original: Different from the RNNs[1, 16, 32, 33] to encode the past trajectories into a hidden state vector, which is prone to loss ofinformation, MANTRA [20] utilizes an external memory to model long-term dependencies.
Revised: Different from the RNNs[1, 16, 32, 33] to encode the past trajectories into a hidden state vector, which is prone to loss ofinformation, MANTRA [20] utilizes an external memory to model long-term dependencies.

Original: It storesthe historical single-agent trajectories in a memory, and encodes the environment information to searched trajectory
Revised: It storesthe historical single-agent trajectories in a memory, and encodes the environment information to searched trajectory

Original: Difference from Previous Works.
Revised: Difference from Previous Works.

Original: Both unimodal and multimodal methods use single or partialaspects of the scene history, disregarding the historical group trajectories.
Revised: Both unimodal and multimodal methods use single or partialaspects of the scene history, disregarding the historical group trajectories.

Original: In our work, we integratethe information from the scene history in a more comprehensive way and propose dedicated modulesto handle different types of information, respectively.
Revised: In our work, we integratethe information from the scene history in a more comprehensive way and propose dedicated modulesto handle different types of information, respectively.

Original: The main differences between our methodand the previous works, especially memory-based methods [15, 20, 21, 31] and clustering-basedmethods [27, 28, 34], are as follows:i) MANTRA [20] and MemoNet [31] consider historical individual trajectories
Revised: The main differences between our methodand the previous works, especially memory-based methods [15, 20, 21, 31] and clustering-basedmethods [27, 28, 34], are as follows:

Original: while our proposed

Original: SHENet

Original: focuses on

Original: historical group trajectories

Original: , which is

Original: more generalizable in different scenarios; In [15], the historical trajectories are not stored for search.ii) [27, 34] group person-neighbors for trajectory prediction; [28] cluster the trajectories into fixed number of categories for trajectory classification; Our SHENet generates representative trajectoriesas reference for individual person trajectory prediction.
Revised: i) MANTRA [20] and MemoNet [31] consider historicalindividual trajectories, while our proposed SHENet focuses on historical group trajectories, which ismore generalizable in different scenarios; In [15], the historical trajectories are not stored for search.ii) [27, 34] group person-neighbors for trajectory prediction; [28] clusters the trajectories into fixednumber of categories for trajectory classification; Our SHENet generates representative trajectoriesas reference for individual person trajectory prediction.

Original:  Method
Revised:  Method

Original: 3.1 Overview
Revised: 3.1 Overview

Original: The architecture of the proposed Scene History Excavating Network (SHENet) is depicted in Figure 2,which consists of two major components: group trajectory bank module (GTB) and cross-modalinteraction module (CMI).
Revised: The architecture of the proposed Scene History Excavating Network (SHENet) is depicted in Figure 2,which consists of two major components: group trajectory bank module (GTB) and cross-modalinteraction module (CMI).

Original: Formally, given all the trajectories Z in observed videos of this scene,a scene image S p and a past trajectory of a target person p in the last T pas time steps, X p =
Revised: Formally, given all the trajectories Z in observed videos of this scene,a scene image S p and a past trajectory of a target person p in the last T pas time steps, X p =

Original: 16: Calculate p
Revised: 16: Calculate p

Original: Y p 17: if d θ then 18: n ←
Revised: Y p 17: if d θ then 18: n ←

Original: Update adding p 20: end
Revised: Update adding p 20: end

Original: 21: if β then 22:
Revised: 21: if β then 22:

Original: 24: end
Revised: 24: end

Original: 25: return
Revised: 25: return

Original: { x tp } 0 t = − T pas +1 ∈ R T pas × 2 , where x tp ∈ R 2 represents the position of p th person at time step t ,
Revised: { x tp } 0 t = − T pas +1 ∈ R T pas × 2 , where x tp ∈ R 2 represents the position of p th person at time step t ,

Original: SHENet requires to predict the future positions of the person ˆ Y p
Revised: SHENet requires to predict the future positions of the person ˆ Y p

Original: = { ˆ y tp } T fut t =1 ∈ R T fut × 2 in the next
Revised: = { ˆ y tp } T fut t =1 ∈ R T fut × 2 in the next

Original: T fut frames, such that ˆ Y p is as close to the ground truth trajectory Y p
Revised: T fut frames, such that ˆ Y p is as close to the ground truth trajectory Y p

Original: = { y tp } T fut t =1 ∈ R T fut × 2 aspossible.
Revised: = { y tp } T fut t =1 ∈ R T fut × 2 aspossible.

Original: The proposed GTB first condenses
Revised: The proposed GTB first condenses

Original: Z as representative group trajectories Z bank .
Revised: Z as representative group trajectories Z bank .

Original: Then, theobserved trajectory X p is used as a key to search a closet representative group trajectory in Z bankas a candidate future trajectory, Y p ′ .
Revised: Then, theobserved trajectory X p is used as a key to search a closet representative group trajectory in Z bankas a candidate future trajectory, Y p ′ .

Original: Meanwhile, the past trajectory X p and a scene image S p areseparately forwarded to the trajectory encoder and the scene encoder to yield the trajectory featureand the scene feature.
Revised: Meanwhile, the past trajectory X p and a scene image S p areseparately forwarded to the trajectory encoder and the scene encoder to yield the trajectory featureand the scene feature.

Original: The encoded features are fed into the cross-modal transformer to learn theoffsets O p between Y p ′ and the ground truth trajectory Y p .
Revised: The encoded features are fed into the cross-modal transformer to learn theoffsets O p between Y p ′ and the ground truth trajectory Y p .

Original: By adding O p to Y p ′ , we obtain our finalprediction ˆ Y p .
Revised: By adding O p to Y p ′ , we obtain our finalprediction ˆ Y p .

Original: In the training phase, if the distance of the Y p to ˆ Y p is higher than a threshold, thetrajectory of the person ( i .
Revised: In the training phase, if the distance of the Y p to ˆ Y p is higher than a threshold, thetrajectory of the person ( i .

Original: e ., X p and Y p ) will be added into the trajectory bank Z bank .
Revised: e ., X p and Y p ) will be added into the trajectory bank Z bank .

Original: After training,the bank is fixed for inference.
Revised: After training,the bank is fixed for inference.

Original: 3.2 Group Trajectory Bank Module
Revised: 3.2 Group Trajectory Bank Module

Original: The group trajectory bank module (GTB) is to construct the representative group trajectories in thescene.
Revised: The group trajectory bank module (GTB) is to construct the representative group trajectories in thescene.

Original: The core functions of GTB are bank initialization, trajectory search, and trajectory update.
Revised: The core functions of GTB are bank initialization, trajectory search, and trajectory update.

Original: Trajectory Bank Initialization.
Revised: Trajectory Bank Initialization.

Original: Due to redundancy of the large amount of recorded trajectories,instead of naively using them, we generate a set of sparse and representative ones as the initialmembers of the trajectory bank.
Revised: Due to redundancy of the large amount of recorded trajectories,instead of naively using them, we generate a set of sparse and representative ones as the initialmembers of the trajectory bank.

Original: Specifically, we denote the trajectories in the training data as
Revised: Specifically, we denote the trajectories in the training data as

Original: Z = { Z } Ni =1 and split each Z
Revised: Z = { Z } Ni =1 and split each Z

Original: i into a pair of observed trajectory X i and future trajectory Y
Revised: i into a pair of observed trajectory X i and future trajectory Y

Original: i , sothat Z is separated into an observed set X
Revised: i , sothat Z is separated into an observed set X

Original: = { X } Ni =1 and a corresponding future set Y =
Revised: = { X } Ni =1 and a corresponding future set Y =

Original: { Y } Ni =1 .
Revised: { Y } Ni =1 .

Original: Afterwards, we calculate the Euclidean distance between each pair of trajectories in {X , Y} , andobtain trajectory clusters through the K-medoids clustering algorithm [23].
Revised: Afterwards, we calculate the Euclidean distance between each pair of trajectories in {X , Y} , andobtain trajectory clusters through the K-medoids clustering algorithm [23].

Original: The initial member of the bank Z bank is the average of the trajectories belonging to the same cluster (see Algorithm 1, Step 1).
Revised: The initial member of the bank Z bank is the average of the trajectories belonging to the same cluster (see Algorithm 1, Step 1).

Original: Each trajectory in Z bank represents a moving pattern of a group of people.
Revised: Each trajectory in Z bank represents a moving pattern of a group of people.

Original: Trajectory Search and Update.
Revised: Trajectory Search and Update.

Original: In the group trajectory bank Z bank , each trajectory can be treatedas a past-future pair.
Revised: In the group trajectory bank Z bank , each trajectory can be treatedas a past-future pair.

Original: Numerically, Z bank =
Revised: Numerically, Z bank =

Original: { Z i } |Z bank | i =
Revised: { Z i } |Z bank | i =

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ; Y i ] } |Z bank | i = , where [; ] representsthe combination of a past trajectory and a future trajectory, and |Z bank | is the number of past-futurepairs in Z bank .
Revised: ; Y i ] } |Z bank | i = , where [; ] representsthe combination of a past trajectory and a future trajectory, and |Z bank | is the number of past-futurepairs in Z bank .

Original: Given a trajectory Z p =
Revised: Given a trajectory Z p =

Original: [ X p ; Y p ] of person p , we use the observed X p as a key tocompute its similarity score to the past trajectories { X i } |Z bank | i = in Z bank and find a representativetrajectory Y p ′ , 1 ≤ p ′ ≤ |Z bank | according to the maximum similarity score (see Algorithm 1, Step2).
Revised: [ X p ; Y p ] of person p , we use the observed X p as a key tocompute its similarity score to the past trajectories { X i } |Z bank | i = in Z bank and find a representativetrajectory Y p ′ , 1 ≤ p ′ ≤ |Z bank | according to the maximum similarity score (see Algorithm 1, Step2).

Original: The similarity function can be formulated as:
Revised: The similarity function can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: By adding the offsets O p (see Equation 2) to the representative trajectory Y p ′ , we obtain our predictedtrajectory ˆ Y p of the observed person (see Figure 2).
Revised: By adding the offsets O p (see Equation 2) to the representative trajectory Y p ′ , we obtain our predictedtrajectory ˆ Y p of the observed person (see Figure 2).

Original: Although the initial trajectory bank works wellfor most cases, to improve the generalizability of the bank Z bank (see Algorithm 1, Step 3), wedecide whether to update Z bank according to the distance threshold θ .
Revised: Although the initial trajectory bank works wellfor most cases, to improve the generalizability of the bank Z bank (see Algorithm 1, Step 3), wedecide whether to update Z bank according to the distance threshold θ .

Original: 3.3 Cross-modal Interaction Module
Revised: 3.3 Cross-modal Interaction Module

Original: This module focuses on the interaction between individual past trajectory and the environmentinformation.
Revised: This module focuses on the interaction between individual past trajectory and the environmentinformation.

Original: It consists of two unimodal encoders to learn the human motion and scene information,respectively, and a cross-modal transformer to model the interaction between them.
Revised: It consists of two unimodal encoders to learn the human motion and scene information,respectively, and a cross-modal transformer to model the interaction between them.

Original: Trajectory Encoder.
Revised: Trajectory Encoder.

Original: The trajectory encoder adopts a multi-head attention structure from a transformer network [29] with N tra self-attention (SA) layers.
Revised: The trajectory encoder adopts a multi-head attention structure from a transformer network [29] with N tra self-attention (SA) layers.

Original: SA layers capture human motionacross different time steps with the size of T pas × N tra × d , and project the motion featuresfrom dimension T pas × N tra d to T pas × d , where d is the embed dimension of the trajectory encoder.
Revised: SA layers capture human motionacross different time steps with the size of T pas × N tra × d , and project the motion featuresfrom dimension T pas × N tra d to T pas × d , where d is the embed dimension of the trajectory encoder.

Original: Thus, we obtain the human motion representation X tra using the trajectory encoder E tra :
Revised: Thus, we obtain the human motion representation X tra using the trajectory encoder E tra :

Original: X tra
Revised: X tra

Original: =
Revised: =

Original: E tra ( X p ) ∈ R T pas × d .
Revised: E tra ( X p ) ∈ R T pas × d .

Original: Scene Encoder.
Revised: Scene Encoder.

Original: Due to the compelling performance of the pretrained Swin Transformer [17] infeature representation, we adopt it as our scene encoder.
Revised: Due to the compelling performance of the pretrained Swin Transformer [17] infeature representation, we adopt it as our scene encoder.

Original: It extracts scene semantic features withthe size of N cls × h × w , where N cls ( N cls = 150 in the scene encoder) is the number of semanticclasses, such as person, road, etc., h and w are the spatial resolution.
Revised: It extracts scene semantic features withthe size of N cls × h × w , where N cls ( N cls = 150 in the scene encoder) is the number of semanticclasses, such as person, road, etc., h and w are the spatial resolution.

Original: In order to make the followedmodule to conveniently fuse the motion representation and the environment information, we reshapethe semantic features from size ( N cls × h × w ) to ( N cls × hw ), and project them from dimension( N cls × hw ) to ( N cls × d ) through a Multi-Layer Perception layer.
Revised: In order to make the followedmodule to conveniently fuse the motion representation and the environment information, we reshapethe semantic features from size ( N cls × h × w ) to ( N cls × hw ), and project them from dimension( N cls × hw ) to ( N cls × d ) through a Multi-Layer Perception layer.

Original: As a result, we obtain the scenerepresentation X sce using the scene encoder E sce : X sce =
Revised: As a result, we obtain the scenerepresentation X sce using the scene encoder E sce : X sce =

Original: E sce ( S p ) ∈
Revised: E sce ( S p ) ∈

Original: R N cls × d .
Revised: R N cls × d .

Original: Cross-modal Transformer.
Revised: Cross-modal Transformer.

Original: Unimodal encoders extract the features from its own modality, however,disregarding the interaction between the human motion and environment information.
Revised: Unimodal encoders extract the features from its own modality, however,disregarding the interaction between the human motion and environment information.

Original: The crossmodal transformer with N cro layers aims to refine the candidate trajectory Y p
Revised: The crossmodal transformer with N cro layers aims to refine the candidate trajectory Y p

Original: ′ (see Section 3.2)by learning this interaction.
Revised: ′ (see Section 3.2)by learning this interaction.

Original: We adopt the two-stream structure: one for capturing the importanthuman motion constrained by environment information, and the other for picking out the environmentinformation related to human motion.
Revised: We adopt the two-stream structure: one for capturing the importanthuman motion constrained by environment information, and the other for picking out the environmentinformation related to human motion.

Original: The cross-attention (CA) layers and self-attention (SA) layersare the main components of the cross-modal transformer (see Figure 3).
Revised: The cross-attention (CA) layers and self-attention (SA) layersare the main components of the cross-modal transformer (see Figure 3).

Original: To capture the importanthuman motion affected by environment and acquire the environment information related to themotion, CA layers treat one modality as query, and the other as key and value to interact with the twomodalities.
Revised: To capture the importanthuman motion affected by environment and acquire the environment information related to themotion, CA layers treat one modality as query, and the other as key and value to interact with the twomodalities.

Original: SA layers serve to facilitate better internal connections, which calculate the similaritybetween an element (query) and other elements (key) in scene-constrained motion or motion-relevantenvironment information.
Revised: SA layers serve to facilitate better internal connections, which calculate the similaritybetween an element (query) and other elements (key) in scene-constrained motion or motion-relevantenvironment information.

Original: Thus, we obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the crossmodal transformer E cro : ˆ X tra =
Revised: Thus, we obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the crossmodal transformer E cro : ˆ X tra =

Original: E cro ( X tra , X sce ) , ˆ X sce =
Revised: E cro ( X tra , X sce ) , ˆ X sce =

Original: E cro ( X sce , X tra ) .
Revised: E cro ( X sce , X tra ) .

Original: To predict theoffsets O p between the searched trajectory Y p ′ and the ground truth trajectory Y p , we take the lastelement ( LE ) h tra ∈ R d of ˆ X tra and the output
Revised: To predict theoffsets O p between the searched trajectory Y p ′ and the ground truth trajectory Y p , we take the lastelement ( LE ) h tra ∈ R d of ˆ X tra and the output

Original: h sce ∈ R
Revised: h sce ∈ R

Original: d of ˆ X sce after the global pooling layer( GPL ).
Revised: d of ˆ X sce after the global pooling layer( GPL ).

Original: Offsets O p ∈ R T fut × 2 can be formulated as follows:
Revised: Offsets O p ∈ R T fut × 2 can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: 2) where [; ] denotes the vector concatenation, and MLP is the Multi-Layer Perception layer.
Revised: 2) where [; ] denotes the vector concatenation, and MLP is the Multi-Layer Perception layer.

Original: 3.4 Training
Revised: 3.4 Training

Original: We train the overall framework of SHENet end-to-end to minimize an objective function.
Revised: We train the overall framework of SHENet end-to-end to minimize an objective function.

Original: Duringtraining, since the scene encoder has been pretrained on ADE20K [36], we freeze its segmentation partand update the parameters of MLP head (see Section 3.3).
Revised: Duringtraining, since the scene encoder has been pretrained on ADE20K [36], we freeze its segmentation partand update the parameters of MLP head (see Section 3.3).

Original: Following the existing works [1, 19, 20, 33],the first regression loss function is the mean squared error (MSE) between the predicted trajectoryand the ground truth trajectory: L tra =
Revised: Following the existing works [1, 19, 20, 33],the first regression loss function is the mean squared error (MSE) between the predicted trajectoryand the ground truth trajectory: L tra =

Original: [Equation]
Revised: [Equation]

Original: The second loss is the curve smoothing (CS) regression loss, which serves to reduce the impactof individual bias.
Revised: The second loss is the curve smoothing (CS) regression loss, which serves to reduce the impactof individual bias.

Original: It calculates the MSE after trajectory smoothing.
Revised: It calculates the MSE after trajectory smoothing.

Original: CS loss can be formulated asfollows:
Revised: CS loss can be formulated asfollows:

Original: [Equation]
Revised: [Equation]

Original: where CS represents the function of curve smoothing [2].
Revised: where CS represents the function of curve smoothing [2].

Original: In a more challenging PAV dataset, we use
Revised: In a more challenging PAV dataset, we use

Original: L cs instead of L tra .
Revised: L cs instead of L tra .

Original:  Experiments
Revised:  Experiments

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We evaluate our method on ETH [24], UCY [13], and PAV datasets.
Revised: We evaluate our method on ETH [24], UCY [13], and PAV datasets.

Original: ETH/UCY datasetsare the benchmark commonly used for human trajectory prediction (HTP).
Revised: ETH/UCY datasetsare the benchmark commonly used for human trajectory prediction (HTP).

Original: The benchmark containsvideos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2 captured at 2.5HZ.
Revised: The benchmark containsvideos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2 captured at 2.5HZ.

Original: Unimodal methods [1, 10, 30, 33] only focus on the trajectory data, however, [16, 20] need toconsider scene information.
Revised: Unimodal methods [1, 10, 30, 33] only focus on the trajectory data, however, [16, 20] need toconsider scene information.

Original: As a result, these multimodal methods [16, 20] use fewer frames thanthe unimodal methods, since a video from UNIV is not available.
Revised: As a result, these multimodal methods [16, 20] use fewer frames thanthe unimodal methods, since a video from UNIV is not available.

Original: We follow [16] to process the dataand predict future n fut = 12 frames given the observed n pas = 8 frames.
Revised: We follow [16] to process the dataand predict future n fut = 12 frames given the observed n pas = 8 frames.

Original: Compared to the ETH/UCY datasets, PAV is more challenging with diverse movement patterns.
Revised: Compared to the ETH/UCY datasets, PAV is more challenging with diverse movement patterns.

Original: Itobtains 3 videos from the MOT15 dataset [11], captured with static cameras.
Revised: Itobtains 3 videos from the MOT15 dataset [11], captured with static cameras.

Original: The video sequencesinclude PETS09-S2L1 (PETS) [4, 11], ADL-Rundle-6 (ADL), and Venice-2 (VENICE), whichprovide sufficient trajectories for the HTP task.
Revised: The video sequencesinclude PETS09-S2L1 (PETS) [4, 11], ADL-Rundle-6 (ADL), and Venice-2 (VENICE), whichprovide sufficient trajectories for the HTP task.

Original: We divide the videos into training (80%) and testing(20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4,200 training sequences and 664/306/650testing sequences, respectively.
Revised: We divide the videos into training (80%) and testing(20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4,200 training sequences and 664/306/650testing sequences, respectively.

Original: We use observed n pas =
Revised: We use observed n pas =

Original: 10 frames to predict future n fut
Revised: 10 frames to predict future n fut

Original: = 50frames, such that we can compare the long-term prediction results of different methods.
Revised: = 50frames, such that we can compare the long-term prediction results of different methods.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: For ETH and UCY datasets, we adopt the standard metrics [1, 7, 16, 20, 33] for HTP: Average Displacement Error (ADE) and Final Displacement
Revised: For ETH and UCY datasets, we adopt the standard metrics [1, 7, 16, 20, 33] for HTP: Average Displacement Error (ADE) and Final Displacement

Original: Error (FDE).
Revised: Error (FDE).

Original: ADE is the average L 2 error between the predicted trajectories and ground truth trajectories over all time steps, and FDE is the L 2 error between the predicted trajectories and ground truth trajectories at the final time step.
Revised: ADE is the average L 2 error between the predicted trajectories and ground truth trajectories over all time steps, and FDE is the L 2 error between the predicted trajectories and ground truth trajectories at the final time step.

Original: The trajectories in PAV are with some jittering phenomena ( e .
Revised: The trajectories in PAV are with some jittering phenomena ( e .

Original: g ., abrupt and sharp turns).
Revised: g ., abrupt and sharp turns).

Original: Thus, a reasonable prediction may produce approximately the same error as an unrealistic prediction by using traditional metrics: ADE and FDE (see Figure 7(a)).
Revised: Thus, a reasonable prediction may produce approximately the same error as an unrealistic prediction by using traditional metrics: ADE and FDE (see Figure 7(a)).

Original: In order to focus on the pattern and shape of the trajectory itself, and reduce the impact of randomness and subjectivity, we propose the CS-Metric: CS-ADE and CS-FDE (illustrated in Figure 4).
Revised: In order to focus on the pattern and shape of the trajectory itself, and reduce the impact of randomness and subjectivity, we propose the CS-Metric: CS-ADE and CS-FDE (illustrated in Figure 4).

Original: CS-ADEis calculated as follows:
Revised: CS-ADEis calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: (4) where CS is the curve smoothing function defined as the same as our L cs in Section 3.4.
Revised: (4) where CS is the curve smoothing function defined as the same as our L cs in Section 3.4.

Original: Similar to
Revised: Similar to

Original: CS-ADE, CS-FDE calculates the final displacement error after trajectory smoothing:
Revised: CS-ADE, CS-FDE calculates the final displacement error after trajectory smoothing:

Original: [Equation]
Revised: [Equation]

Original: Figure 5 shows some samples from the training data, converting rough ground truth trajectories tosmooth ones.
Revised: Figure 5 shows some samples from the training data, converting rough ground truth trajectories tosmooth ones.

Original: Implementation Details.
Revised: Implementation Details.

Original: In SHENet, the initial size of group trajectory bank is set to |Z bank | = 32 .
Revised: In SHENet, the initial size of group trajectory bank is set to |Z bank | = 32 .

Original: Both the trajectory encoder and the scene encoder have 4 self-attention (SA) layers.
Revised: Both the trajectory encoder and the scene encoder have 4 self-attention (SA) layers.

Original: The cross-modaltransformer is with 6 SA layers and cross-attention (CA) layers.
Revised: The cross-modaltransformer is with 6 SA layers and cross-attention (CA) layers.

Original: We set all the embed dimensionsto 512 .
Revised: We set all the embed dimensionsto 512 .

Original: For the trajectory encoder, it learns the human motion information with size of T pas × 

Revised: For the trajectory encoder, it learns the human motion information with size of T pas ×

Original: ( T pas = 8 in ETH/UCY, T pas = 10 in PAV).
Revised: ( T pas = 8 in ETH/UCY, T pas = 10 in PAV).

Original: For the scene encoder, it outputs the semantic featureswith size 150 × 56 × 56 .
Revised: For the scene encoder, it outputs the semantic featureswith size 150 × 56 × 56 .

Original: We reshape the features from size 150 × 56 × 56 to 150 × 3136 , and projectthem from dimension 150 × 3136 to 150 × 512 .
Revised: We reshape the features from size 150 × 56 × 56 to 150 × 3136 , and projectthem from dimension 150 × 3136 to 150 × 512 .

Original: We train the model for 100 epochs on 4 NVIDIA
Revised: We train the model for 100 epochs on 4 NVIDIA

Original: Quadro RTX 6000 GPUs and use the Adam optimizer with a fixed learning rate 1 e − 5 .
Revised: Quadro RTX 6000 GPUs and use the Adam optimizer with a fixed learning rate 1 e − 5 .

Original: More detailsare in the supplementary material.
Revised: More detailsare in the supplementary material.

Original: 4.2 Ablation Studies
Revised: 4.2 Ablation Studies

Original: In Table 1, we evaluate each component of SHENet, including a group trajectory bank (GTB) module,and a cross-modal interaction (CMI) module that contains a trajectory encoder (TE), a scene encoder(SE) and a cross-modal transformer (CMT).
Revised: In Table 1, we evaluate each component of SHENet, including a group trajectory bank (GTB) module,and a cross-modal interaction (CMI) module that contains a trajectory encoder (TE), a scene encoder(SE) and a cross-modal transformer (CMT).

Original: Effect of GTB.
Revised: Effect of GTB.

Original: We first investigate the performance of GTB.
Revised: We first investigate the performance of GTB.

Original: GTB improves FDE by 21.2% on PETScompared to the CMI ( i .
Revised: GTB improves FDE by 21.2% on PETScompared to the CMI ( i .

Original: e ., TE, SE, and CMT), which is a significant improvement, emphasizing theimportance of GTB.
Revised: e ., TE, SE, and CMT), which is a significant improvement, emphasizing theimportance of GTB.

Original: However, GTB (1-st row of Table 1) alone is not enough and performs a littleworse than CMI.
Revised: However, GTB (1-st row of Table 1) alone is not enough and performs a littleworse than CMI.

Original: Thus, we explore the effects of various parts in CMI module.
Revised: Thus, we explore the effects of various parts in CMI module.

Original: Effect of TE and SE.
Revised: Effect of TE and SE.

Original: To evaluate the performance of TE and SE, we concatenate the trajectoryfeatures extracted from TE and the scene features from SE together (3-rd row in Table 1), performancewill improve on ADL and VENICE with smaller movements, compared to TE alone.
Revised: To evaluate the performance of TE and SE, we concatenate the trajectoryfeatures extracted from TE and the scene features from SE together (3-rd row in Table 1), performancewill improve on ADL and VENICE with smaller movements, compared to TE alone.

Original: It indicates thatthe incorporation of environment information into trajectory prediction can improve the accuracy ofthe
Revised: It indicates thatthe incorporation of environment information into trajectory prediction can improve the accuracy ofthe

Original: [Table]
Revised: [Table]

Original: Effect of CMT.
Revised: Effect of CMT.

Original: Compared to the third row of Table 1, CMT (4-th row in Table 1) can result ina promising improvement of the model performance.
Revised: Compared to the third row of Table 1, CMT (4-th row in Table 1) can result ina promising improvement of the model performance.

Original: Notably
Revised: Notably

Original: [Table]
Revised: [Table]

Original: it performs better than the TE and
Revised: it performs better than the TE and

Original: SE concatenated on PETS, improving by 7.4% over ADE.
Revised: SE concatenated on PETS, improving by 7.4% over ADE.

Original: Compared to GTB alone, the full CMIimproves by 12.2% over ADE on average.
Revised: Compared to GTB alone, the full CMIimproves by 12.2% over ADE on average.

Original: 4.3 Comparison with State-of-the-art Methods
Revised: 4.3 Comparison with State-of-the-art Methods

Original: is adopted for evaluation.
Revised: is adopted for evaluation.

Original: [Table]
Revised: [Table]

Original: On ETH/UCY datasets
Revised: On ETH/UCY datasets

Original: [Table]
Revised: [Table]

Original: we compare our model with the state-of-art methods: SS-LSTM [33],
Revised: we compare our model with the state-of-art methods: SS-LSTM [33],

Original: Social-STGCN [22], MANTRA [20], AgentFormer [35], YNet [19].
Revised: Social-STGCN [22], MANTRA [20], AgentFormer [35], YNet [19].

Original: The results are summarized in
Revised: The results are summarized in

Original: Table 2.
Revised: Table 2.

Original: Our model reduces the average FDE from 0.39 to 0.
Revised: Our model reduces the average FDE from 0.39 to 0.

Original: [Table]
Revised: [Table]

Original: and achieves 7.7% improvementcompared with the state-of-the-art method, YNet.
Revised: and achieves 7.7% improvementcompared with the state-of-the-art method, YNet.

Original: Particularly, our model outperforms previousmethods significantly on ETH, when there are large movements of trajectories, and improves the
Revised: Particularly, our model outperforms previousmethods significantly on ETH, when there are large movements of trajectories, and improves the

Original: ADE/FDE by 12.8%/15.3%, respectively.
Revised: ADE/FDE by 12.8%/15.3%, respectively.

Original: To evaluate the performance of our model in long-term predictions, we conduct experiments on PAV,with n pas =
Revised: To evaluate the performance of our model in long-term predictions, we conduct experiments on PAV,with n pas =

Original: 10 observed frames and n fut = 50 future frames of each trajectory.
Revised: 10 observed frames and n fut = 50 future frames of each trajectory.

Original: Table 3 shows theperformance compared with previous HTP methods: SS-LSTM [33], Social-STGCN [22], Next [16],
Revised: Table 3 shows theperformance compared with previous HTP methods: SS-LSTM [33], Social-STGCN [22], Next [16],

Original: MANTRA [20], YNet [19].
Revised: MANTRA [20], YNet [19].

Original: Compared to the state-of-the-art results of YNet, the proposed SHENetachieves repective 3.3%/10.5% improvement over CS-ADE/CS-FDE on average.
Revised: Compared to the state-of-the-art results of YNet, the proposed SHENetachieves repective 3.3%/10.5% improvement over CS-ADE/CS-FDE on average.

Original: Since YNet predictsthe heatmaps of a trajectory, it performs a little better when there are small movements of trajectories, e .
Revised: Since YNet predictsthe heatmaps of a trajectory, it performs a little better when there are small movements of trajectories, e .

Original: g .,
Revised: g .,

Original: VENICE.
Revised: VENICE.

Original: Nevertheless, our method is still competitive in
Revised: Nevertheless, our method is still competitive in

Original: VENICE, and is much better than others approaches on PETS with large movements and intersections.
Revised: VENICE, and is much better than others approaches on PETS with large movements and intersections.

Original: [Table]
Revised: [Table]

Original: In particular, our method improves the CS-FDE by 16.2% on PETS, compared to YNet.
Revised: In particular, our method improves the CS-FDE by 16.2% on PETS, compared to YNet.

Original: We alsoachieve great improvements using the traditional ADE/FDE metrics.
Revised: We alsoachieve great improvements using the traditional ADE/FDE metrics.

Original: The complete results are shownin the supplementary material.4.4 Qualitative Analysis.
Revised: The complete results are shownin the supplementary material.4.4 Qualitative Analysis.

Original: [Figure]
Revised: [Figure]

Original: Figure 6 presents the qualitative results of SHENet and other methods.
Revised: Figure 6 presents the qualitative results of SHENet and other methods.

Original: By comparison, we surprisinglynotice that in an extremely challenging situation where a person walks to the curb and turns back(green curve), all other methods do not deal with it well, while the proposed SHENet can still handleit.
Revised: By comparison, we surprisinglynotice that in an extremely challenging situation where a person walks to the curb and turns back(green curve), all other methods do not deal with it well, while the proposed SHENet can still handleit.

Original: This should be attributed to the effect of our dedicatedly designed historical group trajectory bankmodule.
Revised: This should be attributed to the effect of our dedicatedly designed historical group trajectory bankmodule.

Original: In addition, compared to the memory -based method MANTRA [20], we search trajectories of groups instead of just individuals.
Revised: In addition, compared to the memory -based method MANTRA [20], we search trajectories of groups instead of just individuals.

Original: This is more versatile and can be applied in more challenging scenes.
Revised: This is more versatile and can be applied in more challenging scenes.

Original: Figureincludes the qualitative results of YNet and our SHENet without/with curve smoothing (CS).
Revised: Figureincludes the qualitative results of YNet and our SHENet without/with curve smoothing (CS).

Original: The first row presents the results of using MSE loss L tra .
Revised: The first row presents the results of using MSE loss L tra .

Original: Influenced by the past trajectories with some noise ( e .
Revised: Influenced by the past trajectories with some noise ( e .

Original: g ., abrupt and sharp turns), the predicted trajectory points of YNet are gathered together, and cannot present a clear direction, while our method can provide a potential path based on the historical group trajectories.
Revised: g ., abrupt and sharp turns), the predicted trajectory points of YNet are gathered together, and cannot present a clear direction, while our method can provide a potential path based on the historical group trajectories.

Original: The two predictions are visually distinct, however, the numerical errors (ADE/FDE) are approximately the same.
Revised: The two predictions are visually distinct, however, the numerical errors (ADE/FDE) are approximately the same.

Original: In contrast, the qualitative results of our proposed CS loss L cs are shown in the second row of Figure 7.
Revised: In contrast, the qualitative results of our proposed CS loss L cs are shown in the second row of Figure 7.

Original: We can see that the proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by YNet and our method.
Revised: We can see that the proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by YNet and our method.

Original: Morevisualization results are provided in supplementary material.
Revised: Morevisualization results are provided in supplementary material.

Original:  Conclusion
Revised:  Conclusion

Original: The paper presents SHENet, a method that fully utilizes scene history for HTP.
Revised: The paper presents SHENet, a method that fully utilizes scene history for HTP.

Original: SHENet includesa
Revised: SHENet includesa

Original: GTB module to construct a group trajectory bank from all historical trajectories and retrieve arepresentative trajectory from this bank for an observed person, and a CMI module, which interactsbetween human motion and environment information, to refine this representative trajectory.
Revised: GTB module to construct a group trajectory bank from all historical trajectories and retrieve arepresentative trajectory from this bank for an observed person, and a CMI module, which interactsbetween human motion and environment information, to refine this representative trajectory.

Original: Weachieve the SOTA performance on the HTP benchmarks and our method demonstrates significantimprovements and generalizaibility in the challenging scenes.
Revised: Weachieve the SOTA performance on the HTP benchmarks and our method demonstrates significantimprovements and generalizaibility in the challenging scenes.

Original: However, there are several unexploredaspects in the current framework, e.g., the process of bank construction currently focuses on only human motion.
Revised: However, there are several unexploredaspects in the current framework, e.g., the process of bank construction currently focuses on only human motion.

Original: Future work includes further exploring the trajectory bank using the interact information(human motion and scene information).
Revised: Future work includes further exploring the trajectory bank using the interact information(human motion and scene information).

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes]
Revised: [Yes]

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] All data used in
Revised: [Yes] All data used in

Original: the paper is public; our code are provided
Revised: the paper is public; our code are provided

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you report error bars (e
Revised: Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a)
Revised: (a)

Original: If your work uses existing assets, did you cite the creators?
Revised: If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A] public datasets
Revised: [N/A] public datasets

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?

Original: [N/A] public datasets
Revised: [N/A] public datasets

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A](b)
Revised: [N/A](b)

Original: Did you describe any potential participant risks, with links to Institutional Review
Revised: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix
Revised: A Appendix

Original: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
Revised: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.

Original: This section will often be part of the supplemental material.
Revised: This section will often be part of the supplemental material.