Original: Forecasting Human Trajectory from Scene History
Revised: Forecasting Human Trajectory from Scene History

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Predicting the future trajectory of a person remains a challenging problem, dueto randomness and subjectivity.
Revised: Predicting the future trajectory of a person remains a challenging problem, due to randomness and subjectivity of human movement.

Original: However, the moving patterns of human in aconstrained scenario typically conform to a limited number of regularities to acertain extent, because of the scenario restrictions ( e .
Revised: However, the moving patterns of human in a constrained scenario typically conform to a limited number of regularities to a certain extent, because of the scenario restrictions ( e .

Original: g ., floor plan, roads andobstacles) and person-person or person-object interactivity.
Revised: g ., floor plan, roads, and obstacles) and person-person or person-object interactivity.

Original: Thus, an individualperson in this scenario should follow one of the regularities as well.
Revised: Thus, an individual person in this scenario should follow one of the regularities as well.

Original: In otherwords, a person’s subsequent trajectory has likely been traveled by others.
Revised: In other words, a person’s subsequent trajectory has likely been traveled by others.

Original: Basedon this hypothesis, we propose to forecast a person’s future trajectory by learningfrom the implicit scene regularities.
Revised: Based on this hypothesis, we propose to forecast a person’s future trajectory by learning from the implicit scene regularities.

Original: We call the regularities, inherently derivedfrom the past dynamics of the people and the environment in the scene, scenehistory .
Revised: We call the regularities, inherently derived from the past dynamics of the people and the environment in the scene, scene history .

Original: We categorize scene history information into two types: historical grouptrajectories and individual-surroundings interaction.
Revised: We categorize scene history information into two types: historical group trajectory and individual-surroundings interaction.

Original: To exploit these informationfor trajectory prediction, we propose a novel framework SceneHistory Excavating
Revised: To exploit these two types of information for trajectory prediction, we propose a novel framework Scene

Original:  Network (SHENet), where the scene history is leveraged in a simple yet effectiveapproach.
Revised: History Excavating Network (SHENet), where the scene history is leveraged in a simple yet effective approach.

Original: In particular, we design two components, the group trajectory bankmodule to extract representative group trajectories as the candidate for future path,and the cross-modal interaction module to model the interaction between individualpast trajectory and its surroundings for trajectory refinement, respectively.
Revised: In particular, we design two components: the group trajectory bank module to extract representative group trajectories as the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement.

Original: Inaddition, to mitigate the uncertainty in the evaluation, caused by the aforementionedrandomness and subjectivity, we propose to include smoothness into evaluationmetrics.
Revised: In addition, to mitigate the uncertainty in ground-truth trajectory, caused by the aforementioned randomness and subjectivity of human movement, we propose to include smoothness into the training process and evaluation metrics.

Original: We conduct extensive evaluations to validate the efficacy of proposedframework on ETH, UCY, as well as a new, challenging benchmark dataset PAV,demonstrating superior performance compared to state-of-the-art methods. 
Revised: We conduct extensive evaluations to validate the efficacy of our proposed framework on ETH, UCY, as well as a new, challenging benchmark dataset PAV, demonstrating superior performance compared to state-of-the-art methods. Code is available at: https:/

Original:  Introduction
Revised:  Introduction

Original: Human trajectory prediction (HTP) aims to predict a target person’s future path from a video clip[5, 9, 19, 20].
Revised: Human trajectory prediction (HTP) aims to predict a target person’s future path from a video clip [5, 9, 19, 20].

Original: This is critical for intelligent transportation, as it enables vehicle to perceive the statusof pedestrian in advance so  it can avoid potential collision.
Revised: This is critical for intelligent transportation, as it enables vehicle to perceive the status of pedestrian in advance so that it can avoid potential collision.

Original: Surveillance systems with HTP capabilitycan assist security officers to predict the possible escape path of suspects.
Revised: Surveillance systems with HTP capability can assist security officers to predict the possible escape path of suspects.

Original: Although much work hasbeen done in recent years [1, 9, 12, 14, 18, 33], very fewof them are reliable and generalizable enoughto be applied in real-world scenarios, primarily due to two challenges of the task: randomness andsubjectivity of human movement.
Revised: Although much work has been done in recent years [1, 9, 12, 14, 18, 35], very few are reliable and generalizable enough to be applied in real-world scenarios, primarily due to two challenges of the task: randomness and subjectivity of human movement.

Original: However, most real world scenarios are constrained.
Revised: However, in the constrained real world scenarios, the challenges are not absolutely intractable.

Original: As shown in Figure 1, given previously captured video in this scene, the target person’s  (red box) future trajectorybecomes more predictable as the moving pattern of human typically complies with several underlyingregularities in this scenario, which the target person would follow.
Revised: As shown in Figure 1, given previously-captured video in this scene, the target person’s future trajectory (red box) becomes more predictable as the moving pattern of human typically complies with several underlying regularities in this scenario, which the target person would follow.

Original: Therefore, to predict the trajectory,first we  need to understand these regularities.
Revised: Therefore, to predict the trajectory, we first need to understand these regularities.

Original: We argue that these regularities are implicitly encodedin historical human trajectories (Figure 1 left), individual past trajectory,the surroundings, and theinteraction between them (Figure 1 right), which we refer to as scene history .
Revised: We argue that these regularities are implicitly encoded in historical human trajectories (Figure 1 left), individual past trajectory, surroundings, and interaction between them (Figure 1 right), which we refer to as scene history .

Original: We separatethe historical information into two categories: historical group trajectories (HGT) and individual-surroundings interaction (ISI).
Revised: We separate historical information into two categories: historical group trajectories (HGT) and individual-surroundings interaction (ISI).

Original: HGT refers to the group representatives of all historicaltrajectories in a scene.
Revised: HGT refers to the group representatives of all historical trajectories in a scene.

Original: The reason of resorting to HGT is that given a new target person in the scenario,his/her path is more likely to share more similarities with one of the group trajectories than anyindividual instance of the historical trajectories, due to the aforementioned randomness, subjectivity,and regularities.
Revised: The reason of resorting to HGT is that, given a new target person in the scenario, his/her path is more likely to share more similarity with one of the group trajectories than any individual instance of the historical trajectories, due to the aforementioned randomness, subjectivity, and regularities.

Original: Nevertheless, the group trajectories are less correlated to the individual’s past statusand the corresponding surroundings, influencing the person’s future trajectory as well.
Revised: Nevertheless, the group trajectories are less correlated to the individual’s past status and the corresponding surroundings, influencing the person’s future trajectory as well.

Original: ISI is requiredto more comprehensively leveragethe history information by extracting contextual information.
Revised: ISI is required to more comprehensively leverage history information by extracting contextual information.

Original: Afew existing methods take into account similarity between individual past trajectory and historicaltrajectories.
Revised: Few existing methods have taken into account similarity between individual past trajectory and historical trajectories.

Original: Most attempts [16, 19, 26, 33] explore only individual-surroundings interaction, wheretremendous efforts are spent on modeling the individual trajectory, the semantic information ofenvironment information, and  relationship between them.
Revised: Most attempts [16, 19, 27, 35] explore only individual-surroundings interaction, where tremendous efforts are spent on modeling the individual trajectory, the semantic information of environment information, and the relationship between them.

Original: Although MANTRA [20] models thesimilarity using  encoders trained in a reconstruction manner, it performs the  similarity computationon instance level instead of group level, making it sensitive to the capability of the trained encoder.
Revised: Although MANTRA [20] models the similarity using the encoders trained in a reconstruction manner and MemoNet [32] simplifies the similarity by storing the intention of historical trajectories, they both perform similarity computation on instance level, instead of group level, making it sensitive to the capability of the trained encoder.

Original: Based on the above analysis, we propose a simple yet effective framework, Scene History Excavating
Revised: Based on the above analysis, we propose a simple yet effective framework, Scene History Excavating

Original: Network (SHENet), to leverage HGT and ISI jointly for HTP.
Revised: Network (SHENet), to leverage HGT and ISI jointly for HTP.

Original: In particular, the framework consists oftwo major components: 1) a group trajectory bank (GTB) module, and 2) a cross-modal interaction(CMI) module.
Revised: In particular, the framework consists of two major components: (i) a group trajectory bank (GTB) module, and (ii) a cross-modal interaction (CMI) module.

Original: GTB constructs the representative group trajectories from all historical individualtrajectories and provides a candidate path for future trajectory prediction.
Revised: GTB constructs the representative group trajectories from all historical individual trajectories and provides a candidate path for future trajectory prediction.

Original: CMI encodes the observedindividual trajectory and the surroundings separately, and models their interaction with a cross-modaltransformer for refinement of the searched candidate trajectory.
Revised: CMI encodes the observed individual trajectory and the surroundings separately, and models their interaction using a cross-modal transformer for refinement of the searched candidate trajectory.

Original: In addition,in order to alleviate theevaluation uncertainty from two mentioned characteristics ( i .
Revised: In addition, to alleviate the uncertainty from two mentioned characteristics ( i .

Original: e .,randomness and subjectivity), we introduce curve smoothing (CS) into  current metrics average andfinal displacement error ( i .
Revised: e ., randomness and subjectivity), we introduce curve smoothing (CS) into the training process and current evaluation metrics, average and final displacement errors ( i .

Original: e ., ADE and FDE), resulting in two new metrics CS-ADE and CS-FDE.
Revised: e ., ADE and FDE), resulting in two new metrics CS-ADE and CS-FDE.

Original: Moreover, to facilitate the development of research in HTP, we collect a new challenging dataset withdiverse movement patterns, named PAV.
Revised: Moreover, to facilitate the development of research in HTP, we collect a new challenging dataset with diverse movement patterns, named PAV.

Original: The dataset is obtained by choosing the videos with fixedcamera view and complex human movement from MOT15 dataset [11].
Revised: The dataset is obtained by choosing the videos with fixed camera view and complex human movement from MOT15 dataset [11].

Original: The contributions of the work can be summarized as follows: 1) We introduce group history tosearch individual trajectory for HTP.
Revised: The contributions of the work can be summarized as follows: 1) We introduce group history to search individual trajectory for HTP.

Original: We propose a simple and effective framework, SHENet, tojointly utilize two types of scene history ( i .
Revised: We propose a simple and effective framework, SHENet, to jointly utilize two types of scene history ( i .

Original: e ., historical group trajectories and individual-surroundingsinteraction) for HTP.
Revised: e ., historical group trajectories and individual-surroundings interaction) for HTP.

Original: We construct a new challenging dataset, PAV, and two new metrics consideringrandomness and subjectivity in human moving patterns are proposed to  better benchmark HTPperformance.
Revised: We construct a new challenging dataset, PAV; also, one novel loss function and two new metrics, considering randomness and subjectivity in human moving patterns, are proposed to achieve better benchmark HTP performance.

Original: We conduct comprehensive experiments on ETH, UCY, and PAV to demonstrate thesuperior performance of SHENet and the efficacy of each component.
Revised: We conduct comprehensive experiments on ETH, UCY, and PAV to demonstrate the superior performance of SHENet and the efficacy of each component.

Original:  Related Work
Revised:  Related Work

Original: Unimodal Methods.
Revised: Unimodal Methods.

Original: Unimodal methods rely on learning the regularity of individual motion frompast trajectories in order to predict future trajectories.
Revised: Unimodal methods rely on learning the regularity of individual motion from past trajectories in order to predict future trajectories.

Original: For example, Social LSTM [1] focuses onmodeling the interaction between individual trajectories through a social pooling module.
Revised: For example, Social LSTM [1] focuses on modeling the interaction between individual trajectories through a social pooling module.

Original: STGAT [8]uses an attention module to learn spatial interactions and assign reasonable importance to neighbors.
Revised: STGAT [8] uses an attention module to learn spatial interactions and assign reasonable importance to neighbors.

Original: PIE [25] uses a temporal attention module to calculate the importance of observed trajectories ateach time step.
Revised: PIE [25] uses a temporal attention module to calculate the importance of observed trajectories at each time step.

Original: Giuliari et al.
Revised: Giuliari et al.

Original: [6] use the Transformer networks [3, 29] to predict the trajectoriesof individual people, which weight all observed points in the trajectories according to an attentionmechanism.
Revised: [6] use the Transformer networks [3, 30] to predict the trajectories of individual people, which weight all observed points in the trajectories according to an attention mechanism.

Revised: Xu et al.

Revised: [34] propose a newly designed graph neural network to extract spatial-temporal feature representations of the observed trajectories, along with a knowledge learning module to refine individual-level transferable representations.

Original: These methods consider the scene history in two aspects: the individual’s past trajectoryand the interaction between the individual’s past trajectory and its neighboring trajectories.
Revised: These methods consider the scene history in two aspects: the individual’s past trajectory and the interaction between the individual’s past trajectory and its neighboring trajectories.

Original: Multimodal Methods.
Revised: Multimodal Methods.

Original: Additionally, multimodal methods examine the influence of environmentinformation on HTP.
Revised: Additionally, multimodal methods examine the influence of environment information on HTP.

Original: SS-LSTM [33] proposes a scene interaction module to capture global informationof the scene.
Revised: SS-LSTM [35] proposes a scene interaction module to capture global information of the scene.

Original: Trajectron++ [26] uses a graph structure to model the trajectories and interacts withenvironmental information and other individuals. 
Revised: Trajectron++ [27] uses a graph structure to model the trajectories and interacts with environmental information and other individuals. Zhou et al.

Revised: [39] propose a context encoder to extract local features for each agent, and a global interaction module to aggregate the local context of individual agents.

Original: Liang et al.
Revised: Liang et al.

Original: [16] integrate the interactive modulesinto a person interaction module (person-person, person-object, and person-scene), and add a personbehavior module (person appearance) to encode rich visual information.
Revised: [16] integrate the interactive modules into a person interaction module (person-person, person-object, and person-scene), and add a person behavior module (person appearance) to encode rich visual information.

Original: Different from the RNNs[1, 16, 32, 33] to encode the past trajectories into a hidden state vector, which is prone to loss ofinformation, MANTRA [20] utilizes an external memory to model long-term dependencies.
Revised: Different from the RNNs [1, 16, 33, 35] to encode the past trajectories into a hidden state vector, which is prone to loss of information, MANTRA [20] utilizes an external memory to model long-term dependencies.

Original: It storesthe historical single-agent trajectories in a memory, and encodes the environment information to  searched trajectory 
Revised: It stores the historical single-agent trajectories in a memory, and encodes the environment information to refine a searched trajectory from this memory.

Original: Difference from Previous Works.
Revised: Difference from Previous Works.

Original: Both unimodal and multimodal methods use single or partialaspects of the scene history, disregarding the historical group trajectories.
Revised: Both unimodal and multimodal methods use single or partial aspects of the scene history, disregarding the historical group trajectory.

Original: In our work, we integratethe information from the scene history in a more comprehensive way and propose dedicated modulesto handle different types of information, respectively.
Revised: In our work, we integrate the information from the scene history in a more comprehensive way and propose dedicated modules to handle different types of information, respectively.

Original: The main differences between our methodand the previous works, especially memory-based methods [15, 20, 21, 31] and clustering-basedmethods [27, 28, 34], are as follows: 
Revised: The main differences between our method and the previous works, especially memory-based methods [15, 20, 21, 32] and clustering-based methods [28, 29, 36], are as follows: i) MANTRA [20] and MemoNet [32] consider historical individual trajectories, while our proposed SHENet focuses on historical group trajectories, which is more generalizable in different scenarios; In [15], historical trajectories are not stored for search. Calculate distance d predicted p ground Y p 17: if d θ then 18:

Original: i) MANTRA [20] and MemoNet [31] consider historicalindividual trajectories, while our proposed SHENet focuses on historical group trajectories, which ismore generalizable in different scenarios; In [15], the historical trajectories are not stored for search.ii) [27, 34] group person-neighbors for trajectory prediction; [28] clusters the trajectories into fixednumber of categories for trajectory classification; Our SHENet generates representative trajectoriesas reference for individual person trajectory prediction.
Revised: Update Z bank by adding trajectory p 20: end 21: if β then 22: Cluster newly added trajectories in Z bank , and merge the clustered trajectories into Z bank 24: end if 25: return Z bank ii) [28, 36] group person-neighbors for trajectory prediction; [29] clusters the trajectories into fixed number of categories for trajectory classification; Our SHENet generates representative trajectories as reference for individual person trajectory prediction.

Original:  Method 
Revised:  Method 3.1 Overview

Original: 3.1 Overview

Original: The architecture of the proposed Scene History Excavating Network (SHENet) is depicted in Figure 2,which consists of two major components: group trajectory bank module (GTB) and cross-modalinteraction module (CMI).
Revised: The architecture of the proposed Scene History Excavating Network (SHENet) is depicted in Figure 2, which consists of two major components: group trajectory bank module (GTB) and cross-modal interaction module (CMI).

Original: Formally, given all the trajectories Z in observed videos of this scene,a scene image S p and a past trajectory of a target person p in the last T pas time steps, X p = 16: Calculate p Y p 17: if d θ then 18: n ← Update adding p 20: end
Revised: Formally, given all the trajectories Z in observed videos of this scene, a scene image S p and a past trajectory of a target person p in the last T pas time steps, X p = { x tp } 0 t = − T pas +1 ∈ R T pas × 2 , where x tp ∈ R 2 represents the position of p th person at time step t ,

Original: 21: if β then 22:

Original: 24: end

Original: 25: return

Original: { x tp } 0 t = − T pas +1 ∈ R T pas × 2 , where x tp ∈ R 2 represents the position of p th person at time step t ,

Original: SHENet requires to predict the future positions of the person ˆ Y p
Revised: SHENet requires to predict the future positions of the person ˆ Y p

Original: = { ˆ y tp } T fut t =1 ∈ R T fut × 2 in the next
Revised: = { ˆ y tp } T fut t =1 ∈ R T fut × 2 in the next

Original: T fut frames, such that ˆ Y p is as close to the ground truth trajectory Y p
Revised: T fut frames, such that ˆ Y p is as close to the ground-truth trajectory Y p

Original: = { y tp } T fut t =1 ∈ R T fut × 2 aspossible.
Revised: = { y tp } T fut t =1 ∈ R T fut × 2 as possible.

Original: The proposed GTB first condenses
Revised: The proposed GTB first condenses

Original: Z as representative group trajectories Z bank .
Revised: Z as representative group trajectories Z bank .

Original: Then, theobserved trajectory X p is used as a key to search a closet representative group trajectory in Z bankas a candidate future trajectory, Y p ′ .
Revised: Then, the observed trajectory X p is used as a key to search a closet representative group trajectory in Z bank as a candidate future trajectory, Y p ′ .

Original: Meanwhile, the past trajectory X p and a scene image S p areseparately forwarded to the trajectory encoder and the scene encoder to yield the trajectory featureand the scene feature.
Revised: Meanwhile, the past trajectory X p and a scene image S p are separately forwarded to the trajectory encoder and the scene encoder to yield the trajectory feature and the scene feature, respectively.

Original: The encoded features are fed into the cross-modal transformer to learn theoffsets O p between Y p ′ and the ground truth trajectory Y p .
Revised: The encoded features are fed into the cross-modal transformer to learn the offsets O p between Y p ′ and the ground-truth trajectory Y p .

Original: By adding O p to Y p ′ , we obtain our finalprediction ˆ Y p .
Revised: By adding O p to Y p ′ , we obtain our final prediction ˆ Y p .

Original: In the training phase, if the distance of the Y p to ˆ Y p is higher than a threshold, thetrajectory of the person ( i .
Revised: In the training phase, if the distance of the Y p to ˆ Y p is higher than a threshold, the trajectory of the person ( i .

Original: e ., X p and Y p ) will be added into the trajectory bank Z bank .
Revised: e ., X p and Y p ) will be added into the trajectory bank Z bank .

Original: After training,the bank is fixed for inference.
Revised: After training, the bank is fixed for inference.

Original: 3.2 Group Trajectory Bank Module
Revised: 3.2 Group Trajectory Bank Module

Original: The group trajectory bank module (GTB) is  to construct the representative group trajectories in thescene.
Revised: The group trajectory bank module (GTB) is used to construct the representative group trajectories in the scene.

Original: The core functions of GTB are bank initialization, trajectory search, and trajectory update.
Revised: The core functions of GTB are bank initialization, trajectory search, and trajectory update.

Original: Trajectory Bank Initialization.
Revised: Trajectory Bank Initialization.

Original: Due to redundancy of the large amount of recorded trajectories,instead of naively using them, we generate a set of sparse and representative ones as the initialmembers of the trajectory bank.
Revised: Due to redundancy of the large amount of recorded trajectories, instead of naively using them, we generate a set of sparse and representative ones as the initial members of the trajectory bank.

Original: Specifically, we denote the trajectories in the training data as
Revised: Specifically, we denote the trajectories in the training data as

Original: Z = { Z } Ni =1 and split each Z
Revised: Z = { Z } Ni =1 and split each Z

Original: i into a pair of observed trajectory X i and future trajectory Y
Revised: i into a pair of observed trajectory X i and future trajectory Y

Original: i , sothat Z is separated into an observed set X
Revised: i , so that Z is separated into an observed set X

Original: = { X } Ni =1 and a corresponding future set Y =
Revised: = { X } Ni =1 and a corresponding future set Y =

Original: { Y } Ni =1 .
Revised: { Y } Ni =1 .

Original: Afterwards, we calculate the Euclidean distance between each pair of trajectories in {X , Y} , andobtain trajectory clusters through the K-medoids clustering algorithm [23].
Revised: Afterwards, we calculate the Euclidean distance between each pair of trajectories in {X , Y} , and obtain trajectory clusters through the K-medoids clustering algorithm [23].

Original: The initial member of the bank Z bank is the average of the trajectories belonging to the same cluster (see Algorithm 1, Step 1).
Revised: The initial member of the bank Z bank is the average of the trajectories belonging to the same cluster (see Algorithm 1, Step 1).

Original: Each trajectory in Z bank represents a moving pattern of a group of people.
Revised: Each trajectory in Z bank represents a moving pattern of a group of people.

Original: Trajectory Search and Update.
Revised: Trajectory Search and Update.

Original: In the group trajectory bank Z bank , each trajectory can be treatedas a past-future pair.
Revised: In the group trajectory bank Z bank , each trajectory can be treated as a past-future pair.

Original: Numerically, Z bank =
Revised: Numerically, Z bank =

Original: { Z i } |Z bank | i =
Revised: { Z i } |Z bank | i =

Original: =
Revised: =

Original: [Equation] ; Y i ] } |Z bank | i = , where [; ] representsthe combination of a past trajectory and a future trajectory, and |Z bank | is the number of past-futurepairs in Z bank .
Revised: [ X i ; Y i ] } |Z bank | i = , where [; ] represents the combination of a past trajectory and a future trajectory, and |Z bank | is the number of past-future pairs in Z bank .

Original: Given a trajectory Z p =
Revised: Given a trajectory Z p =

Original: [ X p ; Y p ] of person p , we use the observed X p as a key tocompute its similarity score to the past trajectories { X i } |Z bank | i = in Z bank and find a representativetrajectory Y p ′ , 1 ≤ p ′ ≤ |Z bank | according to the maximum similarity score (see Algorithm 1, Step2).
Revised: [ X p ; Y p ] of person p , we use the observed X p as a key to compute its similarity score to the past trajectories { X i } |Z bank | i = in Z bank and find a representative trajectory Y p ′ , 1 ≤ p ′ ≤ |Z bank | according to the maximum similarity score (see Algorithm 1, Step 2).

Original: The similarity function can be formulated as:
Revised: The similarity function can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: By adding the offsets O p (see Equation 2) to the representative trajectory Y p ′ , we obtain our predictedtrajectory ˆ Y p of the observed person (see Figure 2).
Revised: By adding the offsets O p (see Equation 2) to the representative trajectory Y p ′ , we obtain our predicted trajectory ˆ Y p of the observed person (see Figure 2).

Original: Although the initial trajectory bank works wellfor most cases, to improve the generalizability of the bank Z bank (see Algorithm 1, Step 3), wedecide whether to update Z bank according to the distance threshold θ .
Revised: Although the initial trajectory bank works well for most cases, to improve the generalizability of the bank Z bank (see Algorithm 1, Step 3), we decide whether to update Z bank according to the distance threshold θ .

Original: 3.3 Cross-modal Interaction Module
Revised: 3.3 Cross-modal Interaction Module

Original: This module focuses on the interaction between individual past trajectory and the environmentinformation.
Revised: This module focuses on the interaction between individual past trajectory and the environment information.

Original: It consists of two unimodal encoders to learnthe human motion and scene information,respectively, and a cross-modal transformer to model the interaction between them.
Revised: It consists of two unimodal encoders to learn human motion and scene information, respectively, and a cross-modal transformer to model their interaction.

Original: Trajectory Encoder.
Revised: Trajectory Encoder.

Original: The trajectory encoder adopts a multi-head attention structure from a transformer network [29] with N tra self-attention (SA) layers.
Revised: The trajectory encoder adopts a multi-head attention structure from a transformer network [30] with N tra self-attention (SA) layers.

Original: SA layers capture human motionacross different time steps with the size of T pas × N tra × d , and project the motion featuresfrom dimension T pas × N tra d to T pas × d , where d is the embed dimension of the trajectory encoder.
Revised: SA layers capture human motion across different time steps with the size of T pas × N tra × d , and project the motion features from dimension T pas × N tra d to T pas × d , where d is the embed dimension of the trajectory en-

Original:  Thus, we obtain the human motion representation X tra using the trajectory encoder E tra :
Revised: coder. Thus, we obtain the human motion representation X tra using the trajectory encoder E tra :

Original: X tra
Revised: X tra

Original: =
Revised: =

Original: E tra ( X p ) ∈ R T pas × d .
Revised: E tra ( X p ) ∈ R T pas × d .

Original: Scene Encoder.
Revised: Scene Encoder.

Original: Due to the compelling performance of the pretrained Swin Transformer [17] infeature representation, we adopt it as our scene encoder.
Revised: Due to the compelling performance of the pretrained Swin Transformer [17] in feature representation, we adopt it as our scene encoder.

Original: It extracts scene semantic features withthe size of N cls × h × w , where N cls ( N cls = 150 in the  scene encoder) is the number of semanticclasses, such as person, road, etc., h and w are the spatial resolution.
Revised: It extracts scene semantic features with the size of N cls × h × w , where N cls ( N cls = 150 in the pretrained scene encoder) is the number of semantic classes, such as person and road, and h and w are the spatial resolutions.

Original: In order to make the followedmodule to conveniently fusethe motion representation andthe environment information, we reshapethe semantic features from size ( N cls × h × w ) to ( N cls × hw ), and project them from dimension( N cls × hw ) to ( N cls × d ) through a Multi-Layer Perception layer.
Revised: In order to make the following module to conveniently fuse motion representation and environment information, we reshape semantic features from size ( N cls × h × w ) to ( N cls × hw ), and project them from dimension ( N cls × hw ) to ( N cls × d ) through a Multi-Layer Perception layer.

Original: As a result, we obtain the scenerepresentation X sce using the scene encoder E sce : X sce =
Revised: As a result, we obtain the scene representation X sce using the scene encoder E sce : X sce =

Original: E sce ( S p ) ∈
Revised: E sce ( S p ) ∈

Original: R N cls × d .
Revised: R N cls × d .

Original: Cross-modal Transformer.
Revised: Cross-modal Transformer.

Original: Unimodal encoders extractthe features from its own modality, however,disregarding the interaction betweenthe human motion and environment information.
Revised: Unimodal encoders extract features from its own modality, disregarding the interaction between human motion and environment information.

Original: The crossmodal transformer with N cro layers aims to refine the candidate trajectory Y p ′ (see Section 3.2)by learning this interaction.
Revised: The cross-modal transformer with N cro layers aims to refine the candidate trajectory Y p ′ (see Section 3.2) by learning this interaction.

Original: We adopt the two-stream structure: one for capturing the importanthuman motion constrained by environment information, and the other for picking out the environmentinformation related to human motion.
Revised: We adopt a two-stream structure: one for capturing the important human motion constrained by environment information, and the other for picking out the environment information related to human motion.

Original: The cross-attention (CA) layers and self-attention (SA) layersare the main components of the cross-modal transformer (see Figure 3).
Revised: The cross-attention (CA) layers and self-attention (SA) layers are the main components of the cross-modal transformer (see Figure 3).

Original: To capture the importanthuman motion affected by environment and acquire the environment information related to themotion, CA layers treat one modality as query, and the other as key and value to interact with the twomodalities.
Revised: To capture the important human motion affected by environment and acquire the environment information related to the motion, CA layers treat one modality as query, and the other as key and value to interact with the two modalities.

Original: SA layers serve to facilitate better internal connections, which calculate the similaritybetween an element (query) and other elements (key) in scene-constrained motion or motion-relevantenvironment information.
Revised: SA layers serve to facilitate better internal connections, which calculate the similarity between an element (query) and other elements (key) in scene-constrained motion or motion-relevant environment information.

Original: Thus, we obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the crossmodal transformer E cro : ˆ X tra =
Revised: Thus, we obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the crossmodal transformer E cro : ˆ X tra =

Original: E cro ( X tra , X sce ) , ˆ X sce =
Revised: E cro ( X tra , X sce ) , ˆ X sce =

Original: E cro ( X sce , X tra ) .
Revised: E cro ( X sce , X tra ) .

Original: To predict theoffsets O p between the searched trajectory Y p ′ and the ground truth trajectory Y p , we take the lastelement ( LE ) h tra ∈ R d of ˆ X tra and the output
Revised: To predict the offsets O p between the searched trajectory Y p ′ and the ground-truth trajectory Y p , we take the last element ( LE ) h tra ∈ R d of ˆ X tra and the output

Original: h sce ∈ R
Revised: h sce ∈ R

Original: d of ˆ X sce after the global pooling layer( GPL ).
Revised: d of ˆ X sce after the global pooling layer ( GPL ).

Original: Offsets O p ∈ R T fut × 2 can be formulated as follows:
Revised: Offsets O p ∈ R T fut × 2 can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: 2) where [; ] denotes the vector concatenation, and MLP is the Multi-Layer Perception layer.
Revised: where [; ] denotes the vector concatenation, and MLP is the Multi-Layer Perception layer.

Original: 3.4 Training
Revised: 3.4 Training

Original: We train the overall framework of SHENet end-to-end to minimize an objective function.
Revised: We train the overall framework of SHENet end-to-end to minimize an objective function.

Original: Duringtraining, since the scene encoder has been pretrained on ADE20K [36], we freeze its segmentation partand updatethe parameters of MLP head (see Section 3.3).
Revised: During the training, since the scene encoder has been pretrained on ADE20K [38], we freeze its segmentation part and update parameters of MLP head (see Section 3.3).

Original: Following the existing works [1, 19, 20, 33],the first regression loss function is the mean squared error (MSE) between the predicted trajectoryand the ground truth trajectory: L tra =
Revised: Following the existing works [1, 19, 20, 35], we calculate the mean squared error (MSE) between the predicted trajectory and the ground-truth trajectory on ETH/UCY datasets:

Original: [Equation]
Revised: [Equation]

Original: The second loss is the curve smoothing (CS) regression loss, which serves to reduce the impactof individual bias.
Revised: In a more challenging PAV dataset, we use the curve smoothing (CS) regression loss, which serves to reduce the impact of individual bias.

Original: It calculates the MSE after trajectory smoothing.
Revised: It calculates the MSE after trajectory smoothing.

Original: CS loss can be formulated asfollows:
Revised: CS loss can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where CS represents the function of curve smoothing [2].
Revised: where CS represents the function of curve smoothing [2].

Original: In a more challenging PAV dataset, we use

Original: L cs instead of L tra .

Original:  Experiments 
Revised:  Experiments 4.1 Experimental Setup

Original: 4.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We evaluate our method on ETH [24], UCY [13],and PAV  datasets.
Revised: We evaluate our method on ETH [24], UCY [13], PAV and Stanford Drone Dataset (SDD) [26] datasets.

Original: ETH/UCY datasetsare the benchmark commonly used for human trajectory prediction (HTP).
Revised: ETH/UCY datasets are the benchmark commonly used for human trajectory prediction (HTP).

Original: The benchmark containsvideos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2 captured at 2.5HZ.
Revised: The benchmark contains videos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2 captured at 2.5HZ.

Original: Unimodal methods [1, 10, 30, 33] only focus on the trajectory data, however,  [16, 20] need toconsider scene information.
Revised: Unimodal methods [1, 10, 31, 35] only focus on the trajectory data, however, multimodal methods [16, 20] need to consider scene information.

Original: As a result, these multimodal methods [16, 20] use fewer frames thanthe unimodal methods, since a video from UNIV is not available.
Revised: Since a video from UNIV is not available, [16, 20] use fewer frames than the unimodal methods.

Original: We follow [16] to process the dataand predict future n fut = 12 frames given the observed n pas = 8 frames.
Revised: We follow [16] to process the data and predict future n fut = 12 frames given the observed n pas = 8 frames.

Original: Compared to the ETH/UCY datasets, PAV is more challenging with diverse movement patterns. Itobtains 3 videos from the MOT15 dataset [11], captured with static cameras. The video sequencesinclude PETS09-S2L1 (PETS) [4, 11], ADL-Rundle-6 (ADL), and Venice-2 (VENICE), whichprovide sufficient trajectories for the HTP task.
Revised: Compared to the ETH/UCY datasets, PAV is more challenging with diverse movement patterns, including PETS09-S2L1 (PETS) [4, 11], ADL-Rundle-6 (ADL), and Venice-2 (VENICE), which are captured from static camera and provide sufficient trajectories for the HTP task.

Original: We divide the videos into training (80%) and testing(20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4,200 training sequences and 664/306/650testing sequences, respectively.
Revised: We divide the videos into training (80%) and testing (20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4, training sequences and 664/306/650 testing sequences, respectively.

Original: We useobserved n pas =
Revised: We use n pas =

Original: 10  frames to predict future n fut = 50frames, such that we can compare the long-term prediction results of different methods.
Revised: 10 observed frames to predict future n fut = 50 frames, such that we can compare the long-term prediction results from different methods.

Revised: Different from ETH/UCY and PAV datasets, SDD is a large-scale datatset captured on the university campus in bird’s eye view.

Original: Evaluation metrics.
Revised: It consists of multiple interacting agents ( e . g ., pedestrians, bicyclists, and cars) and diverse scenes ( e . g ., sidewalks and crossroads). Following [19], we use past 8 frames to predict future 12 frames.

Revised: More results conducted on SDD are in the the supplementary material.

Original:  For ETH and UCY datasets, we adopt the standard metrics [1, 7, 16, 20, 33] for HTP: Average Displacement Error (ADE) and Final Displacement
Revised: Evaluation Metrics. For ETH and UCY datasets, we adopt the standard metrics [1, 7, 16, 20, 35] for HTP: Average Displacement Error (ADE) and Final Displacement

Original: Error (FDE).
Revised: Error (FDE).

Original: ADE is the average L 2 error betweenthe predicted trajectories and ground truth trajectories over all time steps, and FDE is the L 2 error between the predicted trajectories and ground truth trajectories at the final time step.
Revised: ADE is the average L 2 error between predicted trajectories and groundtruth trajectories over all time steps, and FDE is the L 2 error between the predicted trajectories and ground-truth trajectories at the final time step.

Original: The trajectories in PAV are with some jittering phenomena ( e .
Revised: The trajectories in PAV are with some jittering phenomena ( e .

Original: g ., abrupt and sharp turns).
Revised: g ., abrupt and sharp turns).

Original: Thus, a reasonable prediction may produce approximately the same error as an unrealistic prediction by using traditional metrics: ADE and FDE (see Figure 7(a)).
Revised: Thus, a reasonable prediction may produce approximately the same error as an unrealistic prediction by using traditional metrics: ADE and FDE (see Figure 7(a)).

Original: In order to focus on the pattern and shape of the trajectory itself, and  reduce the impact of randomness and subjectivity, we propose the CS-Metric: CS-ADE and CS-FDE (illustrated in Figure 4).
Revised: In order to focus on the pattern and shape of the trajectory itself and also reduce the impact of randomness and subjectivity, we propose the CS-Metric: CS-ADE and CS-FDE (illustrated in Figure 4).

Original: CS-ADEis calculated as follows:
Revised: CS-ADE is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: (4) where CS is the curve smoothing function defined as the same as our L cs in Section 3.4.
Revised: where CS is the curve smoothing function defined as the same as our L cs in Section 3.4.

Original: Similar to
Revised: Similar to

Original: CS-ADE, CS-FDE calculates the final displacement error after trajectory smoothing:
Revised: CS-ADE, CS-FDE calculates the final displacement error after trajectory smoothing:

Original: [Equation]
Revised: [Equation]

Original: Figure 5 shows some samples from the training data, converting rough ground truth trajectories tosmooth ones.
Revised: Figure 5 shows some samples from the training data, converting rough ground-truth trajectories to smooth ones.

Original: Implementation Details.
Revised: Implementation Details.

Original: In SHENet, the initial size of group trajectory bank is set to |Z bank | = 32 .
Revised: In SHENet, the initial size of group trajectory bank is set to |Z bank | = 32 .

Original: Both the trajectory encoder and the scene encoder have 4 self-attention (SA) layers.
Revised: Both the trajectory encoder and the scene encoder have 4 self-attention (SA) layers.

Original: The cross-modaltransformer is with 6 SA layers and cross-attention (CA) layers.
Revised: The cross-modal transformer is with 6 SA layers and cross-attention (CA) layers.

Original: We set all the embed dimensionsto 512 .
Revised: We set all the embed dimensions to 512 .

Original: For the trajectory encoder, it learns the human motion information with size of T pas [-× 
-]
Revised: For the trajectory encoder, it learns the human motion information with size of T pas [-× 
-]×( T pas = 8 in ETH/UCY, T pas =

Original: ( T pas = 8 in ETH/UCY, T pas = 10 in PAV).
Revised: 10 in PAV).

Original: For the scene encoder, it outputs the semantic featureswith size 150 × 56 × 56 .
Revised: For the scene encoder, it outputs the semantic features with size 150 × 56 × 56 .

Original: We reshape the features from size 150 × 56 × 56 to 150 × 3136 , and projectthem from dimension 150 × 3136 to 150 × 512 .
Revised: We reshape the features from size 150 × 56 × 56 to 150 × 3136 , and project them from dimension 150 × 3136 to 150 × 512 .

Original: We train themodel for 100 epochs on 4 NVIDIA Quadro RTX 6000 GPUs and use the Adam optimizer with a fixed learning rate 1 e − 5 . More detailsare in the supplementary material.
Revised: We train the

Original: 4.2 Ablation Studies
Revised: 4.2 Ablation Studies

Original: In Table 1, we evaluate each component of SHENet, including a group trajectory bank (GTB) module,and a cross-modal interaction (CMI) module that contains a trajectory encoder (TE), a scene encoder(SE) and a cross-modal transformer (CMT). 
Revised: In Table 1, we evaluate each component of SHENet, including a group trajectory bank (GTB) module, and a cross-modal interaction (CMI) module that contains a trajectory encoder (TE), a scene encoder (SE) and a cross-modal transformer (CMT). [Table]

Original: Effect of GTB.
Revised: Effect of GTB.

Original: We first investigate the performance of GTB.
Revised: We first investigate the performance of GTB.

Original: GTB improves FDE by 21.2% on PETScompared to the CMI ( i .
Revised: GTB improves FDE by 21.2% on PETS compared to the CMI ( i .

Original: e ., TE, SE, and CMT), which is a significant improvement, emphasizing theimportance of GTB.
Revised: e ., TE, SE, and CMT), which is a significant improvement, indicating the importance of GTB.

Original: However, GTB (1-st row of Table 1) alone is not enough and  performs a littleworse than CMI.
Revised: However, GTB (1-st row of Table 1) alone is not enough and even performs a little worse than CMI.

Original: Thus, we explore the effects of various parts in CMI module.
Revised: Thus, we explore the effects of various parts in CMI module.

Original: Effect of TE and SE.
Revised: Effect of TE and SE.

Original: To evaluate the performance of TE and SE, we concatenate the trajectoryfeatures extracted from TE and the scene features from SE together (3-rd row in Table 1), performancewill improve on ADL and VENICE with smaller movements, compared to  TE alone.
Revised: To evaluate the performance of TE and SE, we concatenate the trajectory features extracted from TE and the scene features from SE together (3-rd row in Table 1), and immprove performance on ADL and VENICE with smaller movements, compared to the case of using TE alone.

Original: It indicates thatthe incorporation of environment information into trajectory prediction can improve the accuracy ofthe [Table]
Revised: This indicates that the incorporation of environment information into trajectory prediction can improve the accuracy of the results.

Original: Effect of CMT.
Revised: Effect of CMT.

Original: Compared to the third row of Table 1, CMT (4-th row in Table 1) can result ina promising improvement of the model performance.
Revised: Compared to the third row of Table 1, CMT (4-th row in Table 1) can result in a promising improvement of the model performance.

Original: Notably [Table] it performs better than the TE and
Revised: Notably, it performs better than the TE and

Original: SE concatenated on PETS, improving by 7.4% over ADE.
Revised: SE concatenated on PETS, improving by 7.4% over ADE.

Original: Compared to GTB alone, the full CMIimproves by 12.2% over ADE on average.
Revised: Compared to GTB alone, the full CMI improves by 12.2% over ADE on average.

Original: 4.3 Comparison with State-of-the-art Methods
Revised: 4.3 Comparison with State-of-the-art Methods

Revised: [Table] [Table]

Original: is adopted for evaluation.
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: On ETH/UCY datasets
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original:  we compare our model with the state-of-art methods: SS-LSTM [33],
Revised: [Table] [Table] [Table] On ETH/UCY datasets, we compare our model with the state-of-the-art methods: SS-LSTM [35],

Original: Social-STGCN [22], MANTRA [20], AgentFormer [35], YNet [19].
Revised: Social-STGCN [22], MANTRA [20], AgentFormer [37], YNet [19].

Original: The results are summarized in
Revised: The results are summarized in

Original: Table 2.
Revised: Table 2.

Original: Our model reduces the average FDE from 0.39 to 0. [Table] and achieves 7.7% improvementcompared with the state-of-the-art method, YNet.
Revised: Our model reduces the average FDE from 0.39 to 0.36, and achieves 7.7% improvement compared with the state-of-the-art method, YNet.

Original: Particularly, our model outperforms previousmethods significantly on ETH, when there are large movements of trajectories, and improves the
Revised: Particularly, our model outperforms previous methods significantly on ETH, when there are large movements of trajectories, and improves the

Original: ADE/FDE by 12.8%/15.3%, respectively.
Revised: ADE and FDE by 12.8% and 15.3%, respectively.

Original: To evaluate the performance of our model in long-term predictions, we conduct experiments on PAV,with n pas =
Revised: To evaluate the performance of our model in long-term predictions, we conduct experiments on PAV, with n pas =

Original: 10 observed frames and n fut = 50 future frames of each trajectory.
Revised: 10 observed frames and n fut = 50 future frames of each trajectory.

Original: Table 3 shows theperformance compared with previous HTP methods: SS-LSTM [33], Social-STGCN [22], Next [16],
Revised: Table 3 shows the performance compared with previous HTP methods: SS-LSTM [35], Social-STGCN [22], Next [16],

Original: MANTRA [20], YNet [19].
Revised: MANTRA [20], YNet [19].

Original: Compared to the state-of-the-art results of YNet, the proposed SHENetachieves repective 3.3%/10.5% improvement over CS-ADE/CS-FDE on average.
Revised: Compared to the state-of-the-art results of YNet, the proposed SHENet achieves respective 3.3% and 10.5% improvement over CS-ADE and CS-FDE on average.

Original: Since YNet predictsthe heatmaps of a trajectory, it performs a little better when there are small movements of trajectories, e .
Revised: Since YNet predicts the heatmaps of a trajectory, it performs a little better when there are small movements of trajectories, e .

Original: g .,
Revised: g .,

Original: VENICE.
Revised: VENICE.

Original: Nevertheless, our method is still competitive in VENICE, and is much better than others approaches on PETS with large movements andintersections.
Revised: Nevertheless, our method is still competitive in VENICE, and is much better than others approaches on PETS with large movements and

Original: [Table]
Revised: [Table]

Original:  In particular, our method improves the CS-FDE by 16.2% on PETS, compared to YNet.
Revised: intersections. In particular, our method improves the CS-FDE by 16.2% on PETS, compared to YNet.

Original: We alsoachieve great improvements using the traditional ADE/FDE metrics.
Revised: We also achieve great improvements using the traditional ADE/FDE metrics.

Original: The complete results are shownin the supplementary material.4.4 Qualitative Analysis.
Revised: The complete results are shown in the supplementary material. 4.4 Analysis Distance Threshold θ .

Original: [Figure]
Revised: θ is used to determine the update of trajectory bank. The typical value of θ is set according to the trajectory length. When the ground-truth trajectory is longer in terms of pixel, the absolute value of prediction error is usually larger. However, their relative errors are comparable.

Revised: Thus, the θ is set to be 75% of the training error when the error converges.

Original: Figure 6 presents the qualitative results of SHENet and other methods.
Revised: In our experiments, we set θ =in PETS, and θ = 6 in ADL. The "75% of the [Table] training error" is obtained from the experimental result, as shown in Table 4.

Revised: Cluster Number in K-medoids.

Revised: We study the effect of setting different number of initial clusters K , as shown in Table 5 .

Revised: We can note that the initial number of clusters is not sensitive to prediction results, especially when the initial number of clusters is 24-36.

Revised: Therefore, we can set K to 32 in our experiments.

Revised: [Table]

Revised: Analysis for Bank Complexity.

Revised: The time complexity of the searching and updating are O ( N ) and O (1) .

Revised: Their space complexity is O ( N ) .

Revised: The group trajectory number N ≤ 1000 .

Revised: The time complexity of the clustering process is O ( β 2 + MKβ ) , and the space complexity is O ( β 2 + Mβ ) .

Revised: β is the number of trajectories for clustering.

Revised: K is the number of clusters, and M is the number of iterations for the clustering methods.

Revised: 4.5 Qualitative Results.

Original:  By comparison, we surprisinglynotice that in an extremely challenging situation where a person walks to the curb and turns back(green curve), all other methods do not deal with it well, while the proposed SHENet can still handleit.
Revised: Figure 6 presents the qualitative results of SHENet and other methods. By comparison, we surprisingly notice that in an extremely challenging situation where a person walks to the curb and turns back (green curve), all other methods do not deal with it well, while our proposed SHENet can still handle it.

Original: This should be attributed to the effect of our dedicatedly designed historical group trajectory bankmodule.
Revised: This should be attributed to the effect of our dedicatedly designed historical group trajectory bank module.

Original: In addition, compared to the memory -based method MANTRA [20], we search trajectories of groups instead of just individuals.
Revised: In addition, compared to the memory -based method MANTRA [20], we search trajectories of groups, instead of just individuals.

Original: This is more versatile and can be applied in more challenging scenes.
Revised: This is more versatile and can be applied in more challenging scenes.

Original: Figureincludes the qualitative results of YNet and our SHENet without/with curve smoothing (CS).
Revised: Figureincludes the qualitative results of YNet and our SHENet without/with curve smoothing (CS).

Original: The first row presents the results of using MSE loss L tra .
Revised: The first row presents the results of using MSE loss L tra .

Original: Influenced by the past trajectories with some noise ( e .
Revised: Influenced by the past trajectories with some noise ( e .

Original: g ., abrupt and sharp turns), the predicted trajectory points of YNet are gathered together, and cannot present a clear direction, while our method can provide a potential path based on the historical group trajectories.
Revised: g ., abrupt and sharp turns), the predicted trajectory points of YNet are gathered together, and cannot present a clear direction, while our method can provide a potential path based on the historical group trajectories.

Original: The two predictions are visually distinct, however, the numerical errors (ADE/FDE) are approximately the same.
Revised: The two predictions are visually distinct, however, the numerical errors (ADE/FDE) are approximately the same.

Original: In contrast, the qualitative results of our proposed CS loss L cs are shown in the second row of Figure 7.
Revised: In contrast, the qualitative results of our proposed CS loss L cs are shown in the second row of Figure 7.

Original: We can see that the proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by YNet and our method.
Revised: We can see that our proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by YNet and our method.

Original: Morevisualization results are provided in supplementary material.
Revised: More visualization results are provided in supplementary material.

Original:  Conclusion
Revised:  Conclusion

Original: The paper presents SHENet, a  method that fully utilizes scene history for HTP.
Revised: The paper presents SHENet, a novel method that fully utilizes scene history for HTP.

Original: SHENet includesa GTB module to construct a group trajectory bank from all historical trajectories and retrieve arepresentative trajectory from this bank for an observed person, and  a CMI module, which interactsbetween human motion and environment information, to refine this representative trajectory.
Revised: SHENet includes a GTB module to construct a group trajectory bank from all historical trajectories and retrieve a representative trajectory from this bank for an observed person, and also a CMI module (which interacts between human motion and environment information) to refine this representative trajectory.

Original: Weachieve the SOTA performance on the HTP benchmarks and our method demonstrates significantimprovements and generalizaibility in the challenging scenes.
Revised: We achieve the SOTA performance on the HTP benchmarks and our method demonstrates significant improvements and generalizaibility in the challenging scenes.

Original: However, there are  several unexploredaspects in the current framework, e.g., the process of bank construction currently focuses on only human motion.
Revised: However, there are still several unexplored aspects in the current framework, e.g., the process of bank construction currently focusing on only human motion.

Original: Future work includes further exploring the trajectory bank using the interact information(human motion and scene information). 
Revised: Future work includes further exploring the trajectory bank using the interact information (human motion and scene information). Acknowledgments and Disclosure of Funding

Revised: This work is supported by the National Key R&D Program of China (2021ZD0111100) and sponsored by Shanghai Rising-Star Program.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] • Did you include the license to the code and datasets?
Revised: [Yes] • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] All data used in the paper is public; our code are provided (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] All data used in the paper is public; our code are provided (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes](c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] public datasets (e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
Revised: [N/A] public datasets (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] public datasets 
Revised: [N/A] public datasets 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A](b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix
Revised: Supplementary Material

Revised: Section A provides additional details for the method.

Original: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
Revised: The experiment details are described in Section B, including experimental setup and results. We also present more qualitative results in Section C. A Method details A.1 Cross-modal Interaction

Revised: Trajectory Encoder.

Original: This section will often be part of  the supplemental material.
Revised: The trajectory encoder aims to learn the human motion representation via a transformer structure. The observed trajectory of person p is X p with the size of T pas × 2 ( e . g ., T pas = 10 in PAV).

Revised: We obtain the human motion with the size of 10 × 4 × 512 using N tra

Revised: =self-attention layers, where 512 is the embed dimension.

Revised: We reshape the motion features from size 10 × 4 × 512 to 10 × 2048 and project it from dimension 10 × 2048 to 10 × 512 .

Revised: Thus, we obtain the human motion representation X tra on PAV dataset with the size of 10 × 512 .

Revised: Scene Encoder.

Revised: The scene encoder is to extract the environment information.

Revised: The scene semantic features are with the size of 150 × 56 × 56 , where 150 is the number of semantic classes.

Revised: We reshape the semantic features from size ( 150 × 56 × 56 ) to ( 150 × 3 , 136 ), and project them from dimension ( 150 × 3 , 136 ) to ( 150 × 512 ) through a Multi-Layer Perception layer.

Revised: As a result, we obtain the scene representation X sce with the size of 150 × 512 .

Revised: Cross-modal Transformer.

Revised: The cross-modal transformer with N cro = 6 layers aims to learn the interaction between human motion and environment information.

Revised: We obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the cross-modal transformer.

Revised: The size of ˆ X tra and ˆ X sce are 10 ×and 150 × 512 , respectively.

Revised: We take the last element ( LE ) h tra ∈ R 512 of ˆ X tra and the output h sce ∈ R 512 of ˆ X sce after the global pooling layer ( GPL ), and get the offsets O p ∈ R 50 ×

Revised: ( T fut = 50 in PAV) through a Multi-Layer Perception layer.

Revised: A.2 Training

Revised: We use the curve smoothing (CS) regression loss L cs to reduce the impact of randomness and subjectivity.

Revised: The quadratic Bézier curve is adopted to smooth the trajectory, which can be formulated as follows:

Revised: [Equation]

Revised: where z − T pas +1 p is the starting point of the trajectory, and z T fut p is the destination of the trajectory.

Revised: z T ′ p is the control point of this trajectory, which can be calculated as follows:

Revised: [Equation]

Revised: Then we divide the time period t ∈ [0 , 1] equidistantly, and get the smoothed trajectory Z p =

Revised: [ X p ; Y p ] = { z tp } T fut t = − T pas +1 .

Revised: B Experiments B.1 Experimental Setup

Revised: Dataset Details.

Revised: ETH/UCY datasets are the benchmark commonly used for human trajectory prediction.

Revised: The benchmark contains videos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2.

Revised: Following [18], we sample the frames at 2.5 HZ and predict future n fut = 12 frames given the observed n pas = 8 frames.

Revised: We use the preprocessed data provided by YNet [18], which converts the raw data from word coordinate into image pixel space.

Revised: We use the leave-one-scene-out strategy with 4 scenes for training and the remaining scene for testing.

Revised: PAV is a more challenging dataset with diverse movement patterns, which includes 3 videos PETS, ADL, and VENICE.

Revised: We divide the videos into training (80%) and testing (20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4,200 training sequences and 664/306/650 testing sequences, respectively.

Revised: We use observed n pas =

Revised: 10 frames to predict future n fut = 50 frames.

Revised: Evaluation Metrics.

Revised: For ETH and UCY datasets, we adopt the standard metrics ( i .

Revised: e ., ADE and FDE).

Revised: Due to the limitations discussed in Section 4.1, we introudce curve smoothing (CS) into current metrics on PAV dataset, and thus we propose CS-ADE and CS-FDE.

Revised: The curve smoothing function is defined as the same as Equation 6.

Revised: B.2 Experimental Results

Revised: PAV without CS.

Revised: We conduct experiments on PAV using the traditional ADE/FDE metrics.

Revised: Tableshows the quantitative result of our method and previous human trajectory prediction methods.

Revised: Compared to the state-of-the-art results of YNet, the proposed SHENet achieves 5.1%/4.0% improvement over ADE/FDE on average.

Revised: In particular, our method improves the FDE by 13.6% on PETS.

Revised: [Table]

Revised: Social-STGCN [22] 63.76 / 159.

Revised: Next [16] 51.78 / 109.

Revised: MANTRA [20]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: YNet [19] 53.46 /

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: SHENet (Ours) 46.31 / 101.

Revised: ETH/UCY without Video Data.

Revised: We also conduct the experiments on ETH/UCY without using the video data, shown in Table 7.

Revised: Since MANTRA didn’t conduct experiments on ETH/UCY, we use the results of MANTRA reported in the work [32].

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: MemoNet [32] 0.40 / 0.

Revised: SHENet (Ours) 0.37 / 0.

Revised: From Table 7, we can note that our method achieves the comparable performance without using the video data.

Revised: SDD.

Revised: We also report the performance of different methods on SDD in Table 8.

Revised: It shows that our model performs a little bit worse than the results of YNet [19] and MemoNet [32].

Revised: Nevertheless, our

Revised: [Table]

Revised: method performs better than previous baselines (such as PECNet [18], MANTRA [32]).

Revised: Consequently, our method can achieve reasonable performance in bird-eye-view scenario.

Revised: C Qualitative Results

Revised: Figure 8 presents the visualization of clustered trajectories.

Revised: We can see that similar trajectories are clustered into the same group.

Revised: Each group trajectory represents a moving pattern of a group of people.

Revised: Figure 9 illustrates the qualitative results of SHENet and other methods.

Revised: Our method is superior to others when it comes to human crossing intersections ( e .

Revised: g ., the first row in Figure 9) or turning ( e .

Revised: g ., the second row in Figure 9).

Revised: Figure 10 includes the qualitative results of our SHENet without/with curve smoothing (CS).

Revised: The first row presents the results of using MSE loss L tra .

Revised: We can note that our SHENet can not provide a correct path from scene history, since there exists some noise ( e .

Revised: g ., abrupt and sharp turns) in past trajectories.

Revised: Thus to reduce the impact of noise, we use the CS loss

Revised: L cs to train our model.

Revised: In contrast, the qualitative results of using L cs are shown in the second row of Figure 10.

Revised: We can see that the proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by our method.

Revised: [Figure]