Original: Forecasting Human Trajectory from Scene History
Revised: Forecasting Human Trajectory from Scene History

Original: Abstract
Revised: Abstract

Original: Predicting the future trajectory of a person remains a challenging problem, due to randomness and subjectivity of human movement.
Revised: Predicting the future trajectory of a person remains a challenging problem, due to randomness and subjectivity of human movement.

Original: However, the moving patterns of human in a constrained scenario typically conform to a limited number of regularities to a certain extent, because of the scenario restrictions ( e .
Revised: However, the moving patterns of human in a constrained scenario typically conform to a limited number of regularities to a certain extent, because of the scenario restrictions ( e .

Original: g ., floor plan, roads, and obstacles) and person-person or person-object interactivity.
Revised: g ., floor plan, roads, and obstacles) and person-person or person-object interactivity.

Original: Thus, an individual person in this scenario should follow one of the regularities as well.
Revised: Thus, an individual person in this scenario should follow one of the regularities as well.

Original: In other words, a person’s subsequent trajectory has likely been traveled by others.
Revised: In other words, a person’s subsequent trajectory has likely been traveled by others.

Original: Based on this hypothesis, we propose to forecast a person’s future trajectory by learning from the implicit scene regularities.
Revised: Based on this hypothesis, we propose to forecast a person’s future trajectory by learning from the implicit scene regularities.

Original: We call the regularities, inherently derived from the past dynamics of the people and the environment in the scene, scene history .
Revised: We call the regularities, inherently derived from the past dynamics of the people and the environment in the scene, scene history .

Original: We categorize scene history information into two types: historical group trajectory and individual-surroundings interaction.
Revised: We categorize scene history information into two types: historical group trajectory and individual-surroundings interaction.

Original: To exploit these two types of information for trajectory prediction, we propose a novel framework Scene
Revised: To exploit these two types of information for trajectory prediction, we propose a novel framework Scene

Original: History Excavating Network (SHENet), where the scene history is leveraged in a simple yet effective approach.
Revised: History Excavating Network (SHENet), where the scene history is leveraged in a simple yet effective approach.

Original: In particular, we design two components: the group trajectory bank module to extract representative group trajectories as the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement.
Revised: In particular, we design two components: the group trajectory bank module to extract representative group trajectories as the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement.

Original: In addition, to mitigate the uncertainty in ground-truth trajectory, caused by the aforementioned randomness and subjectivity of human movement, we propose to include smoothness into the training process and evaluation metrics.
Revised: In addition, to mitigate the uncertainty in ground-truth trajectory, caused by the aforementioned randomness and subjectivity of human movement, we propose to include smoothness into the training process and evaluation metrics.

Original: We conduct extensive evaluations to validate the efficacy of our proposed framework on ETH, UCY, as well as a new, challenging benchmark dataset PAV, demonstrating superior performance compared to state-of-the-art methods.
Revised: We conduct extensive evaluations to validate the efficacy of our proposed framework on ETH, UCY, as well as a new, challenging benchmark dataset PAV, demonstrating superior performance compared to state-of-the-art methods.

Original: Code is available at: https:/
Revised: Code is available at: https:/

Original:  Introduction
Revised:  Introduction

Original: Human trajectory prediction (HTP) aims to predict a target person’s future path from a video clip [5, 9, 19, 20].
Revised: Human trajectory prediction (HTP) aims to predict a target person’s future path from a video clip [5, 9, 19, 20].

Original: This is critical for intelligent transportation, as it enables vehicle to perceive the status of pedestrian in advance so that it can avoid potential collision.
Revised: This is critical for intelligent transportation, as it enables vehicle to perceive the status of pedestrian in advance so that it can avoid potential collision.

Original: Surveillance systems with HTP capability can assist security officers to predict the possible escape path of suspects.
Revised: Surveillance systems with HTP capability can assist security officers to predict the possible escape path of suspects.

Original: Although much work has been done in recent years [1, 9, 12, 14, 18, 35], very few are reliable and generalizable enough to be applied in real-world scenarios, primarily due to two challenges of the task: randomness and subjectivity of human movement.
Revised: Although much work has been done in recent years [1, 9, 12, 14, 18, 35], very few are reliable and generalizable enough to be applied in real-world scenarios, primarily due to two challenges of the task: randomness and subjectivity of human movement.

Original: However, in the constrained real world scenarios, the challenges are not absolutely intractable.
Revised: However, in the constrained real world scenarios, the challenges are not absolutely intractable.

Original: As shown in Figure 1, given previously-captured video in this scene, the target person’s future trajectory (red box) becomes more predictable as the moving pattern of human typically complies with several underlying regularities in this scenario, which the target person would follow.
Revised: As shown in Figure 1, given previously-captured video in this scene, the target person’s future trajectory (red box) becomes more predictable as the moving pattern of human typically complies with several underlying regularities in this scenario, which the target person would follow.

Original: Therefore, to predict the trajectory, we first need to understand these regularities.
Revised: Therefore, to predict the trajectory, we first need to understand these regularities.

Original: We argue that these regularities are implicitly encoded in historical human trajectories (Figure 1 left), individual past trajectory, surroundings, and interaction between them (Figure 1 right), which we refer to as scene history .
Revised: We argue that these regularities are implicitly encoded in historical human trajectories (Figure 1 left), individual past trajectory, surroundings, and interaction between them (Figure 1 right), which we refer to as scene history .

Original: We separate historical information into two categories: historical group trajectories (HGT) and individual-surroundings interaction (ISI).
Revised: We separate historical information into two categories: historical group trajectories (HGT) and individual-surroundings interaction (ISI).

Original: HGT refers to the group representatives of all historical trajectories in a scene.
Revised: HGT refers to the group representatives of all historical trajectories in a scene.

Original: The reason of resorting to HGT is that, given a new target person in the scenario, his/her path is more likely to share more similarity with one of the group trajectories than any individual instance of the historical trajectories, due to the aforementioned randomness, subjectivity, and regularities.
Revised: The reason of resorting to HGT is that, given a new target person in the scenario, his/her path is more likely to share more similarity with one of the group trajectories than any individual instance of the historical trajectories, due to the aforementioned randomness, subjectivity, and regularities.

Original: Nevertheless, the group trajectories are less correlated to the individual’s past status and the corresponding surroundings, influencing the person’s future trajectory as well.
Revised: Nevertheless, the group trajectories are less correlated to the individual’s past status and the corresponding surroundings, influencing the person’s future trajectory as well.

Original: ISI is required to more comprehensively leverage history information by extracting contextual information.
Revised: ISI is required to more comprehensively leverage history information by extracting contextual information.

Original: Few existing methods have taken into account similarity between individual past trajectory and historical trajectories.
Revised: Few existing methods have taken into account similarity between individual past trajectory and historical trajectories.

Original: Most attempts [16, 19, 27, 35] explore only individual-surroundings interaction, where tremendous efforts are spent on modeling the individual trajectory, the semantic information of environment information, and the relationship between them.
Revised: Most attempts [16, 19, 27, 35] explore only individual-surroundings interaction, where tremendous efforts are spent on modeling the individual trajectory, the semantic information of environment information, and the relationship between them.

Original: Although MANTRA [20] models the similarity using the encoders trained in a reconstruction manner and MemoNet [32] simplifies the similarity by storing the intention of historical trajectories, they both perform similarity computation on instance level, instead of group level, making it sensitive to the capability of the trained encoder.
Revised: Although MANTRA [20] models the similarity using the encoders trained in a reconstruction manner and MemoNet [32] simplifies the similarity by storing the intention of historical trajectories, they both perform similarity computation on instance level, instead of group level, making it sensitive to the capability of the trained encoder.

Original: Based on the above analysis, we propose a simple yet effective framework, Scene History Excavating
Revised: Based on the above analysis, we propose a simple yet effective framework, Scene History Excavating

Original: Network (SHENet), to leverage HGT and ISI jointly for HTP.
Revised: Network (SHENet), to leverage HGT and ISI jointly for HTP.

Original: In particular, the framework consists of two major components: (i) a group trajectory bank (GTB) module, and (ii) a cross-modal interaction (CMI) module.
Revised: In particular, the framework consists of two major components: (i) a group trajectory bank (GTB) module, and (ii) a cross-modal interaction (CMI) module.

Original: GTB constructs the representative group trajectories from all historical individual trajectories and provides a candidate path for future trajectory prediction.
Revised: GTB constructs the representative group trajectories from all historical individual trajectories and provides a candidate path for future trajectory prediction.

Original: CMI encodes the observed individual trajectory and the surroundings separately, and models their interaction using a cross-modal transformer for refinement of the searched candidate trajectory.
Revised: CMI encodes the observed individual trajectory and the surroundings separately, and models their interaction using a cross-modal transformer for refinement of the searched candidate trajectory.

Original: In addition, to alleviate the uncertainty from two mentioned characteristics ( i .
Revised: In addition, to alleviate the uncertainty from two mentioned characteristics ( i .

Original: e ., randomness and subjectivity), we introduce curve smoothing (CS) into the training process and current evaluation metrics, average and final displacement errors
Revised: e ., randomness and subjectivity), we introduce curve smoothing (CS) into the training process and current evaluation metrics, average and final displacement errors

Original: ( i .
Revised: ( i .

Original: e ., ADE and FDE), resulting in two new metrics
Revised: e ., ADE and FDE), resulting in two new metrics

Original: CS-ADE and CS-FDE.
Revised: CS-ADE and CS-FDE.

Original: Moreover, to facilitate the development of research in HTP, we collect a new challenging dataset with diverse movement patterns, named PAV.
Revised: Moreover, to facilitate the development of research in HTP, we collect a new challenging dataset with diverse movement patterns, named PAV.

Original: The dataset is obtained by choosing the videos with fixed camera view and complex human movement from MOT15 dataset [11].
Revised: The dataset is obtained by choosing the videos with fixed camera view and complex human movement from MOT15 dataset [11].

Original: The contributions of the work can be summarized as follows: 1) We introduce group history to search individual trajectory for HTP.
Revised: The contributions of the work can be summarized as follows: 1) We introduce group history to search individual trajectory for HTP.

Original: We propose a simple and effective framework, SHENet, to jointly utilize two types of scene history ( i .
Revised: We propose a simple and effective framework, SHENet, to jointly utilize two types of scene history ( i .

Original: e ., historical group trajectories and individual-surroundings interaction) for HTP.
Revised: e ., historical group trajectories and individual-surroundings interaction) for HTP.

Original: We construct a new challenging dataset, PAV; also, one novel loss function and two new metrics, considering randomness and subjectivity in human moving patterns, are proposed to achieve better benchmark HTP performance.
Revised: We construct a new challenging dataset, PAV; also, one novel loss function and two new metrics, considering randomness and subjectivity in human moving patterns, are proposed to achieve better benchmark HTP performance.

Original: We conduct comprehensive experiments on
Revised: We conduct comprehensive experiments on

Original: ETH, UCY, and PAV to demonstrate the superior performance of SHENet and the efficacy of each component.
Revised: ETH, UCY, and PAV to demonstrate the superior performance of SHENet and the efficacy of each component.

Original:  Related Work
Revised:  Related Work

Original: Unimodal Methods.
Revised: Unimodal Methods.

Original: Unimodal methods rely on learning the regularity of individual motion from past trajectories in order to predict future trajectories.
Revised: Unimodal methods rely on learning the regularity of individual motion from past trajectories in order to predict future trajectories.

Original: For example, Social LSTM [1] focuses on modeling the interaction between individual trajectories through a social pooling module.
Revised: For example, Social LSTM [1] focuses on modeling the interaction between individual trajectories through a social pooling module.

Original: STGAT [8] uses an attention module to learn spatial interactions and assign reasonable importance to neighbors.
Revised: STGAT [8] uses an attention module to learn spatial interactions and assign reasonable importance to neighbors.

Original: PIE [25] uses a temporal attention module to calculate the importance of observed trajectories at each time step.
Revised: PIE [25] uses a temporal attention module to calculate the importance of observed trajectories at each time step.

Original: Giuliari et al.
Revised: Giuliari et al.

Original: [6] use the Transformer networks [3, 30] to predict the trajectories of individual people, which weight all observed points in the trajectories according to an attention mechanism.
Revised: [6] use the Transformer networks [3, 30] to predict the trajectories of individual people, which weight all observed points in the trajectories according to an attention mechanism.

Original: Xu et al.
Revised: Xu et al.

Original: [34] propose a newly designed graph neural network to extract spatial-temporal feature representations of the observed trajectories, along with a knowledge learning module to refine individual-level transferable representations.
Revised: [34] propose a newly designed graph neural network to extract spatial-temporal feature representations of the observed trajectories, along with a knowledge learning module to refine individual-level transferable representations.

Original: These methods consider the scene history in two aspects: the individual’s past trajectory and the interaction between the individual’s past trajectory and its neighboring trajectories.
Revised: These methods consider the scene history in two aspects: the individual’s past trajectory and the interaction between the individual’s past trajectory and its neighboring trajectories.

Original: Multimodal Methods.
Revised: Multimodal Methods.

Original: Additionally, multimodal methods examine the influence of environment information on HTP.
Revised: Additionally, multimodal methods examine the influence of environment information on HTP.

Original: SS-LSTM [35] proposes a scene interaction module to capture global information of the scene.
Revised: SS-LSTM [35] proposes a scene interaction module to capture global information of the scene.

Original: Trajectron++ [27] uses a graph structure to model the trajectories and interacts with environmental information and other individuals.
Revised: Trajectron++ [27] uses a graph structure to model the trajectories and interacts with environmental information and other individuals.

Original: Zhou et al.
Revised: Zhou et al.

Original: [39] propose a context encoder to extract local features for each agent, and a global interaction module to aggregate the local context of individual agents.
Revised: [39] propose a context encoder to extract local features for each agent, and a global interaction module to aggregate the local context of individual agents.

Original: Liang et al.
Revised: Liang et al.

Original: [16] integrate the interactive modules into a person interaction module (person-person, person-object, and person-scene), and add a person behavior module (person appearance) to encode rich visual information.
Revised: [16] integrate the interactive modules into a person interaction module (person-person, person-object, and person-scene), and add a person behavior module (person appearance) to encode rich visual information.

Original: Different from the RNNs [1, 16, 33, 35] to encode the past trajectories into a hidden state vector, which is prone to loss of information, MANTRA [20] utilizes an external memory to model long-term dependencies.
Revised: Different from the RNNs [1, 16, 33, 35] to encode the past trajectories into a hidden state vector, which is prone to loss of information, MANTRA [20] utilizes an external memory to model long-term dependencies.

Original: It stores the historical single-agent trajectories in a memory, and encodes the environment information to refine a searched trajectory from this memory.
Revised: It stores the historical single-agent trajectories in a memory, and encodes the environment information to refine a searched trajectory from this memory.

Original: Difference from Previous Works.
Revised: Difference from Previous Works.

Original: Both unimodal and multimodal methods use single or partial aspects of the scene history, disregarding the historical group trajectory.
Revised: Both unimodal and multimodal methods use single or partial aspects of the scene history, disregarding the historical group trajectory.

Original: In our work, we integrate the information from the scene history in a more comprehensive way and propose dedicated modules to handle different types of information, respectively.
Revised: In our work, we integrate the information from the scene history in a more comprehensive way and propose dedicated modules to handle different types of information, respectively.

Original: The main differences between our method and the previous works, especially memory-based methods [15, 20, 21, 32] and clustering-based methods [28, 29, 36], are as follows: i) MANTRA [20] and MemoNet [32] consider historical individual trajectories, while our proposed SHENet focuses on historical group trajectories, which is more generalizable in different scenarios;
Revised: The main differences between our method and the previous works, especially memory-based methods [15, 20, 21, 32] and clustering-based methods [28, 29, 36], are as follows: i) MANTRA [20] and MemoNet [32] consider historical individual trajectories, while our proposed SHENet focuses on historical group trajectories, which is more generalizable in different scenarios;

Original: In [15], historical trajectories are not stored for search.
Revised: In [15], historical trajectories are not stored for search.

Original: Calculate distance d predicted p ground
Revised: Calculate distance d predicted p ground

Original: Y p 17: if d θ then 18:
Revised: Y p 17: if d θ then 18:

Original: Update
Revised: Update

Original: Z bank by adding trajectory p 20: end
Revised: Z bank by adding trajectory p 20: end

Original: 21: if β then 22:
Revised: 21: if β then 22:

Original: Cluster newly added trajectories in Z bank , and merge the clustered trajectories into Z bank 24: end if 25: return Z bank ii) [28, 36] group person-neighbors for trajectory prediction; [29] clusters the trajectories into fixed number of categories for trajectory classification; Our SHENet generates representative trajectories as reference for individual person trajectory prediction.
Revised: Cluster newly added trajectories in Z bank , and merge the clustered trajectories into Z bank 24: end if 25: return Z bank ii) [28, 36] group person-neighbors for trajectory prediction; [29] clusters the trajectories into fixed number of categories for trajectory classification; Our SHENet generates representative trajectories as reference for individual person trajectory prediction.

Original:  Method 3.1 Overview
Revised:  Method 3.1 Overview

Original: The architecture of the proposed Scene History Excavating Network (SHENet) is depicted in Figure 2, which consists of two major components: group trajectory bank module (GTB) and cross-modal interaction module (CMI).
Revised: The architecture of the proposed Scene History Excavating Network (SHENet) is depicted in Figure 2, which consists of two major components: group trajectory bank module (GTB) and cross-modal interaction module (CMI).

Original: Formally, given all the trajectories Z in observed videos of this scene, a scene image S p and a past trajectory of a target person p in the last T pas time steps, X p = { x tp } 0 t = − T pas +1 ∈ R T pas × 2 , where x tp ∈ R 2 represents the position of p th person at time step t ,
Revised: Formally, given all the trajectories Z in observed videos of this scene, a scene image S p and a past trajectory of a target person p in the last T pas time steps, X p = { x tp } 0 t = − T pas +1 ∈ R T pas × 2 , where x tp ∈ R 2 represents the position of p th person at time step t ,

Original: SHENet requires to predict the future positions of the person ˆ Y p
Revised: SHENet requires to predict the future positions of the person ˆ Y p

Original: = { ˆ y tp } T fut t =1 ∈ R T fut × 2 in the next
Revised: = { ˆ y tp } T fut t =1 ∈ R T fut × 2 in the next

Original: T fut frames, such that ˆ Y p is as close to the ground-truth trajectory Y p
Revised: T fut frames, such that ˆ Y p is as close to the ground-truth trajectory Y p

Original: = { y tp } T fut t =1 ∈ R T fut × 2 as possible.
Revised: = { y tp } T fut t =1 ∈ R T fut × 2 as possible.

Original: The proposed GTB first condenses
Revised: The proposed GTB first condenses

Original: Z as representative group trajectories Z bank .
Revised: Z as representative group trajectories Z bank .

Original: Then, the observed trajectory X p is used as a key to search a closet representative group trajectory in Z bank as a candidate future trajectory, Y p ′ .
Revised: Then, the observed trajectory X p is used as a key to search a closet representative group trajectory in Z bank as a candidate future trajectory, Y p ′ .

Original: Meanwhile, the past trajectory X p and a scene image S p are separately forwarded to the trajectory encoder and the scene encoder to yield the trajectory feature and the scene feature, respectively.
Revised: Meanwhile, the past trajectory X p and a scene image S p are separately forwarded to the trajectory encoder and the scene encoder to yield the trajectory feature and the scene feature, respectively.

Original: The encoded features are fed into the cross-modal transformer to learn the offsets O p between Y p ′ and the ground-truth trajectory Y p .
Revised: The encoded features are fed into the cross-modal transformer to learn the offsets O p between Y p ′ and the ground-truth trajectory Y p .

Original: By adding O p to Y p ′ , we obtain our final prediction ˆ Y p .
Revised: By adding O p to Y p ′ , we obtain our final prediction ˆ Y p .

Original: In the training phase, if the distance of the Y p to ˆ Y p is higher than a threshold, the trajectory of the person ( i .
Revised: In the training phase, if the distance of the Y p to ˆ Y p is higher than a threshold, the trajectory of the person ( i .

Original: e ., X p and Y p ) will be added into the trajectory bank Z bank .
Revised: e ., X p and Y p ) will be added into the trajectory bank Z bank .

Original: After training, the bank is fixed for inference.
Revised: After training, the bank is fixed for inference.

Original: 3.2 Group Trajectory Bank Module
Revised: 3.2 Group Trajectory Bank Module

Original: The group trajectory bank module (GTB) is used to construct the representative group trajectories in the scene.
Revised: The group trajectory bank module (GTB) is used to construct the representative group trajectories in the scene.

Original: The core functions of GTB are bank initialization, trajectory search, and trajectory update.
Revised: The core functions of GTB are bank initialization, trajectory search, and trajectory update.

Original: Trajectory Bank Initialization.
Revised: Trajectory Bank Initialization.

Original: Due to redundancy of the large amount of recorded trajectories, instead of naively using them, we generate a set of sparse and representative ones as the initial members of the trajectory bank.
Revised: Due to redundancy of the large amount of recorded trajectories, instead of naively using them, we generate a set of sparse and representative ones as the initial members of the trajectory bank.

Original: Specifically, we denote the trajectories in the training data as
Revised: Specifically, we denote the trajectories in the training data as

Original: Z = { Z } Ni =1 and split each Z
Revised: Z = { Z } Ni =1 and split each Z

Original: i into a pair of observed trajectory X i and future trajectory
Revised: i into a pair of observed trajectory X i and future trajectory

Original: Y
Revised: Y

Original: i , so that Z is separated into an observed set X
Revised: i , so that Z is separated into an observed set X

Original: = { X } Ni =1 and a corresponding future set Y =
Revised: = { X } Ni =1 and a corresponding future set Y =

Original: { Y } Ni =1 .
Revised: { Y } Ni =1 .

Original: Afterwards, we calculate the Euclidean distance between each pair of trajectories in {X , Y} , and obtain trajectory clusters through the K-medoids clustering algorithm [23].
Revised: Afterwards, we calculate the Euclidean distance between each pair of trajectories in {X , Y} , and obtain trajectory clusters through the K-medoids clustering algorithm [23].

Original: The initial member of the bank Z bank is the average of the trajectories belonging to the same cluster (see Algorithm 1, Step 1).
Revised: The initial member of the bank Z bank is the average of the trajectories belonging to the same cluster (see Algorithm 1, Step 1).

Original: Each trajectory in Z bank represents a moving pattern of a group of people.
Revised: Each trajectory in Z bank represents a moving pattern of a group of people.

Original: Trajectory Search and Update.
Revised: Trajectory Search and Update.

Original: In the group trajectory bank Z bank , each trajectory can be treated as a past-future pair.
Revised: In the group trajectory bank Z bank , each trajectory can be treated as a past-future pair.

Original: Numerically, Z bank =
Revised: Numerically, Z bank =

Original: { Z i } |Z bank | i =
Revised: { Z i } |Z bank | i =

Original: =
Revised: =

Original: [ X i ; Y i ] } |Z bank | i = , where [; ] represents the combination of a past trajectory and a future trajectory, and |Z bank | is the number of past-future pairs in Z bank .
Revised: [ X i ; Y i ] } |Z bank | i = , where [; ] represents the combination of a past trajectory and a future trajectory, and |Z bank | is the number of past-future pairs in Z bank .

Original: Given a trajectory Z p =
Revised: Given a trajectory Z p =

Original: [ X p ; Y p ] of person p , we use the observed X p as a key to compute its similarity score to the past trajectories { X i } |Z bank | i = in Z bank and find a representative trajectory Y p ′ , 1 ≤ p ′ ≤ |Z bank | according to the maximum similarity score (see Algorithm 1, Step 2).
Revised: [ X p ; Y p ] of person p , we use the observed X p as a key to compute its similarity score to the past trajectories { X i } |Z bank | i = in Z bank and find a representative trajectory Y p ′ , 1 ≤ p ′ ≤ |Z bank | according to the maximum similarity score (see Algorithm 1, Step 2).

Original: The similarity function can be formulated as:
Revised: The similarity function can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: By adding the offsets O p (see Equation 2) to the representative trajectory Y p ′ , we obtain our predicted trajectory ˆ Y p of the observed person (see Figure 2).
Revised: By adding the offsets O p (see Equation 2) to the representative trajectory Y p ′ , we obtain our predicted trajectory ˆ Y p of the observed person (see Figure 2).

Original: Although the initial trajectory bank works well for most cases, to improve the generalizability of the bank Z bank (see Algorithm 1, Step 3), we decide whether to update Z bank according to the distance threshold θ .
Revised: Although the initial trajectory bank works well for most cases, to improve the generalizability of the bank Z bank (see Algorithm 1, Step 3), we decide whether to update Z bank according to the distance threshold θ .

Original: 3.3 Cross-modal Interaction Module
Revised: 3.3 Cross-modal Interaction Module

Original: This module focuses on the interaction between individual past trajectory and the environment information.
Revised: This module focuses on the interaction between individual past trajectory and the environment information.

Original: It consists of two unimodal encoders to learn human motion and scene information, respectively, and a cross-modal transformer to model their interaction.
Revised: It consists of two unimodal encoders to learn human motion and scene information, respectively, and a cross-modal transformer to model their interaction.

Original: Trajectory Encoder.
Revised: Trajectory Encoder.

Original: The trajectory encoder adopts a multi-head attention structure from a transformer network [30] with N tra self-attention (SA) layers.
Revised: The trajectory encoder adopts a multi-head attention structure from a transformer network [30] with N tra self-attention (SA) layers.

Original: SA layers capture human motion across different time steps with the size of T pas × N tra × d , and project the motion features from dimension T pas × N tra d to T pas × d , where d is the embed dimension of the trajectory en-
Revised: SA layers capture human motion across different time steps with the size of T pas × N tra × d , and project the motion features from dimension T pas × N tra d to T pas × d , where d is the embed dimension of the trajectory en-

Original: coder.
Revised: coder.

Original: Thus, we obtain the human motion representation X tra using the trajectory encoder E tra :
Revised: Thus, we obtain the human motion representation X tra using the trajectory encoder E tra :

Original: X tra
Revised: X tra

Original: =
Revised: =

Original: E tra ( X p ) ∈ R T pas × d .
Revised: E tra ( X p ) ∈ R T pas × d .

Original: Scene Encoder.
Revised: Scene Encoder.

Original: Due to the compelling performance of the pretrained Swin Transformer [17] in feature representation, we adopt it as our scene encoder.
Revised: Due to the compelling performance of the pretrained Swin Transformer [17] in feature representation, we adopt it as our scene encoder.

Original: It extracts scene semantic features with the size of N cls × h × w , where N cls ( N cls = 150 in the pretrained scene encoder) is the number of semantic classes, such as person and road, and h and w are the spatial resolutions.
Revised: It extracts scene semantic features with the size of N cls × h × w , where N cls ( N cls = 150 in the pretrained scene encoder) is the number of semantic classes, such as person and road, and h and w are the spatial resolutions.

Original: In order to make the following module to conveniently fuse motion representation and environment information, we reshape semantic features from size ( N cls × h × w ) to ( N cls × hw ), and project them from dimension ( N cls × hw ) to ( N cls × d ) through a Multi-Layer Perception layer.
Revised: In order to make the following module to conveniently fuse motion representation and environment information, we reshape semantic features from size ( N cls × h × w ) to ( N cls × hw ), and project them from dimension ( N cls × hw ) to ( N cls × d ) through a Multi-Layer Perception layer.

Original: As a result, we obtain the scene representation X sce using the scene encoder E sce : X sce =
Revised: As a result, we obtain the scene representation X sce using the scene encoder E sce : X sce =

Original: E sce ( S p ) ∈
Revised: E sce ( S p ) ∈

Original: R N cls × d .
Revised: R N cls × d .

Original: Cross-modal Transformer.
Revised: Cross-modal Transformer.

Original: Unimodal encoders extract features from its own modality, disregarding the interaction between human motion and environment information.
Revised: Unimodal encoders extract features from its own modality, disregarding the interaction between human motion and environment information.

Original: The cross-modal transformer with N cro layers aims to refine the candidate trajectory Y p ′ (see Section 3.2) by learning this interaction.
Revised: The cross-modal transformer with N cro layers aims to refine the candidate trajectory Y p ′ (see Section 3.2) by learning this interaction.

Original: We adopt a two-stream structure: one for capturing the important human motion constrained by environment information, and the other for picking out the environment information related to human motion.
Revised: We adopt a two-stream structure: one for capturing the important human motion constrained by environment information, and the other for picking out the environment information related to human motion.

Original: The cross-attention (CA) layers and self-attention (SA) layers are the main components of the cross-modal transformer (see Figure 3).
Revised: The cross-attention (CA) layers and self-attention (SA) layers are the main components of the cross-modal transformer (see Figure 3).

Original: To capture the important human motion affected by environment and acquire the environment information related to the motion,
Revised: To capture the important human motion affected by environment and acquire the environment information related to the motion,

Original: CA layers treat one modality as query, and the other as key and value to interact with the two modalities.
Revised: CA layers treat one modality as query, and the other as key and value to interact with the two modalities.

Original: SA layers serve to facilitate better internal connections, which calculate the similarity between an element (query) and other elements (key) in scene-constrained motion or motion-relevant environment information.
Revised: SA layers serve to facilitate better internal connections, which calculate the similarity between an element (query) and other elements (key) in scene-constrained motion or motion-relevant environment information.

Original: Thus, we obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the crossmodal transformer E cro : ˆ X tra =
Revised: Thus, we obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the crossmodal transformer E cro : ˆ X tra =

Original: E cro ( X tra , X sce ) , ˆ X sce =
Revised: E cro ( X tra , X sce ) , ˆ X sce =

Original: E cro ( X sce , X tra ) .
Revised: E cro ( X sce , X tra ) .

Original: To predict the offsets O p between the searched trajectory Y p ′ and the ground-truth trajectory Y p , we take the last element ( LE ) h tra ∈ R d of ˆ X tra and the output
Revised: To predict the offsets O p between the searched trajectory Y p ′ and the ground-truth trajectory Y p , we take the last element ( LE ) h tra ∈ R d of ˆ X tra and the output

Original: h sce ∈ R
Revised: h sce ∈ R

Original: d of ˆ X sce after the global pooling layer ( GPL ).
Revised: d of ˆ X sce after the global pooling layer ( GPL ).

Original: Offsets O p ∈ R T fut × 2 can be formulated as follows:
Revised: Offsets O p ∈ R T fut × 2 can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where [; ] denotes the vector concatenation, and MLP is the Multi-Layer Perception layer.
Revised: where [; ] denotes the vector concatenation, and MLP is the Multi-Layer Perception layer.

Original: 3.4 Training
Revised: 3.4 Training

Original: We train the overall framework of SHENet end-to-end to minimize an objective function.
Revised: We train the overall framework of SHENet end-to-end to minimize an objective function.

Original: During the training, since the scene encoder has been pretrained on ADE20K [38], we freeze its segmentation part and update parameters of MLP head (see Section 3.3).
Revised: During the training, since the scene encoder has been pretrained on ADE20K [38], we freeze its segmentation part and update parameters of MLP head (see Section 3.3).

Original: Following the existing works [1, 19, 20, 35], we calculate the mean squared error (MSE) between the predicted trajectory and the ground-truth trajectory on ETH/UCY datasets:
Revised: Following the existing works [1, 19, 20, 35], we calculate the mean squared error (MSE) between the predicted trajectory and the ground-truth trajectory on ETH/UCY datasets:

Original: [Equation]
Revised: [Equation]

Original: In a more challenging PAV dataset, we use the curve smoothing (CS) regression loss, which serves to reduce the impact of individual bias.
Revised: In a more challenging PAV dataset, we use the curve smoothing (CS) regression loss, which serves to reduce the impact of individual bias.

Original: It calculates the MSE after trajectory smoothing.
Revised: It calculates the MSE after trajectory smoothing.

Original: CS loss can be formulated as follows:
Revised: CS loss can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where CS represents the function of curve smoothing [2].
Revised: where CS represents the function of curve smoothing [2].

Original:  Experiments 4.1 Experimental Setup
Revised:  Experiments 4.1 Experimental Setup

Original: Datasets.
Revised: Datasets.

Original: We evaluate our method on ETH [24], UCY [13], PAV and Stanford Drone Dataset (SDD) [26] datasets.
Revised: We evaluate our method on ETH [24], UCY [13], PAV and Stanford Drone Dataset (SDD) [26] datasets.

Original: ETH/UCY datasets are the benchmark commonly used for human trajectory prediction (HTP).
Revised: ETH/UCY datasets are the benchmark commonly used for human trajectory prediction (HTP).

Original: The benchmark contains videos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2 captured at 2.5HZ.
Revised: The benchmark contains videos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2 captured at 2.5HZ.

Original: Unimodal methods [1, 10, 31, 35] only focus on the trajectory data, however, multimodal methods [16, 20] need to consider scene information.
Revised: Unimodal methods [1, 10, 31, 35] only focus on the trajectory data, however, multimodal methods [16, 20] need to consider scene information.

Original: Since a video from UNIV is not available, [16, 20] use fewer frames than the unimodal methods.
Revised: Since a video from UNIV is not available, [16, 20] use fewer frames than the unimodal methods.

Original: We follow [16] to process the data and predict future n fut = 12 frames given the observed n pas = 8 frames.
Revised: We follow [16] to process the data and predict future n fut = 12 frames given the observed n pas = 8 frames.

Original: Compared to the ETH/UCY datasets, PAV is more challenging with diverse movement patterns, including PETS09-S2L1 (PETS) [4, 11], ADL-Rundle-6 (ADL), and Venice-2 (VENICE), which are captured from static camera and provide sufficient trajectories for the HTP task.
Revised: Compared to the ETH/UCY datasets, PAV is more challenging with diverse movement patterns, including PETS09-S2L1 (PETS) [4, 11], ADL-Rundle-6 (ADL), and Venice-2 (VENICE), which are captured from static camera and provide sufficient trajectories for the HTP task.

Original: We divide the videos into training (80%) and testing (20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4, training sequences and 664/306/650 testing sequences, respectively.
Revised: We divide the videos into training (80%) and testing (20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4, training sequences and 664/306/650 testing sequences, respectively.

Original: We use n pas =
Revised: We use n pas =

Original: 10 observed frames to predict future n fut = 50 frames, such that we can compare the long-term prediction results from different methods.
Revised: 10 observed frames to predict future n fut = 50 frames, such that we can compare the long-term prediction results from different methods.

Original: Different from ETH/UCY and PAV datasets, SDD is a large-scale datatset captured on the university campus in bird’s eye view.
Revised: Different from ETH/UCY and PAV datasets, SDD is a large-scale datatset captured on the university campus in bird’s eye view.

Original: It consists of multiple interacting agents ( e .
Revised: It consists of multiple interacting agents ( e .

Original: g ., pedestrians, bicyclists, and cars) and diverse scenes ( e .
Revised: g ., pedestrians, bicyclists, and cars) and diverse scenes ( e .

Original: g ., sidewalks and crossroads).
Revised: g ., sidewalks and crossroads).

Original: Following [19], we use past 8 frames to predict future 12 frames.
Revised: Following [19], we use past 8 frames to predict future 12 frames.

Original: More results conducted on SDD are in the the supplementary material.
Revised: More results conducted on SDD are in the the supplementary material.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: For ETH and UCY datasets, we adopt the standard metrics [1, 7, 16, 20, 35] for HTP: Average Displacement Error (ADE) and Final Displacement
Revised: For ETH and UCY datasets, we adopt the standard metrics [1, 7, 16, 20, 35] for HTP: Average Displacement Error (ADE) and Final Displacement

Original: Error (FDE).
Revised: Error (FDE).

Original: ADE is the average L 2 error between predicted trajectories and groundtruth trajectories over all time steps, and
Revised: ADE is the average L 2 error between predicted trajectories and groundtruth trajectories over all time steps, and

Original: FDE is the L 2 error between the predicted trajectories and ground-truth trajectories at the final time step.
Revised: FDE is the L 2 error between the predicted trajectories and ground-truth trajectories at the final time step.

Original: The trajectories in PAV are with some jittering phenomena ( e .
Revised: The trajectories in PAV are with some jittering phenomena ( e .

Original: g ., abrupt and sharp turns).
Revised: g ., abrupt and sharp turns).

Original: Thus, a reasonable prediction may produce approximately the same error as an unrealistic prediction by using traditional metrics: ADE and FDE (see Figure 7(a)).
Revised: Thus, a reasonable prediction may produce approximately the same error as an unrealistic prediction by using traditional metrics: ADE and FDE (see Figure 7(a)).

Original: In order to focus on the pattern and shape of the trajectory itself and also reduce the impact of randomness and subjectivity, we propose the CS-Metric: CS-ADE and CS-FDE (illustrated in Figure 4).
Revised: In order to focus on the pattern and shape of the trajectory itself and also reduce the impact of randomness and subjectivity, we propose the CS-Metric: CS-ADE and CS-FDE (illustrated in Figure 4).

Original: CS-ADE is calculated as follows:
Revised: CS-ADE is calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: where CS is the curve smoothing function defined as the same as our L cs in Section 3.4.
Revised: where CS is the curve smoothing function defined as the same as our L cs in Section 3.4.

Original: Similar to
Revised: Similar to

Original: CS-ADE, CS-FDE calculates the final displacement error after trajectory smoothing:
Revised: CS-ADE, CS-FDE calculates the final displacement error after trajectory smoothing:

Original: [Equation]
Revised: [Equation]

Original: Figure 5 shows some samples from the training data, converting rough ground-truth trajectories to smooth ones.
Revised: Figure 5 shows some samples from the training data, converting rough ground-truth trajectories to smooth ones.

Original: Implementation Details.
Revised: Implementation Details.

Original: In SHENet, the initial size of group trajectory bank is set to |Z bank | = 32 .
Revised: In SHENet, the initial size of group trajectory bank is set to |Z bank | = 32 .

Original: Both the trajectory encoder and the scene encoder have 4 self-attention (SA) layers.
Revised: Both the trajectory encoder and the scene encoder have 4 self-attention (SA) layers.

Original: The cross-modal transformer is with 6 SA layers and cross-attention (CA) layers.
Revised: The cross-modal transformer is with 6 SA layers and cross-attention (CA) layers.

Original: We set all the embed dimensions to 512 .
Revised: We set all the embed dimensions to 512 .

Original: For the trajectory encoder, it learns the human motion information with size of T pas ×( T pas = 8 in ETH/UCY, T pas =
Revised: For the trajectory encoder, it learns the human motion information with size of T pas ×( T pas = 8 in ETH/UCY, T pas =

Original: 10 in PAV).
Revised: 10 in PAV).

Original: For the scene encoder, it outputs the semantic features with size 150 × 56 × 56 .
Revised: For the scene encoder, it outputs the semantic features with size 150 × 56 × 56 .

Original: We reshape the features from size 150 × 56 × 56 to 150 × 3136 , and project them from dimension 150 × 3136 to 150 × 512 .
Revised: We reshape the features from size 150 × 56 × 56 to 150 × 3136 , and project them from dimension 150 × 3136 to 150 × 512 .

Original: We train the
Revised: We train the

Original: 4.2 Ablation Studies
Revised: 4.2 Ablation Studies

Original: In Table 1, we evaluate each component of SHENet, including a group trajectory bank (GTB) module, and a cross-modal interaction (CMI) module that contains a trajectory encoder (TE), a scene encoder (SE) and a cross-modal transformer (CMT).
Revised: In Table 1, we evaluate each component of SHENet, including a group trajectory bank (GTB) module, and a cross-modal interaction (CMI) module that contains a trajectory encoder (TE), a scene encoder (SE) and a cross-modal transformer (CMT).

Original: [Table]
Revised: [Table]

Original: Effect of GTB.
Revised: Effect of GTB.

Original: We first investigate the performance of GTB.
Revised: We first investigate the performance of GTB.

Original: GTB improves FDE by 21.2% on PETS compared to the CMI ( i .
Revised: GTB improves FDE by 21.2% on PETS compared to the CMI ( i .

Original: e ., TE, SE, and CMT), which is a significant improvement, indicating the importance of GTB.
Revised: e ., TE, SE, and CMT), which is a significant improvement, indicating the importance of GTB.

Original: However, GTB (1-st row of Table 1) alone is not enough and even performs a little worse than CMI.
Revised: However, GTB (1-st row of Table 1) alone is not enough and even performs a little worse than CMI.

Original: Thus, we explore the effects of various parts in CMI module.
Revised: Thus, we explore the effects of various parts in CMI module.

Original: Effect of TE and SE.
Revised: Effect of TE and SE.

Original: To evaluate the performance of TE and SE, we concatenate the trajectory features extracted from TE and the scene features from SE together (3-rd row in Table 1), and immprove performance on ADL and VENICE with smaller movements, compared to the case of using TE alone.
Revised: To evaluate the performance of TE and SE, we concatenate the trajectory features extracted from TE and the scene features from SE together (3-rd row in Table 1), and improve performance on ADL and VENICE with smaller movements, compared to the case of using TE alone.

Original: This indicates that the incorporation of environment information into trajectory prediction can improve the accuracy of the results.
Revised: This indicates that the incorporation of environment information into trajectory prediction can improve the accuracy of the results.

Original: Effect of CMT.
Revised: Effect of CMT.

Original: Compared to the third row of Table 1, CMT (4-th row in Table 1) can result in a promising improvement of the model performance.
Revised: Compared to the third row of Table [Table] CMT (4-th row in Table 1) can result in a promising improvement of the model performance.

Original: Notably, it performs better than the TE and
Revised: Notably, it performs better than the TE and

Original: SE concatenated on PETS, improving by 7.4% over ADE.
Revised: SE concatenated on PETS, improving by 7.4% over ADE.

Original: Compared to GTB alone, the full CMI improves by 12.2% over ADE on average.
Revised: Compared to GTB alone, the full CMI improves by 12.2% over ADE on average.

Original: 4.3 Comparison with State-of-the-art Methods
Revised: 4.3 Comparison with State-of-the-art Methods

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: On ETH/UCY datasets, we compare our model with the state-of-the-art methods: SS-LSTM [35],
Revised: On ETH/UCY datasets, we compare our model with the state-of-the-art methods: SS-LSTM [35],

Original: Social-STGCN [22], MANTRA [20], AgentFormer [37], YNet [19].
Revised: Social-STGCN [22], MANTRA [20], AgentFormer [37], YNet [19].

Original: The results are summarized in
Revised: The results are summarized in

Original: Table 2.
Revised: Table 2.

Original: Our model reduces the average FDE from 0.39 to 0.36, and achieves 7.7% improvement compared with the state-of-the-art method, YNet.
Revised: Our model reduces the average FDE from 0.39 to 0.36, and achieves 7.7% improvement compared with the state-of-the-art method, YNet.

Original: Particularly, our model outperforms previous methods significantly on ETH, when there are large movements of trajectories, and improves the
Revised: Particularly, our model outperforms previous methods significantly on ETH, when there are large movements of trajectories, and improves the

Original: ADE and FDE by 12.8% and 15.3%, respectively.
Revised: ADE and FDE by 12.8% and 15.3%, respectively.

Original: To evaluate the performance of our model in long-term predictions, we conduct experiments on PAV, with n pas =
Revised: To evaluate the performance of our model in long-term predictions, we conduct experiments on PAV, with n pas =

Original: 10 observed frames and n fut = 50 future frames of each trajectory.
Revised: 10 observed frames and n fut = 50 future frames of each trajectory.

Original: Table 3 shows the performance compared with previous HTP methods: SS-LSTM [35], Social-STGCN [22], Next [16],
Revised: Table 3 shows the performance compared with previous HTP methods: SS-LSTM [35], Social-STGCN [22], Next [16],

Original: MANTRA [20], YNet [19].
Revised: MANTRA [20], YNet [19].

Original: Compared to the state-of-the-art results of YNet, the proposed SHENet achieves respective 3.3% and 10.5% improvement over CS-ADE and CS-FDE on average.
Revised: Compared to the state-of-the-art results of YNet, the proposed SHENet achieves respective 3.3% and 10.5% improvement over CS-ADE and CS-FDE on average.

Original: Since
Revised: Since

Original: YNet predicts the heatmaps of a trajectory, it performs a little better when there are small movements of trajectories, e .
Revised: YNet predicts the heatmaps of a trajectory, it performs a little better when there are small movements of trajectories, e .

Original: g .,
Revised: g .,

Original: VENICE.
Revised: VENICE.

Original: Nevertheless, our method is still competitive in VENICE, and is much better than others approaches on PETS with large movements and
Revised: Nevertheless, our method is still competitive in VENICE, and is much better than others approaches on PETS with large movements and

Original: [Table]
Revised: [Table]

Original: intersections.
Revised: intersections.

Original: In particular, our method improves the CS-FDE by 16.2% on PETS, compared to YNet.
Revised: In particular, our method improves the CS-FDE by 16.2% on PETS, compared to YNet.

Original: We also achieve great improvements using the traditional ADE/FDE metrics.
Revised: We also achieve great improvements using the traditional ADE/FDE metrics.

Original: The complete results are shown in the supplementary material.
Revised: The complete results are shown in the supplementary material.

Original: 4.4 Analysis
Revised: 4.4 Analysis

Original: Distance Threshold θ .
Revised: Distance Threshold θ .

Original: θ is used to determine the update of trajectory bank.
Revised: θ is used to determine the update of trajectory bank.

Original: The typical value of θ is set according to the trajectory length.
Revised: The typical value of θ is set according to the trajectory length.

Original: When the ground-truth trajectory is longer in terms of pixel, the absolute value of prediction error is usually larger.
Revised: When the ground-truth trajectory is longer in terms of pixel, the absolute value of prediction error is usually larger.

Original: However, their relative errors are comparable.
Revised: However, their relative errors are comparable.

Original: Thus, the θ is set to be 75% of the training error when the error converges.
Revised: Thus, the θ is set to be 75% of the training error when the error converges.

Original: In our experiments, we set θ =in PETS, and θ = 6 in ADL.
Revised: In our experiments, we set θ =in PETS, and θ = 6 in ADL.

Original: The "75% of the
Revised: The "75% of the

Original: [Table]
Revised: [Table]

Original: training error" is obtained from the experimental result, as shown in Table 4.
Revised: training error" is obtained from the experimental result, as shown in Table 4.

Original: Cluster Number in K-medoids.
Revised: Cluster Number in K-medoids.

Original: We study the effect of setting different number of initial clusters K , as shown in Table 5 .
Revised: We study the effect of setting different number of initial clusters K , as shown in Table 5 .

Original: We can note that the initial number of clusters is not sensitive to prediction results, especially when the initial number of clusters is 24-36.
Revised: We can note that the initial number of clusters is not sensitive to prediction results, especially when the initial number of clusters is 24-36.

Original: Therefore, we can set K to 32 in our experiments.
Revised: Therefore, we can set K to 32 in our experiments.

Original: [Table]
Revised: [Table]

Original: Analysis for Bank Complexity.
Revised: Analysis for Bank Complexity.

Original: The time complexity of the searching and updating are O ( N ) and O (1) .
Revised: The time complexity of the searching and updating are O ( N ) and O (1) .

Original: Their space complexity is O ( N ) .
Revised: Their space complexity is O ( N ) .

Original: The group trajectory number N ≤ 1000 .
Revised: The group trajectory number N ≤ 1000 .

Original: The time complexity of the clustering process is O ( β 2 + MKβ ) , and the space complexity is O ( β 2 + Mβ ) .
Revised: The time complexity of the clustering process is O ( β 2 + MKβ ) , and the space complexity is O ( β 2 + Mβ ) .

Original: β is the number of trajectories for clustering.
Revised: β is the number of trajectories for clustering.

Original: K is the number of clusters, and M is the number of iterations for the clustering methods.
Revised: K is the number of clusters, and M is the number of iterations for the clustering methods.

Original: 4.5 Qualitative Results.
Revised: 4.5 Qualitative Results.

Original: Figure 6 presents the qualitative results of SHENet and other methods.
Revised: Figure 6 presents the qualitative results of SHENet and other methods.

Original: By comparison, we surprisingly notice that in an extremely challenging situation where a person walks to the curb and turns back (green curve), all other methods do not deal with it well, while our proposed SHENet can still handle it.
Revised: By comparison, we surprisingly notice that in an extremely challenging situation where a person walks to the curb and turns back (green curve), all other methods do not deal with it well, while our proposed SHENet can still handle it.

Original: This should be attributed to the effect of our dedicatedly designed historical group trajectory bank module.
Revised: This should be attributed to the effect of our dedicatedly designed historical group trajectory bank module.

Original: In addition, compared to the memory -based method MANTRA [20], we search trajectories of groups, instead of just individuals.
Revised: In addition, compared to the memory -based method MANTRA [20], we search trajectories of groups, instead of just individuals.

Original: This is more versatile and can be applied in more challenging scenes.
Revised: This is more versatile and can be applied in more challenging scenes.

Original: Figureincludes the qualitative results of YNet and our SHENet without/with curve smoothing (CS).
Revised: Figureincludes the qualitative results of YNet and our SHENet without/with curve smoothing (CS).

Original: The first row presents the results of using MSE loss L tra .
Revised: The first row presents the results of using MSE loss L tra .

Original: Influenced by the past trajectories with some noise ( e .
Revised: Influenced by the past trajectories with some noise ( e .

Original: g ., abrupt and sharp turns), the predicted trajectory points of YNet are gathered together, and cannot present a clear direction, while our method can provide a potential path based on the historical group trajectories.
Revised: g ., abrupt and sharp turns), the predicted trajectory points of YNet are gathered together, and cannot present a clear direction, while our method can provide a potential path based on the historical group trajectories.

Original: The two predictions are visually distinct, however, the numerical errors (ADE/FDE) are approximately the same.
Revised: The two predictions are visually distinct, however, the numerical errors (ADE/FDE) are approximately the same.

Original: In contrast, the qualitative results of our proposed CS loss L cs are shown in the second row of Figure 7.
Revised: In contrast, the qualitative results of our proposed CS loss L cs are shown in the second row of Figure 7.

Original: We can see that our proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by YNet and our method.
Revised: We can see that our proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by YNet and our method.

Original: More visualization results are provided in supplementary material.
Revised: More visualization results are provided in supplementary material.

Original:  Conclusion
Revised:  Conclusion

Original: The paper presents SHENet, a novel method that fully utilizes scene history for HTP.
Revised: The paper presents SHENet, a novel method that fully utilizes scene history for HTP.

Original: SHENet includes a GTB module to construct a group trajectory bank from all historical trajectories and retrieve a representative trajectory from this bank for an observed person, and also a CMI module (which interacts between human motion and environment information) to refine this representative trajectory.
Revised: SHENet includes a GTB module to construct a group trajectory bank from all historical trajectories and retrieve a representative trajectory from this bank for an observed person, and also a CMI module (which interacts between human motion and environment information) to refine this representative trajectory.

Original: We achieve the SOTA performance on the HTP benchmarks and our method demonstrates significant improvements and generalizaibility in the challenging scenes.
Revised: We achieve the SOTA performance on the HTP benchmarks and our method demonstrates significant improvements and generalizaibility in the challenging scenes.

Original: However, there are still several unexplored aspects in the current framework, e.g., the process of bank construction currently focusing on only human motion.
Revised: However, there are still several unexplored aspects in the current framework, e.g., the process of bank construction currently focusing on only human motion.

Original: Future work includes further exploring the trajectory bank using the interact information (human motion and scene information).
Revised: Future work includes further exploring the trajectory bank using the interact information (human motion and scene information).

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work is supported by the National Key R&D Program of China (2021ZD0111100) and sponsored by Shanghai Rising-Star Program.
Revised: This work is supported by the National Key R&D Program of China (2021ZD0111100) and sponsored by Shanghai Rising-Star Program.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] All data used in the paper is public; our code are provided (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] All data used in the paper is public; our code are provided (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] public datasets (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] public datasets (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] public datasets 5.
Revised: [N/A] public datasets 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] Supplementary Material
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Section A provides additional details for the method.

Original: The experiment details are described in Section

Original: B, including experimental setup and results.

Original: We also present more qualitative results in Section C.

Original: A Method details A.1 Cross-modal Interaction

Original: Trajectory Encoder.

Original: The trajectory encoder aims to learn the human motion representation via a transformer structure.

Original: The observed trajectory of person p is X p with the size of T pas × 2 ( e .

Original: g .,

Original: T pas = 10 in PAV).

Original: We obtain the human motion with the size of 10 × 4 × 512 using N tra

Original: =self-attention layers, where 512 is the embed dimension.

Original: We reshape the motion features from size 10 × 4 × 512 to 10 × 2048 and project it from dimension 10 × 2048 to 10 × 512 .

Original: Thus, we obtain the human motion representation X tra on PAV dataset with the size of 10 × 512 .

Original: Scene Encoder.

Original: The scene encoder is to extract the environment information.

Original: The scene semantic features are with the size of 150 × 56 × 56 , where 150 is the number of semantic classes.

Original: We reshape the semantic features from size ( 150 × 56 × 56 ) to ( 150 × 3 , 136 ), and project them from dimension ( 150 × 3 , 136 ) to ( 150 × 512 ) through a Multi-Layer Perception layer.

Original: As a result, we obtain the scene representation X sce with the size of 150 × 512 .

Original: Cross-modal Transformer.

Original: The cross-modal transformer with N cro = 6 layers aims to learn the interaction between human motion and environment information.

Original: We obtain the inter-modal representation ( ˆ X tra , ˆ X sce ) via the cross-modal transformer.

Original: The size of ˆ X tra and ˆ X sce are 10 ×and 150 × 512 , respectively.

Original: We take the last element ( LE ) h tra ∈ R 512 of ˆ X tra and the output h sce ∈ R 512 of ˆ X sce after the global pooling layer ( GPL ), and get the offsets O p ∈ R 50 ×

Original: ( T fut = 50 in PAV) through a Multi-Layer Perception layer.

Original: A.2 Training

Original: We use the curve smoothing (CS) regression loss L cs to reduce the impact of randomness and subjectivity.

Original: The quadratic Bézier curve is adopted to smooth the trajectory, which can be formulated as follows:

Original: [Equation]

Original: where z − T pas +1 p is the starting point of the trajectory, and z T fut p is the destination of the trajectory.

Original: z T ′ p is the control point of this trajectory, which can be calculated as follows:

Original: [Equation]

Original: Then we divide the time period t ∈ [0 , 1] equidistantly, and get the smoothed trajectory Z p =

Original: [ X p ; Y p ] = { z tp } T fut t = − T pas +1 .

Original: B Experiments B.1 Experimental Setup

Original: Dataset Details.

Original: ETH/UCY datasets are the benchmark commonly used for human trajectory prediction.

Original: The benchmark contains videos from 5 scenes, including ETH, HOTEL, UNIV, ZARA1, and ZARA2.

Original: Following [19], we sample the frames at 2.5 HZ and predict future n fut = 12 frames given the observed n pas = 8 frames.

Original: We use the preprocessed data provided by YNet [19], which converts the raw data from word coordinate into image pixel space.

Original: We use the leave-one-scene-out strategy with 4 scenes for training and the remaining scene for testing.

Original: PAV is a more challenging dataset with diverse movement patterns, which includes 3 videos PETS, ADL, and VENICE.

Original: We divide the videos into training (80%) and testing (20%) sets, and PETS/ADL/VENICE contain 2,370/2,935/4,200 training sequences and 664/306/650 testing sequences, respectively.

Original: We use observed n pas =

Original: 10 frames to predict future n fut = 50 frames.

Original: Evaluation Metrics.

Original: For ETH and UCY datasets, we adopt the standard metrics ( i .

Original: e ., ADE and FDE).

Original: Due to the limitations discussed in Section 4.1, we introudce curve smoothing (CS) into current metrics on PAV dataset, and thus we propose CS-ADE and CS-FDE.

Original: The curve smoothing function is defined as the same as Equation 6.

Original: B.2 Experimental Results

Original: PAV without CS.

Original: We conduct experiments on PAV using the traditional ADE/FDE metrics.

Original: Tableshows the quantitative result of our method and previous human trajectory prediction methods.

Original: Compared to the state-of-the-art results of YNet, the proposed SHENet achieves 5.1%/4.0% improvement over ADE/FDE on average.

Original: In particular, our method improves the FDE by 13.6% on PETS.

Original: [Table]
Revised: [N/A]

Original: Social-STGCN [22] 63.76 / 159.

Original: Next [16] 51.78 / 109.

Original: MANTRA [20]

Original: [Table]

Original: [Table]

Original: [Table]

Original: YNet [19] 53.46 /

Original: [Table]

Original: [Table]

Original: [Table]

Original: SHENet (Ours) 46.31 / 101.

Original: ETH/UCY without Video Data.

Original: We also conduct the experiments on ETH/UCY without using the video data, shown in Table 7.

Original: Since MANTRA didn’t conduct experiments on ETH/UCY, we use the results of MANTRA reported in the work [32].

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: MemoNet [32] 0.40 / 0.

Original: SHENet (Ours) 0.37 / 0.

Original: From Table 7, we can note that our method achieves the comparable performance without using the video data.

Original: SDD.

Original: We also report the performance of different methods on SDD in Table 8.

Original: It shows that our model performs a little bit worse than the results of YNet [19] and MemoNet [32].

Original: Nevertheless, our

Original: [Table]

Original: method performs better than previous baselines (such as PECNet [18], MANTRA [32]).

Original: Consequently, our method can achieve reasonable performance in bird-eye-view scenario.

Original: C Qualitative Results

Original: Figure 8 presents the visualization of clustered trajectories.

Original: We can see that similar trajectories are clustered into the same group.

Original: Each group trajectory represents a moving pattern of a group of people.

Original: Figure 9 illustrates the qualitative results of SHENet and other methods.

Original: Our method is superior to others when it comes to human crossing intersections ( e .

Original: g ., the first row in Figure 9) or turning ( e .

Original: g ., the second row in Figure 9).

Original: Figure 10 includes the qualitative results of our SHENet without/with curve smoothing (CS).

Original: The first row presents the results of using MSE loss L tra .

Original: We can note that our SHENet can not provide a correct path from scene history, since there exists some noise ( e .

Original: g ., abrupt and sharp turns) in past trajectories.

Original: Thus to reduce the impact of noise, we use the CS loss

Original: L cs to train our model.

Original: In contrast, the qualitative results of using L cs are shown in the second row of Figure 10.

Original: We can see that the proposed CS significantly reduces the impact of randomness and subjectivity, and produces reasonable predictions by our method.

Original: [Figure]