Original: Contact-aware Human Motion Forecasting
Revised: Contact-aware Human Motion Forecasting

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In this paper, we tackle the task of scene-aware 3D human motion forecasting,which consists of predicting future human poses given a 3D scene and a past humanmotion.
Revised: In this paper, we tackle the task of scene-aware 3D human motion forecasting,which consists of predicting future human poses given a 3D scene and a past humanmotion.

Original: A key challenge of this task is to ensure consistency between the humanand the scene, accounting for human-scene interactions.
Revised: A key challenge of this task is to ensure consistency between the humanand the scene, accounting for human-scene interactions.

Original: Previous attempts to doso model such interactions only implicitly, and thus tend to produce artifacts suchas “ghost motion" because of the lack of explicit constraints between the localposes and the global motion.
Revised: Previous attempts to doso model such interactions only implicitly, and thus tend to produce artifacts suchas “ghost motion" because of the lack of explicit constraints between the localposes and the global motion.

Original: Here, by contrast, we propose to explicitly modelthe human-scene contacts.
Revised: Here, by contrast, we propose to explicitly modelthe human-scene contacts.

Original: To this end, we introduce distance-based contact mapsthat capture the contact relationships between every joint and every 3D scenepoint at each time instant.
Revised: To this end, we introduce distance-based contact mapsthat capture the contact relationships between every joint and every 3D scenepoint at each time instant.

Original: We then develop a two-stage pipeline that first predictsthe future contact maps from the past ones and the scene point cloud, and thenforecasts the future human poses by conditioning them on the predicted contactmaps.
Revised: We then develop a two-stage pipeline that first predictsthe future contact maps from the past ones and the scene point cloud, and thenforecasts the future human poses by conditioning them on the predicted contactmaps.

Original: During training, we explicitly encourage consistency between the globalmotion and the local poses via a prior defined using the contact maps and futureposes.
Revised: During training, we explicitly encourage consistency between the globalmotion and the local poses via a prior defined using the contact maps and futureposes.

Original: Our approach outperforms the state-of-the-art human motion forecastingand human synthesis methods on both synthetic and real datasets.
Revised: Our approach outperforms the state-of-the-art human motion forecastingand human synthesis methods on both synthetic and real datasets.

Original:  Introduction
Revised:  Introduction

Original: Human motion prediction has a broad application potential covering human robot interaction [13],autonomous driving [19], virtual/augmented reality (AR/VR) [24] and animation [26].
Revised: Human motion prediction has a broad application potential covering human robot interaction [17],autonomous driving [23], virtual/augmented reality (AR/VR) [28] and animation [31].

Original: As such, ithas been an active research topic for decades [2, 23, 25, 30].
Revised: As such, ithas been an active research topic for decades [4, 27, 30, 35].

Original: Nevertheless, most methods [15, 1,18, 27, 7, 14, 17, 3] disregard the fact that humans evolve in 3D environments, thus ignoring thehuman-scene interactions.
Revised: Nevertheless, most methods [19, 1,22, 32, 9, 18, 21, 5] disregard the fact that humans evolve in 3D environments, thus ignoring thehuman-scene interactions.

Original: By contrast, in this work, we tackle the task of scene-aware human motionforecasting which aims to incorporate the scene context to predict future 3D human motions.
Revised: By contrast, in this work, we tackle the task of scene-aware human motionforecasting which aims to incorporate the scene context to predict future 3D human motions.

Original: While some recent works in human motion prediction [4, 6] and human synthesis [28] have startedto explore the use of scene context, they do so implicitly, by taking an embedding of a 2D sceneimage [4] a 3D scene [28], or a specific object [6] as an additional input to their model.
Revised: While some recent works in human motion prediction [6, 8] and human synthesis [33] have startedto explore the use of scene context, they do so implicitly, by taking an embedding of a 2D sceneimage [6] a 3D scene [33], or a specific object [8] as an additional input to their model.

Original: Whilesuch embeddings encode valuable information, they do not provide precise cues to help placing thehuman in the scene.
Revised: Whilesuch embeddings encode valuable information, they do not provide precise cues to help placing thehuman in the scene.

Original: In the context of synthesizing a static human body in a scene, some efforts havenonetheless been made to incorporate more precise information, such as the distance between everyscene point and the closest vertex on body surface [31], or a semantic scene label, e.g., floor, sofa,at every vertex on the 3D body mesh [9].
Revised: In the context of synthesizing a static human body in a scene, some efforts havenonetheless been made to incorporate more precise information, such as the distance between everyscene point and the closest vertex on body surface [36], or a semantic scene label, e.g., floor, sofa, atevery vertex on the 3D body mesh [12].

Original: However, while such representations allow one to place ahuman in the scene, they remain under-constrained for motion prediction.
Revised: However, while such representations allow one to place ahuman in the scene, they remain under-constrained for motion prediction.

Original: Specifically, they imposeneither temporal consistency, nor consistency between local pose changes and global motion.
Revised: Specifically, they imposeneither temporal consistency, nor consistency between local pose changes and global motion.

Original: Assuch, they yield artifacts such as “ghost motion".
Revised: Assuch, they yield artifacts such as “ghost motion".

Original: In this paper, we address this by explicitly modeling the contact between the human body jointsand the scene.
Revised: In this paper, we address this by explicitly modeling the contact between the human body jointsand the scene.

Original: To this end, we introduce per-joint contact maps that encode the distance betweeneach joint and every 3D scene point.
Revised: To this end, we introduce per-joint contact maps that encode the distance betweeneach joint and every 3D scene point.

Original: Such contact maps constrain both the global motion and thelocal human pose, thus avoiding the “ghost motion" issue.
Revised: Such contact maps constrain both the global motion and thelocal human pose, thus avoiding the “ghost motion" issue.

Original: As illustrated in Fig 1, our human motion
Revised: As illustrated in Fig 1, our human motion

Original: [Figure]
Revised: [Figure]

Original: prediction framework therefore consists of a two-stage pipeline that first predicts the future contactmaps and then forecasts the future human poses by conditioning them on the predicted maps.
Revised: prediction framework therefore consists of a two-stage pipeline that first predicts the future contact maps and then forecasts the future human poses by conditioning them on the predicted maps.

Original: Specifically, we obtain a compact temporal representation of the historical contact maps using asimilar Discrete Cosine Transform (DCT) encoding strategy to that of [18] for human motion, anduse a feed-forward 3D scene encoding network, i.e., PVCNN [16], to predict the future contact mapsfrom thescene point cloud, pasthuman motion and the historical contact maps.
Revised: Specifically, we obtain a compact temporal representation of the historical contact maps using asimilar Discrete Cosine Transform (DCT) encoding strategy to that of [22] for human motion, anduse a feed-forward 3D scene encoding network, i.e , PVCNN [20], to predict the future contact maps the past motion and the historical contact maps.

Original: To then forecasthumanmotion, we retainthe closestcontact points for every human joint according to the predictedcontact maps and forecast the global translations and then the local poses given these points and thepast human motion. Consistency between the global translations and the local poses is achieved via aprior defined using the predictedhuman poses and the contact points.
Revised: then forecasthuman we retain closest predicted

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: (i)
Revised: (i)

Original: We propose to predict per-joint contact maps thatcapture fine-grained human-scene interactions to avoid generating unrealistic human motions.
Revised: We introduce a distance-based per-joint contactmap that captures fine-grained human-scene interactions to avoid generating unrealistic human motions.

Original: (ii) Weintroduce a two-stage pipeline that first predicts the future contact maps and then forecasts the futurehuman poses.
Revised: (ii) We further propose a two-stage pipeline whose first stage models the temporal dependenciesof past contact maps and predicts the future ones, and whose second stage forecasts the future humanmotion conditioned on these contact maps.

Original: Our experiments on both synthetic and real datasets demonstrate the benefits of ourapproach over the state-of-the-art human motion forecasting and human motion synthesis methods.
Revised: Our experiments on both synthetic and real datasetsdemonstrate the benefits of our approach over the state-of-the-art human motion forecasting andhuman motion synthesis methods.

Original:  Related Work
Revised:  Related Work

Original: Human motion prediction.
Revised: Human motion prediction.

Original: Modeling 3D human motion has been a long-standing research goal [2,23, 25, 30].
Revised: Modeling 3D human motion has been a long-standing research goal [4,27, 30, 35].

Original: While traditional methods [2, 30], relying on either Hidden Markov Models [2] or the
Revised: While traditional methods [4, 35], relying on either Hidden Markov Models [4] or the

Original: Gaussian process latent variable model [30], can tackle periodic and simple non-periodic motions,such as walking and golf swing, more complex motions have been shown to be better modeledvia deep learning frameworks [15, 1,  18, 27, 7, 14, 17, 3], which can be roughly categorized intofeed-forward models [15, 18, 17] and recurrent networks [1, 27, 7, 14, 3] according to their temporalspatial encoding strategies.
Revised: Gaussian process latent variable model [35], can tackle periodic and simple non-periodic motions,such as walking and golf swing, more complex motions have been shown to be better modeledvia deep learning frameworks [19, 1, 22, 32, 9, 18, 21, 5], which can be roughly categorized intofeed-forward models [19, 22, 21] and recurrent networks [1, 32, 9, 18, 5] according to their temporalspatial encoding strategies.

Original: Despite the success of these methods at forecasting complex motions,they typically only predict local poses, disregarding global motion and any scene information.
Revised: Despite the success of these methods at forecasting complex motions,they typically only predict local poses, disregarding global motion and any scene information.

Original: Inthis paper, we seek to predict future human motions that are consistent with the 3D scenes they areperformed in.
Revised: Inthis paper, we seek to predict future human motions that are consistent with the 3D scenes they areperformed in.

Original: Recently, a few works [6, 4] have started to incorporate scene context in motion forecasting.
Revised: Recently, a few works [8, 6] have started to incorporate scene context in motion forecasting.

Original: Inparticular, Corona et al .
Revised: Inparticular, Corona et al .

Original: [6] introduced a semantic-graph model that extracts a joint embedding of thehuman pose and an object of interest, such as a cup.
Revised: [8] introduced a semantic-graph model that extracts a joint embedding of thehuman pose and an object of interest, such as a cup.

Original: This method, however, is ill-suited to modelinteractions with the whole scene itself, for example the floor or stairs that the person touches whilewalking.
Revised: This method, however, is ill-suited to modelinteractions with the whole scene itself, for example the floor or stairs that the person touches whilewalking.

Original: In [4], Cao et al .
Revised: In [6], Cao et al .

Original: proposed a multi-stage pipeline that breaks down the motion forecastinginto three sub-tasks: predicting a 2D goal, planning a 2D and 3D path, forecasting the 3D posesfollowing the path.
Revised: proposed a multi-stage pipeline that breaks down the motion forecastinginto three sub-tasks: predicting a 2D goal, planning a 2D and 3D path, forecasting the 3D posesfollowing the path.

Original: To this end, they extracted a scene representation using 2D images and let their [Table] [Equation] [Table]
Revised: To this end, they extracted a scene representation using 2D images and let theirmodel learn the scene constraints implicitly.

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Figure]

Original: model learn the scene constraints implicitly.

Original: This strategy, however, cannot handle scene occlusionsand does not enforce consistency between the local and global motion.
Revised: This strategy, however, cannot handle scene occlusionsand does not enforce consistency between the local and global motion.

Original: More importantly, both thesemethods aim to learn interactions implicitly.
Revised: More importantly, both these methods aim to learn interactions implicitly.

Original: By contrast, our contact maps explicitly encode thehuman-scene interactions, leading to direct constraints at the human joint level.
Revised: By contrast, our contact maps explicitly encode thehuman-scene interactions, leading to direct constraints at the human joint level.

Original: Scene-aware human synthesis.
Revised: Scene-aware human synthesis.

Original: Synthesizing a realistic human in a 3D scene has recently gainedan increasing popularity [31, 32, 9, 29, 28].
Revised: Synthesizing a realistic human in a 3D scene has recently gainedan increasing popularity [36, 37, 12, 34, 33, 10].

Original: A cornerstone in the success of these methods is themodeling of human-scene interactions.
Revised: A cornerstone in the success of these methodsis the modeling of human-scene interactions.

Original: To achieve this, many of these works [32, 29, 28] followa similar approach to [4], modeling the human-scene interactions implicitly, via either Generative Adversarial Networks (GANs) [29], or Variational Autoencoders (VAEs) [32, 28].
Revised: To achieve this, many of these works [37, 34, 33]follow a similar approach to [6], modeling the human-scene interactions implicitly, via either Generative Adversarial Networks (GANs) [34], or Variational Autoencoders (VAEs) [37, 33].

Original: Two methods [31,9] nonetheless exploit explicit representations of human-scene interaction.
Revised: Threemethods [36, 12, 10] nonetheless exploit explicit representations of human-scene interaction.

Original: In particular, POSA [9]uses a body-centric representation of the human-scene interaction where a semantic scene label, e.g.,floor, sofa, is assigned to every human mesh vertex.
Revised: Inparticular, POSA [12] uses a body-centric representation of the human-scene interaction where asemantic scene label, e.g., floor, sofa, is assigned to every human mesh vertex.

Original: This label encodes the contact probability to thescene surface and the corresponding semantic scene label.
Revised: This label encodes thecontact probability to the scene surface and the corresponding semantic scene label.

Original: However, this semantic representationdoes not provide any information about the 3D location of the human body, and is thus ill-suited tothe motion forecasting task.
Revised: However, thissemantic representation does not provide any information about the 3D location of the human body,and is thus ill-suited to the motion forecasting task.

Original: In PLACE [31], Zhang et al .
Revised: In PLACE [36], Zhang et al .

Original: introduce a contact representation basedon Basic Point Sets (BSP) 
Revised: introduce a contactrepresentation based on Basic Point Sets (BSP) [26].

Original: Specifically, given a set of basic 3D scene points, they representthe human-scene interaction using the minimum distance from every such point to the human bodysurface.
Revised: Specifically, given a set of basic 3D scenepoints, they represent the human-scene interaction using the minimum distance from every such pointto the human body surface.

Original: As in the semantic-based case, this strategy only gives a weak prior on the human pose, as itdoes not explicitly defines which joint should be in contact with which scene point.
Revised: As in the semantic-based case, this strategy only gives a weak prior onthe human pose, as it does not explicitly defines which joint should be in contact with which scenepoint.

Revised: In SAMP [10], their approach only models a coarse interaction of the human with a givenobject in the final frame by predicting the final root location and orientation.

Revised: Such coarse interactionhowever cannot constrain the poses at intermediate frames.

Original: By contrast, ourper-joint contact maps provide a more detailed contact information for every human joint.
Revised: By contrast, our per-joint contact mapsprovide a more detailed contact information for every human joint at each future frame.

Revised: Hand-object interaction Although hand-object contact relationships have already been studiedfor the task of grasping [2, 29, 3, 14], existing methods cannot be naively applied to human-sceneinteractions because their object-centric contact relationships tend to be static across time.

Revised: Forexample, when we are using a hammer, we will grasp the handle tightly, and thus the contact regionbetween our palms and the hammer does not change across time.

Revised: By contrast, our human-scenecontact maps changes across the frames for almost all human activities.

Revised: This motivates us to proposea distance-based per-joint contact map at each frame.

Original:  Approach
Revised:  Approach

Original: Let us now introduce our approach to scene-aware 3D human motion forecasting.
Revised: Let us now introduce our approach to scene-aware 3D human motion forecasting.

Original: Following previouswork [18], a sequence of P past human poses is represented as X =
Revised: Following previouswork [22], a sequence of P past human poses is represented as X =

Original: [ x 1 , x 2 , · · · , x P ] ∈ R P × J × 3 ,where x p ∈ R J × 3 encodes the 3D locations of all J joints at time p in a global reference frame.
Revised: [ x 1 , x 2 , · · · , x P ] ∈ R P × J × 3 ,where x p ∈ R J × 3 encodes the 3D locations of all J joints at time p in a global reference frame.

Original: The3D scene is represented as a set of 3D points S ∈ R N × 3 .
Revised: The3D scene is represented as a set of 3D points S ∈ R N × 3 .

Original: Given the historical motion X and the3D scene S , our goal is then to forecast T future human poses Y =
Revised: Given the historical motion X and the3D scene S , our goal is then to forecast T future human poses Y =

Original: [ x P +1 , x P +2 , · · · , x P + T ] ∈ 97 R T
Revised: [ x P +1 , x P +2 , · · · , x P + T ] ∈ 109 R T

Original: × J × 3 .
Revised: × J × 3 .

Original: To this end, we introduce a two-stage pipeline that first predicts future human-scene contactmaps, and then forecasts the future poses by conditioning them on these contact maps.
Revised: To this end, we introduce a two-stage pipeline that first predicts future human-scene contactmaps, and then forecasts the future poses by conditioning them on these contact maps.

Original: An overviewof our pipeline is shown in Fig.
Revised: An overviewof our pipeline is shown in Fig.

Original: Below, we present our contact representation, and our pipeline topredict future contact maps and future motion.
Revised: Below, we present our contact representation, and our pipeline topredict future contact maps and future motion.

Original: 3.1 Per-joint Scene Contact Map
Revised: 3.1 Per-joint Scene Contact Map

Original: We represent human-scene contact using the distances between the human joints and the scene points.
Revised: We represent human-scene contact using the distances between the human joints and the scene points.

Original: Specifically, given a human pose x and a 3D scene S , we first compute the per-joint distance mapd ∈ R J × N , where each entry d jn encodes the ℓ 2 distance between the j -th human joint and n -thscene point, i.e.,
Revised: Specifically, given a human pose x and a 3D scene S , we first compute the per-joint distance mapd ∈ R J × N , where each entry d jn encodes the ℓ 2 distance between the j -th human joint and n -thscene point, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where j ∈ { 1 , 2 , · · · , J } and n ∈ { 1 , 2 , · · · , N } .
Revised: where j ∈ { 1 , 2 , · · · , J } and n ∈ { 1 , 2 , · · · , N } .

Original: In the distance map, the scene points that are less relevant to a human joint, because they are far awayfrom it, have a higher value.
Revised: In the distance map, the scene points that are less relevant to a human joint, because they are far awayfrom it, have a higher value.

Original: This will tend to give them more influence when used in a deep neuralnetwork and may in turn cause issues when training our contact prediction module.
Revised: This will tend to give them more influence when used in a deep neuralnetwork and may in turn cause issues when training our contact prediction module.

Original: To address this,we normalize the distance map to obtain a continuous contact map c ∈ R J × N whose elements aredefined as
Revised: To address this,we normalize the distance map to obtain a continuous contact map c ∈ R J × N whose elements aredefined as

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Figure]

Original: where the constant σ is the normalizing factor.
Revised: where the constant σ is the normalizing factor.

Original: In such a contact map, closer scene points have highervalues than far-away ones, whose values will be very close to zero.
Revised: In such a contact map, closer scene points have highervalues than far-away ones, whose values will be very close to zero.

Original: Given the 3D scene S and past human poses X , we compute the sequence of past contact maps
Revised: Given the 3D scene S and past human poses X , we compute the sequence of past contact maps

Original: C =
Revised: C =

Original: [ c 1 , c 2 , · · · , c P ] ∈ R P × J × N as described above.
Revised: [ c 1 , c 2 , · · · , c P ] ∈ R P × J × N as described above.

Original: Our goal then is to predict the future contactmaps M =
Revised: Our goal then is to predict the future contactmaps M =

Original: [ c P +1 , c P +2 , · · · , c P + T ] ∈ R T × J × N .
Revised: [ c P +1 , c P +2 , · · · , c P + T ] ∈ R T × J × N .

Original: In the next section, we introduce a contact mapprediction module to do so.
Revised: In the next section, we introduce a contact mapprediction module to do so.

Original: 3.2 Contact Prediction Network
Revised: 3.2 Contact Prediction Network

Original: Since our contact maps are based on distances, they are smooth over time.
Revised: Since our contact maps are based on distances, they are smooth over time.

Original: Therefore, similarly to [18]for human motion, we adopt a temporal encoding strategy based on the Discrete Cosine Transform(DCT).
Revised: Therefore, similar to [22]for human motion, we adopt a temporal encoding strategy based on the Discrete Cosine Transform(DCT).

Original: With the DCT, a sequence is represented as a linear combination of a set of pre-defined cosinebases.
Revised: With the DCT, a sequence is represented as a linear combination of a set of pre-defined cosinebases.

Original: By discarding the high-frequency components, the DCT provides a compact representation,which nicely captures the smoothness of the sequence.
Revised: By discarding the high-frequency components, the DCT provides a compact representation,which nicely captures the smoothness of the sequence.

Original: More formally, let us denote a P -frame sequence of contact values between the j -th human joint andthe n -th scene point as ˜ c
Revised: More formally, let us denote a P -frame sequence of contact values between the j -th human joint andthe n -th scene point as ˜ c

Original: jn
Revised: jn

Original: =
Revised: =

Original: [ c 1 jn , c 2 jn , · · · c Pjn ] .
Revised: [ c 1 jn , c 2 jn , · · · c Pjn ] .

Original: This sequence can be fully represented using P
Revised: This sequence can be fully represented using P

Original: DCT coefficients, with the l -th one given by
Revised: DCT coefficients, with the l -th one given by

Original: [Equation]
Revised: [Equation]

Original: where l ∈ { 1 , 2 , · · · P } and δ ij denotes the Kronecker delta function, i.e.,
Revised: where l ∈ { 1 , 2 , · · · P } and δ ij denotes the Kronecker delta function, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Given these DCT coefficients, the sequence in the original space can be obtained via the Inverse
Revised: Given these DCT coefficients, the sequence in the original space can be obtained via the Inverse

Original: Discrete Cosine Transform (IDCT) as
Revised: Discrete Cosine Transform (IDCT) as

Original: [Equation]
Revised: [Equation]

Original: where L is the number of DCT coefficients used.
Revised: where L is the number of DCT coefficients used.

Original: For a lossless reconstruction, L = P , but when thesequence is smooth, one can ignore the coefficients of the high-frequency DCT bases and thus use
Revised: For a lossless reconstruction, L = P , but when thesequence is smooth, one can ignore the coefficients of the high-frequency DCT bases and thus use

Original: L < P at a negligible loss.
Revised: L < P at a negligible loss.

Original: Recall that our goal is to predict the future T contact maps M =
Revised: Recall that our goal is to predict the future T contact maps M =

Original: [ c P +1 , c P +2 , · · · , c P + T ] fromthe past P ones C
Revised: [ c P +1 , c P +2 , · · · , c P + T ] fromthe past P ones C

Original: =
Revised: =

Original: [ c 1 , c 2 , · · · , c P ] .
Revised: [ c 1 , c 2 , · · · , c P ] .

Original: To leverage the DCT representation, we then reformulatethis problem as learning a mapping from the DCT coefficients of the past contact maps to those ofthe future ones.
Revised: To leverage the DCT representation, we then reformulate this problem as learning a mapping from the DCT coefficients of the past contact maps to those ofthe future ones.

Original: Specifically, following the padding strategy of [18], we first repeat the last contactmap c P T times to create a sequence of P + T contact maps C ′
Revised: Specifically, following the padding strategy of [22], we first repeat the last contactmap c P T times to create a sequence of P + T contact maps C ′

Original: =
Revised: =

Original: [ c 1 , c 2 , · · · , c P , c P , · · · , c P ] ∈ 138 R ( P + T ) × J × N .
Revised: [ c 1 , c 2 , · · · , c P , c P , · · · , c P ] ∈ 150 R ( P + T ) × J × N .

Original: We then compute the DCT coefficients of this sequence and define it as H ∈ R
Revised: We then compute the DCT coefficients of this sequence and define it as H ∈ R

Original: L × J × N .
Revised: L × J × N .

Original: Note that for each joint we only retain the first L DCT coefficients.
Revised: Note that for each joint we only retain the first L DCT coefficients.

Original: Our goal then is to learn a residualbetween these DCT coefficients H and those of the real sequence [ c 1 , c 2 , · · · , c P + T ] .
Revised: Our goal then is to learn a residualbetween these DCT coefficients H and those of the real sequence [ c 1 , c 2 , · · · , c P + T ] .

Original: We formulate this as a point-cloud encoding problem.
Revised: We formulate this as a point-cloud encoding problem.

Original: That is, for each scene point, we regardthe corresponding DCT coefficients of all joints as its feature.
Revised: That is, for each scene point, we regardthe corresponding DCT coefficients of all joints as its feature.

Original: Thus, every scene point has aninitial feature vector of size LJ .
Revised: Thus, every scene point has aninitial feature vector of size LJ .

Original: Our contact prediction network then takes as input the 3D scenepoints S , the DCT features H and the past human poses X , and outputs the future contact mapsˆ C
Revised: Our contact prediction network then takes as input the 3D scenepoints S , the DCT features H and the past human poses X , and outputs the future contact mapsˆ C

Original: = [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] as
Revised: = [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] as

Original: [Equation]
Revised: [Equation]

Original: (6) where F represents the trainable point-cloud processing model.
Revised: where F represents the trainable point-cloud processing model and G x is the GRU encoder.

Original: Specifically, as shown in Fig. 2, we use the Point-Voxel CNN (PVCNN)
Revised: Specifically, as shown in Fig. 2, we use the Point-Voxel CNN (PVCNN)

Original: [16] to encode the 3Dscene with its DCT feature vectors.
Revised: [20] to encode the 3Dscene with its DCT feature vectors.

Original: The PVCNN was designed to process a 3D point cloud.
Revised: The PVCNN was designed to process a 3D point cloud.

Original: Itincorporates voxel-based convolutions and point-based representations, leading to a memory- andcomputation-efficient structure for 3D data.
Revised: Itincorporates voxel-based convolutions and point-based representations, leading to a memory- andcomputation-efficient structure for 3D data.

Original: We adapt the PVCNN to also take as input the pasthuman poses X , encoded by a Gated Recurrent Unit (GRU) [5], to predict a residual of the DCTcoefficients.
Revised: We adapt the PVCNN to also take as input the pasthuman poses X , encoded by a Gated Recurrent Unit (GRU) [7], to predict a residual of the DCTcoefficients.

Original: We then obtain the predicted contacted maps ˆ C
Revised: We then obtain the predicted contacted maps ˆ C

Original: =
Revised: =

Original: [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] via the IDCT.
Revised: [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] via the IDCT.

Original: Note that, here, we seek not only to predict the future contact maps but also to recover the past ones.
Revised: Note that, here, we seek not only to predict the future contact maps but also to recover the past ones.

Original: To this end, we use the average ℓ 2 loss between the ground-truth contact maps and the predicted ones.
Revised: To this end, we use the average ℓ 2 loss between the ground-truth contact maps and the predicted ones.

Original: Formally, this loss is defined as
Revised: Formally, this loss is defined as

Original: [Equation]
Revised: [Equation]

Original: 3.3 Motion Forecasting Network
Revised: 3.3 Motion Forecasting Network

Original: Given the future contact maps M and the past human motion X , our human motion forecastingmodule aims to predict the future human poses Y .
Revised: Given the future contact maps M and the past human motion X , our human motion forecastingmodule aims to predict the future human poses Y .

Original: To this end, we first extract the closest contactscene point to each joint from the contact maps.
Revised: To this end, we first extract the closest contactscene point to each joint from the contact maps.

Original: Given these contact points, we first use a simpleneural network to predict the future path (global translations) and then forecast the future local poseswith an RNN-based model.
Revised: Given these contact points, we first use a simpleneural network to predict the future path (global translations) and then forecast the future local poseswith an RNN-based model.

Original: Let us discuss these steps in more detail.
Revised: Let us discuss these steps in more detail.

Original: Contact points.
Revised: Contact points.

Original: Since our focus now is to predict the 3D location of human joints, we propose toretain the most relevant scene point i.e., the closest one to each joint.
Revised: Since our focus now is to predict the 3D location of human joints, we propose toretain the most relevant scene point i.e., the closest one to each joint.

Original: Specifically, given the contactmap at time step p c p ∈ R J × N and the 3D scene points S ∈ R N × 3 , we would like to find the scenepoint that is closest to each human joint.
Revised: Specifically, given the contactmap at time step p c p ∈ R J × N and the 3D scene points S ∈ R N × 3 , we would like to find the scenepoint that is closest to each human joint.

Original: Let us denote the resulting contact points as q p ∈ R J × 4 ,each row of which stores the 3D location of the scene point closest to the corresponding human jointtogether with a binary value indicating whether the joint truly is in contact with the scene or not.
Revised: Let us denote the resulting contact points as q p ∈ R J × 4 ,each row of which stores the 3D location of the scene point closest to the corresponding human jointtogether with a binary value indicating whether the joint truly is in contact with the scene or not.

Original: More formally, the contact point for joint j is computed as [Equation]
Revised: More formally, the contact point for joint j is computed asq pj =  

Original: , 1] , where k = argmax n = { 1 , 2 , ··· ,N

Original: [Equation] c j,n , if c j,k > ϵ [Equation] otherwise ,(8) where S k is the 3D location of the k -th scene point, and ϵ is a threshold to determine whetherthe joint is in contact with the scene or not.
Revised: [ S k , 1] , where k = argmax n = { 1 , 2 , ··· ,N } c j,n , if c j,k > ϵ [0 , 0 , 0 , 0] otherwise , where S k is the 3D location of the k -th scene point, and ϵ is a threshold to determine whetherthe joint is in contact with the scene or not.

Original: We compute such contact points for the entire futuresequence, which yields a sequence of contact points Q =
Revised: We compute such contact points for the entire futuresequence, which yields a sequence of contact points Q =

Original: [ q P +1 , q P +2 , · · · , q P + T ] ∈ R T
Revised: [ q P +1 , q P +2 , · · · , q P + T ] ∈ R T

Original: × J × 4 ,where q p ∈ R J × 4 is the contact points at the p -th time step.
Revised: × J × 4 ,where q p ∈ R J × 4 is the contact points at the p -th time step.

Original: Motion forecasting.
Revised: Motion forecasting.

Original: As shown in Fig 2, our forecasting model first predicts the global translation ineach frame and then the local motion given the global translations.
Revised: As shown in Fig 2, our forecasting model first predicts the global translation ineach frame and then the local motion given the global translations.

Original: Specifically, given the past motion
Revised: Specifically, given the past motion

Original: X and the contact points Q , we use a simple multilayer perceptron (MLP) to predict the future globaltranslations Y root ∈ R T × 3 as
Revised: X and the contact points Q , we use a simple multilayer perceptron (MLP) to predict the future globaltranslations Y root ∈ R T × 3 as

Original: [Equation]
Revised: [Equation]

Original: Y

Original: [Equation]
Revised: [Equation]

Original: (9) where M represents the MLP.
Revised: where M represents the MLP and ˜

Revised: G x is the GRU to encode the past motion.

Original: The global translations are then feed into a GRU to predict the local pose at each future timestep.
Revised: The global translations are then fed into a GRU to predict the local pose at each future timestep.

Original: Assuming that the past and future local pose sequences are represented by X local =[ x local, x local, · · · , x local P ] ∈ R P × ( J − 1) × 3 and Y local =
Revised: Assuming that the past and future local pose sequences are represented by X local =[ x local, x local, · · · , x local P ] ∈ R P × ( J − 1) × 3 and Y local =

Original: [ x local P +1 , x local P +2 , · · · , x local P + T ] ∈ R T × ( J − 1) × 3 ,respectively, this can be expressed as
Revised: [ x local P +1 , x local P +2 , · · · , x local P + T ] ∈ R T × ( J − 1) × 3 ,respectively, this can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where G denotes the GRU, p ∈ { P + 1 , P + 2 , · · · , P + T } , and ˆ x local p ∈ R [Equation] [Equation] 3 , ˆ x
Revised: where G denotes the GRU to predict the furture poses, p ∈ { P + 1 , P + 2 , · · · , P + T } , andˆ x local p ∈ R (

Original: [Equation]
Revised: [Equation]

Original:  p ∈ R [-
 andq-] p ∈ R J × 4 are the local human pose, global translation and contact points at time p , respectively.
Revised: − 1) × 3 , ˆ x root p ∈ R [-
 andq-]3 and q p ∈ R J × 4 are the local human pose, global translation andcontact points at time p , respectively.

Original: The global translation and local motion prediction modules are trained jointly.
Revised: The global translation and local motion prediction modules are trained jointly.

Original: To this end, we use 3loss terms.
Revised: To this end, we use 3loss terms.

Original: The first one is a global translation loss defined as
Revised: The first one is a global translation loss defined as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where x root p ∈ R 3 and ˆ x root p ∈ R 3 are the ground-truth and predicted global translations at time p .
Revised: where x root p ∈ R 3 and ˆ x root p ∈ R 3 are the ground-truth and predicted global translations at time p .

Original: The second loss accounts for the local human pose prediction and is expressed as
Revised: The second loss accounts for the local human pose prediction and is expressed as

Original: [Equation]
Revised: [Equation]

Original:  where x local pj ∈ R 3 and ˆ x local pj ∈ R 3 are the ground-truth and predicted local positions of the j -th jointat time p .
Revised: 11) where x local pj ∈ R 3 and ˆ x local pj ∈ R 3 are the ground-truth and predicted local positions of the j -th jointat time p .

Original: Finally, our third loss term encodes a contact prior based on the contact points.
Revised: Finally, our third loss term encodes a contact prior based on the contact points.

Original: We define it as
Revised: We define it as

Original: [Equation]
Revised: [Equation]

Original: where ˆ x pj ∈ R 3 is the predicted location of joint j at time p , obtained by adding the predicted globaltranslation at time p to the corresponding local pose.
Revised: where ˆ x pj ∈ R 3 is the predicted location of joint j at time p , obtained by adding the predicted globaltranslation at time p to the corresponding local pose.

Original: q pj [1:3] ∈ R 3 and q pj 4 ∈ { 0 , 1 } are the 3Dcoordinates of the contact scene point and the indicator value, respectively.
Revised: q pj [1:3] ∈ R 3 and q pj 4 ∈ { 0 , 1 } are the 3Dcoordinates of the contact scene point and the indicator value, respectively.

Original: The overall loss is then expressed as
Revised: The overall loss is then expressed as

Original: [Equation]
Revised: [Equation]

Original: We use a stage-wise training strategy where the contact prediction network and motion forecastingnetwork are trained separately.
Revised: We use a stage-wise training strategy where the contact prediction network and motion forecastingnetwork are trained separately.

Original: During training, the motion forecasting network is given the groundtruth contact points as input.
Revised: During training, the motion forecasting network is given the groundtruth contact points as input.

Original: At test time, we first compute the contact points from the predictedcontact maps and then use these contact points to forecast the future human motion.
Revised: At test time, we first compute the contact points from the predictedcontact maps and then use these contact points to forecast the future human motion.

Original:  Experiments
Revised:  Experiments

Original: [Figure]

Original: 4.1 Datasets
Revised: 4.1 Datasets

Original: We evaluate our method on two datasets, GTA-IM [4] and PROX [8].
Revised: We evaluate our method on two datasets, GTA-IM [6] and PROX [11].

Original: GTA-IM.
Revised: GTA-IM.

Original: The GTA Indoor Motion dataset [4] is a large-scale synthetic dataset that captures humanscene interactions.
Revised: The GTA Indoor Motion dataset [6] is a large-scale synthetic dataset that captures humanscene interactions.

Original: It consists of 50 different characters performing various activities in 7 differentscenes.
Revised: It consists of 50 different characters performing various activities in 7 differentscenes.

Original: Each scene is a building, and each building has several rooms on one or more floors.
Revised: Each scene is a building, and each building has several rooms on one or more floors.

Original: Thedataset contains around 1 million RGB-D frames together with the corresponding 3D human poses.
Revised: Thedataset contains around 1 million RGB-D frames together with the corresponding 3D human poses.

Original: We use 4 of the scenes as our training set (“r001",“r002",“r003",“r006") and the remaining 3 as ourtest set (“r010",“r011",“r013") 1 .
Revised: We use 4 of the scenes as our training set (“r001",“r002",“r003",“r006") and the remaining 3 as ourtest set (“r010",“r011",“r013") 1 .

Original: To obtain the 3D point clouds of the different scenes, we registertheir depth maps from different videos sequences with the ground-truth camera extrinsic matrices.
Revised: To obtain the 3D point clouds of the different scenes, we registertheir depth maps from different videos sequences with the ground-truth camera extrinsic matrices.

Original: Following [4], we use 21 out of the 98 human joints provided by the dataset.
Revised: Following [6], we use 21 out of the 98 human joints provided by the dataset.

Original: The videos run at 30Hz.
Revised: The videos run at 30Hz.

Original: We train our models to observe the past 30 time steps (1 second) and predict the future 60 time steps(2 seconds). 
Revised: We train our models to observe the past 30 time steps (1 second) and predict the future 60 time steps(2 seconds). [Table] [Table]

Original: PROX.
Revised: PROX.

Original: Proximal Relationships with Object eXclusion (PROX) [8] is a real dataset captured using a
Revised: Proximal Relationships with Object eXclusion (PROX) [11] is a real dataset captured using a

Original: Kinect-One sensor.
Revised: Kinect-One sensor.

Original: It comprises 12 different scenes with 20 subjects interacting with the scenes.
Revised: It comprises 12 different scenes with 20 subjects interacting with the scenes.

Original: Thedataset also provides SMPL-X parameters [21] as the ground-truth human pose and shape in eachframe.
Revised: Thedataset also provides SMPL-X parameters [25] as the ground-truth human pose and shape in eachframe.

Original: Since these parameters were obtained by a frame-wise fitting algorithm, the motion sequences [Table] [Table] are jittery and thus ill-suited to our task.
Revised: Since these parameters were obtained by a frame-wise fitting algorithm, the motion sequencesare jittery and thus ill-suited to our task.

Original: We therefore refine the dataset via a simple temporaloptimization process to generate smooth motions.
Revised: We therefore refine the dataset via a simple temporaloptimization process to generate smooth motions.

Original: More details are provided in the supplementarymaterial.
Revised: More details are provided in the supplementarymaterial.

Original: Following [28], we use 8 scenes for training (“N3Library”, “MPH112”, “MPH11”, “MPH8”,“BasementSittingBooth”, “N0Sofa”, “N3Office”, “Werkraum”) and 4 scenes for testing (“MPH16”,“MPH1Library”, “N0SittingBooth”, “N3OpenArea”).
Revised: Following [33], we use 8 scenes for training (“N3Library”, “MPH112”, “MPH11”, “MPH8”,“BasementSittingBooth”, “N0Sofa”, “N3Office”, “Werkraum”) and 4 scenes for testing (“MPH16”,“MPH1Library”, “N0SittingBooth”, “N3OpenArea”).

Original: We use the 22 body joints of SMPL-X model.
Revised: We use the 22 body joints of SMPL-X model.

Original: As for GTA-IM, the frame-rate of this dataset is 30 Hz, and we train our models to take the past 30time steps as input and predict the future 60 steps.
Revised: As for GTA-IM, the frame-rate of this dataset is 30 Hz, and we train our models to take the past 30time steps as input and predict the future 60 steps.

Original: 4.2 Metrics, Baselines & Implementation
Revised: 4.2 Metrics, Baselines & Implementation

Original: Metrics.
Revised: Metrics.

Original: We use the Mean Per Joint Position Error (MPJPE) [10] to evaluate both the globaltranslations (path error) and the local motion (pose error).
Revised: We use the Mean Per Joint Position Error (MPJPE) [13] to evaluate both the globaltranslations (path error) and the local motion (pose error).

Original: Baselines.
Revised: Baselines.

Original: We compare our method with two human motion prediction models (LTD [18] and
Revised: We compare our method with two human motion prediction models (LTD [22] and

Original: DMGNN [14]) and one scene-aware human motion synthesis method (SLT [28]).
Revised: DMGNN [18]) and one scene-aware human motion synthesis method (SLT [33]).

Original: LTD [18] is a representative feed-forward method based on Graph Convolutional Networks (GCNs) [12].
Revised: LTD [22] is a representative feed-forward method based on Graph Convolutional Networks (GCNs) [16].

Original: DMGNN [14]is a state-of-the-art RNN-based approach for human motion prediction.
Revised: DMGNN [18]is a state-of-the-art RNN-based approach for human motion prediction.

Original: SLT [28] is a stage-wiseapproach to synthesize long-term human motions.
Revised: SLT [33] is a stage-wiseapproach to synthesize long-term human motions.

Original: We used the official implementations of LTD [18]and DMGNN [14] to train them on our datasets.
Revised: We used the official implementations of LTD [22]and DMGNN [18] to train them on our datasets.

Original: For SLT [28], we adapted the official code to ourtask e.g., we modified their model so as to take the past motion as input.
Revised: For SLT [33], we adapted the official code to ourtask e.g., we modified their model so as to take the past motion as input.

Original: The detail of these changesare provided in the supplementary material.
Revised: The detail of these changesare provided in the supplementary material.

Original: Implementation details.
Revised: Implementation details.

Original: Our models are implemented in Pytorch [20] and trained using the
Revised: Our models are implemented in Pytorch [24] and trained using the

Original: ADAM [11] optimizer.
Revised: ADAM [15] optimizer.

Original: Both our contact prediction network and motion forecasting one are trainedfor 50 epochs with learning rates of 0.0005 and 0.001, respectively.
Revised: Both our contact prediction network and motion forecasting one are trainedfor 50 epochs with learning rates of 0.0005 and 0.001, respectively.

Original: The training of each network takesabout 12 hours on a 24GB NVIDIA RTX3090Ti GPU.
Revised: The training of each network takesabout 12 hours on a 24GB NVIDIA RTX3090Ti GPU.

Original: For both datasets, the normalizing factor σ ,the number of DCT coefficients L and the contact threshold ϵ are set to 0 .
Revised: For both datasets, the normalizing factor σ , [Figure] the number of DCT coefficients L and the contact threshold ϵ are set to 0 .

Original: 2 , 20 and 0 .
Revised: 2 , 20 and 0 .

Original: 32 , respectively.
Revised: 32 , respectively.

Original: For the motion forecasting network, the loss weights ( λ 1 , λ 2 , λ 3 ) are set to (1 . 0 , 1 . 0 , 0 . 1) for bothdatasets.
Revised: For the motion forecasting network, the loss weights ( λ 1 , λ 2 , λ 3 ) are set to (1 . 0 , 1 . 0 , 0 . 1) for bothdatasets.

Original: Additional implementation details are given in the supplementary material.
Revised: Additional implementation details are given in the supplementary material.

Original: [Figure]

Original: 4.3 Results
Revised: 4.3 Results

Original: Quantitative results.
Revised: Quantitative results.

Original: We provide quantitative results on GTA-IM and PROX in Table 1 and 2,respectively.
Revised: We provide quantitative results on GTA-IM and PROX in Table 1 and 2,respectively.

Original: Our approach outperforms the baselines for 3D paths and poses on both datasets acrossalmost all time steps by a large margin.
Revised: Our approach outperforms the baselines for 3D paths and poses on both datasets acrossalmost all time steps by a large margin.

Original: Specifically, the baseline models either perform well for the3D path but comparatively poorly for the local poses, e.g., SLT [28] with an average path error of121.

Revised: Specifically, the baseline models either perform well for the3D path but comparatively poorly for the local poses, e.g., SLT [33] with an average path error of121.

Original: mm but the highest pose error on GTA-IM, or the reverse, e.g., DMGNN [14].
Revised: mm but the highest pose error on GTA-IM, or the reverse, e.g., DMGNN [18].

Original: By contrast, ourmodels produce more accurate 3D paths and poses than those of the baselines.
Revised: By contrast, ourmodels produce more accurate 3D paths and poses than those of the baselines.

Original: As an ablation study, we trained our human motion forecasting network without contact maps (“Oursw/o contact”) and observe an increase of up-to 10mm in the mean path error and 7mm in the meanpose error.
Revised: As an ablation study, we trained our human motion forecasting network without contact maps (“Oursw/o contact”) and observe an increase of up-to 10mm in the mean path error and 7mm in the meanpose error.

Original: By contrast, using the ground-truth contact maps (“Ours w/ GT contact”) yields a furtherperformance boost, especially on the mean path error with a decrease of up-to 104mm.
Revised: By contrast, using the ground-truth contact maps (“Ours w/ GT contact”) yields a furtherperformance boost, especially on the mean path error with a decrease of up-to 104mm.

Original: This indicatesthe effectiveness of conditioning the motion predictions on per-joint contact maps.
Revised: This indicatesthe effectiveness of conditioning the motion predictions on per-joint contact maps.

Original: Qualitative results.
Revised: Qualitative results.

Original: We show our contact maps of four joints (left foot, left elbow, right foot and rightelbow) in Fig.
Revised: We show our contact maps of four joints (left foot, left elbow, right foot and rightelbow) in Fig.

Original: Our model predicts accurate contact maps for diverse motions, such as “walking”(top) and “sitting down” (bottom).
Revised: Our model predicts accurate contact maps for diverse motions, such as “walking”(top) and “sitting down” (bottom).

Original: Note that, for the sample from GTA-IM [4] where the subject isabout to walk around a corner, our contact maps precisely capture the contacts between the left elbowand the wall (shown as purple points on the wall), leading to accurate human motion predictions.
Revised: Note that, for the sample from GTA-IM [6] where the subject isabout to walk around a corner, our contact maps precisely capture the contacts between the left elbowand the wall (shown as purple points on the wall), leading to accurate human motion predictions.

Original: We further compare our results with those of the baselines in Fig.
Revised: We further compare our results with those of the baselines in Fig.

Original: We restrict this comparison to
Revised: We restrict this comparison to

Original: DMGNN [14] and SLT [28], which are quantitatively more accurate than LTD [18].
Revised: DMGNN [18] and SLT [33], which are quantitatively more accurate than LTD [22].

Original: The completecomparison is included in the supplementary material.
Revised: The completecomparison is included in the supplementary material.

Original: Due to the lack of explicit constraints onglobal motion and local movements, the baseline methods tend to produce unrealistic human motions,such as motions with almost no local movements but large global translations, i.e., ghost motion(highlighted with a red box), walking through a sofa (highlighted with green boxes).
Revised: Due to the lack of explicit constraints onglobal motion and local movements, the baseline methods tend to produce unrealistic human motions,such as motions with almost no local movements but large global translations, i.e., ghost motion(highlighted with a red box), walking through a sofa (highlighted with green boxes).

Original: Thanks to ourper-joint contact maps, our results are more plausible and closer to the ground truth.
Revised: Thanks to ourper-joint contact maps, our results are more plausible and closer to the ground truth.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have introduced a framework for scene-aware human motion forecasting thatencourages consistency between global motion and local poses by exploiting human-scene contacts.
Revised: In this paper, we have introduced a framework for scene-aware human motion forecasting thatencourages consistency between global motion and local poses by exploiting human-scene contacts.

Revised: [Figure]

Original: To this end, we have proposed a per-joint contact map representation that captures the contactrelationships between every human joint and the scene points.
Revised: To this end, we have proposed a per-joint contact map representation that captures the contactrelationships between every human joint and the scene points.

Original: Our model consists of two stages.
Revised: Our model consists of two stages.

Original: We [Figure] first predict the per-joint contact maps given the motion history, and then forecast the future globaltranslations and local poses given the estimated future contact maps.
Revised: Wefirst predict the per-joint contact maps given the motion history, and then forecast the future globaltranslations and local poses given the estimated future contact maps.

Original: Thanks to the explicit constraintsprovided by our per-joint contact maps, our method yields more plausible and more accurate futurehuman motions than the state-of-the-art motion prediction or scene-aware human synthesis strategies.
Revised: Thanks to the explicit constraintsprovided by our per-joint contact maps, our method yields more plausible and more accurate futurehuman motions than the state-of-the-art motion prediction or scene-aware human synthesis strategies.

Original:  One limitation of our work is that the quality of our predicted motions depends on that of the contactmaps.
Revised: Limitations & Societal Impacts One limitation of our work is that the quality of our predictedmotions depends on that of the contact maps.

Original: As evidenced by our results with ground-truth contact maps, improving the contact mappredictions translates in better motion predictions.
Revised: As evidenced by our results with ground-truth contactmaps, improving the contact map predictions translates in better motion predictions.

Original: This will therefore be one of our future researchdirections.
Revised: This willtherefore be one of our future research directions.

Revised: Additionally, in many applications where thehuman shape matters, our joint-based contact map may not be enough to regularize the human surface.

Revised: In our future work, we would like to extend our contact map to human surface as well.

Original: Furthermore, in real application scenario, a potential risk of our method is that it maypredict future motions that do not obey real physical rules, e.g., motion with imbalanced forces.
Revised: Furthermore,in real application scenario, a potential risk of our method is that it may predict future motions thatdo not obey real physical rules, e.g., motion with imbalanced forces.

Original: Forexample, in the scenario of human-robot interaction where the agent i.e., a robot, needs to plan itsactions according to the future human motion, such physically unrealistic future motion may lead tounsafe situations such as collision.
Revised: For example, in the scenarioof human-robot interaction where the agent i.e., a robot, needs to plan its actions according to thefuture human motion, such physically unrealistic future motion may lead to unsafe situations such ascollision.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] see Section 
Revised: [Yes] see Section 1(b)

Original: (b) Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] see Section
Revised: [Yes] see Section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] seeSection
Revised: [Yes] seeSection

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] Data and
Revised: [Yes] Data and

Original: implementation details are in Section 4 and the supplementary material.
Revised: implementation details are in Section 4 and the supplementary material.

Original: We will release
Revised: We will release

Original: our source code upon the acceptance of this paper.
Revised: our source code upon the acceptance of this paper.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] see Section 4 and the supplementary material
Revised: [Yes] see Section 4 and the supplementary material(c)

Original: (c) Did you report error bars (e
Revised: Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary material
Revised: [Yes] See the supplementary material

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] see the supplementary material
Revised: [Yes] see the supplementary material

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re 
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board ) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: ) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]