Original: Contact-aware Human Motion Forecasting
Revised: Contact-aware Human Motion Forecasting

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In this paper, we tackle the task of scene-aware 3D human motion forecasting,which consists of predicting future human poses given a 3D scene and a past humanmotion.
Revised: In this paper, we tackle the task of scene-aware 3D human motion forecasting, which consists of predicting future human poses given a 3D scene and a past human motion.

Original: A key challenge of this task is to ensure consistency between the humanand the scene, accounting for human-scene interactions.
Revised: A key challenge of this task is to ensure consistency between the human and the scene, accounting for human-scene interactions.

Original: Previous attempts to doso model such interactions only implicitly, and thus tend to produce artifacts suchas “ghost motion" because of the lack of explicit constraints between the localposes and the global motion.
Revised: Previous attempts to do so model such interactions only implicitly, and thus tend to produce artifacts such as “ghost motion" because of the lack of explicit constraints between the local poses and the global motion.

Original: Here, by contrast, we propose to explicitly modelthe human-scene contacts.
Revised: Here, by contrast, we propose to explicitly model the human-scene contacts.

Original: To this end, we introduce distance-based contact mapsthat capture the contact relationships between every joint and every 3D scenepoint at each time instant.
Revised: To this end, we introduce distance-based contact maps that capture the contact relationships between every joint and every 3D scene point at each time instant.

Original: We then develop a two-stage pipeline that first predictsthe future contact maps from the past ones and the scene point cloud, and thenforecasts the future human poses by conditioning them on the predicted contactmaps.
Revised: We then develop a two-stage pipeline that first predicts the future contact maps from the past ones and the scene point cloud, and then forecasts the future human poses by conditioning them on the predicted contact maps.

Original: During training, we explicitly encourage consistency between the globalmotion and the local poses via a prior defined using the contact maps and futureposes.
Revised: During training, we explicitly encourage consistency between the global motion and the local poses via a prior defined using the contact maps and future poses.

Original: Our approach outperforms the state-of-the-art human motion forecastingand human synthesis methods on both synthetic and real datasets.
Revised: Our approach outperforms the state-of-the-art human motion forecasting and human synthesis methods on both synthetic and real datasets.

Revised: Our code is available at https://github.com/wei-mao-2019/ContAwareMotionPred .

Original:  Introduction
Revised:  Introduction

Original: Human motion prediction has a broad application potential covering human robot interaction [17],autonomous driving [23], virtual/augmented reality (AR/VR) [28] and animation [31].
Revised: Human motion prediction has a broad application potential covering human robot interaction [17], autonomous driving [23], virtual/augmented reality (AR/VR) [28] and animation [31].

Original: As such, ithas been an active research topic for decades [4, 27, 30, 35].
Revised: As such, it has been an active research topic for decades [4, 27, 30, 35].

Original: Nevertheless, most methods [19, 1,22, 32, 9, 18, 21, 5] disregard the fact that humans evolve in 3D environments, thus ignoring thehuman-scene interactions.
Revised: Nevertheless, most methods [19, 1, 22, 32, 9, 18, 21, 5] disregard the fact that humans evolve in 3D environments, thus ignoring the human-scene interactions.

Original: By contrast, in this work, we tackle the task of scene-aware human motionforecasting which aims to incorporate the scene context to predict future 3D human motions.
Revised: By contrast, in this work, we tackle the task of scene-aware human motion forecasting, which aims to incorporate the scene context to predict future 3D human motions.

Original: While some recent works in human motion prediction [6, 8] and human synthesis [33] have startedto explore the use of scene context, they do so implicitly, by taking an embedding of a 2D sceneimage [6] a 3D scene [33], or a specific object [8] as an additional input to their model.
Revised: While some recent works in human motion prediction [6, 8] and human synthesis [33] have started to explore the use of scene context, they do so implicitly, by taking an embedding of a 2D scene image [6], a 3D scene [33], or a specific object [8] as an additional input to their model.

Original: Whilesuch embeddings encode valuable information, they do not provide precise cues to help placing thehuman in the scene.
Revised: While such embeddings encode valuable information, they do not provide precise cues to help placing the human in the scene.

Original: In the context of synthesizing a static human body in a scene, some efforts havenonetheless been made to incorporate more precise information, such as the distance between everyscene point and the closest vertex on body surface [36], or a semantic scene label, e.g., floor, sofa, atevery vertex on the 3D body mesh [12].
Revised: In the context of synthesizing a static human body in a scene, some efforts have nonetheless been made to incorporate more precise information, such as the distance between every scene point and the closest vertex on body surface [36], or a semantic scene label, e.g., floor, sofa, at every vertex on the 3D body mesh [12].

Original: However, while such representations allow one to place ahuman in the scene, they remain under-constrained for motion prediction.
Revised: However, while such representations allow one to place a human in the scene, they remain under-constrained for motion prediction.

Original: Specifically, they imposeneither temporal consistency, nor consistency between local pose changes and global motion.
Revised: Specifically, they impose neither temporal consistency, nor consistency between local pose changes and global motion.

Original: Assuch, they yield artifacts such as “ghost motion".
Revised: As such, they yield artifacts such as “ghost motion".

Original: In this paper, we address this by explicitly modeling the contact between the human body jointsand the scene.
Revised: In this paper, we address this by explicitly modeling the contact between the human body joints and the scene.

Original: To this end, we introduce per-joint contact maps that encode the distance betweeneach joint and every 3D scene point.
Revised: To this end, we introduce per-joint contact maps that encode the distance between each joint and every 3D scene point.

Original: Such contact maps constrain both the global motion and thelocal human pose, thus avoiding the “ghost motion" issue.
Revised: Such contact maps constrain both the global motion and the local human pose, thus avoiding the “ghost motion" issue.

Original: As illustrated in Fig 1, our human motion
Revised: As illustrated in Fig 1, our human motion

Original: [Figure]
Revised: [Figure]

Original: prediction framework therefore consists of a two-stage pipeline that first predicts the future contact maps and then forecasts the future human poses by conditioning themon the predicted maps.
Revised: prediction framework therefore consists of a two-stage pipeline that first predicts the future contact maps and then forecasts the future human poses by conditioning them predicted maps.

Original: Specifically, we obtain a compact temporal representation of the historical contact maps using asimilar Discrete Cosine Transform (DCT) encoding strategy to that of [22] for human motion, anduse a feed-forward 3D scene encoding network, i.e , PVCNN [20], to predict the future contact maps  the  past  motion and the historical contact maps.
Revised: Specifically, we obtain a compact temporal representation of the historical contact maps using a similar Discrete Cosine Transform (DCT) encoding strategy to that of [22] for human motion, and use a feed-forward 3D scene encoding network, i.e., PVCNN [20], to predict the future contact maps from the scene point cloud, past human motion and the historical contact maps.

Original:  then forecasthuman we retain  closest  predicted 
Revised: To then forecast human motion, we retain the closest contact points for every human joint according to the predicted contact maps and forecast the global translations and then the local poses given these points and the past human motion. Consistency between the global translations and the local poses is achieved via a prior defined using the predicted human poses and the contact points.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: (i)
Revised: (i)

Original: We introduce a distance-based per-joint contactmap that captures fine-grained human-scene interactions to avoid generating unrealistic human motions.
Revised: We introduce a distance-based per-joint contact map that captures fine-grained human-scene interactions to avoid generating unrealistic human motions.

Original: (ii) We further propose a two-stage pipeline whose first stage models the temporal dependenciesof past contact maps and predicts the future ones, and whose second stage forecasts the future humanmotion conditioned on these contact maps.
Revised: (ii) We further propose a two-stage pipeline whose first stage models the temporal dependencies of past contact maps and predicts the future ones, and whose second stage forecasts the future human motion conditioned on these contact maps.

Original: Our experiments on both synthetic and real datasetsdemonstrate the benefits of our approach over the state-of-the-art human motion forecasting andhuman motion synthesis methods.
Revised: Our experiments on both synthetic and real datasets demonstrate the benefits of our approach over the state-of-the-art human motion forecasting and human motion synthesis methods.

Original:  Related Work
Revised:  Related Work

Original: Human motion prediction.
Revised: Human motion prediction.

Original: Modeling 3D human motion has been a long-standing research goal [4,27, 30, 35].
Revised: Modeling 3D human motion has been a long-standing research goal [4, 27, 30, 35].

Original: While traditional methods [4, 35], relying on either Hidden Markov Models [4] or the
Revised: While traditional methods [4, 35], relying on either Hidden Markov Models [4] or the

Original: Gaussian process latent variable model [35], can tackle periodic and simple non-periodic motions,such as walking and golf swing, more complex motions have been shown to be better modeledvia deep learning frameworks [19, 1, 22, 32, 9, 18, 21, 5], which can be roughly categorized intofeed-forward models [19, 22, 21] and recurrent networks [1, 32, 9, 18, 5] according to their temporalspatial encoding strategies.
Revised: Gaussian process latent variable model [35], can tackle periodic and simple non-periodic motions, such as walking and golf swing, more complex motions have been shown to be better modeled via deep learning frameworks [19, 1, 22, 32, 9, 18, 21, 5], which can be roughly categorized into feed-forward models [19, 22, 21] and recurrent networks [1, 32, 9, 18, 5] according to their temporalspatial encoding strategies.

Original: Despite the success of these methods at forecasting complex motions,they typically only predict local poses, disregarding global motion and any scene information.
Revised: Despite the success of these methods at forecasting complex motions, they typically only predict local poses, disregarding global motion and any scene information.

Original: Inthis paper, we seek to predict future human motions that are consistent with the 3D scenes they areperformed in.
Revised: In this paper, we seek to predict future human motions that are consistent with the 3D scenes they are performed in.

Original: Recently, a few works [8, 6] have started to incorporate scene context in motion forecasting.
Revised: Recently, a few works [8, 6] have started to incorporate scene context in motion forecasting.

Original: Inparticular, Corona et al .
Revised: In particular, Corona et al .

Original: [8] introduced a semantic-graph model that extracts a joint embedding of thehuman pose and an object of interest, such as a cup.
Revised: [8] introduced a semantic-graph model that extracts a joint embedding of the human pose and an object of interest, such as a cup.

Original: This method, however, is ill-suited to modelinteractions with the whole scene itself, for example the floor or stairs that the person touches whilewalking.
Revised: This method, however, is ill-suited to model interactions with the whole scene itself, for example the floor or stairs that the person touches while walking.

Original: In [6], Cao et al .
Revised: In [6], Cao et al .

Original: proposed a multi-stage pipeline that breaks down the motion forecastinginto three sub-tasks: predicting a 2D goal, planning a 2D and 3D path, forecasting the 3D posesfollowing the path.
Revised: proposed a multi-stage pipeline that breaks down the motion forecasting into three sub-tasks: predicting a 2D goal, planning a 2D and 3D path, forecasting the 3D poses following the path.

Original: To this end, they extracted a scene representation using 2D images and let theirmodel learn the scene constraints implicitly.
Revised: To this end, they extracted a scene representation using 2D images and let their model learn the scene constraints implicitly.

Original: This strategy, however, cannot handle scene occlusionsand does not enforce consistency between the local and global motion.
Revised: This strategy, however, cannot handle scene occlusions and does not enforce consistency between the local and global motion.

Original: More importantly, both these methods aim to learn interactions implicitly.
Revised: More importantly, both these methods aim to learn interactions implicitly.

Original: By contrast, our contact maps explicitly encode thehuman-scene interactions, leading to direct constraints at the human joint level.
Revised: By contrast, our contact maps explicitly encode the human-scene interactions, leading to direct constraints at the human joint level.

Original: Scene-aware human synthesis.
Revised: Scene-aware human synthesis.

Original: Synthesizing a realistic human in a 3D scene has recently gainedan increasing popularity [36, 37, 12, 34, 33, 10].
Revised: Synthesizing a realistic human in a 3D scene has recently gained an increasing popularity [36, 37, 12, 34, 33, 10].

Original: A cornerstone in the success of these methodsis the modeling of human-scene interactions.
Revised: A cornerstone in the success of these methods is the modeling of human-scene interactions.

Original: To achieve this, many of these works [37, 34, 33]follow a similar approach to [6], modeling the human-scene interactions implicitly, via either Generative Adversarial Networks (GANs) [34], or Variational Autoencoders (VAEs) [37, 33].
Revised: To achieve this, many of these works [37, 34, 33] follow a similar approach to [6], modeling the human-scene interactions implicitly, via either Generative Adversarial Networks (GANs) [34], or Variational Autoencoders (VAEs) [37, 33].

Original: Threemethods [36, 12, 10] nonetheless exploit explicit representations of human-scene interaction.
Revised: Three methods [36, 12, 10] nonetheless exploit explicit representations of human-scene interaction.

Original: Inparticular, POSA [12] uses a body-centric representation of the human-scene interaction where asemantic scene label, e.g., floor, sofa, is assigned to every human mesh vertex.
Revised: In particular, POSA [12] uses a body-centric representation of the human-scene interaction where a semantic scene label, e.g., floor, sofa, is assigned to every human mesh vertex.

Original: This label encodes thecontact probability to the scene surface and the corresponding semantic scene label.
Revised: This label encodes the contact probability to the scene surface and the corresponding semantic scene label.

Original: However, thissemantic representation does not provide any information about the 3D location of the human body,and is thus ill-suited to the motion forecasting task.
Revised: However, this semantic representation does not provide any information about the 3D location of the human body, and is thus ill-suited to the motion forecasting task.

Original: In PLACE [36], Zhang et al .
Revised: In PLACE [36], Zhang et al .

Original: introduce a contactrepresentation based on Basic Point Sets (BSP) [26].
Revised: introduce a contact representation based on Basic Point Sets (BSP) [26].

Original: Specifically, given a set of basic 3D scenepoints, they represent the human-scene interaction using the minimum distance from every such pointto the human body surface.
Revised: Specifically, given a set of basic 3D scene points, they represent the human-scene interaction using the minimum distance from every such point to the human body surface.

Original: As in the semantic-based case, this strategy only gives a weak prior onthe human pose, as it does not explicitly defines which joint should be in contact with which scenepoint.
Revised: As in the semantic-based case, this strategy only gives a weak prior on the human pose, as it does not explicitly defines which joint should be in contact with which scene point.

Original: In SAMP [10], their approach only models a coarse interaction of the human with a givenobject in the final frame by predicting the final root location and orientation.
Revised: In SAMP [10], their approach only models a coarse interaction of the human with a given object in the final frame by predicting the final root location and orientation.

Original: Such coarse interactionhowever cannot constrain the poses at intermediate frames.
Revised: Such coarse interaction however cannot constrain the poses at intermediate frames.

Original: By contrast, our per-joint contact mapsprovide a more detailed contact information for every human joint at each future frame.
Revised: By contrast, our per-joint contact maps provide a more detailed contact information for every human joint at each future frame.

Original: Hand-object interaction Although hand-object contact relationships have already been studiedfor the task of grasping [2, 29, 3, 14], existing methods cannot be naively applied to human-sceneinteractions because their object-centric contact relationships tend to be static across time.
Revised: Hand-object interaction Although hand-object contact relationships have already been studied for the task of grasping [2, 29, 3, 14], existing methods cannot be naively applied to human-scene interactions because their object-centric contact relationships tend to be static across time.

Original: Forexample, when we are using a hammer, we will grasp the handle tightly, and thus the contact regionbetween our palms and the hammer does not change across time.
Revised: For example, when we are using a hammer, we will grasp the handle tightly, and thus the contact region between our palms and the hammer does not change across time.

Original: By contrast, our human-scenecontact maps changes across the frames for almost all human activities.
Revised: By contrast, our human-scene contact maps changes across the frames for almost all human activities.

Original: This motivates us to proposea distance-based per-joint contact map at each frame.
Revised: This motivates us to propose a distance-based per-joint contact map at each frame.

Original:  Approach
Revised:  Approach

Original: Let us now introduce our approach to scene-aware 3D human motion forecasting.
Revised: Let us now introduce our approach to scene-aware 3D human motion forecasting.

Original: Following previouswork [22], a sequence of P past human poses is represented as X =
Revised: Following previous work [22], a sequence of P past human poses is represented as X =

Original: [ x 1 , x 2 , · · · , x P ] ∈ R P × J × 3 ,where x p ∈ R J × 3 encodes the 3D locations of all J joints at time p in a global reference frame.
Revised: [ x 1 , x 2 , · · · , x P ] ∈ R P × J × 3 , where x p ∈ R J × 3 encodes the 3D locations of all J joints at time p in a global reference frame.

Original: The3D scene is represented as a set of 3D points S ∈ R N × 3 .
Revised: The 3D scene is represented as a set of 3D points S ∈ R N × 3 .

Original: Given the historical motion X and the3D scene S , our goal is then to forecast T future human poses Y =
Revised: Given the historical motion X and the 3D scene S , our goal is then to forecast T future human poses Y =

Original: [ x P +1 , x P +2 , · · · , x P + T ] ∈109 R T × J × 3 .
Revised: [ x P +1 , x P +2 , · · · , x P + T ] ∈ R T × J × 3 .

Original: To this end, we introduce a two-stage pipeline that first predicts future human-scene contactmaps, and then forecasts the future poses by conditioning them on these contact maps.
Revised: To this end, we introduce a two-stage pipeline that first predicts future human-scene contact maps, and then forecasts the future poses by conditioning them on these contact maps.

Original: An overviewof our pipeline is shown in Fig.
Revised: An overview of our pipeline is shown in Fig.

Original: Below, we present our contact representation, and our pipeline topredict future contact maps and future motion.
Revised: Below, we present our contact representation, and our pipeline to predict future contact maps and future motion.

Original: 3.1 Per-joint Scene Contact Map
Revised: 3.1 Per-joint Scene Contact Map

Original: We represent human-scene contact using the distances between the human joints and the scene points.
Revised: We represent human-scene contact using the distances between the human joints and the scene points.

Original: Specifically, given a human pose x and a 3D scene S , we first compute the per-joint distance mapd ∈ R J × N , where each entry d jn encodes the ℓ 2 distance between the j -th human joint and n -thscene point, i.e.,
Revised: Specifically, given a human pose x and a 3D scene S , we first compute the per-joint distance map d ∈ R J × N , where each entry d jn encodes the ℓ 2 distance between the j -th human joint and n -th scene point, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where j ∈ { 1 , 2 , · · · , J } and n ∈ { 1 , 2 , · · · , N } .
Revised: where j ∈ { 1 , 2 , · · · , J } and n ∈ { 1 , 2 , · · · , N } .

Original: In the distance map, the scene points that are less relevant to a human joint, because they are far awayfrom it, have a higher value.
Revised: In the distance map, the scene points that are less relevant to a human joint, because they are far away from it, have a higher value.

Original: This will tend to give them more influence when used in a deep neuralnetwork and may in turn cause issues when training our contact prediction module.
Revised: This will tend to give them more influence when used in a deep neural network and may in turn cause issues when training our contact prediction module.

Original: To address this,we normalize the distance map to obtain a continuous contact map c ∈ R J × N whose elements aredefined as
Revised: To address this, we normalize the distance map to obtain a continuous contact map c ∈ R J × N whose elements are [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Figure]

Original: [Table]
Revised: Conditioned on these contact points, our motion forecasting network forecasts the global translations ˆ Y root and then the local poses ˆ Y local . defined as

Original: [Figure]
Revised: [Equation]

Original: where the constant σ is the normalizing factor.
Revised: where the constant σ is the normalizing factor.

Original: In such a contact map, closer scene points have highervalues than far-away ones, whose values will be very close to zero.
Revised: In such a contact map, closer scene points have higher values than far-away ones, whose values will be very close to zero.

Original: Given the 3D scene S and past human poses X , we compute the sequence of past contact maps
Revised: Given the 3D scene S and past human poses X , we compute the sequence of past contact maps

Original: C =
Revised: C =

Original: [ c 1 , c 2 , · · · , c P ] ∈ R P × J × N as described above.
Revised: [ c 1 , c 2 , · · · , c P ] ∈ R P × J × N as described above.

Original: Our goal then is to predict the future contactmaps M =
Revised: Our goal then is to predict the future contact maps M =

Original: [ c P +1 , c P +2 , · · · , c P + T ] ∈ R T × J × N .
Revised: [ c P +1 , c P +2 , · · · , c P + T ] ∈ R T × J × N .

Original: In the next section, we introduce a contact mapprediction module to do so.
Revised: In the next section, we introduce a contact map prediction module to do so.

Original: 3.2 Contact Prediction Network
Revised: 3.2 Contact Prediction Network

Original: Since our contact maps are based on distances, they are smooth over time.
Revised: Since our contact maps are based on distances, they are smooth over time.

Original: Therefore, similar to [22]for human motion, we adopt a temporal encoding strategy based on the Discrete Cosine Transform(DCT).
Revised: Therefore, similar to [22] for human motion, we adopt a temporal encoding strategy based on the Discrete Cosine Transform (DCT).

Original: With the DCT, a sequence is represented as a linear combination of a set of pre-defined cosinebases.
Revised: With the DCT, a sequence is represented as a linear combination of a set of pre-defined cosine bases.

Original: By discarding the high-frequency components, the DCT provides a compact representation,which nicely captures the smoothness of the sequence.
Revised: By discarding the high-frequency components, the DCT provides a compact representation, which nicely captures the smoothness of the sequence.

Original: More formally, let us denote a P -frame sequence of contact values between the j -th human joint andthe n -th scene point as ˜ c
Revised: More formally, let us denote a P -frame sequence of contact values between the j -th human joint and the n -th scene point as ˜ c

Original: jn
Revised: jn

Original: =
Revised: =

Original: [ c 1 jn , c 2 jn , · · · c Pjn ] .
Revised: [ c 1 jn , c 2 jn , · · · c Pjn ] .

Original: This sequence can be fully represented using P
Revised: This sequence can be fully represented using P

Original: DCT coefficients, with the l -th one given by
Revised: DCT coefficients, with the l -th one given by

Original: [Equation]
Revised: [Equation]

Original: where l ∈ { 1 , 2 , · · · P } and δ ij denotes the Kronecker delta function, i.e.,
Revised: where l ∈ { 1 , 2 , · · · P } and δ ij denotes the Kronecker delta function, i [Equation] e.,

Original: [Equation]
Revised: [Equation]

Original: Given these DCT coefficients, the sequence in the original space can be obtained via the Inverse
Revised: Given these DCT coefficients [Equation] the sequence in the original space can be obtained via the Inverse

Original: Discrete Cosine Transform (IDCT) as
Revised: Discrete Cosine Transform (IDCT) as

Original: [Equation]
Revised: [Equation]

Original: where L is the number of DCT coefficients used.
Revised: where L is the number of DCT coefficients used.

Original: For a lossless reconstruction, L = P , but when thesequence is smooth, one can ignore the coefficients of the high-frequency DCT bases and thus use
Revised: For a lossless reconstruction, L = P , but when the sequence is smooth, one can ignore the coefficients of the high-frequency DCT bases and thus use

Original: L < P at a negligible loss.
Revised: L < P at a negligible loss.

Original: Recall that our goal is to predict the future T contact maps M =
Revised: Recall that our goal is to predict the future T contact maps M =

Original: [ c P +1 , c P +2 , · · · , c P + T ] fromthe past P ones C =
Revised: [ c P +1 , c P +2 , · · · , c P + T ] from the past P ones C =

Original: [ c 1 , c 2 , · · · , c P ] .
Revised: [ c 1 , c 2 , · · · , c P ] .

Original: To leverage the DCT representation, we then reformulate this problem as learning a mapping from the DCT coefficients of the past contact maps to those ofthe future ones.
Revised: To leverage the DCT representation, we then reformulate this problem as learning a mapping from the DCT coefficients of the past contact maps to those of the future ones.

Original: Specifically, following the padding strategy of [22], we first repeat the last contactmap c P T times to create a sequence of P + T contact maps C ′
Revised: Specifically, following the padding strategy of [22], we first repeat the last contact map c P T times to create a sequence of P + T contact maps C ′

Original: =
Revised: =

Original: [ c 1 , c 2 , · · · , c P , c P , · · · , c P ] ∈150 R ( P + T )
Revised: [ c 1 , c 2 , · · · , c P , c P , · · · , c P ] ∈ R ( P + T )

Original: × J × N .
Revised: × J × N .

Original: We then compute the DCT coefficients of this sequence and define it as H ∈ R
Revised: We then compute the DCT coefficients of this sequence and define it as H ∈ R

Original: L × J × N .
Revised: L × J × N .

Original: Note that for each joint we only retain the first L DCT coefficients.
Revised: Note that for each joint we only retain the first L DCT coefficients.

Original: Our goal then is to learn a residualbetween these DCT coefficients H and those of the real sequence [ c 1 , c 2 , · · · , c P + T ] .
Revised: Our goal then is to learn a residual between these DCT coefficients H and those of the real sequence [ c 1 , c 2 , · · · , c P + T ] .

Original: We formulate this as a point-cloud encoding problem.
Revised: We formulate this as a point-cloud encoding problem.

Original: That is, for each scene point, we regardthe corresponding DCT coefficients of all joints as its feature.
Revised: That is, for each scene point, we regard the corresponding DCT coefficients of all joints as its feature.

Original: Thus, every scene point has aninitial feature vector of size LJ .
Revised: Thus, every scene point has an initial feature vector of size LJ .

Original: Our contact prediction network then takes as input the 3D scenepoints S , the DCT features H and the past human poses X , and outputs the future contact mapsˆ C
Revised: Our contact prediction network then takes as input the 3D scene points S , the DCT features H and the past human poses X , and outputs the future contact maps ˆ C

Original: = [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] as
Revised: = [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] as

Original: [Equation]
Revised: [Equation]

Original:  where F represents the trainable point-cloud processing model and G x is the GRU encoder.
Revised: (6) where F represents the trainable point-cloud processing model and G x is the GRU encoder.

Original: Specifically, as shown in Fig. 2, we use the Point-Voxel CNN (PVCNN) [20] to encode the 3Dscene with its DCT feature vectors.
Revised: Specifically, as shown in Fig. 2, we use the Point-Voxel CNN (PVCNN) [20] to encode the 3D scene with its DCT feature vectors.

Original: The PVCNN was designed to process a 3D point cloud.
Revised: The PVCNN was designed to process a 3D point cloud.

Original: Itincorporates voxel-based convolutions and point-based representations, leading to a memory- andcomputation-efficient structure for 3D data.
Revised: It incorporates voxel-based convolutions and point-based representations, leading to a memory- and computation-efficient structure for 3D data.

Original: We adapt the PVCNN to also take as input the pasthuman poses X , encoded by a Gated Recurrent Unit (GRU) [7], to predict a residual of the DCTcoefficients.
Revised: We adapt the PVCNN to also take as input the past human poses X , encoded by a Gated Recurrent Unit (GRU) [7], to predict a residual of the DCT coefficients.

Original: We then obtain the predicted contacted maps ˆ C
Revised: We then obtain the predicted contacted maps ˆ C

Original: =
Revised: =

Original: [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] via the IDCT.
Revised: [ˆ c 1 , ˆ c 2 , · · · , ˆ c P + T ] via the IDCT.

Original: Note that, here, we seek not only to predict the future contact maps but also to recover the past ones.
Revised: Note that, here, we seek not only to predict the future contact maps but also to recover the past ones.

Original: To this end, we use the average ℓ 2 loss between the ground-truth contact maps and the predicted ones.
Revised: To this end, we use the average ℓ 2 loss between the ground-truth contact maps and the predicted ones.

Original: Formally, this loss is defined as
Revised: Formally, this loss is defined as

Original: [Equation]
Revised: [Equation]

Original: 3.3 Motion Forecasting Network
Revised: 3.3 Motion Forecasting Network

Original: Given the future contact maps M and the past human motion X , our human motion forecastingmodule aims to predict the future human poses Y .
Revised: Given the future contact maps M and the past human motion X , our human motion forecasting module aims to predict the future human poses Y .

Original: To this end, we first extract the closest contactscene point to each joint from the contact maps.
Revised: To this end, we first extract the closest contact scene point to each joint from the contact maps.

Original: Given these contact points, we first use a simpleneural network to predict the future path (global translations) and then forecast the future local poseswith an RNN-based model.
Revised: Given these contact points, we first use a simple neural network to predict the future path (global translations) and then forecast the future local poses with an RNN-based model.

Original: Let us discuss these steps in more detail.
Revised: Let us discuss these steps in more detail.

Original: Contact points.
Revised: Contact points.

Original: Since our focus now is to predict the 3D location of human joints, we propose toretain the most relevant scene point i.e., the closest one to each joint.
Revised: Since our focus now is to predict the 3D location of human joints, we propose to retain the most relevant scene point i.e., the closest one to each joint.

Original: Specifically, given the contactmap at time step p c p ∈ R J × N and the 3D scene points S ∈ R N × 3 , we would like to find the scenepoint that is closest to each human joint.
Revised: Specifically, given the contact map at time step p c p ∈ R J × N and the 3D scene points S ∈ R N × 3 , we would like to find the scene point that is closest to each human joint.

Original: Let us denote the resulting contact points as q p ∈ R J × 4 ,each row of which stores the 3D location of the scene point closest to the corresponding human jointtogether with a binary value indicating whether the joint truly is in contact with the scene or not.
Revised: Let us denote the resulting contact points as q p ∈ R J × 4 , each row of which stores the 3D location of the scene point closest to the corresponding human joint together with a binary value indicating whether the joint truly is in contact with the scene or not.

Original: More formally, the contact point for joint j is computed asq pj =  
Revised: More formally, the contact point for joint j is computed as

Original: [ S k , 1] , where k = argmax n = { 1 , 2 , ··· ,N } c j,n , if c j,k > ϵ [0 , 0 , 0 , 0] otherwise , where S k is the 3D location of the k -th scene point, and ϵ is a threshold to determine whetherthe joint is in contact with the scene or not.
Revised: [Equation] , 1] , where k = argmax n = { 1 , 2 , ··· ,N } c j,n , if c j,k > ϵ [0 , 0 , 0 , 0] otherwise , where S k is the 3D location of the k -th scene point, and ϵ is a threshold to determine whether the joint is in contact with the scene or not.

Original: We compute such contact points for the entire futuresequence, which yields a sequence of contact points Q =
Revised: We compute such contact points for the entire future sequence, which yields a sequence of contact points Q =

Original: [ q P +1 , q P +2 , · · · , q P + T ] ∈ R T
Revised: [ q P +1 , q P +2 , · · · , q P + T ] ∈ R T

Original: × J × 4 ,where q p ∈ R J × 4 is the contact points at the p -th time step.
Revised: × J × 4 , where q p ∈ R J × 4 is the contact points at the p -th time step.

Original: Motion forecasting.
Revised: Motion forecasting.

Original: As shown in Fig 2, our forecasting model first predicts the global translation ineach frame and then the local motion given the global translations.
Revised: As shown in Fig 2, our forecasting model first predicts the global translation in each frame and then the local motion given the global translations.

Original: Specifically, given the past motion
Revised: Specifically, given the past motion

Original: X and the contact points Q , we use a simple multilayer perceptron (MLP) to predict the future globaltranslations Y root ∈ R T × 3 as[Equation]
Revised: X and the contact points Q , we use a simple multilayer perceptron (MLP) to predict the future global translations Y root ∈ R T × 3 as

Original: [Equation]
Revised: [Equation]

Original: where M represents the MLP and ˜
Revised: where M represents the MLP and ˜

Original: G x is the GRU to encode the past motion.
Revised: G x is the GRU to encode the past motion.

Original: The global translations are then fed into a GRU to predict the local pose at each future timestep.
Revised: The global translations are then fed into a GRU to predict the local pose at each future time step.

Original: Assuming that the past and future local pose sequences are represented by X local =[ x local, x local, · · · , x local P ] ∈ R P ×( J − 1) × 3 and Y local =
Revised: Assuming that the past and future local pose sequences are represented by X local = [ x local, x local, [Equation] · , x local P ] ∈ R P ×

Original: [ x local P +1 , x local P +2 , · · · , x localP + T ] ∈ R T × ( J − 1) × 3 ,respectively, this can be expressed as
Revised: [Equation] [Equation] 3 and Y local = [ x local

Original: [Equation]
Revised: [Equation]

Original: where G denotes the GRU to predict the furture poses, p ∈ { P + 1 , P + 2 , · · · , P + T } ,andˆ xlocal p ∈ R (
Revised: 1 , x [Equation] [Equation] , x

Revised: [Equation]

Revised: ∈

Original: [Equation]
Revised: [Equation]

Revised: , respectively, this can be expressed as

Original: − 1) × 3 ,  ˆ x root p ∈ R  3 and q p ∈
Revised: [Equation] where G denotes the GRU to predict the furture poses, p ∈ { P + 1 , P + 2 , · · · , P + T } , and ˆ x local p ∈ R ( J [Equation] 1) × 3 ,

Original:  R 
Revised: [Equation] x root [Equation] ∈ R 3 and q p ∈

Original:  J × 4 are the local human pose, global translation andcontact points at time p , respectively.
Revised: R J × 4 are the local human pose, global translation and contact points at time p , respectively.

Original: The global translation and local motion prediction modules are trained jointly.
Revised: The global translation and local motion prediction modules are trained jointly.

Original: To this end, we use 3loss terms.
Revised: To this end, we useloss terms.

Original: The first one is a global translation loss defined as
Revised: The first one is a global translation loss defined as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where x root p ∈ R 3 and ˆ x root p ∈ R 3 are the ground-truth and predicted global translations at time p .
Revised: where x root p ∈ R 3 and ˆ x root p ∈ R 3 are the ground-truth and predicted global translations at time p .

Original: The second loss accounts for the local human pose prediction and is expressed as
Revised: The second loss accounts for the local human pose prediction and is expressed as

Original: [Equation]
Revised: [Equation]

Original: 11) where x local pj ∈ R 3 and ˆ x local pj ∈ R 3 are the ground-truth and predicted local positions of the j -th jointat time p .
Revised: [Equation] where x local pj ∈ R 3 and ˆ x local pj ∈ R 3 are the ground-truth and predicted local positions of the j -th joint at time p .

Original: Finally, our third loss term encodes a contact prior based on the contact points.
Revised: Finally, our third loss term encodes a contact prior based on the contact points.

Original: We define it as
Revised: We define it as

Original: [Equation]
Revised: [Equation]

Original: where ˆ x pj ∈ R 3 is the predicted location of joint j at time p , obtained by adding the predicted globaltranslation at time p to the corresponding local pose.
Revised: where ˆ x pj ∈ R 3 is the predicted location of joint j at time p , obtained by adding the predicted global translation at time p to the corresponding local pose.

Original: q pj [1:3] ∈ R 3 and q pj 4 ∈ { 0 , 1 } are the 3Dcoordinates of the contact scene point and the indicator value, respectively.
Revised: q pj [1:3] ∈ R 3 and q pj 4 ∈ { 0 , 1 } are the 3D coordinates of the contact scene point and the indicator value, respectively.

Original: The overall loss is then expressed as
Revised: The overall loss is then expressed as

Original: [Equation]
Revised: [Equation]

Original: We use a stage-wise training strategy where the contact prediction network and motion forecastingnetwork are trained separately.
Revised: We use a stage-wise training strategy where the contact prediction network and motion forecasting network are trained separately.

Original: During training, the motion forecasting network is given the groundtruth contact points as input.
Revised: During training, the motion forecasting network is given the groundtruth contact points as input.

Original: At test time, we first compute the contact points from the predictedcontact maps and then use these contact points to forecast the future human motion.
Revised: At test time, we first compute the contact points from the predicted contact maps and then use these contact points to forecast the future human motion.

Original:  Experiments 
Revised:  Experiments 4.1 Datasets

Original: 4.1 Datasets

Original: We evaluate our method on two datasets, GTA-IM [6] and PROX [11].
Revised: We evaluate our method on two datasets, GTA-IM [6] and PROX [11].

Original: GTA-IM.
Revised: GTA-IM.

Original: The GTA Indoor Motion dataset [6] is a large-scale synthetic dataset that captures humanscene interactions.
Revised: The GTA Indoor Motion dataset [6] is a large-scale synthetic dataset that captures humanscene interactions.

Original: It consists of 50 different characters performing various activities in 7 differentscenes.
Revised: It consists of 50 different characters performing various activities in 7 different scenes.

Original: Each scene is a building, and each building has several rooms on one or more floors. 
Revised: Each scene is a building, and each building has several rooms on one or more floors. The

Original: Thedataset contains around 1 million RGB-D frames together with the corresponding 3D human poses.
Revised: [Table] [Table] dataset contains around 1 million RGB-D frames together with the corresponding 3D human poses.

Original: We use 4 of the scenes as our training set (“r001",“r002",“r003",“r006") and the remaining 3 as ourtest set (“r010",“r011",“r013") 1 .
Revised: We use 4 of the scenes as our training set (“r001",“r002",“r003",“r006") and the remaining 3 as our test set (“r010",“r011",“r013") 1 .

Original: To obtain the 3D point clouds of the different scenes, we registertheir depth maps from different videos sequences with the ground-truth camera extrinsic matrices.
Revised: To obtain the 3D point clouds of the different scenes, we register their depth maps from different videos sequences with the ground-truth camera extrinsic matrices.

Original: Following [6], we use 21 out of the 98 human joints provided by the dataset.
Revised: Following [6], we use 21 out of the 98 human joints provided by the dataset.

Original: The videos run at 30Hz.
Revised: The videos run at 30Hz.

Original: We train our models to observe the past 30 time steps (1 second) and predict the future 60 time steps(2 seconds).[Table] [Table]
Revised: We train our models to observe the past 30 time steps (1 second) and predict the future 60 time steps (2 seconds).

Original: PROX.
Revised: PROX.

Original: Proximal Relationships with Object eXclusion (PROX)
Revised: Proximal Relationships with Object eXclusion (PROX)

Original: [11] is a real dataset captured using a
Revised: [11] is a real dataset captured using a

Original: Kinect-One sensor.
Revised: Kinect-One sensor.

Original: It comprises 12 different scenes with 20 subjects interacting with the scenes.
Revised: It comprises 12 different scenes with 20 subjects interacting with the scenes.

Original: Thedataset also provides SMPL-X parameters [25] as the ground-truth human pose and shape in eachframe.
Revised: The dataset also provides SMPL-X parameters [25] as the ground-truth human pose and shape in each frame.

Original: Since these parameters were obtained by a frame-wise fitting algorithm, the motion sequencesare jittery and thus ill-suited to our task.
Revised: Since these parameters were obtained by a frame-wise fitting algorithm, the motion sequences are jittery and thus ill-suited to our task.

Original: We therefore refine the dataset via a simple temporaloptimization process to generate smooth motions.
Revised: We therefore refine the dataset via a simple temporal optimization process to generate smooth motions.

Original: More details are provided in the supplementarymaterial.
Revised: More details are provided in the supplementary material.

Original: Following [33], we use 8 scenes for training (“N3Library”, “MPH112”, “MPH11”, “MPH8”,“BasementSittingBooth”, “N0Sofa”, “N3Office”, “Werkraum”) and 4 scenes for testing (“MPH16”,“MPH1Library”, “N0SittingBooth”, “N3OpenArea”).
Revised: Following [33], we use 8 scenes for training (“N3Library”, “MPH112”, “MPH11”, “MPH8”, “BasementSittingBooth”, “N0Sofa”, “N3Office”, “Werkraum”) and 4 scenes for testing (“MPH16”, “MPH1Library”, “N0SittingBooth”, “N3OpenArea”).

Original: We use the 22 body joints of SMPL-X model.
Revised: We use the 22 body joints of SMPL-X model.

Original: As for GTA-IM, the frame-rate of this dataset is 30 Hz, and we train our models to take the past 30time steps as input and predict the future 60 steps.
Revised: As for GTA-IM, the frame-rate of this dataset is 30 Hz, and we train our models to take the pasttime steps as input and predict the future 60 steps.

Original: 4.2 Metrics, Baselines & Implementation
Revised: 4.2 Metrics, Baselines & Implementation

Original: Metrics.
Revised: Metrics.

Original: We use the Mean Per Joint Position Error (MPJPE) [13] to evaluate both the globaltranslations (path error) and the local motion (pose error).
Revised: We use the Mean Per Joint Position Error (MPJPE) [13] to evaluate both the global translations (path error) and the local motion (pose error).

Original: Baselines.
Revised: Baselines.

Original: We compare our method with two human motion prediction models (LTD [22] and
Revised: We compare our method with two human motion prediction models (LTD [22] and

Original: DMGNN [18]) and one scene-aware human motion synthesis method (SLT [33]).
Revised: DMGNN [18]) and one scene-aware human motion synthesis method (SLT [33]).

Original: LTD [22] is a representative feed-forward method based on Graph Convolutional Networks (GCNs) [16].
Revised: LTD [22] is a representative feed-forward method based on Graph Convolutional Networks (GCNs) [16].

Original: DMGNN [18]is a state-of-the-art RNN-based approach for human motion prediction.
Revised: DMGNN [18] [Figure] is a state-of-the-art RNN-based approach for human motion prediction.

Original: SLT [33] is a stage-wiseapproach to synthesize long-term human motions.
Revised: SLT [33] is a stage-wise approach to synthesize long-term human motions.

Original: We used the official implementations of LTD [22]and DMGNN [18] to train them on our datasets.
Revised: We used the official implementations of LTD [22] and DMGNN [18] to train them on our datasets.

Original: For SLT [33], we adapted the official code to ourtask e.g., we modified their model so as to take the past motion as input.
Revised: For SLT [33], we adapted the official code to our task e.g., we modified their model so as to take the past motion as input.

Original: The detail of these changesare provided in the supplementary material.
Revised: The detail of these changes are provided in the supplementary material.

Original: Implementation details.
Revised: Implementation details.

Original: Our models are implemented in Pytorch [24] and trained using the
Revised: Our models are implemented in Pytorch [24] and trained using the

Original: ADAM [15] optimizer.
Revised: ADAM [15] optimizer.

Original: Both our contact prediction network and motion forecasting one are trainedfor 50 epochs with learning rates of 0.0005 and 0.001, respectively.
Revised: Both our contact prediction network and motion forecasting one are trained for 50 epochs with learning rates of 0.0005 and 0.001, respectively.

Original: The training of each network takesabout 12 hours on a 24GB NVIDIA RTX3090Ti GPU and the evaluation of one sample takes around90 ms during testing.
Revised: The training of each network takes about 12 hours on a 24GB NVIDIA RTX3090Ti GPU and the evaluation of one sample takes around 90 ms during testing.

Original: For both datasets, the normalizing factor σ , the number of DCT coefficients
Revised: For both datasets, the normalizing factor σ , the number of DCT coefficients

Original: [Figure]

Original: L and the contact threshold ϵ are set to 0 .
Revised: L and the contact threshold ϵ are set to 0 .

Original: 2 , 20 and 0 .
Revised: 2 , 20 and 0 .

Original: 32 , respectively.
Revised: 32 , respectively.

Original: For the motion forecastingnetwork, the loss weights ( λ 1 , λ 2 , λ 3 ) are set to (1 . 0 , 1 . 0 , 0 . 1) for both datasets.
Revised: For the motion forecasting network, the loss weights ( λ 1 , λ 2 , λ 3 ) are set to (1 . 0 , 1 . 0 , 0 . 1) for both datasets.

Original: For each motionsequence, we randomly sample 5000 scene points that are within 2.5 meters away from the root jointof the last observed pose.
Revised: For each motion sequence, we randomly sample 5000 scene points that are within 2.5 meters away from the root joint of the last observed pose.

Original: Additional implementation details are given in the supplementary material.
Revised: Additional implementation details are given in the supplementary material.

Original: 4.3 Results
Revised: 4.3 Results

Original: Quantitative results.
Revised: Quantitative results.

Original: We provide quantitative results on GTA-IM and PROX in Table 1 and 2,respectively.
Revised: We provide quantitative results on GTA-IM and PROX in Table 1 and 2, respectively.

Original: Our approach outperforms the baselines for 3D paths and poses on both datasets acrossalmost all time steps by a large margin.
Revised: Our approach outperforms the baselines for 3D paths and poses on both datasets across almost all time steps by a large margin.

Original: Specifically, the baseline models either perform well for the3D path but comparatively poorly for the local poses, e.g., SLT [33] with an average path error [-of121.
 mm-] but the highest pose error on GTA-IM, or the reverse, e.g., DMGNN [18].
Revised: Specifically, the baseline models either perform well for the 3D path but comparatively poorly for the local poses, e.g., SLT [33] with an average path error [-of121.
 mm-]of 121.8mm but the highest pose error on GTA-IM, or the reverse, e.g., DMGNN [18].

Original: By contrast, ourmodels produce more accurate 3D paths and poses than those of the baselines.
Revised: By contrast, our models produce more accurate 3D paths and poses than those of the baselines.

Original: As an ablation study, we trained our human motion forecasting network without contact maps (“Oursw/o contact”) and observe an increase of up-to 10mm in the mean path error and 7mm in the meanpose error.
Revised: As an ablation study, we trained our human motion forecasting network without contact maps (“Ours w/o contact”) and observe an increase of up-to 10mm in the mean path error and 7mm in the mean pose error.

Original: By contrast, using the ground-truth contact maps (“Ours w/ GT contact”) yields a furtherperformance boost, especially on the mean path error with a decrease of up-to 104mm.
Revised: By contrast, using the ground-truth contact maps (“Ours w/ GT contact”) yields a further performance boost, especially on the mean path error with a decrease of up-to 104mm.

Original: This indicatesthe effectiveness of conditioning the motion predictions on per-joint contact maps.
Revised: This indicates the effectiveness of conditioning the motion predictions on per-joint contact maps.

Original: Qualitative results.
Revised: Qualitative results.

Original: We show our contact maps of four joints (left foot, left elbow, right foot and rightelbow) in Fig.
Revised: We show our contact maps of four joints (left foot, left elbow, right foot and right elbow) in Fig.

Original: Our model predicts accurate contact maps for diverse motions, such as “walking”(top) and “sitting down” (bottom).
Revised: Our model predicts accurate contact maps for diverse motions, such as “walking” (top) and “sitting down” (bottom).

Original: Note that, for the sample from GTA-IM [6] where the subject isabout to walk around a corner, our contact maps precisely capture the contacts between the left elbowand the wall (shown as purple points on the wall), leading to accurate human motion predictions. 
Revised: Note that, for the sample from GTA-IM [6] where the subject is about to walk around a corner, our contact maps precisely capture the contacts between the left elbow and the wall (shown as purple points on the wall), leading to accurate human motion predictions. [Figure]

Original: We further compare our results with those of the baselines in Fig.
Revised: d e v i [Figure]

Original: We restrict this comparison to
Revised: S i

Original: DMGNN [18] and SLT [33], which are quantitatively more accurate than LTD [22].
Revised: [Figure] e v i e [Figure]

Original: The completecomparison is included in the supplementary material.
Revised: p v i e w

Original: Due to the lackof explicit constraints onglobal motion and local movements, the baseline methodstend to produce unrealistic human motions,such as motions with almost no local movements but large global translations, i.e., ghost motion(highlighted with a red box), walking through a sofa (highlighted with green boxes). Thanks to ourper-joint contact maps, our results are more plausible and closer to the ground truth.
Revised: LTD supplementary lack constraints on global motion and local movements baseline methods

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we have introduced a framework for scene-aware human motion forecasting thatencourages consistency between global motion and local poses by exploiting human-scene contacts.
Revised: In this paper we have introduced a framework for scene-aware human motion forecasting that encourages consistency between global motion and local poses by exploiting human-scene contacts.

Original: [Figure]

Original: To this end, we have proposed a per-joint contact map representation that captures the contactrelationships between every human joint and the scene points.
Revised: To this end we have proposed a per-joint contact map representation that captures the contact relationships between every human joint and the scene points.

Original: Our model consists of two stages.
Revised: Our model consists of two stages.

Original: Wefirst predict the per-joint contact maps given the motion history, and then forecast the future globaltranslations and local poses given the estimated future contact maps.
Revised: We first predict the per-joint contact maps given the motion history and then forecast the future global translations and local poses given the estimated future contact maps.

Original: Thanks to the explicit constraintsprovided by our per-joint contact maps, our method yields more plausible and more accurate future human motions than the state-of-the-art motion prediction or scene-aware human synthesis strategies.
Revised: Thanks to the explicit constraints provided by our per-joint contact maps, our method yields more plausible and more accurate future human motions than the state-of-the-art motion prediction or scene-aware human synthesis strategies.

Original: Limitations & Societal Impacts One limitation of our work is that the quality of our predictedmotions depends on that of the contact maps.
Revised: Limitations & Societal Impacts One limitation of our work is that the quality of our predicted motions depends on that of the contact maps.

Original: As evidenced by our results with ground-truth contactmaps, improving the contact map predictions translates in better motion predictions.
Revised: As evidenced by our results with ground-truth contact maps, improving the contact map predictions translates in better motion predictions.

Original: This willtherefore be one of our future research directions.
Revised: This will therefore be one of our future research directions.

Original: Additionally, in many applications where thehuman shape matters, our joint-based contact map may not be enough to regularize the human surface.
Revised: Additionally, in many applications where the human shape matters, our joint-based contact map may not be enough to regularize the human surface.

Original: In our future work, we would like to extend our contact map to human surface.
Revised: In our future work we would like to extend our contact map to human surface.

Original: Furthermore, in realapplication, a potential risk of our method is that it may predict future motions that do not obey realphysical rules, e.g., motion with imbalanced forces.
Revised: Furthermore, in real application, a potential risk of our method is that it may predict future motions that do not obey real physical rules, e g., motion with imbalanced forces.

Original: For example, in the scenario of human-robotinteraction where the agent i.e., a robot, needs to plan its actions according to the future humanmotion, such physically unrealistic future motion may lead to unsafe situations such as collision. 
Revised: For example, in the scenario of human-robot interaction where the agent i e., a robot, needs to plan its actions according to the future human motion, such physically unrealistic future motion may lead to unsafe situations such as collision. Acknowledgements

Revised: This research was supported in part by the Australia Research Council DECRA Fellowship (DE180100628) and ARC Discovery Grant (DP200102274).

Revised: The authors would like to thank

Revised: NVIDIA for the donated GPU (Titan V).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] see Section
Revised: [Yes] see Section(b)

Original: (b) Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] see Section
Revised: [Yes] see Section(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] seeSection (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] see Section(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Data andimplementation details are in Section 4 and the supplementary material.
Revised: [Yes] Data and implementation details are in Section 4 and the supplementary material.

Original: We will release our source code upon the acceptance of this paper.
Revised: Our code is available at https://github.com/wei-mao-2019/ContAwareMotionPred .

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] see Section 4 and the supplementary material (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] see Section 4 and the supplementary material (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary material 
Revised: [Yes] See the supplementary material 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: Yes (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] see the supplementary material (c) Did you include any new assets either in the supplemental material or as a URL?[No] [Table]
Revised: [Yes] see the supplementary material (c) Did you include any new assets either in the supplemental material or as a URL?

Original:  (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... ) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A Did you describe any potential participant risks, with links to Institutional Review ) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A
Revised: [N/A]