HUMANISE : Language-conditioned Human Motion Generation in 3D Scenes
Anonymous Author(s)
AffiliationAddress email
Sit on the couch
Walk to the refrigerator
Lie down on the bed Stand up from the table far away from the door
Sit on the toilet
Abstract
Learning to generate diverse scene-aware and goal-oriented human motions
in3D scenes remains challenging due to the mediocre characteristics of the existingdatasets on Human-Scene Interaction (HSI); they only have limited scale/qualityand lack semantics.
To fill in the gap, we propose a large-scale and semantic-richsynthetic HSI dataset, denoted as HUMANISE , by aligning the captured humanmotion sequences with various 3D indoor scenes.
We automatically annotatethe aligned motions with language descriptions that depict the action and theindividual interacting objects; e.g ., sit on the armchair near the desk.
HUMANISE thus enables a new generation task, language-conditioned human motion generationin 3D scenes .
The proposed task is challenging as it requires joint modelingof the 3D scene, human motion, and natural language.
To tackle this task, wepresent a novel scene-and-language conditioned generative model that can produce3D human motions of the desirable action interacting with the specified objects.
Our experiments demonstrate that our model generates diverse and semanticallyconsistent human motions in 3D scenes.
We believe our proposed dataset andmodel can facilitate the HSI research, and will release them for research purposes.
Introduction
Imagine instructing a virtual human or robot to “sit on the armchair near the desk.”
Upon acceptingthe instruction, the agent needs to comprehend its semantic meaning ( i.e ., sit ) and understand thesurrounding environment ( i.e ., the armchair near the desk ) to generate desired interactions with the3D scene.
The vision community has been trying to attain this goal by automating the realistic humanmotion generation but still struggles in connecting the instructions to the 3D human motions.
Specifically, existing works generate plausible human poses [Zhang et al.
, 2020b;a] or motions [Starkeet al., 2019;
Wang et al., 2021a; Cao et al., 2020; Wang et al., 2021b; Hassan et al., 2021a] in 3Dscenes by adopting learning-based methods.
Training these learning-based models exploits existing
HSI datasets: PROX [Hassan et al., 2019] and GTA-IM [Cao et al., 2020].
Although these datasetscontain 3D people moving in 3D indoor scenes, they suffer from two fundamental limitations thatprevent existing methods from learning instruction-aware 3D human motions.
• Limited scale and quality .
PROX [Hassan et al., 2019] only contains 12 indoor scenes withjittering human motion.
The synthetic GTA-IM [Cao et al., 2020] only has 49 scenes and lacks 3Dhuman shapes and diverse movement styles.
• Absence of scene and language semantics .
PROX [Hassan et al., 2019] and GTA-IM [Cao et al.
,2020] both suffer from incomplete 3D semantic segmentation annotations and the lack of human
action labels/descriptions, making it infeasible to generate instruction-aware human motions.
To tackles the issues above, we propose a large-scale and semantic-rich synthetic HSI dataset,
HUMANISE (as shown in Fig. 1), by aligning the captured human motion sequences [Mahmood et al.
,2019] with the scanned indoor scenes [Dai et al., 2017].
Specifically, for an action-specific motionsequence ( e.g ., sit ), we first sample an interacting target object ( e.g ., armchair ) and the possibleinteracting positions on the object surface.
Next, we sample the valid translation and orientationparameters by employing the collision and contact constraints, such that the interactions between thetransformed human and the scene are physically plausible and visually natural.
We automaticallyannotate the synthesized motion sequences with template-based language descriptions; for instance,sit on the armchair near the desk .
To specify the interacting objects unambiguously, we adopt theobject referential utterances in Sr3D [Achlioptas et al., 2020].
In total, HUMANISE contains 4,892high-quality motion sequences in 512 indoor scenes.
Each motion segment has rich semantics aboutthe action type and the corresponding interacting objects, specified by the language description.
Based on HUMANISE , we solve a new task: language-conditioned human motion generation in3D scenes .
It requires the model to generate diverse and physically plausible human motions withdesignated action types and interacting 3D objects that are specified by the language descriptions.
This task is more challenging than previous motion generation tasks: (i) Both the 3D scene and thelanguage description pose constraints on motion generation.
(ii)
The generated human motions shouldperform the correct action and be precisely grounded near the target location according to the languagedescriptions.
(iii)
The generated human motions should be realistic and physically plausible withinthe 3D scenes.
Therefore, an ideal model should equip with capabilities of generation , multi-modalunderstanding , and language grounding .
Due to the demanding nature of this new task, we utilize the conditional variational auto-encoder(cVAE)
[Sohn et al., 2015] framework to generate human motions conditioned on the given sceneand language description.
The scene and language conditions are learned with separate encodersand fused with the self-attention mechanism to generate the conditional embedding.
The inputmotion is encoded with a recurrent module and decoded with a transformer decoder to generatehuman motions.
We further design two auxiliary losses for target object grounding and actionspecific generation.
The qualitative and quantitative results show that our model generates diverseand semantically consistent human motions in 3D scenes; it outperforms the baselines on variousevaluation metrics.
By evaluating on downstream applications, further experiments demonstrate that
HUMANISE alleviates the limitations of existing HSI datasets.
This paper makes three primary contributions: (i)
We propose a large-scale and semantic-rich synthetic
HSI dataset, HUMANISE , that contains human motions aligned with 3D scenes and correspondinglanguage descriptions.
(ii)
We introduce a new task of language-conditioned human motion generationin 3D scenes that requires a holistic and joint understanding of scenes, humans, and language.
(iii)
We develop a generative model that produces diverse and semantically consistent human motionsgiven the 3D scene and language description.
Related Work
Conditional Human Motion Synthesis Human motion synthesis can be conditioned on past motion[Li et al., 2018;
Barsoum et al., 2018; Yuan and Kitani, 2020; Cao et al., 2020; Xie et al., 2021], 3Dscenes [Starke et al., 2019; Wang et al., 2021a;b; Hassan et al., 2021a], objects [Chao et al., 2019;
Taheri et al., 2020], action labels [Guo et al., 2020;
Petrovich et al., 2021], and language descriptions[Ahuja and Morency, 2019; Ghosh et al., 2021; Lin and Amer, 2018].
Given a furnished 3D indoorscene, Zhang et al.
[2020b;a]; Hassan et al.
[2021b;a]; Wang et al.
[2021a;b] generate physicallyplausible single-frame human pose or long-term human motions.
Due to the lack of semanticsconstraints, these methods either generate random/uncontrollable human motions or need targetpositions as the input.
Conditioned on informative textual descriptions, Lin and Amer [2018]; Ahujaand Morency [2019]; Ghosh et al.
[2021] generate human motions that meet the input descriptions ofhow a human motion is performed.
However, these works focus only on motions alone and neglectthe scene context.
In this work, we incorporate conditions on both scenes and language descriptions,generating human motion sequences consistent with the specified action and location.
Human-Scene Interaction (HSI)
Datasets Real and synthetic datasets have been proposed tostudy HSI related topics.
PiGraphs [Savva et al., 2016] is captured with RGB-D videos and containsestimated human skeletons in 30 scanned scenes.
However, it suffers from inaccurately reconstructedscenes, noisy human poses, and a low frame rate.
PROX [Hassan et al., 2019] proposes new pipelinesto reconstruct human motions from monocular RGB-D sequences captured in 12 indoor scenes.
Nevertheless, the gap between their pose fitting results and high-quality, large-scale MoCap data( e.g ., AMASS [Mahmood et al., 2019]) is still non-trivial.
GTA-IM [Cao et al., 2020]
exploitsthe game engine to collect a synthetic dataset of human skeletons, but with limited human-sceneinteractions and scene diversities.
In comparison, our HUMANISE dataset synthesizes the HumanScene Interaction (HSI) by aligning the high-quality motions in AMASS [Mahmood et al., 2019] withreal-world 3D scenes in ScanNet
[Dai et al., 2017].
As a result, HUMANISE contains high-qualityand realistic human-scene interactions with high diversity ( i.e ., 512 scenes, 4 , 892 motions) and richsemantics captured in the auto-generated language descriptions.
Language Grounding in 3D Scenes Grounding 3D objects with language has been explored in 3Dreference [Achlioptas et al., 2020; Thomason et al., 2022], 3D question answering (QA)
[Das et al.
Ye et al., 2021; Azuma et al., 2021], and embodied navigation [Anderson et al., 2018; Gordonet al., 2019; Gan et al., 2020].
With the language description condition, we aim to generate humanmotions that interact with the specified object in the scene.
However, locating the interacting objectsin the 3D scene with referential descriptions is still challenging [Roh et al., 2022;
Chen et al., 2020]due to the noisy scanned scenes, diverse expressions, and complex spatial reasoning.
To learn betterconditional embedding that facilitates to locate and ground according to language, we introduce amulti-model conditional module that learns a joint embedding from textual descriptions and 3D pointclouds.
We further enhance the model’s spatial and action grounding capabilities by introducing twoauxiliary losses of locating the interacting 3D objects and action-specific generation.
The HUMANISE Dataset
A large-scale and high-quality dataset with rich semantic information is in need to facilitate theresearch in human motion synthesis and HSI.
In this work, we propose a synthetic dataset byleveraging the existing datasets in human motion ( i.e ., AMASS [Mahmood et al., 2019]), 3D indoorscene ( i.e ., ScanNet
[Dai et al., 2017]), and additional annotations of these datasets ( i.e ., motionaction label in BABEL [Punnakkal et al., 2021] and 3D referential descriptions in Sr3D [Achlioptaset al., 2020]).
Our novel automatic synthesis pipeline avoids expensive hardware and time-consuminglabor for realistic capture.
Fig.
2 shows more samples in HUMANISE .
Of note, our dataset synthesispipeline can generalize to other datasets and actions; please refer to Sup.
Mat.
for more details.
Below, we start by describing to align the motion with the 3D scene in Sec. 3.1, followed by how toautomatically generate the language description in Sec. 3.2.
Sec.
3.3 provides dataset statistics.
3.1 Motion Alignment
The idea of synthesizing HUMANISE is straightforward: Automatically aligning captured humanmotions with 3D indoor scenes.
Given a human motion sequence and its action label, we first sample
[Figure]
an interacting target object by the predefined category labels, and possible interacting positions onthe object surface in the 3D scene.
Next, we sample the valid global translation T and rotation Rparameters by employing the collision and contact constraints.
We align the human motion with the3D scene through a transformation with T and R , while all the local poses remain unchanged.
Collision Constraint We treat the point cloud of the 3D scene as a 3-dimensional k-d tree.
Collisionavoidance of human motions is equivalent to finding the closest point to the queried human vertex inthe scene by searching through the k-d tree.
Collision happens as the distance between the humanvertex and the closet point in the scene is less than a threshold.
A valid motion should avoid havingcollisions with the objects or walls for all the poses along the trajectory.
Contact Constraint We decompose this constraint into the foot-ground support constraint and theproximity between the human and the interacting object.
The support constraint is employed with adistance threshold between foot and the ground plane.
The human-object proximity differs amongactions; we design the following proximity constraints to make the interaction realistic.
• Sit
We sample a sittable object ( e.g ., a chair) as interacting objects and uniformly sample 3Dpoints on the object surface as potential contact points.
We assume the final pose of the sittingmotion should contact the sittable object on the hip area, such that the distances between the hip
vertices and the contact point should be less than a threshold.
• Stand-up Since the stand-up motion interacts with the sittable object at the starting frame, weenforce the similar constraints as in sit on the starting pose, instead of the final pose.
• Walk The language description in HUMANISE instructs the walking motion to a target locationin the scene, usually to an object; e.g ., walk to the refrigerator .
We sample dense positions near thespecified object on the floor as the potential targets for the final pose of walking motion.
All thebody poses along the walking trajectory should satisfy the feet-ground support and avoid collisions.
• Lie-down We select objects with a large flat surface ( e.g ., bed) as the interacting objects.
Thetranslation T and rotation R are sampled such that the final pose of the lie-down motion is in
contact with the object’s top surface.
Apart from the non-collision and contact constraint, we
require the human body to lie within the object area.
Diverse Affordance We can sample motions representing diverse 3D affordance with the motionalignment algorithm.
For instance, stand up from the coffee table , lie down on the office table and siton the toilet .
HUMANISE contains these Human-Object Interactions (HOIs) that are not frequentlycaptured in daily activity or previous HSI datasets.
They encode meaningful and versatile object/sceneaffordances which introduce both potentials and challenges for the model learning
3.2 Language Description Generation
The motion descriptions in HUMANISE depict the action type and the singular interacting object/location unambiguously.
To automatically generate these language descriptions, we design acompositional template following Sr3D [Achlioptas et al., 2020]:
[Equation]
For instance, sit on the armchair near the desk.
The template contains four placeholders, of whichspatial-relation and anchor are optional placeholders.
The action is taken from the motion labels in
BABEL [Punnakkal et al., 2021], whereas the target represents the interacting object.
To uniquelyrefer to a target object, a spatial-relation is defined between the target and a surrounding object( anchor ).
We utilize the referential utterances in Sr3D [Achlioptas et al., 2020], which defines fivetypes of spatial relations: horizontal proximity, vertical proximity, between, allocentric, and support.
3.3 Dataset Statistics
In total, HUMANISE includes 4,892 human motion sequences in 512 3D scenes.
We choose thefour most representative interactive indoor actions, i.e ., sit (1299), stand up (936), lie down (169),and walk (2488).
Tab. 1 compares HUMANISE against existing HSI datasets, wherein HUMANISEshows advantages in clip number, scene variety, and human pose quality.
It is the only dataset withrich semantic information of HSIs.
Fig.
2 shows more examples from HUMANISE .
[Table]
✗ ✓
Language-conditioned Human Motion Generation in 3D Scenes
4.1 Problem Definition
Given a 3D scene and a language description, our goal is to generate a realistic human motionsequence that is both semantically consistent with the language description and physically plausiblein the scene.
We denote the given scene as S ∈ R N × 6 , representing an RGB-colored point cloudof N points.
The language description is a tokenized word sequence of length D , denoted as
L 1: D = [ w 1 , · · · , w D ] .
We represent the human pose and shape in the motion sequence with the
SMPL-X
[Pavlakos et al., 2019] body model.
Specifically, the SMPL-X body mesh M ∈ R 10475 × 3 isparameterized by M = F ( t, r, β, p ) , where t ∈ R 3 is the global translation, r ∈ R 6 is a continuousrepresentation [Zhou et al., 2019] of the global orientation, β ∈ R 10 is the body shape parameters,p ∈ R J × 3 is J joint rotations in axis-angle, and F is a differentiable linear blend skinning function.
As we focus on producing the plausible human position and pose, we use the default mean bodyshape for β and denote the parameters of interest as Θ = { t, r, p } .
4.2 Method
We propose a novel generative model to generate human motions conditioned on the given sceneand language description.
The overall architecture is illustrated in Fig.
Specifically, we employ aconditional variational auto-encoder (cVAE) framework to model the probability p (Θ 1: T | S, L 1: D ) ,where T is the length of the motion sequence.
Below, we describe the details of each module.
Condition Module The condition module takes input from two modalities ( i.e ., the given scene Sand the language description L ) and outputs a joint conditional embedding.
To process the 3D scene,we employ the Point Transformer
[Zhao et al., 2021] to produce point-level features [ u 1 , · · · , u N ′ ] ,where the transition-down module reduces the cardinality of the scene point cloud from N points to
N ′ points.
For the language description L 1: D , we use the pre-trained BERT [Kenton and Toutanova,2019] to extract the word-level embedding [ v 1 , · · · , v T ] .
The point-level features and the word-level features are concatenated and fed into the feature fusionmodule, a single-layer self-attention module.
In this way, both the point-level and the word-level
... wwww 1 ... w 1 ... ww t w T
BERT vv t v T
Point Transformer uu n u N ’ [CLS] sit chair
Self-attention Layer f s f w
FC Layer z c
Auxiliary Tasks
Motion Encoder ΘΘ t Θ T μ, Σ
Sampling from N (μ, Σ)
Motion Decoder z t
T ΘΘ t Θ T ^ ^ ^
[Figure]
features attend to the features from two modalities by self-attention.
The updated point features areconcatenated and forwarded to a fully-connected (FC) layer to obtain the scene feature f s .
We takethe updated [ CLS ] feature as the language feature f w .
The scene and language features are finallyconcatenated and mapped to a conditional latent embedding z c with FC layers.
Motion Encoder We first use a bidirectional GRU to obtain a sequence-level feature of the inputmotion Θ 1: T .
Next, the output is concatenated with the conditional embedding z c , followed by an
MLP layer to predict the Gaussian distribution parameters ( i.e ., µ and Σ ).
Finally, we sample a latentvector z from the distribution using the reparameterization trick [Kingma and Welling, 2013].
Motion Decoder Following Petrovich et al.
[2021], we use a transformer decoder to generate asequence of parameters (cid:98) Θ 1: T in a given duration T .
Specifically, we use T sinusoidal positionalembeddings to query the concatenation of the sampled latent z and the conditional embedding z c .
The outputs of the transformer decoder are mapped into body meshes with the followed differentiable
SMPL-X model.
Of note, although the duration T equals to the length of the input motion sequencein the training phase, it can be arbitrary values during inference.
4.3 Training Loss
In our work, the training loss contains reconstruction loss, KL divergence loss, and two auxiliarylosses for target object grounding and action-specific generation, formulated as
[Equation]
where α denotes the loss weight of each term.
Reconstruction Loss Our reconstruction loss is formulated as
[Equation]
where L t , L r , and L p are ℓ 1 distance between the predicted body parameters and the ground truth ( i.e .,global translation t , global rotation r , and joint rotations p ).
L v is an additional term for regressingthe positions of the vertices on the SMPL-X body mesh.
In practice, the vertex set can be all 10475vertices on SMPL-X body mesh or selected sparse markers [Zhang et al., 2021].
KL Divergence Loss Denoting the learned distribution of latent z as ψ = N ( µ, Σ) , we regularizeψ to be similar to the normal distribution ϕ
= N (0 , I ) , i.e ., L kl =
D KL ( ψ ∥ ϕ ) .
Auxiliary Loss
Notably, the learning of the conditional latent embedding z c is crucial to ourgenerative model, as it represents compound information of the action, the interacting object, and the3D scene context from two different modalities.
As such, rather than simply maximizing the negativeevidenced lower bound (ELBO), we introduce two auxiliary tasks for locating the interacting targetobject and generating action-specific motions.
For the first auxiliary task, we directly use a FC layer on top of the latent z c to regress the targetobject’s center.
For the second auxiliary task, we use another FC layer followed by softmax to mapthe latent z c to the probability of each action.
Specifically, we use the MSE loss L o to regress thetarget object center and the cross-entropy loss term L a to classify the action category.
4.4 Implementation Details
We train our generative model on HUMANISE for 150 epochs using Adam [Kingma and Ba, 2014]and a fixed learning rate of 0.0001.
For hyper-parameters, we empirically set α kl =
α o = 0 .
1 ,α a = 0 .
5 , α r = 1 . 0 , and α p = α v = 10 . 0 .
We set the dimension of global condition latent z c to 512and latent z to 256 .
The hidden state size is set to 256 in the single-layer GRU motion encoder.
Thetransformer motion decoder contains two standard layers with the 512 hidden state size.
We train ourmodel with a batch size of 24 on a V100 GPU.
It takes about 16 hours to converge.
Point Transformer Our point transformer module that processes the scene point cloud adopts theoriginal architecture [Zhao et al., 2021].
For each scene, we down-sample N = 32768 points fromthe original scanned scene and obtain N ′ =
128 points with a 512 -D feature for each point afterapplying the point transformer.
We pre-train the point transformer with a semantic segmentation taskon ScanNet
[Dai et al., 2017] dataset, whose train-test split is the same as HUMANISE .
Experiments
We benchmark our proposed task, language-conditioned human motion generation in 3D scenes on
HUMANISE , and describe the detailed settings, baselines, analyses, and ablative studies.
Dataset Splitting We split motions in HUMANISE according to the original scene IDs and split in
ScanNet [Dai et al., 2017], resulting in 4131 motions in 427 scenes for training and 761 motions in85 scenes for testing.
5.1 Experimental Setting
To evaluate the model performance, we conduct experiments in both the action-specific and actionagnostic settings.
In the action-specific setting, we train and evaluate our model on HUMANISEsubsets of each action without the auxiliary task of classifying the action category.
In the actionagnostic setting, we train our full model on the whole dataset with auxiliary tasks.
Ablative Baselines We compare our model with three ablative baselines to verify the significanceof the auxiliary tasks, the feature fusing module, and the scene encoder.
(i)
We remove the auxiliarytasks for locating the target interacting object and recognizing the action category, denoted as w/o aux.loss .
We train this model on the full HUMANISE dataset.
(ii) Instead of feeding the scene and thelanguage feature into the self-attention layer, we directly concatenate them as the global conditionalfeature.
We denote this model as w/o self-att .
(iii)
We replace the point transformer with PointNet++[Qi et al., 2017]
( i.e ., PointNet++ Enc. ), also pertained with the semantic segmentation task.
We trainand test the second and the third ablation models on the walk subset.
5.2 Evaluation Metrics
Reconstruction Metrics Following Wang et al.
[2021a], we introduce the following metrics toevaluate the model’s reconstruction capability.
We report the ℓ 1 error ( × 100 ) between the predicted
SMPL-X parameters and the ground truth, including global translation t , global orientation r , andbody pose p .
We also compute the Mean Per Joint Position Error (MPJPE)
[Ionescu et al., 2013] and
Mean Per Vertex Position Error (MPVPE) in millimeters between the reconstructed SMPL-X bodymesh and the ground truth to evaluate the reconstruction in a more intuitive and fine-grained manner.
We report these metrics by averaging the sequence due to the different motion durations.
Generation Metrics As we hope our generated motion to interact with the correct object specifiedby the language description, we introduce a body-to-goal distance metric ( goal dist. ).
It is theshortest distance (in meters) from the target object to the interacting human body, computed as
Max(Min(Ψ + M ( O )) , 0) , where Ψ + M ( · ) is the positive signed distance field of body mesh M , and
O ⊂ S is the target object point set.
Of note, we use the human body in the first frame to compute
R e f e r e n ce
G e n e r a ti on
A tt e n ti on
Walk to the bed
Sit on the sofa chair that is close to the tv
Stand up from the coffee table close to the table
Lie on the table
G e n e r a ti on
G e n e r a ti on the body-to-goal distance for stand up action and use the last frame for other actions.
We generate tenrandomly sampled motion sequences for each case in the test split and report the average distance.
Perceptual Study We perform human perceptual studies to evaluate the generation results in termsof the overall quality and action-semantic accuracy.
Given the rendered motion and the languagedescription, a worker is asked to score from 1 to 5 for (i) the overall generation quality , and (ii)whether the generated motions are performing the action specified by the description.
A higher valueindicates that the generated result is more plausible with the given scene and language description.
We randomly generate samples in 20 scenarios for each model, and three workers score each sample.
5.3 Results and Discussion
Tab. 2 tabulates the quantitative results of both reconstruction and generation.
Fig.
4 and Fig.
5 showsome qualitative results.
Below we discuss some significant observations.
• As seen in Tab. 2, our model performs well in the action-specific setting, e
g ., walk , sit , and stand up
Fig.
4 also shows
our
model generates human motions that are
visually appealing
and semantically
consistent with
the language description.
Furthermore, the attention visualizations show that the[ CLS ] token attends to the point cloud features around the target interacting objects.
Meanwhile,we also notice that the performance drops in the lie down action, which might be caused by theunbalanced data distribution; the lie down subset is only 10% of
the sit subset. • The action-agnostic setting is more complex and difficult than the action-specific setting, but theauxiliary tasks improve the model’s capability in spatial and action grounding.
From Tab. 2, our fullmodel all actions outperforms the one w/o aux.
loss by a large margin in both reconstruction andgeneration.
The significant difference in goal dist.
and action score show that the proposed auxiliarytasks can help our generative model to locate the target 3D object and generate action-specificmotions.
Qualitative results in Fig.
5 further verifies this observation.•
The ablative experiments on the feature fusing module and scene encoder validate our model designs.
For the action walk , our full model outperforms the model w/o self-att.
in all generation metrics.
We conclude that the transformer layers learn better conditional embedding by attending to features
[Table]
[Figure]
across the scene and language modalities.
The difference on goal dist.
compared with PointNet++
Enc. shows our better grounding performance through finer-grained scene understanding.
Please refer to Sup.
Mat.
for additional experiments, qualitative results, and failure case discussions.
5.4 HUMANISE for Downstream Task
We further demonstrate the quality of HUMANISE by another existing downstream task proposedin Wang et al.
[2021a]: human motion synthesis in 3D scenes .
This task only considers the sceneconditioned generation, and the language annotations in HUMANISE are not used in this setting.
Specifically, we train the RouteNet and PoseNet in Wang et al.
[2021a] using HUMANISE with tenepochs and fine-tune them using PROX [Hassan et al., 2019] with another ten epochs.
For a faircomparison, we reproduce the results of the same model trained on PROX for 20 epochs under thesame setting.
We evaluate both models on the test split of PROX.
The reconstruction errors with metrics from Wang et al.
[2021a] are reported in Tab. 3.
The modelpre-trained on HUMANISE outperforms the model trained only with PROX on all evaluation metricsby a large margin, except for the orientation.
It shows our dataset can alleviate the limitations ofexisting HSI datasets and potentially support more applications in HSI related topics.
[Table]
Conclusion
In this work, we propose a large-scale and semantic-rich HSI dataset, HUMANISE .
It contains diverseand physically plausible interactions equipped with language descriptions.
HUMANISE enables anew task: language-conditioned human motion generation in 3D scene.
We further design a sceneand-language conditioned generative model that produces diverse and semantically consistent humanmotions based on HUMANISE .
Limitations Our work is primarily limited on two aspects.
(i)
The human motions are relativelyshort since action-specific motions usually last within 5 seconds.
A large-scale, long-duration HSIdataset is in need for the scene understanding community.
(ii)
The problem of language grounding in3D scenes is critical for our task but far from being solved, as also revealed in recent 3D groundingworks [Achlioptas et al., 2020; Thomason et al., 2022].
A robust grounding module could helpgenerate more meaningful human motions based on language description.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] The dataset contribution
is illustrated in Sec.
and
Tab. 1.
The task contribution is shown in Fig
The model contribution is analysed in
Sec. 5 and Tab. 2.
(b) Did you describe the limitations of your work?
[Yes] Please refer to Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Pleaserefer to Sup.
Mat. .
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We provide
the code, sample data
and trained checkpoint in the Sup.
Mat. .
This will reproduce the
qualitative results from Fig.
4 and will roughly reproduce the quantitative result from
Tab. 2.
The full dataset is too large to include in the Sup.
Mat. , but we will release the
full dataset upon acceptance of this paper.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please refer to Sec. 4.4 and Sec.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No] We are aware of the requirements but cannot make it due
to time limit.
We include the code in Sup.
Mat. for reproduction and will add it in the
later version.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Sec. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] We mention it in the Sup.
Mat. .
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[Yes] We include the details of human evaluation in the Sup.
Mat. .
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]