Embodied Scene-aware Human Pose Estimation
Anonymous Author(s) AffiliationAddress email
Abstract
We propose embodied scene-aware human pose estimation where we estimate 3Dposes based on a simulated agent’s proprioception and scene awareness, alongwith external third-person observations.
Unlike prior methods that often resortto multistage optimization, non-causal inference, and complex contact modelingto estimate human pose and human scene interactions, our method is one stage,causal, and recovers global 3D human poses in a simulated environment.
Since2D third-person observations are coupled with the camera pose, we propose todisentangle the camera pose and use a multi-step projection gradient defined in theglobal coordinate frame as the movement cue for our embodied agent.
Leveraginga physics simulation and prescanned scenes ( e.g., 3D mesh), we simulate ouragent in everyday environments (libraries, offices, bedrooms, etc. ) and equip ouragent with environmental sensors to intelligently navigate and interact with scenegeometries.
Our method also relies only on 2D keypoints and can be trained onsynthetic datasets derived from popular human motion databases.
To evaluate, weuse the popular H36M and PROX datasets and, for the first time, achieve a successrate of 96.7% on the challenging PROX dataset without ever using PROX motionsequences for training.
Introduction
We consider a 3D human pose estimation setting where a calibrated camera is situated within a prescanned environment (with known 3D geometries such as meshes).
Such a setting has a wide range of applications in mixed reality telepresence, animation and gaming, at-home robotics, videosurveillance, etc. , where a single fixed RGB camera is often used to observe human movement.
However, even under such constraints, severe occlusion, low-textured garments, and human-sceneinteractions often cause pose estimation methods to fail catastrophically.
Especially when estimatingglobal 3D human pose estimation, state-of-the-art methods often resort to expensive multistage batchoptimization [5], motion prior or infilling [26, 42], and depth information to recover reasonableglobal translation and orientation.
Consequently, these methods cannot be applied in a streamingfashion and have limited use cases in time-critical applications.
Multistage optimization can alsosuffer from failure of convergence, where the optimizer cannot find a good solution due to eitherocclusion, numerical instability, or bad initialization.
Effective use of scene information is also critical for estimating human pose.
Unfortunately, the lackof datasets with paired ground truth 3D body pose and annotated contact labels make it difficult toincorporate scene information to estimate pose.
Thus, popular methods often resort to minimizingthe distance between predefined contact points in the body and the scene [5], using vertex-basedproximity [43] as a proxy scene representation, or semantic labels as affordance cues [15].
Thesemethods, while effective in modeling human-scene interactions, are expensive to compute and heavilyrely on an batch optimization procedure to achieve good performance.
More complex scenes are alsoharder to model in a batch optimization framework, as the number of contact constraints increasessignificantly as the number of geometries increases.
We hypothesize that embodiment may be a key element in estimating accurate absolute 3D humanposes.
Here we define embodiment as having a tangible existence in a simulated environment inwhich the agent can act and perceive.
In particular, humans move according to their current bodyjoint configuration (proprioception), movement destination (goal), and an understanding of theirenvironment’s physical laws (scene awareness and embodiment).
Under these guiding principles, wepropose embodied scene-aware human pose estimation as an alternative to optimization-basedand regression-based methods.
Concretely, we estimate poses by controlling a physically simulatedagent equipped with environmental sensors to mimic visual observations in a streaming fashion.
Ateach timestep, our agent receives movement goal from a third-person observation and computesits torque actuation based on (1) current agent state: body pose, velocities, etc. , (2) comparisonbetween its current pose and visual observation, and (3) environmental features from the agent’spoint of view.
Acting in a physics simulator, our agent can receive timely feedback about physicalplausibility, contact, and scene geometry to compute the next time frame pose estimate.
While thereexist prior methods that also use simulated characters, we draw the distinction in how the next-frametarget pose is computed.
Previous work often directly regresses the kinematic pose [3, 40] withoutconsidering the current state of the body and environmental signals.
As a result, when the kinematicpose estimation method fails due to occlusion, these embodied agents will receive unreasonable targetposes.
Our method, on the other hand, predicts body velocities by comparing the current body posewith visual observations.
When 2D observations are occluded, body velocities will be computedbased on environment cues and current body states.
Thus, our method is more robust to occlusionand can recover from consecutive frames of missing body part observations.
Our method is also ableto estimate pose for long-term (several minutes), in sub-real-time (10 fps), while creating physicallyrealistic human scene interactions.
To achieve this, we propose a multistep projection gradient (MPG) to provide movement cues to ourembodied agent from 2D observations.
At each time step, MPG performs an iterative gradient descentstarting from the current pose of the humanoid to exploit the temporal coherence of human movementand better avoid local minima.
MPG also directly provides temporal body and root movement signalsin the world coordinate frame to our agent, so our method does not need to explicitly take the 3Dcamera pose as input.
To control a simulated character, we extend the Universal Humanoid Controller(UHC) and the kinematic policy proposed in previous work [18] to support the control of humanoidswith different body shapes in environments with diverse geometries.
Finally, we equip our agentwith an occupancy-based environmental sensor to learn about affordances and avoidance.
Since only2D keypoint coordinates are needed by our method to estimate body motion, our model can also belearned from purely synthetic data.
To summarize, our contribution is as follows: 1) we propose embodied scene-aware human poseestimation, where we control a simulated scene-aware humanoid to mimic visual observations andrecover accurate global 3D human pose and human-scene interactions; 2) we formulate a multistepprojection gradient(MPG) to provide camera-agnostic movement signal for an embodied agent to match 2D observations; and 3) we achieve state-of-the-art result on the challenging PROX datasetwhile our method remains sub-real-time, causal, and trained on synthetic data derived from popularhuman motion databases.
Related Work
Local and Absolute 3D Human Pose Estimation.
It is popular to estimate the 3D human pose inthe body local coordinate system (relative to the root)
[1, 9, 10, 11, 12, 13, 14, 17, 21, 22, 24, 25, 32,34] where translation is discarded and root rotation is relative to the camera coordinate frame.
Amongthese methods, some focus on lifting 2D keypoints to 3D [1, 24], while others use a template 3Dhuman mesh (the SMPL [16] body model and its extensions [23, 27]) and jointly recover body shapeand joint angles.
Among these methods, some [13, 32, 40] make use of iterative optimization to betterfit 2D keypoints.
SPIN [13] devises an optimization-in-the-loop fitting procedure to provide a betterstarting point for a pose regressor.
NeuralGD
[32] and SimPOE [40] both utilize a learned gradientupdate to iteratively refine body pose and shape, where the optimization starts from either zero pose(NeuralGD) or estimated kinematic pose (SimPOE).
Compared to these methods, our MPG startsthe optimization process from previous time-step pose to leverage the temporal coherence in humanmotion.
While the estimating pose in the local coordinate frame is useful in certain applications,it is often desirable to recover the pose in the absolute/global space to fully leverage the estimatedpose for animation or telepresence.
However, recovering the absolute 3D body pose (which includesroot translation) is challenging due to limb articulation, occlusion, and depth ambiguity.
Amongthe works that attempt to estimate the absolute human pose [21, 24, 25, 28], most take a two-stageapproach where the root-relative body pose is first regressed from a single image, and then a roottranslation module is used to estimate the absolute depth from image-level features.
In comparisonto these methods, we leverage physical laws and motion cues to autoregressively recover both rootmovement and body pose.
Scene-aware Human Pose and Motion Modeling The growing interest in jointly modeling humanpose and human-scene interactions has led to a growing popularity in datasets containing both humanand scenes [6, 20, 36].
In this work, we focus on the PROX [6] dataset, where a single human isrecorded moving in daily indoor environments such as offices, libraries, bedrooms, etc.
Due to thediversity of motion and human-scene interaction (sitting on chairs, lying on beds, etc.) and lack ofground-truth training data, the Prox dataset remains highly challenging.
Consequently, state-of-the-artmethods are mainly optimization-based [6, 26, 41] while leveraging the full set of observations (scenescans, camera pose, depth).
HuMoR [26] and LEMO [41] achieves impressive result by usinglarge-scale human motion datasets to learn strong prior of human movement to constrain or infillthe missing human motion.
However, multistage optimization can often become unstable due toocclusion or failures in 2D keypoint detections.
Physics-based humanoid control for pose estimation Physical laws guard every aspect of humanmotion and there is growing interest in enforcing physics prior in human pose estimation [3, 7, 18,30, 31, 37, 38, 39, 40].
Some methods prefer to use physics as a postprocessing step and refineinitial kinematic pose estimates [30, 31, 37], while ours follows the line of work where a simulatedcharacter is controlled to estimate pose inside a physics simulation [3, 7, 18, 40].
Although bothapproaches have their respective strong suites, utilizing a physics simulation simplifies the processof incorporating unknown scene geometries for contact and can empower real-time pose estimationapplications.
Among simulated character control-based methods, Neural Mocon [7] utilizes samplingbased control and relies on the learned distribution of human motion from the training split of thedataset to acquire quality motion samples.
SimPOE [40] first extracts the kinematic pose usingoff-the-shelf method [11] and then uses simulated agent for imitation.
PoseTriplet [3] proposes usinga self-supervised framework to co-evolve a pose estimator, imitator, and hallucinator.
Comparedto ours, SimPoE and PoseTriplet
both relies on a kinematic pose estimator as backbones [11, 24]and requires well-observed 2D keypoints for reliable pose estimates.
They also require lengthypolicy training time and do not model any scene geometries.
Our work also extends Kin_poly [18],where a simulated character is used to perform the first-person pose estimation.
While Kin_polyuses egocentric motion as the only motion cue we proposes a multi-step Projection Gradient toleverage thrid-person observations.
Compared to their object-pose based scene feature, we use anoccupancy-based scene feature inspired by the graphics community [4, 33] to better model affordance. [Figure]
Method
Given a monocular video I 1: T taken using a calibrated camera in a known environment, our aim isto estimate, track, and simulate 3D human motion in metric space.
We denote the 3D human poseas q t ≜ ( r R t , r T t , θ t ) , consisting of the 3D root position r T t , the orientation r R t and the angles of thebody joints θ t .
As a notation convention, we use (cid:101) · to represent kinematic poses (without physicssimulation), (cid:98) · to denote ground-truth quantities from Motion Capture (MoCap), and normal symbolswithout accents for values from the physics simulation.
In Sec 3.1, we will first summarize ourextension to the Universal Humanoid Controller [18] to support controlling people with differentbody proportions.
Then, in Sec.3.2, we will describe the state space of our character’s control policy:the multistep projection gradient (MPG) as a feature and occupancy scene feature.
Finally, we willdetail the training procedure to learn our embodied pose estimator using synthetic datasets in Sec 3.3.
Our pose estimation pipeline is visualized at Fig.2.
3.1 Preliminaries: Physics-based Character and Universal Humanoid Controller (UHC)
To simulate people with varying body shapes, we first use an automatic simulated character creationprocess similar to SimPoE [40] to generate human bodies with realistic joint limits, body proportions,and other physical properties based on the popular human mesh model SMPL [16].
In SMPL, bodyjoints are recovered from mesh vertices and a joint regressor, so length of each limb may varydepending on body pose.
This is undesirable as computing a large number of vertices (6890) istime-consuming and varying body limb length is physically inaccurate.
Therefore, we fix the lengthsof the limbs using those calculated from T-pose (zeroing the angles of the joints) and use forwardkinematics to recover the 3D location of the body joints J t .
Given the shape of the body β and thepose q t , we define the forward kinematics function as FK β ( q t ) → J t .
To learn a universal humanoid controller that can perform everyday human motion with differentbody types, we follow the standard formulation in physics-based character and model controllinga humanoid for motion imitation as a Markov Decision Process (MDP) defined as a tuple M =⟨S , A , T , R, γ ⟩ of states, actions, transition dynamics, reward function, and discount factor.
The states ∈ S , reward r ∈ R , and transition dynamics T are determined by the physics simulator, whilethe action a ∈ A is given by the control policy π UHC ( a t | s t , (cid:98) q t , β ) .
Here we extend the Universal
Humanoid Controller proposed in Kin_poly [18] condition the control policy on the current simulationstate s t , the reference motion (cid:98) q t , and the body shape β .
We employ Proximal Policy Optimization(PPO)
[29] to find the optimal control policy π UHC to maximize the expected discounted reward
E (cid:104)(cid:80) Tt =1 γ t − 1 r t (cid:105) .
For more information on the UHC network architecture and training procedure,please refer to Kin_poly [18] and our Appendix.
3.2 Embodied Scene-Aware Human Pose Estimation
To estimate the global body pose from the image observations, we control a simulated character tofollow 2D keypoint observations sequentially.
Assuming a known camera pose and scene geometry,we first use an off-the-shelf 2D keypoint pose estimator, DEKR [2], to extract 2D keypoints kp 2D t at aper-frame level.
Then, we use an off-the-shelf pose and body shape estimator, HuMoR [26] to extractthe first-frame 3D pose (cid:101) q 0 and body shape β from RGB observations as initialization.
Given (cid:101) q 0 andβ , we initialize our simulated character inside pre-scanned scene, and then we attempt to estimateposes which matches the observed 2D keypoints in a streaming fashion.
Multi-step Projection Gradient (MPG) as a Motion Feature To match the motion of the 3Dsimulated humanoids to 2D keypoint observations, it is essential to provide 3D information about howposes should be modified in the global coordinate frame (coordinate frame of the scene geometry andthe 3D humanoid).
As 2D keypoints are coupled with camera pose (the same body pose correspondsto different 2D keypoints based on the camera pose), it is problematic to directly input 2D observationswithout a proper coordinate transform.
To avoid this issue, prior arts that use simulated charactercontrol [3, 40] often estimate the kinematic human pose in the camera coordinates using a kinematicpose estimator.
However, since the up-vector of the camera is not always aligned with the directionof gravity, corrections to the character’s z direction are often needed for a successful simulation [3].
To decouple the pose of the camera from the pose estimation pipeline, we propose using a multi-stepprojection gradient (MPG) feature as an alternative to an estimated kinematic pose feature as an inputto the policy.
Intuitively, the MPG is a modified (learned) sum of 2D projection loss gradients thatencodes how much the 3D pose should be adjusted for a better fit.
Given the current global state ofthe humanoid q t and the camera projection function π cam , we define MPG ϕ as:
[Equation]
− q 0 t = [Equation] [Equation] ) , where q k +1 t
[Equation]
and L proj =
∥
π cam ( FK β ( q t ))
− kp 2D t +1 ∥ 2 is the L 2 reprojection loss with respect to the observed 2Dkeypoints kp 2D t +1 .
s is the step size of the gradient and k indexes into the total number of gradientdescent steps K .
The 2D project loss gradient ∂ L ∂ q t represents how the root pose and body joints shouldchange to better match the 2D keypoints of the next frame and contains rich geometric informationindependent of the camera projection function π cam .
Due to noise in 2D keypoint detection, occlusion, and depth ambiguity, a single instantaneousprojection gradient ∂ L proj ∂ q kt can be erroneous and noisy, especially the gradient of root translation andorientation.
Thus, we augment ∂ L proj ∂ q kt with a geometric and learning-based transform G which isintended to remove noise.
In this way, the MPG feature is an aggregation of a collection of modifiedgradients to compute a more stable estimate of the 2D projection loss gradient.
Inspired by PoseTriplet [3], we use a temporal convolution model (TCN) similar to VideoPose3D [24]to regress the camera-space root orientation from 2D keypoints.
We note that while TCN may produceunreasonable pose estimate during server occlusion, its root orientation estimation remains stable.
Wecompute the camera space root orientation using a causal TCN F TCN and transforms it into the worldcoordinate system root orientation r ′ R t using the inverse camera rotation: r ′ R t = F TCN ( kp 2D t − µ : t +1 ) .
We then compute the difference between current root rotation r R t and r ′ R t as an additional movementgradient signal.
Given the current 3D pose q kt and the projection function π cam , we can obtain a betterroot translation r Tt by treating the obtained 3D keytpoints as fixed on a rigid body and solving fortranslation using linear least squares.
Combining the F TCN model and the geometric transformationstep, we obtain the function G to obtain a better root translation and orientation gradient.
By takingmultiple steps, we avoid getting stuck in local minima and obtain better gradient estimates.
For amore complete formulation of MPG, see our Appendix.
Scene Representation To provide our embodied agent with an understanding of its environment,we use a voxel-based occupancy map to indicate the location and size of the scene geometries, similarto [4, 33].
Occupancy provides a basic level scene and object awareness to our agent and is robust to changing scene geometries.
[-; for t ← 1 ...T do s t ← ( q t , ˙ q t ) ϕ t ← (cid:80) Kk =
-]
Concretely, we create a cube of 2 meters in diameter as the sensingrange of our agent’s surroundings, and creates a 16 by 16 by 16 evenly spaced grid.
At the center ofeach point on the grid, we sample the signed distance function of the scene F s to acquire how far thesampling point is from the geometries of the scene.
Then, we threshold the gird feature to provide afiner scene information.
In time step t , for threshold σ = 0 .
05 , the value for point p
i in the grid canbe computed as:
[Equation]
As the scanned meshes provided by the PROX dataset often have incorrect volumes, we use anautomatic geometric fitting procedure to approximate the scene by simple shapes such as rectanglesand cylinders.
The simplified scene can be visualized together in Equ.2.
For more information aboutour scene simplification process, please refer to Appendix.
Scene-aware Kinematic Policy To train and control our embodied agent in the physics simulator,we extend a policy network similar to Kin_poly [18], with the important distinction that our kinematicpolicy is scene-aware and performs third-person pose estimation.
Under the guiding principle ofembodied human pose estimation, our kinematic policy π KIN computes residual 3D movement ˜˙ q tbased on proprioception q t , movement goal ϕ t , and scene awareness s t :
[Equation]
Specifically, the input q t contains the current body joint configurations, ϕ t provides the movementsignal per step, while s t provides awareness to the agent’s surroundings.
Within π KIN , an agentcentric transform T AC is used to transform q t , ϕ t and s t into the agent-centirc coordinate system.
The output ˜˙ q t , is defined as ˜˙ q t ≜ (˜˙ r R t , ˜˙ r T t , ˜˙ θ t ) , which computes the velocity in root translation ˜˙ r R t andorientation ˜˙ r T t , as well as the joint angles ˜˙ θ t .
Notice that ˜˙ q t computes the residual body movementinstead of the the full kinematic body pose ˜
q t +1 and establishes a better connection between themovement feature ϕ t .
During inference, π KIN works in tandem with π UHC and computes the nextstep target pose ˜
q t +1 for the humanoid controller to mimic.
In turn, π KIN receives timely feedbackabout physical realism, contact, and scene features based on physics simulation and agent’s state.
3.3 Learning the embodied human pose estimator
Since π KIN relies only on 2D keypoints, we can leverage the large-scale human motion data baseand train our pose estimation network using synthetic data.
We use the AMASS [19], H36M [8],and Kin_poly [18] datasets to dynamically generate 2D keypoints and 3D pose pairs for training.
At the beginning of each episode, we sample a motion sequence and randomly change its directionand starting position in a valid range, similar to the procedure defined in NeuralGD [32].
Then weuse a randomly selected camera projection function π cam to project 3D keypoints onto 2D imagesto obtain paired 2D keypoints and human motion sequences.
During training, we initialize theagent with the ground-truth 3D location and pose.
The H36M and Kin_poly dataset contains simplehuman-scene interactions, such as sitting on chairs and stepping on boxes.
To better learn objectaffordance and collision avoidance, we also randomly sample scenes from the PROX dataset and pairthem with motions from the AMASS dataset.
Although these motion sequences do not correspond tothe sampled scene, the embodied agent will hit the scene geometries while trying to follow the 2Dobservations.
We then stop the simulation when the agent deviates too far away from the ground-truthmotion to learn scene affordance and avoidance.
We train the kinematic policy as a per-step model together with physics simulation, following thedynamics-regulated training procedure defined in [18].
Combining our UHC and our kinematicpolicy, we can leverage the pre-learned motor skills from the UHC and let the kinematic policy act directly in a simulated physical space to obtain feedback on physical plausibility.
The procedure fordynamics-regulated training is described in Alg.
In each episode, we connect the UHC and thekinematic policy using the result of the UHC and the physics simulation q t +1 to compute the inputfeatures to the kinematic policy.
Using q t +1 verifies that (cid:101) q t can be followed by our controller andinforms our π KIN of the current humanoid state to encourage the policy to adjust its predictions toimprove stability.
Here we only train our π KIN using supervised learning, as we have third-personpose estimation is a more well-posed problem and requires less exploration than estimating pose fromonly egocentric videos [18].
For each motion sequence, our loss function is defined as L 1 distancebetween the predicted kinematic pose and the ground truth plus a prior loss using a leaned motiontransition prior p θ and the encoder q ϕ from HuMoR [26].
The prior term encourages the generatedmotion to be smoother:
[Equation]
Experiments
Implementation details.
We use the MuJoCo [35] physics simulator and use the simplified scenegeometries as environments.
The training process or our embodied pose estimator takes around 1 dayon a RTX 3090 with 30 CPU threads.
During inference, our network is causal and runs at 10 FPS onan Intel desktop CPU (not counting the time for 2D keypoint pose estimation, which runs at around30 FPS [2]).
For more implementation details and network architecture, please refer to Appendix.
Our Universal Humanoid Controller is trained on the training split of AMASS datasetwhich contains high-quality SMPL parameters of 11402 sequences curated from the various MoCapdatasets.
For training our pose estimator, we use motions from the AMASS training split, the Kin_poly dataset, and H36M dataset.
The Kin_poly dataset contains 266 motion sequences of typesitting, stepping, pushing, and avoiding, and H36M dataset consists of 210 indoor videos with 51different unique actions.
Notice that we only use the motion sequences from these dataset and not thepaired videos.
For evaluation, we use videos from the PROX [6] dataset and the test split of H36M.
Metrics and Baselines.
We compare against a series of state-of-the-art methods HyperIK
MeTRAbs [28], VIBE [11], and SimPOE [40] on the H36M dataset.
On the PROX dataset, wecompare against SOTA methods HuMoR [26], LEMO [41], and HyberIK with RootNet [21].
Weuse the official implementation for the above methods, all of which are under the MIT license.
We use pose-based and plausibility-based metrics for evaluation.
To evaluate pose estimation whenthere is ground truth MoCap pose, we report the absolute/global and root-relative mean per jointposition error:
A-MPJPE and MPJPE.
On the PROX dataset, since there is no ground truth 3Dpose annotation, we report: success rate SR (whether the optimization procedure converges andwhether the simulated agents loses balance), average chamfer distance from the point cloud (ACD,acceleration (ACC), and scene penetration frequency Freq. and distance Pen..
For a full definition ofour evaluation metrics, please refer to Appendix.
4.1 Results
[Table]
[Table]
Results on H36M
We report pose estimation onthe H36M dataset at Table1.
Here we can see thatour method achieves reasonable pose estimation resulton both root-relative andglobal pose metrics.
Noticethat our experiment on the
H36M dataset is meant tobe as a sanity check on our pose estimation framework and is not a fair comparison.
Our method onlyuses a sparse set of detected 2D keypoints (12 keypoints) as input and does not directly use any imagefeature.
We also evaluate on a different subset of 3D keypoints, as our use of forward kinematics FK β causes our joint definition to be different from prior works.
Results on PROX Table 2 shows our result on the PROX dataset where we compare against SOTAmulti-stage batch optimization based methods LEMO [41] and HuMoR [26].
We also report theresult using HyperIK with RootNet [14] to show that severe occlusion and human scene interactionon the Prox dataset is also challenging to SOTA regression methods.
From Table 2 we can seethat our method achieves best ACD (chamfer distance) without using any depth information, and iscompetitive on all other metrics against both RGB and RGB-D based methods.
Notice that LEMOand HuMoR both directly optimizes chamfer distance and ground penetration in their optimizationformulation, our method only relies 2D keypoints and laws of physics to achieve plausible poseestimates.
Since motion is best seen in videos, please refer to our supplementary video and appendixfor quantitative comparison.
[Table]
4.2 Ablation
In Table 3, we ablate the main components of our framework.
Row 1 (R1) corresponds to not usingany scene awareness; R2 uses MPG with only one time-step, which results in nosier gradients; R3trains a model without using the geometric translation step in G ; R4 does not use the TCN model
F TCN in G .
Comparing R3 and R4 with R5, we can see that the learned function G is essentialto obtain better root translation and orientation movement signal, without which the agent driftssignificantly. R2 also showcases the importance of taking the gradient step multiples times to obtain better gradients.
R1 demonstrates that the scene-awareness further improves our result, where ouragent learns important affordance information about chairs and sofas to improve sauces rate.
[Table]
Discussions
[Table]
Failure Cases While our method can estimate human pose and human-scene interaction from the PROX dataset with high success rate, there is still much room for improvement.
We simplify allthe humanoids and scene geometries as rigid bodies and do not model any soft materials such ashuman skin, cushions, or beds.
As a result, our sitting and lying motion can often result in jitterymotion, since contact occurs only in a very small surface area on the simulated body and hard surfaces.
Prolonged occlusion can also cause our agent to lose track of the 2D keypoints and result in erroneoustranslation estimation. Therefore, our agent can get stuck between chairs and tables.
Please refer toour supplementary and Appendix for visualization of failure cases.
Conclusion and Future Work We introduce embodied scene-aware human pose estimation, wherewe formulate pose estimation in a known environment by controlling a simulated agent to follow2D visual observations.
Based on the 2D observation, its current body pose, and environmentalfeatures, our embodied agent acts in a simulated environment to estimate pose and interact withthe environment.
A multistep projection gradient (MPG) is proposed to provide global movementcues to the agent by comparing the current body pose and target 2D observations.
The experimentalresult shows that our method, trained only on synthetic dataset derived from popular motion datasetbases, can estimate accurate global 3D body pose on the challenging PROX dataset while using only
RGB input.
Future work includes providing a movement prior for the embodied agent to intelligentlysample body motion based on scene features to better deal with occlusion. Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec.5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
In the
supplementary material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of
all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data
and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code included
in supplemental material
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Included in supplemental material
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]