S TOCHASTIC M ULTI -P ERSON 3D M OTION F ORECASTING
A BSTRACT
This paper aims to deal with the ignored real-world complexity in prior work on human motion forecasting, emphasizing the social properties of multi-person motion, the diversity of motion and social interactions, and the complexity of articulated motion.
To this end, we introduce a novel task of stochastic multi-person 3D motion forecasting.
We propose a dual-level generative modeling framework that separately models independent individual motion at the local level and social interactions at the global level.
Notably, this dual-level modeling mechanism can be achieved within a shared generative model, through introducing learnable latent codes that represent intents of future motion and switching the codes’ modes of operation at different levels.
Our framework is general, and we instantiate it with various multi-person forecasting models.
Extensive experiments on CMU-Mocap,
MuPoTS-3D, and SoMoF benchmarks show that our approach produces diverse and accurate multi-person predictions, significantly outperforming the state of the art.
Code is available at https://github.com/Sirui-Xu/DuMMF .
I NTRODUCTION
One of the hallmarks of human intelligence is the ability to predict the evolution of the physical world over time given historical information.
For example, humans naturally anticipate the flow of people in public areas, react, and plan their own behavior based on social rules, such as avoiding collisions.
Effective forecasting of human motion has thus become a crucial task in computer vision and robotics, such as in autonomous driving (Paden et al., 2016) and robot navigation (Rudenko et al., 2018).
This task, however, is challenging: first, human motion are structured with respect to both body physics and social norms, and highly dependent on the surrounding environment and its changes; second, human motion are inherently uncertain and multimodal, especially in the long-term horizon.
Previous work on human motion forecasting often focuses on simplified scenarios.
Perhaps the most widely adopted setting is on stochastic local motion prediction of a single person (Mao et al., 2021;
Yuan & Kitani, 2020), which ignores human interactions with the environment and other people in the environment.
Another related task is deterministic multi-person motion forecasting (Wang et al., 2021b; Adeli et al., 2020; 2021;
Guo et al., 2022).
However, it does not take into account the diversity of individual movements and social interactions.
In addition, stochastic forecasting of human trajectories in crowds (Alahi et al., 2014) has shown progress in modeling social interactions, e.g. , with the use of attention models (Kosaraju et al., 2019;
Vemula et al., 2018; Zhang et al., 2019) and spatial-temporal graph models (Huang et al., 2019; Ivanovic & Pavone, 2019; Salzmann et al., 2020; Yu et al., 2020).
Nevertheless, this task only considers motion and interactions at the trajectory level.
Modeling articulated 3D poses involves richer human-like social interactions than trajectory forecasting which only needs to account for trajectory collisions.
To overcome these limitations, we introduce a novel task of stochastic multi-person 3D motion forecasting , aiming to jointly tackle the aforementioned aspects ignored in the previous work – the social properties of multi-person motion, the multi-modality of motion and social interactions, and the complexity of articulated motion.
As shown in Figure 1, this task requires the predicted multiple
[Figure]
motion sequences of multi-person to satisfy the following conditions – (a) single-person fidelity: for example, all single-person motion should be continuous, and the articulated properties should be preserved under the physical rules; (b) multi-person fidelity: the predicted motion should be socially-aware, with the consideration of the interactions between predictions from different people; and (c) overall diversity: the movement of the human body should be as varied as possible, but within the constraints of conditions (a) and (b) above.
Due to the substantially increased complexity of our task, it becomes challenging to optimize all three objectives simultaneously.
We observe simply extending existing work such as on deterministic motion forecasting cannot address the proposed task.
This difficulty motivates us to adopt a divideand-conquer strategy, together with the observation that single-person fidelity and multi-person fidelity can be viewed as relatively independent goals, while there is an inherent trade-off between fidelity and diversity.
Therefore, we propose a Du al-level generative modeling framework for
M ulti-person M otion F orecasting (DuMMF).
At the local level, we model motion for different people independently under relaxed conditions, thus satisfying single-person fidelity and diversity.
Meanwhile, at the global level, we model social interactions by considering the correlation between all motion, thereby further improving multi-person fidelity.
Notably, this dual-level modeling mechanism can be achieved within a shared generative model, through simply switching the modes of operation of the motion intent codes ( i.e. , latent codes of the generative model) at different levels.
By optimizing these codes with level-specific objectives, we produce diverse and realistic multi-person predictions.
Our contributions can be summarized as follows.
(a) To the best of our knowledge, we are the first to tackle the task of stochastic multi-person 3D motion forecasting.
(b) We propose a simple yet effective dual-level learning framework to address this task.
(c) We formulate discrete and learnable social intents in dual levels to improve the realism and diversity of predictions.
(d) Our framework is general and can be operationalized with different types of multi-person motion forecasting models.
Notably, it can be generalized to more-person scenarios that are unseen in training data (Sec. G).
R ELATED W ORK
Stochastic Human Motion Forecasting.
There have been many advances in stochastic human motion forecasting, many of which (Walker et al., 2017; Yan et al., 2018; Barsoum et al., 2018;
Kundu et al., 2019) are based on the improvement and adaptation of deep generative models such as variational autoencoders (VAEs) (Kingma & Welling, 2013), generative adversarial networks (GANs) (Goodfellow et al., 2014), and normalizing flows (NFs)
(Rezende & Mohamed, 2015).
Some recent approaches (Bhattacharyya et al., 2018; Dilokthanakul et al., 2016; Gurumurthy et al., 2017; Yuan & Kitani, 2019; 2020; Zhang et al., 2021; Mao et al., 2021; Xu et al., 2022b; Petrovich et al., 2022) have put more emphasis on the promotion of diversity.
Yuan & Kitani (2020) introduce remapping in the latent space and ensure that the samples in this space are sufficiently diverse.
Mao et al.
(2021) propose a framework that sequentially generates the different parts of a pose for better controllability of diversity.
Although these methods can generate very diverse human motions, most of them are limited to local motion and ignore the global trajectory.
Some of the motions they produce are actually unreal; for example, there may be severe foot skating when global trajectories are considered.
Predicting human motion under the constraint of scene context (Cao et al., 2020;
Hassan et al., 2021; Zhang & Tang, 2022) has recently been explored, where the effect of global trajectories and scenes on human motion is considered.
Instead of predicting single-person movement, our work focuses on diverse multi-person movements and social interactions.
Multi-Person Forecasting.
So far, research on multi-person forecasting has mainly focused on global trajectory forecasting (Helbing & Molnar, 1995; Mehran et al., 2009; Pellegrini et al., 2009;
Yamaguchi et al., 2011; Zhou et al., 2012; Alahi et al., 2014; 2016; Lee et al., 2017; Gupta et al., 2018; Sadeghian et al., 2019; Amirian et al., 2019; Kosaraju et al., 2019; Sun et al., 2020;
Mangalam et al., 2020; Sun et al., 2021; Kothari et al., 2021; Tsao et al., 2022).
Gupta et al.
(2018) utilize a winner-takes-all loss (Rupprecht et al., 2017) with a socially-aware GAN.
We also adapt these two loss functions into our dual-level modeling.
Kothari et al.
(2021) introduce discrete social anchors and redefine the distribution of trajectories by the predefined anchors.
Unlike their hand-crafted anchors, our discrete social intent codes are learnable components obtained through dual-level optimization.
Some recent attempts address deterministic multi-person 3D motion forecasting.
Adeli et al.
(2020; 2021) use additional contextual information to aid multi-person forecasting and propose a Social
Motion Forecasting (SoMoF) Benchmark.
Guo et al.
(2022) utilize cross-attention for two-person motion forecasting, while Wang et al.
(2021b) introduce a multi-range transformer architecture that can be generalized to more persons.
Please refer to Sec.
B of the Appendix for additional discussion.
M ETHODOLOGY
In this section, we explain the proposed dual-level generative modeling framework (DuMMF) for our task.
As illustrated in Figure 2, our key insight is to decouple the modeling of independent individual movements at the local level and social interactions at the global level (Sec. 3.1).
Within a shared forecasting model, we achieve this by (a) introducing learnable latent codes that represent intents of future movement (Sec. 3.2), (b) switching the codes’ modes of operation at different levels (Sec. 3.
and Sec. 3.2), and (c) training with level-specific objectives (Sec. 3.3).
Our framework is general and can be operationalized with different types of motion forecasting models, and we summarize the multi-person motion predictors used in this paper (Sec. 3.4).
Problem Formulation.
We denote the input motion sequence of length T h for N persons in a scene as { X n } Nn =1 , where X i [ t ] is the pose of i -th person at time step t .
We aim to predict M future motion sequences of length T p , denoted as {{ (cid:98) Y mn } Nn =1 } Mm =1 ,
where (cid:98) Y ji
=
[ (cid:98) Y ji [ T h + 1] , . .
, (cid:98) Y ji [ T h + T p ]] is the j -th predicted motion of i -th person.
We use the 3D coordinates to represent the absolute joint position of V joints, hence ∀ i, j, t, X i [ t ] , (cid:98) Y ji [ t ] ∈ R V × 3 .
We assume to be given the ground truth motion of N persons as { Y n } Nn =1 .
Our goal is to forecast multiple realistic yet diverse future motions, such that (a) all the M predictions represent human-like motion sequences, simultaneously satisfying single-person fidelity and multi-person fidelity ; (b) the predictions are diverse ( overall diversity ); and (c) one of the predicted sequences is as close to the ground truth as possible.
3. D UAL -L EVEL S TOCHASTIC M ULTI -P ERSON M OTION F ORECASTING : O VERVIEW
Basic Generative Modeling Framework.
Stochastic multi-person future motion can be modeled as a joint distribution using deep generative models (Goodfellow et al., 2014).
Accordingly, we denote this joint distribution of the future motion of N persons as p ( Y 1 , Y 2 , . . .
, Y N | X 1 , X 2 , . . .
, X N ) , where all future movements { Y n } N n =1 are conditioned on the past sequences { X n } N n =1 of all per-
[Figure]
T h T h − T h T h −
[Figure]
sons.
Typically, we can use a latent code z ∼ p ( z ) to reparameterize this joint distribution as p ( { Y n } Nn =1 |{ X n } Nn =1 ) =
(cid:82) p ( { Y n } Nn =1 | z , { X n } Nn =1 ) p ( z )d z .
Here, the latent code z can be interpreted as a social intent that guides the future movements of multiple persons.
With this social intent z sampled from the given distribution p ( z ) , a deterministic neural network G θ can map the historical conditions { X n } Nn =1 to a prediction { (cid:98) Y n } Nn =1 under the latent distribution p θ ( { Y n } Nn =1 | z , { X n } Nn =1 ) , which is formulated as
[Equation]
Given the complexity of our task, it becomes challenging to simultaneously ensure all objectives ( i.e. , the fidelity of a single person, the fidelity of multiple persons, and the overall diversity).
To overcome this difficulty, we introduce a dual-level modeling mechanism that explicitly decomposes the task objectives into local modeling of independent individual movements and global modeling of social interactions.
Notably, we achieve this by simply switching the modes of operation for the latent codes z w.r.t.
different levels of modeling, without any change to the model architecture G .
Local-Level Modeling: Individual Motion.
At this level, the generative model G θ models all human bodies as independent of each other, and we aim to improve the overall diversity and the single-person fidelity , alleviating problems such as predicting static motion and foot skating.
Here, the joint distribution of future human motions can be rewritten in the form of all single-person marginal distributions, i.e. , p ( { Y n } Nn [Equation] |{ X n
[Equation]
Nn [Equation] [Equation]
Nn =1 p ( Y n | X n ) .
To this end, as shown in Figure 2(a), we consider leveraging N different individual intents z 1 , z 2 , . . .
, z N independently drawn from p ( z ) to generate independent future movements, denoted as
[Equation]
Global-Level Modeling: Social Interactions.
Going beyond the local individual level, the generative model G θ at the global level takes into account the social behavior of multiple people to model their joint distribution.
The goal is to further improve the multi-person fidelity , mitigating issues such as human collisions.
As illustrated in Figure 2(b), to maintain the network architecture G unchanged, we still use N individual intents as input.
However, different from the local level, we constrain these
N individual intents to be the same, representing social intents that stand for correlations between the intents of multiple persons.
Formally, we have
[Equation]
Note that, without additional constraints, this dual-level modeling scheme by itself is not guaranteed to enforce the latent codes to behave in the designed manner.
To this end, we introduce learnable latent intent codes
z (Sec. 3.2), jointly optimize the codes z and the forecasting model G guided by the level-specific training objectives (Sec. 3.3).
3. D ISCRETE L EARNABLE H UMAN I NTENT C ODES
Intuitively, an arbitrary, albeit identical, individual intent in Eq.
3 may not adequately lead to a valid social intent.
We thus hypothesize that a social intent is formed when all individual intents are the same and belong within some range of “options.”
This can typically be achieved through discrete choice models (DCM) (Aguirregabiria & Mira, 2010; Ryan & Gerard, 2003;
Bhat et al., 2008; Leonardi, 1984) – an effective tool that predicts choices from a set of available options created by hand-crafted rules.
However, the intent options for social interactions are more subtle and difficult to define manually than the intent in other applications such as for trajectories (Kothari et al., 2021).
Therefore, we use a set of learnable codes z d ∈ { Z i } Mi =1 to represent social intents, similar to Xu et al.
(2022b), while having different training strategies specific to this new task, as we will introduce in Sec.
Our motivations are: (a) Subject to physical constraints and social laws, the intents of future movements should share some deterministic patterns.
For example, all intents should avoid imminent collisions anticipated from the history, even if these intents refer to different motions.
We assume that such deterministic properties shared by social intents can be represented by a set of shareable codes learned directly from the data.
(b) It will be easier for the predictor to identify and implement different levels of functionality by jointly optimizing discrete intents and the predictor.
To further enhance the expressiveness of the codes, we retain the original continuous Gaussian noise z c ∼ p ( z ) of the generative model G θ and bundle the discrete intent with the noise to represent the final intent, as shown in Figure 2(c).
Now the global-level modeling of social interactions in Eq.
3 is reformulated as
[Equation]
And correspondingly, the local-level modeling of individual motion in Eq.
2 becomes
[Equation]
3. T RAINING U NDER THE G UIDANCE OF L EVEL -S PECIFIC O BJECTIVES
We jointly train both levels of individual movement modeling and social interaction modeling, but with each level guided by its own objective.
In each forward pass, we explicitly produce different output predictions from different intents of the two levels.
And then, in the backward pass, the discrete intent codes z d are optimized separately at different levels, while the parameters θ of the forecasting model G are updated based on the fused losses from the two levels.
Local-Level Training.
At the local level, we train the model without considering the social interactions.
Given the multi-person motion data ( { X n } Nn =1 , { Y n } N n =1 ) , we first randomly sample the discrete intent codes and merge them with independently sampled random intent codes into M × N different latent codes {{ z mn } Nn =1 } Mm =1 .
We then use each intent and the historical motion of each person to recover M future motion sequences {{ (cid:98) Y mn } Nn =1 } Mm =1 .
Local-level objectives fall into two categories, considering single-person fidelity and overall diversity respectively.
We define the following objectives for constraining single-person fidelity: (a) local reconstruction loss L lR ; (b) limb loss L L ; (c) local GAN loss L lGAN .
To promote overall diversity, we leverage the following objectives: (a) multi-modal reconstruction loss L mmR ; (b) diversity-promoting loss L D .
Global-Level Training.
We train the model to facilitate the modeling of social interactions at the global level.
The difference from the local setting is that now we incorporate the discrete and continuous codes into only M distinct latent codes {{ z m } Nn =1 } Mm =1 , which means that the latent codes z m of all N individuals are the same for a certain m -th prediction.
Correspondingly, we introduce the following objectives to facilitate multi-person fidelity: (a) global reconstruction loss
L gR ; (b) global GAN loss L gGAN .
Please refer to the Appendix (Sec. C) for more detail on the above level-specific learning objectives.
We demonstrate that these learning objectives are important, and in some cases critical, to the fidelity and diversity of multi-person motion (see Sec.
G of the Appendix).
[Table]
Inference.
We only use the global level where we sample the same intent for all people in the scene during inference.
Note that the uncertainty of human motion substantially increases with the time horizons of the forecasting.
So instead of predicting a fixed number of diverse outputs (Mao et al., 2021; Yuan & Kitani, 2020), the number here should grow with the length of the prediction.
To this end, we employ autoregressive inference (Wang et al., 2021b) and propose progressive diversity .
Given the historical motion sequence { X n } Nn =1 and M sampled intent codes, the model outputs
M predictions {{ (cid:98) Y mn } Nn =1 } Mm =1 .
And then, given a combination of history, the previous predictions {{ X n , Y mn } Nn =1 } Mm =1 , and M new intent codes as input, the model outputs M 2 predictions {{ (cid:98) Y mn } Nn =1 } M 2 m =1 , etc.
(see Figure 1(c)).
3. N ETWORK A RCHITECTURE
Our dual-level modeling framework in conjunction with the discrete learnable intent codes is general and, in principle, does not rely on specific network architectures.
To demonstrate this, we combine our framework with various types of deterministic multi-person motion predictors, yielding significant improvements across all predictors (see Sec. 4).
As shown in Figure 2, we abstract the encoder of the multi-person motion predictor into two parts: the local part is responsible for encoding single-person motion, while the global part is responsible for encoding multi-person motion and its interactions.
A summary of multi-person predictors used in the paper is given in Table A of the Appendix.
And we further detail how we adapt a state-of-the-art deterministic multi-person predictor (Wang et al., 2021b) to our framework in Sec.
D of the Appendix.
E XPERIMENTS
Datasets.
In the main paper, we show the evaluation on two motion capture datasets, CMUMocap (CMU) and MuPoTS-3D (Mehta et al., 2018).
CMU-Mocap consists of movement sequences with up to two subjects for each scene.
It contains 2235 recordings performed by 144 different subjects, eight of which include double-person motion.
We directly adopt these two-person motions for comparisons in two-person scenarios.
For 3-person scenarios, we follow Wang et al.
(2021b)
for the train/test split and the preprocess to mix single-person and double-person motion together to synthesize a 3-person motion in each scene.
Moreover, we construct an even more challenging scenario where we mix up to 9 people per scene.
Please refer to Sec.
H of the Appendix for the detail of this generalized setting.
We use 15-joint skeleton coordinates with global translation as the representation of the data.
Unlike Wang et al.
(2021b), we convert all length units to meters as suggested in CMU-Mocap dataset.
MuPoTS-3D consists of more than 8,000 frames with up to three subjects.
We convert the data to the same 15-joint human skeleton and length units as CMU-Mocap and evaluate the generalization on MuPoTS-3D of a model trained only on CMU-Mocap.
We also report our performance on the SoMoF benchmark (Adeli et al., 2020; 2021) in Sec.
G of the Appendix.
Metrics.
For evaluating multi-person motion accuracy and diversity, we adopt the common metrics used in stochastic forecasting (Mao et al., 2021; Yuan & Kitani, 2020; 2019; Salzmann et al., 2020;
Kothari et al., 2021) as follows.
For accuracy measurement, we follow the Best-of-N (BoN) evaluation and use (a) Average Displacement Error (ADE) : the average ℓ 2 distance over time between the
[Table]
[Figure]
ground truth and the prediction closest to the ground truth; (b) Final Displacement Error (FDE) : the ℓ 2 distance between the final pose of the ground truth and the last predicted pose closest to the ground truth.
For diversity measurement, we employ (c) Final Pairwise Distance (FPD) : the average ℓdistance between all predicted final pose pairs.
We disentangle the local pose and the global trajectory of the motion and measure their accuracy and diversity separately by defining the following metrics: rootADE , rootFDE , poseADE , poseFDE , rootFPD , and poseFPD .
To measure three different aspects including single-person fidelity, multi-person fidelity, and overall diversity comprehensively, we provide a summary and analysis of all metrics for this novel task in Table F and Sec.
F of the
Appendix.
We include results on all above metrics in Sec.
G of the Appendix.
Baselines.
Given that we propose a new task, there is no readily available baseline from existing work.
For comparison purposes, we customize several baselines from deterministic multi-person motion forecasting, which we summarized in Table.
A).
And we extend them to stochastic forecasting.
(a)
RNN+SC-MPF (Adeli et al., 2020):
We use Social Pooling (Alahi et al., 2016) to integrate the multiperson information.
We follow the implementation in their paper and reproduce the results.
For the
Social Pooling module, we select the maximum pooling for its best performance as ablated in Adeli et al.
(b) Transformer+SC-MPF : For a fair comparison, we introduce a transformer-based encoder architecture to combine with the Social Pooling module.
(c) XIA (Guo et al., 2022):
We follow their implementation to use Cross Interaction Attention (XIA) to encode 2-person information.
(d) MRT (Wang et al., 2021b):
As mentioned in Sec.
3.4, we use this as our main architecture for its state-of-the-art performance.
We further extend MRT to 2-person scenarios to compare with XIA .
For a fair comparison, we use their respective encoders, but we use the same transformer-based decoder for the three baselines (b), (c), and (d).
For stochastic forecasting baselines, we formulate conditional generative adversarial networks (CGAN) (Mirza & Osindero, 2014) with noise input and all learning objectives that we introduced at the local and global levels, including two GAN losses L gGAN and
L lGAN , without dual-level modeling and discrete intents.
Implementation Details.
We train the model to predict a 15-frame sequence of 3 people given the real 15, 30, and 45 historical frames at 15Hz, since our encoder (RNN/Transformer) accepts different input lengths.
The local and global encoders and the decoder all consist of 3-layer transformers (or RNNs), where we set the dimension of feature to 128.
The model is trained with a batch size of 32 .
As illustrated in Sec. 3.3, the two levels of modeling result in different outputs and have their own learning objectives.
For individual modeling, the weight of each loss term ( λ lR , λ L , λ lGAN , λ mmR , λ D ) is (0 . 01 , 0 . 0005 , 0 . 0005 , 0 . 0001 , 0 . 0001) .
For social modeling, the weight for each loss term ( λ gR , λ gGAN ) is (1 , 0 . 0001) .
For evaluation, we use 15 real frames as historical motion and recursively predict the next 15 frames 3 times, as illustrated in Sec. 3.3.
Thus, given the number of intents to be M , the model outputs M, M 2 , M 3 different predictions in sequence.
Additional implementation details are provided in Sec.
E of the Appendix.
Quantitative Results.
We compare our method with MRT (Wang et al., 2021b) in Table 1.
We show that the improvement is significant by performing each experiment five times with different random seeds and reporting error bars.
When the number of intents is one, equivalent to a deterministic setting, our method marginally outperforms MRT for all metrics on CMU-Mocap and also generalizes better on MuPoTS-3D, suggesting the generality of our approach, which is also helpful for deterministic processes.
As the number of intents increases, our method consistently provide more accurate and diverse results, especially in long-term predictions.
In Table 2, we demonstrate that our DuMMF framework benefits all predictor variants.
We observe that the simple combination of deterministic predictors and CGAN results in very low diversity and accuracy.
By contrast, our full approach significantly outperforms CGAN baselines on both diversity and accuracy across all backbones.
In
Table D of the Appendix, we further show that DuMMF achieves the best generalization results across all predictors on MuPoTS-3D, suggesting the generality and superiority of our approach.
Ablation: Effectiveness of Dual-Level Modeling.
Tables 2 and 3 show the effectiveness of our dual-level framework.
First, we investigate the settings with only single-person motion modeling or only social interaction modeling.
In Table 3, compared with our full method, modeling independent multi-person motion (‘w/o Social’) provides higher diversity but leads to inaccurate poses, since
[Table] [Table]
w/o Discrete
[Table] [Table]
[Table] [Table]
[Table] [Table]
w/
[Table]
Continuous
[Table] [Table]
the social restrictions are not considered.
With only social interaction modeling, the model (‘w/o
Individual’) cannot output sufficiently diverse predictions, which also makes predictions inaccurate, as more varied outputs have a better chance of covering the ground truth.
Note that the results of ‘CGAN’ and ‘w/o Separation’ in Table 2 and ‘w/o Separation’ in Table 3 are worse since they simply use all the learning objectives together without separating two-level modeling, while ‘w/o Separation’ is slightly better due to the use of discrete intents.
Under our dual-level framework with level-specific motion intents and learning objectives, the model can better integrate the advantages of the two levels, thus leading to improved accuracy and diversity.
Ablation: Effectiveness of Discrete Human Intent.
In Tables 2 and 3, we would like to show that discrete human intents are effective and crucial.
We observe the best results when using both discrete and continuous intents, indicating that they are complementary.
In the absence of discrete intents (‘w/o Discrete’), the performance is only comparable with the baseline (‘CGAN’).
And the improvements of dual-level modeling is pronounced in the case with discrete intent (‘Full’ vs.
‘w/o Separation’).
This further suggests that discrete learnable intents are critical to ensure that the advantages of both levels are integrated during training.
The performance without continuous intent (‘w/o Continuous’) is slightly worse than the full method.
Still, discrete intents only support a finite number of outputs, which is limited and not what we want.
In Sec.
G of the Appendix, we further analyze the effect of the number of discrete intents to the stochastic forecasting.
Qualitative Results.
Consistent with the quantitative evaluation above, we observe that our method provides diverse multi-person motion, and produces predictions closer to the ground truth compared with the deterministic method MRT in Figure 3.
In Table 3, ‘w/o Social’ results in extremely high diversity, especially in trajectories.
In Figure 4, we qualitatively show that our generated results reflects real-world diversity of social interactions.
Furthermore, we provide qualitative results for 6 - and 9 -person scenarios in Figure A and Figure B of the Appendix.
Please refer to Sec.
H of the
Appendix for more detail on the more-person setting.
Limitation.
Our dual-level framework with level-specific losses enables the model to produce highquality and diverse predictions, but we still observe artifacts such as foot skating in some predicted motion.
This is because we are not intrinsically modeling articulated motion, but only constraining it by some loss functions.
This is a common problem with learning-based methods.
Therefore, we consider exploiting a physical simulator to further improve the plausibility of the predicted motion.
C ONCLUSION
We propose a new task, namely, stochastic multi-person 3D motion forecasting.
To simultaneously achieve single-person fidelity, social realism, and overall diversity, we introduce a dual-level generative modeling framework (DuMMF) with learnable latent intent codes.
Compared with prior work on deterministic or single-person prediction, our model learns to generate diverse and realistic human motions and interactions.
Ethics Statement.
Our proposed technique is useful in many applications, such as self-driving to avoid crowds.
The potential negative societal impact includes: (a) our approach can be used to synthesize highly realistic human motion, which might lead to the spread of false information; (b) our approach requires real behavioral information as input, and we are concerned that this may expose the identity information and violate people’s privacy.
However, on the positive side, our model operates on the processed human skeleton representation, which contains much less identification information compared to the raw data.
Acknowledgement.
This work was supported in part by NSF Grant 2106825, NIFA award 2020-67021-32799, the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for
Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign through the NCSA Fellows program, the IBM-Illinois Discovery Accelerator Institute, the Illinois-Insper Partnership and the
Amazon Research Award.
This work used NVIDIA GPUs at NCSA Delta through allocation CIS220014 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
A V ISUALIZATION V IDEO
In addition to Figure 3 and Figure 4 in the main paper and more qualitative results in this Appendix (Figure A and B), we provide a video to demonstrate more comprehensive visualizations of multi-person 3D motion forecasting at https://sirui-xu.github.io/DuMMF/images/demo.mp4 .
In this video, we illustrate that our method DuMMF generates diverse multi-person motion and social interactions, as well as taking into ac- count both single-person and multi-person fidelity.
We also show that our model is scalable and provide effective predictions in more challenging scenarios with a significantly increased number of people and associated more complex interactions.
We also highlight the impact and effectiveness of our dual-level modeling framework.
B A DDITIONAL D ISCUSSION ON R ELATED W ORK
As we demonstrate in the main paper, our proposed stochastic multi-person motion forecasting needs to simul- taneously take into account single-person pose fidelity, consistency of pose and trajectory, social interactions between poses, and overall diversity of motion, while prior work including stochastic multi-person trajectory forecasting (Alahi et al., 2014;
Kothari et al., 2020; 2021) and deterministic multi-person motion forecasting (Adeli et al., 2020; 2021;
Guo et al., 2022; Wang et al., 2021b) focuses on simplified scenarios.
Our problem reveals real-world complexity with substantially increased, multi-faced challenges which have not been, and cannot be, jointly tackled by any of the prior literature.
In particular, in our stochastic scenario, the fidelity and interactions need to be both satisfied and diversified for all the predictions, which is very challenging and cannot be addressed by a simple extension of existing work on stochastic multi-person trajectory forecasting and deterministic multi-person motion forecasting as shown in Sec.
G.
For deterministic multi-person motion forecasting , in Sec. 2 of the main paper, we have highlighted our key difference with (Adeli et al., 2020; 2021;
Guo et al., 2022), and here we discuss in more detail.
(Adeli et al., 2020) introduces a social pooling module to integrate social information.
Adeli et al.
(2021) proposes a graph attentional network to jointly model human-human and human-object interactions.
Both methods utilize additional contextual information to aid deterministic prediction with their proposed Social Motion Forecasting (SoMoF) benchmark (Adeli et al., 2020; 2021).
Guo et al.
(2022) proposes a cross-interaction attention mechanism to predict cross dependencies between two pose sequences, making it applicable only to 2-person scenarios.
There are many methods achieving strong performance with simple and non-transformer-based frameworks on the SoMoF leaderboard.
Specifically, futuremotion_ICCV21 (Wang et al., 2021a) develops a simple but effective framework based on a combination of graph convolutional networks and multiple motion optimization techniques.
DViTA (Parsaeifard et al., 2021) decouples the human pose into a trajectory and local pose and employs a VAE to learn a representation of the local pose dynamics.
LSTMV_LAST (Saadat et al.) proposes a sequence-to-sequence LSTM via keypoint-based representation.
Complementing Sec.
2 of the main paper, we here discuss in more depth the trajectory-level multi-person forecasting .
Trajectron++ (Salzmann et al., 2020) constructs a spatio-temporal scene graph where nodes represent individuals and edges represent their interactions, and global and local modeling is embedded in this graph structure.
EvolveGraph (Li et al., 2020) introduces an effective dynamic mechanism to evolve interaction graphs, which can flexibly model dynamically changing interactions.
Tra2tra (Xu et al., 2021) introduces a global social spatio-temporal attention neural network to encode both spatial interactions and temporal features.
GroupNet (Xu et al., 2022a) employs a multi-scale hypergraph neural network that models group-based interactions and facilitates more comprehensive relational reasoning.
T-GNN (Xu et al., 2022c) introduces a transferable graph neural network that allows not only trajectory prediction but also domain alignment of potential distribution differences.
MID (Gu et al., 2022) employs a diffusion model to model the variation of indeterminacy for trajectory prediction.
We follow to abstract the encoder into local and global part as Figure 2(d).
As we use the same decoder for baselines (Sec. 4 of the main paper), we only discuss encoders of three predictors in this paper.
Note that XIA (Guo et al., 2022) can only be applied to two-person scenarios.
[Table]
C A DDITIONAL D ETAILS OF L EVEL -S PECIFIC O BJECTIVES
We use ∆ to represent the residual of the motion sequence.
For example, ∆ (cid:98) Y ji
=
[ (cid:98) Y ji [ T h +1]
− X
i
[ T h ] , (cid:98) Y ji [ T h + 2] −
(cid:98) Y ji [ T h + 1] , . .
, (cid:98) Y ji [ T h + T p ] − (cid:98)
Y ji [ T h + T p − 1]] .
Local-Level Objectives.
We adopt the multiple output loss (Guzmán-rivera et al., 2012) and extend it to the local reconstruction loss of multiple people L lR , which is used to optimize the most accurate prediction of each person while maintaining diversity.
We highlight the structure of the human skeleton by introducing the limb loss (Mao et al., 2021)
L L .
Specifically,
[Table]
N N
[Table]
n
[Table]
[Equation]
where the vector L n represents the ground truth distance between all pairs of joints that are physically connected in the n -th human body and (cid:98) L mn includes the limb length for all the T p poses in (cid:98) Y mn .
A GAN loss (Kocabas et al., 2020) is leveraged to train the model and the local discriminator D l for individual body realism.
Suppose { Y ∗ n } Nn =1 is the set of real motion clips sampled from the data, we have the following.
[Equation]
We further develop a multimodal reconstruction loss L mmR to provide additional supervision for all outputs {{ ∆ (cid:98) Y mn } Nn =1 } Mm =1 .
We first construct pseudo future motion { (cid:101) Y pi } Pp =1 for each historical sequence X i .
Different from (Yuan & Kitani, 2020; Mao et al., 2021), we additionally consider translation T ∈ R 3 and rotation R ∈ R 3 × 3 of the pose.
Specifically, given a threshold ϵ , we cluster future motion with a similar start pose and train the model with their residuals as
[Equation]
To explicitly encourage diversity, we adopt a diversity-promoting loss (Yuan & Kitani, 2020), which directly promotes the pairwise distance between the predictions of a single person.
We decompose this loss into two parts, promoting the diversity of local pose and the global root separately.
Supposing that (cid:98) Y mn ( l ) and (cid:98) Y mn ( g ) are the local pose and the global root joint extracted from the global pose (cid:98) Y mn , respectively, and α and β are two hyperparameters, this diversity-promoting loss is denoted as
[Equation]
Global-Level Objectives.
Since in this setting we treat N individuals as a whole, the reconstruction loss is reformulated as
[Equation]
[Table] [Table]
The global GAN loss is further leveraged to promote the realism of social motion, where the global discriminator D g takes the motion of all N people as input.
Suppose { Y ∗∗ n } Nn =1 is the multi-person motion clip sampled from the data, and we have
[Equation]
[Table]
D A DDITIONAL D ETAILS OF N ETWORK A RCHITECTURE
Below we detail the main multi-person predictor MRT (Wang et al., 2021b)
we use and how it integrates with our DuMMF framework for the proposed task.
Encoder.
The encoder consists of a local-range transformer encoder and a global-range transformer encoder.
The local encoder takes historical motion residuals ∆ X i of a single person as input and extracts a feature for each person.
The global encoder takes historical motion { X n } Nn =1 of all persons as input and runs only once to obtain global features.
Decoder.
To generate the future motion of the i -th person, we concatenate the local features of the i -th person with the global features as the final encodings, which are used as keys and values for the transformerbased decoder.
The decoder takes the last pose of the historical motion
X i [ T h ] and the sampled intent codes { z cn + z dn } Nn =1 as the query.
Note that Wang et al.
(2021b) only uses the encoder-decoder attention.
On the contrary, we introduce the decoder-decoder attention (Vaswani et al., 2017) to further incorporate motion and intent.
Local Discriminator.
To ensure the fidelity of the motion, especially to address the problem of foot skating , where the feet appear to slide in the ground plane, we concatenate the predicted motion of each person Y ji with the feet velocities ∆ F ji as input to a local discriminator D l .
The local discriminator uses an architecture similar to the local encoder, with additional fully connected layers to output a fidelity score for each motion.
Global Discriminator.
To ensure the realism of social interactions and avoid motion collisions, we propose a global discriminator D g that encodes all motion of N people { Y jn } Nn =1 at the same time and outputs a fidelity score.
The global discriminator uses an architecture similar to the global encoder, with additional fully connected layers.
Incorporate of DuMMF and the Transformer.
Our dual-level modeling framework operates primarily in the decoding stage.
Specifically, we get the key K and the value V of the transformer-based (Vaswani et al., 2017) decoder, from the same concatenated encodings H obtained by the local and global encoders.
The query Q is from the end pose (at time step T h )
X i [ T h ] of i -th person’s historical sequence X i .
Based on the dual-level modeling mechanism discussed in Sec.
3.1 of the main paper, we use the i -th intent code z di from the discrete set { Z m } Mm =1 and the continuous intent code z ci to generate the i -th prediction, leading to a total of M predictions.
The discrete and continuous intent codes are combined to get the key, value, and query as follows:
[Equation]
where W z is a linear layer.
We then apply linear layers W K , W V , W Q as follows:
[Equation]
Finally, L stacked standard transformer decoder layers are applied and we get the predicted motion of the i -th person.
We also follow Wang et al.
(2021b) to adopt Discrete Cosine Transform (DCT) to the inputs and Inverse
Discrete Cosine Transform (IDCT) to the outputs.
Please refer to Wang et al.
(2021b)
for more details.
E A DDITIONAL I MPLEMENTATION D ETAILS
Here, we provide more details on the implementation of our method DuMMF.
The two hyperparameters ( α, β ) in diversity promoting loss (Sec. C) are set to (50 , 100) .
The model is trained using a batch size of 32 for 50 epoch, with 6000 training examples per epoch.
We use ADAM (Kingma & Ba, 2014) to train the model.
The code is based on PyTorch.
On one NVIDIA GeForce GTX TITAN X GPU, training an epoch takes approximately 5 minutes.
For license, CMU-Mocap (CMU) is free for all users; MuPoTS-3D (Mehta et al., 2018) is for noncommercial purposes.
Part of our code is based on AMCParser (MIT license), attention-is-all- you-need-pytorch (MIT license), and MRT (Wang et al., 2021b)
(not specified), and XIA (Guo et al., 2022) (GPL license).
[Table]
[Figure]
F S UMMARY OF E VALUATION M ETRICS
In the main paper, we mainly focused on evaluating the fidelity and diversity of the predicted multi-person motion based on Average Displacement Error (ADE) , Final Displacement Error (FDE) , and Final Pairwise
[Table]
[Table]
Distance (FPD) , and briefly discussed other metrics.
Here, we explain these and additional metrics in detail and also provide a systematic summary in Table B for better understanding.
Additional comparisons based on these metrics are shown in the following sections.
As summarized in Table B, we group the metrics into thee types, with each type evaluating different aspects of predicted motion (discussed in Sec.
4 of the main paper) – single-person fidelity , multi-person fidelity , and overall diversity .
Specifically, Local Average Displacement Error (lADE) and Local Final Displacement Error (lFDE) are proposed to further evaluate the single -person fidelity.
They compute the average distance between the individual ground truth and the individual prediction closest to the individual ground truth.
Note that ADE (or lADE) and
FDE (or lFDE) only measure the best predictions in all outputs.
While we can average ADE (or lADE) and FDE (or lFDE) by computing the distance between multiple predictions and a single ground truth, this way of assessing the overall prediction quality cannot reflect motion realism.
For example, a very realistic but diverse set of outputs may have poor average ADE and FDE.
Therefore, we do not use such metrics for our evaluation.
For diversity evaluation, a common metric is Average Pairwise Distance (APD) (Mao et al., 2021; Yuan & Kitani, 2020; 2019) that is the average ℓ 2 distance between all predicted motion pairs.
In this paper, we formulate diverse forecasting as producing more forecasts over time (see Sec. 3.3).
This progressive generation is actually closer to reality because the multi-modality of motion should be more pronounced after further time.
However, in this case, APD cannot reflect the diversity well, since many predictions will share the same previous segment.
Therefore, we only examine the diversity of the last pose (FPD), as the last pose should not be the same.
Moreover, we introduce three tailored metrics to evaluate specific aspects of predicted motion, which correspond to the unique challenges in multi-person motion forecasting as discussed in the main paper.
) Foot et al.
Trajectory considered collision al.
trajectories
G A DDITIONAL E XPERIMENTAL R ESULTS
Additional Quantitative Results.
In Table C, we use lADE and lFDE to compare single-person fidelity, and we observe that our model also significantly outperforms the baseline.
Additional Comparisons on SoMoF Benchmark.
We investigate the performance of our framework on the Social Movement Prediction Challenge (SoMoF) (Adeli et al., 2020; 2021).
We specifically use the 3DPW dataset (von Marcard et al., 2018), where we use labeled trajectories and poses (13-joint human skeleton), but we do not use videos of given scenes as input.
We discard the multi-modal reconstruction loss since the 3DPW provided by SoMoF benchmark is relatively small.
In Table E, we provide results on our implemented
[Figure]
Start
End Poses
[Figure]
DuMMF (Ours)
[Figure]
deterministic prediction and compare with baselines directly reported from the leaderboard on the SoMoF benchmark.
In Table F, we use SoMoF benchmark for stochastic multi-person forecasting.
Note that ADE and FDE require access to ground truth data, which is not publicly available from SoMoF benchmark.
Thus, we report the results on validation set.
We observe a similar performance as on CMU-Mocap (CMU) and
MuPoTS-3D (Mehta et al., 2018), specifically, using DuMMF, the model has significantly better accuracy and diversity in stochastic multi-person forecasting.
Additional Comparison with Single-Person Forecasting Methods.
In Table G, we provide a comparison by applying a stochastic single-person forecasting baseline.
For a fair comparison, we formulate this single-person forecasting architecture (named SRT) with a transformer encoder-decoder adapted from MRT (Wang et al.,
[Table] [Table]
2021b)
[Table]
where we modified the encoder and decoder to be individually independent.
Table G illustrates that (1) the modeling of social interactions is crucial, since the single-person forecasting baseline has much worse performance; (2) the improvement from our DuMMF for the multi-person forecasting model is much higher than that for the single-person forecasting model
[Table]
Additional Ablation on Impact of Learning Objectives.
In Table H, we evaluate the impact of each loss term within the dual-level framework.
In general, using all loss functions yields the best results, since its results are either the best or the second best.
We observe that local and global discriminators not only make predictions more accurate, but also more diverse.
Note that the reconstruction losses L lR and L gR optimize only the most accurate prediction of all the outputs.
It is important to provide supervision for other predictions that are not the best.
We observe that limb loss L L is crucial, as it is the only loss function that provides supervision for all outputs .
The multi-modal reconstruction loss also has a large performance impact, since it provides supervision for more than one output.
Additional Analysis of the Number of Discrete Latent Codes.
Note that the number of discrete latent codes is not restricted to the same number as the number of predictions per second.
We chose them to be the same for simplicity and a better trade-off between prediction performance and training efficiency.
We find that this setup also achieved the best performance.
To ensure that all predictions come from different discrete codes, the number of discrete latent codes should not be less than the number of predictions.
In Table I, we provide an ablation study when the number of predictions per second is 5.
If the number of discrete intents is greater than 5, we randomly select a discrete code without replacement (excluding any of the previously selected intents) to generate a prediction.
We observe that both the accuracy and diversity of the predictions decrease as the number of intents increases.
An explanation of such behavior could be: When the number of discrete latent codes is the same as the number of predictions, each code is explicitly and fully optimized for a particular prediction, leading to the best prediction accuracy and diversity; whereas, when the number of intents increases, the random selection strategy may hurt performance, as the probability of selecting the best five discrete codes decreases.
We hypothesize that a better training and selection strategy might improve performance with more discrete codes.
[Table]
[Equation]
[Table]
H G ENERALIZABILITY AND S CALABILITY : E VALUATION ON M ORE -P ERSON S CENARIOS
Datasets with More People per Scene.
In the main paper, we constructed multi-person motion data with 3 people per scene.
Here, we construct more challenging datasets with significantly increased number of people – up to 6 or 9 people; this also increases the complexity of social interactions.
Specifically, we first sample 3 -person motion data from the original 3 -person test data in CMU-Mocap (CMU), and then compose them together to generate 6 -person or 9 -person motion data.
We use handcraft rules (Kothari et al., 2020) to filter out scenes with trajectory collisions.
Instead of retraining the model in the novel setting with more people per scene (Wang et al., 2021b), we directly evaluate the model trained on 3-person motion data , and test if our model is able to scale to scenarios with more people.
Note that here we use MRT (Wang et al., 2021b) as the backbone of DuMMF framework.
Qualitative Results.
Similar to Sec.
4 of the main paper, we provide visualizations for 6 -person and 9 -person scenarios in Figure A and Figure B of the end poses of predictions of 3 second.
We randomly select a prediction and visualize the corresponding full motion sequence as well.
Note that our model is trained on only 3 -person data without any fine-tuning on 6 -person or 9 -person data, but it still performs well on more-person data, suggesting the generalizability and scalability of our approach.