Long Range Sequence Generation via Multiresolution Adversarial Training
Abstract
We present a multi-resolution approach for generative modeling of spatiotemporal sequences.
Our approach focuses on a key challenge of modeling long-term temporal dependencies that is common in complex spatiotemporal phenomena.
For instance, realistic modeling of basketball players behavior requires capturing long-term goals and how they inﬂuence short-term decisions.
Our multi-resolution approach has several attractive properties.
First, it is completely unsupervised, and requires no additional labeling of high-level semantics such as long-term goals.
Second, the multi-resolution aspect allows us to model conditional distributions beyond forward sampling, such as conditioning on future behavior.
Finally, our approach integrates generative adversarial training, which enables us to learn generative models that signiﬁcantly outperform conventional generative sequence modeling.
We validate the effectiveness of our model on synthetic sequences and spatiotemporal basketball player trajectory generation.
Introduction
Generative models for sequential data have broad applications in audio synthesis [7], machine translation [12], video generation [11] and sports analytics [14].
For example, given the movement trajectories from professional basketball team play, generative models can extract latent representations that imitate the decision making process of multiple players in spatiotemporal environments.
However, designing generative models for long-range spatiotemporal sequences is highly challenging, as generative modeling requires estimating the joint probability over the entire sequence which is susceptible to covariant shift and compounding errors.
Moreover, long-range sequences contain long-term dependencies which pose difﬁculties to many existing models such as RNNs [8].
In contrast to the traditional maximum likelihood objective, generative adversarial networks (GAN) [2] introduce a discriminator to the loss function which has sparked a new paradigm of generative modeling.
[3] proposed Generative Adversarial Imitation Learning (GAIL) to extend GAN to the sequential setting by using an equivalence between maximum entropy inverse reinforcement learning (IRL) and GANs.
[12] further modiﬁed the reward structure and only gives a single reward when the sequence is completed.
But these models only use simple generators, which limit their ability to model long-range sequences.
In order to generate long-range trajectories [14] proposes to use manually deﬁned macro goals from trajectories as weak labels to train a hierarchical RNN.
[13] further extends this idea to the multi-agent setting with a hierarchical variational RNN.
However, while using macro goals can signiﬁcantly reduce the search space, obtaining the macro goals from trajectories using e g. stationary points, can be ad-hoc and expensive.
To avoid using weak labels, we propose a novel hierarchical framework for generating long-range spatiotemporal sequences from raw data.
The top level generator in this hierarchy operates at the coarsest resolution and the lower level generators inherit the previous predictions to reﬁne the sequence.
By training all levels of the generator using a GAN loss, our framework generates realistic long-range trajectories and outperforms baselines on several spatiotemporal metrics that we use to characterize real data (e.g., path length, step size change).
Our method bears afﬁnity to other multiresolution generative models such as Progressive
Growing GAN [5] and multiscale autoregressive density estimation [9].
1.1 Background
GAIL Given expert trajectories τ E
= { ( s 0 , a 0 ) , ..., ( s T , a T ) } ∼ π E , where π E corresponds to an unknown expert policy, we aim to approximate the expert policy by learning a generator π θ .
To do so,
GAIL introduces another discriminator D ω which outputs a score for every ( s t , a t ) pair indicating the probability of that pair coming from the true distribution.
GAIL deﬁnes the optimization objective
[Equation]
where a generator G and a discriminator D play a two-player minimax game.
We update discriminator parameters using backpropagation, and generator parameters using policy gradient as follows:
[Equation]
[Table]
Multiresolution Adversarial Training 2.1 Multiresolution Generator
To extract hierarchical structures from a sequence in an unsupervised way, multiresolution generative training ( MAT ) introduces a multiresolution generator .
For a sequence of length T , the multiresolution generator consists of N = (cid:98) logT (cid:99) levels of sequential models M 1 , M 2 , ..., M N , each of which operate at increasing temporal resolutions.
• M 1 predicts every n 1 = 2 N − 1 steps: p ( x 1+ jn 1 | x 1 , x 1+ n 1 ...x 1+( j − 1) n 1 ) =
g 1 ( h 1 j ) , h 1 j = f 1 ( x 1+( j − 1) n 1 , h 1 j − 1 ) .
• M k , 1 < k ≤ N predicts every n k = 2 N − k steps, working recursively: – if j mod 2 == 0 :
copy the states from the top level and update hidden states: x 1+ jn k = x 1+( j/ 2) n k − 1 , h k j = f k ( x 1+( j − 1) n k , h k j − 1 ) .
– Otherwise: interpolate the states from past and future, and update hidden states: p ( x 1+ jn k | x 1 , x 1+ n k ..., x 1+( j +1) n k ) =
g k ( h kj , x 1+( j +1) n k ) , h kj =
f k
( x 1+( j − 1) n k , h kj − 1 ) .
The dependency graph of this multiresolution generator is shown in the ﬁgure 1.
At each level, the sequence model only need to capture the temporal dependency of a particular resolution.
2.2 Multiresolution Adversarial Training
One way to optimize the objective in Eqn 1 is to use policy gradients, but this procedure can be expensive and tends to suffer from high variance and sample complexity [4].
Instead, MAT takes a model-based approach and assumes the environment models are known and differentiable.
Hence, we can use the "reparameterization trick" [6] to differentiate the generator objective with respect to the policy parameters.
Similar ideas have been shown in [1, 10] to be more stable and sample-efﬁcient.
Formally, MAT uses the multiresolution generator G with parameters θ , and a discriminator D parameterized by ω .
We are given some trajectory data drawn from an expert policy τ E ∼ π E , where τ E = { x 1 , x 2 , ..., x T } .
Denote the subsequence s t = { x 1 , x 2 , ..., x t } , 1 ≤ t ≤ T − 1 .
The generator produces a learner’s policy π θ .
The discriminator D outputs the probability that the state-action pair ( s t , x t +1 ) comes from the data rather than the generator.
Thus our training objective function is:
[Equation]
Experiments 3.1 Trigonometric functions
Trigonometric functions are common benchmarks to evaluate the performance of time-series models.
When multiple trigonometric functions f i ( x ) of different scales and frequencies are stacked together, they can form complicated sequential patterns.
To evaluate MAT , we model f ( x ) = (cid:80) i f i ( x ) , where [Equation] [Equation] [Equation]
(cid:19) .
We sampled 5000 sequences of length 129 with random initial points.
MAT uses fewer weights (80,746) than the baseline, a single-layer RNN (182,658).
As shown in Figure 2, as the sequence length increases, RNN fails to capture the pattern, while MAT can successfully recover the ground truth sequence.
3.2 Basketball Behavior Cloning
Data We applied MAT on the tracking data of professional basketball players (107,146 training and 13,845 test sequences).
Each sequence has the ( x, y ) -coordinates of 5 offensive basketball player trajectories for 50 frames (sampled at 6.25Hz for 8 seconds) and takes place in the left half-court.
Training data for sub-models of different levels are sampled at different rates.
For example, we use frames (0, 16, 32, 48) for M 1 , frames (0, 8, 16, 24, 32, 40, 48) for M 2 , etc.
Baselines and Metrics We compare our model with the following 4 baselines:
• Plain RNN model with 400 2-layer GRU cells for the hidden state, 1,581,220 params.
• Plain RNN model with adversarial training , 1,581,220 params.
[Table]
• Hierarchical Variational RNN model [13], macro-goals required , 4,372,190 params. • H-RNN model [13], with adversarial training , macro-goals required , 3,285,280 params.
To evaluate the quality of generated samples, we use 4 metrics as proxies to compare the generated trajectories with real data: (1) Average trajectory length measures the macro strategies of players.
Average out-of-bound rate measures whether the model learns the "rule" of the game.
(3) Average step size change quantiﬁes the relationship between consecutive actions, and (4) Max-Min path difference to analyze the cooperation within a team.
Results Table 1 shows the quantitative comparison of different models using our metrics.
RNN statistics signiﬁcantly deviate from the ground truth, but greatly improve with adversarial training.
HVRNN uses "macro-goals", and performs reasonably w.r.t macro metrics including average path length and max-min path difference.
However, the big step size changes lead to unnatural trajectories.
With adversarial training, issues were alleviated but the models require manually crated macro-goals.
Figure 3 (1-4) shows a sample trajectory generated by MAT at different levels.
The top level generates the macro goals.
At lower levels, the model further reﬁnes the trajectories given the macro goals.
Note that based on predictions of the previous level, our model can capture new spatiotemporal structures of that level without restricting itself to the previous level.
Figure 3 (5-8) compares the trajectories generated by 3 different models and the ground truth.
MAT has more natural step changes, and there are some interesting twists that can only be found in expert and MAT trajectories.
Conclusions
In this work, we propose a novel deep generative model for long-range sequences, namely, multiresolution adversarial training ( MAT ).
By coupling a multiresolution generator with adversarial objective function, MAT can generate higher-quality long-range sequences without human-deﬁned hierarchies.
We demonstrate the effectiveness of MAT using both synthetic data as well as real-world basketball play trajectories.
Experiment results suggest MAT can capture complicated temporal structure and generate more realistic long-range sequences.