Representing Spatial Trajectories as Distributions
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We introduce a representation learning framework for spatial trajectories.
Werepresent partial observations of trajectories as probability distributions in a learnedlatent space, which characterize the uncertainty about unobserved parts of thetrajectory.
Our framework allows us to obtain samples from a trajectory forany continuous point in time—both interpolating and extrapolating.
Our ﬂexibleapproach supports directly modifying speciﬁc attributes of a trajectory, such as itspace, as well as combining different partial observations into single representations.
Experiments show our method’s superiority over baselines in prediction tasks.
Introduction
The visual world is full of objects moving around in predictable ways.
Examples of these spatialtrajectories include human motion, such as people dancing or exercising; objects moving, such asa ball rolling; trajectories of cars and bicycles; or animal migration patterns.
Evidence suggeststhat the human perceptual system encodes motion into high-level neural codes that represent themotion holistically, going beyond the speciﬁc input observations [22].
Humans use this abstractrepresentation for downstream tasks like inferring intention [6].
Computer vision systems likely needsimilar mechanisms to encode trajectories and motions into global representations.
Representation learning has been transformative in other domains such as images and text for itsability to obtain high-level representations that reorganize the information in the input, and are betterat downstream tasks than the original signals.
A global representation of trajectories would allowus to evaluate a trajectory at any point in time, even ones not yet observed.
However, modelingtrajectories presents a series of challenges for representation learning.
First, in real-time scenarios,the future of the trajectory is never observed.
Second, temporal and spatial occlusions may impedeobserving part of a trajectory.
Third, trajectories are by nature continuous in time.
And ﬁnally, atrajectory-level metric is usually not well deﬁned and application-dependent.
We propose a representation learning framework for trajectories that deals with all these challengesin a uniﬁed way.
Our key contribution is the representation of a partial observation of a trajectory as a probability distribution in a learned latent space, that represents all the possible trajectories the observation could have been sampled from.
Our framework’s simplicity and generality allows it to be it does not constrain the input-space metric, accepts observations of different lengths and at any (irregularly sampled) point in time, can be implemented using different families of latent space distributions and is capable of performing inference-time tasks for which it has not been explicitly trained.
Our experiments on human movement datasets show that our method can accurately predict the past and future of a trajectory segment as well as the interpolation between two different segments, outperforming autoregressive baselines.
Additionally, it can do so for any continuous point in time
We also show how we can modify given trajectories by manipulating their representations
[Figure]
Method2.1 Framework, Deﬁnitions and Notation
The input to our framework is a sequence of samples obtained from a (continuous in time andinﬁnite) spatial trajectory u , which we deﬁne as the continuous temporal evolution of a set of spatialcoordinates.
We call each sample a point x , which lives in the input space R K .
We call the sequenceof points, together with the times t at which they were sampled, a segment s , which can be understoodas a partial observation of u .
We deﬁne a distance metric  between points x in the input space.
Our goal is to transform these measurements of motion s into a representation z that will be useful fordownstream tasks.
We deﬁne a latent space R N of trajectories z .
Each z in this space represents thefull extent of a trajectory, both in time and in space.
We use Q to represent probability distributionsover trajectories z , in the latent space.
We deﬁne a distance function D between distributions oftrajectories, which assumes an underlying distance function d between trajectories z .
We use an encoder ⇥ to encode every segment s to a probability distribution Q ( · ; s ) = ⇥
( s )over trajectories, where Q ( z ; s ) represents the probability that s was sampled from the trajectoryrepresented by z .
Additionally, we can decode a trajectory z at a speciﬁc time t by using a decoder  ,obtaining a point x =  ( z, t ) .
takes any continuous t as input.
See Fig.
2 for a schematic.
2.2 Representation Learning
When observing a segment s , one may have some uncertainty about the speciﬁc trajectory it wassampled from.
For instance, a segment showing a person jumping may correspond to a trajectorythat continues with the person falling, or to a trajectory that proceeds with them doing a backﬂipand landing on their feet, but it will not belong to a trajectory of a person swimming.
Therefore,we represent the segment as a distribution over trajectories, where Q represents the likelihood of atrajectory given the segment.
During training, the goal is to learn this mapping from the input space(segments of trajectories) to the latent space (distributions over trajectories).
Concretely, given two segments s a , s b that have been obtained from the same underlying trajectory,we want some z to exist such that its likelihood under the distributions Q a and Q b representing eachof the segments is high.
To encourage this, we train the model to maximize the overlap betweenthe distributions Q a and Q b .
Similarly, we minimize the overlap between (the distribution representations of) segments sampled from different trajectories, under the assumption that no trajectoryz exists that contains both segments.
Speciﬁcally, we minimize a self-supervised triplet loss:
[Equation]
where ↵ is a margin hyperparameter, and T is a set of triplets: for every segment i in the dataset, wedeﬁne several triplets by sampling pairs consisting of a positive segment k + (such that ( i, k + ) is apositive pair) and a negative segment k  (such that ( i, k  ) is a negative pair).
[Figure]
In addition to learning representations of trajectories, we also wish to be able decode them back toinput-space points.
For this to be achieved, we train a decoder  that allows us to obtain the speciﬁcvalue of any trajectory at any continuous time t .
In order to train the decoder  , we sample trajectoriesz ⇠ Q ( · ; s ) from each segment representation, and decode them at speciﬁc time-steps t which werecontained in s , obtaining a prediction ˆ x t =
( z, t ) for which we have ground truth x t .
There is nouncertainty in this prediction, as x t was part of the segment s in the ﬁrst place; the decoder is onlyexplicitly trained for reconstruction, not extrapolation.
We train the decoder via regression, usingthe point-wise distance  .
Note that we are never explicitly deﬁning a trajectory-level distance inthe input space, as the goal of the framework is to learn this distance.
The reconstruction loss ismathematically deﬁned as:
[Equation]
where N is the number of segments in the dataset, and T i is the number of points in segment s i .
Weminimize Eqs.
(1) and (2) jointly and end-to-end.
We implement the encoder ⇥ using a Transformer
Encoder architecture [54], and the decoder  using a ResNet [17].
See Appendix C for more details.
2.3 Exploiting the Structure of Trajectories
Our structured representation of trajectories allows us
to deﬁne positive and negative pairs of segmentsfor Eq.
(1) in different ways.
Next, we describe three complementary approaches that do not requireany change in the loss or architecture: they simply add new positive and negative pairs to Eq. (1).They are beneﬁcial in order to obtain harder negatives and improve the training process.•
Re-encoding Given a trajectory z , we can decode it into any set of times t , obtaining a newsegment. This segment can be (re-)encoded using ⇥ , and a representation Q can be obtained forit, resulting in a new positive or negative for other segment representations.•
Intersection An intersection of two distributions Q in the latent space will represent all thetrajectories that have a high likelihood for both intersected segments.
Note that an intersection inthe latent space is a union in the input space: the intersection constrains the possible trajectoriesto those that are consistent simultaneously for the two segments.
Similarly, an intersection in theinput space (assuming an overlap between segments) is a union in the latent space.• Input-space relationships The initial segments can have different relationships, such as precedence, containment, or overlap [3].
In our experiments, we sample three segments for every trajectory: a past segment ( P in Fig. 3), a future segment ( F ) whose starting time comes right afterthe end of the past segment, and a combination segment ( C ), which contains both the past and the
PF
PF PF
[Figure]
P
CP CP
C
C FP
FP
CF CF
O O
F F
Input Space
Latent Space
Legend
P
F
C
FP
CP
CF
O
Past
Future
Combination
Future given past Past given future
Combination given past
Combination given future
Other future segments. In the latent space, the intersection of the past and future segments should be [Figure] equal to the representation of the combination segment.
[Figure]
We exemplify a combination of these possibilities in Fig.
We differentiate between soft and hardpositives and negatives, where hard ones are deemed more relevant and we give them a more importantrole in the optimization.
In practice, we sample hard positives and negatives more often than soft ones. [Figure]
2.4 Comparing Distributions
[Figure]
Eq.
(1) uses the distance function D to compare distributions of trajectories.
In this section, weintroduce two different ways of designing D , resulting in different intuitions about the latent space.
Symmetric Distance
If two segments can belong to the same trajectory, the distributions Qrepresenting each segment should be similar and close to each other, and the (symmetric) distance Dbetween them should be small.
For example, in Fig.
3, the representations of the past P and future Fsegments belonging to the same trajectory should be hard positives.
Conditional Instead of computing a distance or a similarity, we compute the probability that asegment s a belongs to the same trajectory as another segment s b .
We model this as a conditionalprobability P ( Q a | Q b ) .
There are four possibilities:
• P ( Q a | Q b ) = 1 , when s b includes s a , like the combination segment C including the past P .
• 0 < P ( Q a | Q b ) < 1 , when Q a is possible but not necessary given by Q b , like P and F in Fig.
• 0 < P ( Q b | Q a ) < 1 , deﬁned in a similar manner.
• P ( Q a | Q b ) = 0 , for unrelated segments, like C and O .
We treat the ﬁrst case as a hard positive, the next two as soft positives, and the last case as anegative.
Note that under this interpretation, a past P and a future F from the same trajectory arenot hard positives, but soft positives: one does not fully deﬁne the other.
This approach resultsin probability values that we either maximize (positives) or minimize (negatives), so we deﬁne D ( A, B ) = 1  P ( A | B ) .
The previous approaches require a way of computing either a distance between the distributions Q , ora conditional probability between them.
In the next section, we show two families of distributions forwhich these can be deﬁned.
We list all negative and positive pairs in Appendix C.1.
2.5 Trajectory Segments as Distributions
In order to obtain Q , the encoder ⇥ predicts the parameters of a distribution family.
Conditions forthe distribution families are: 1) we can sample from it in a differentiable way, 2) we can parameterizeit, 3) we can compute, in closed form, an intersection that returns a distribution from the same family,and 4) we can compute either a similarity function or a conditional probability, or both (see Sec.2.4).
Next, we introduce two distribution families that meet the previous criteria.
Normal distributions We use uncorrelated multivariate normal distributions, and parameterizethem with a mean µ and a standard deviation  .
We compute the intersection as the product of twonormal distributions, which remains normal when the dimensions are uncorrelated (see Appendix D).
We use the symmetrized Kullback-Leibler (KL) divergence between distributions as a distancefunction.
This distance is not a proper metric; alternatives are discussed in Appendix D. Normaldistributions assume an underlying Euclidean distance metric d between trajectories z .
Box embeddings Box embeddings [55] represent objects with high-dimensional products-ofintervals (or boxes), parameterized by their two extreme vertices.
The intersection between boxembeddings is well deﬁned and results in another box embedding.
This makes them a natural choiceto represent conditional probabilities, which can be computed as P ( A | B ) =
Vol( A \ B )  Vol( B ) ,where Vol is the volume of the box, and \ represents the intersection operation.
These operationsare straightforward to compute.
Boxes are not actual distributions, as they need not integrate to one.
However, they are easily normalized by dividing by their volume, and therefore they can be treatedas distributions for all the practical purposes required in our framework ( i.e. sampling, where weapproximate the boxes with a uniform distribution).
Symmetric distance functions can also be deﬁnedon box embeddings; we deﬁne a few in Appendix D.2.
In both cases, we use the reparameterization trick [24] in order to sample from the distributions whilekeeping gradient information.
We found the best-performing option was using box embeddings underthe conditional scenario; the values reported in Section 3.2 use this setting.
2.6 Inference
Once trained, our decoder  is able to decode a trajectory at any continuous time t , including timesthat were not part of the input.
For example, our framework can decode a future segment given aninput past segment, by sampling from its representation, and evaluating that sample at some futuretimes.
This future segment will not necessarily be equal to the ground truth future segment (in case itexists), because a single past can have multiple futures.
Overall, this results in a framework that is capable of doing 1) future and past prediction, by decoding asegment at times outside of its range; 2) continuous reconstruction given a discrete input, by decodingat any continuous time t ; 3) interpolation between two segments, by decoding trajectories in theirlatent-space intersection; 4) comparison of different trajectories, by comparing their representationsz ; and 5) modifying existing trajectories, by manipulating the latent space.
All the previous tasks arepossible without explicitly training to do any of them.
We show examples in Section 3.
Experiments
3.1 Datasets
Our method is generally designed for any kind of spatial trajectory.
For our experiments, we selecteddata adhering to the following criteria.
First, there has to be uncertainty in the trajectory whengiven just a segment (for instance, the future is not fully speciﬁed given the past).
Second, theprediction should not require external contextual information.
Context can be seamlessly addedto our architecture, but it involves additional task-speciﬁc engineering decisions, and we want ourevaluation to be orthogonal to them. Similarly, we avoid trajectories that require highly-engineered
[Table]
VRNN [13]
[Table]
Trajectron++ uni.
[Table]
Trajectron++ [44]
[Table]
TrajRep (ours, ablation)
[Table]
+ re-encoding (ours)
[Table]
VRNN [13]
[Table]
Trajectron++ uni.
[Table]
Trajectron++ [44]
[Table]
ours, ablation)
[Table]
+ re
[Table]
point-level distances  .
Finally, we prefer our trajectories to be obtained from real-world data.
For allthe previous reasons, we implement our framework on human movement datasets .
Speciﬁcally, we extract keypoints from human action datasets using OpenPose [10].
For every video,we keep the most salient human trajectories.
This results in sequences of dimension [ L, 25 , 2] , where
L is the number of frames in the trajectory, 25 is the number of joints in a human skeleton extractedby OpenPose, and 2 corresponds to the number of spatial coordinates for every joint.
We refer to thewhole skeleton at every time-step —the combination of all joints, resulting in a K = 50 -dimensionalvector— as a point .
As a distance function  between points ( i.e. skeletons) we use l 2 -norm distanceper-joint, and average across all visible joints.
We extract human movement trajectories from the
FineGym [45], Diving48 [29] and FisV [57] datasets, which correspond to gymnastics, diving andﬁgure skating, respectively.
For each of the datasets, we experiment with short sequences (up to 10 time-steps, or slightly over onesecond) and long ones (up to 30 time-steps, representing slightly under four seconds of the trajectory),and report results for both.
We provide more details on the dataset creation in Appendix B.
3.2 Quantitative Experiments
Baselines and ablations As baselines, we select trajectory-modeling methods that are capable ofencoding uncertainty about the future.
Variational RNNs [13] extend recurrent neural networks(RNNs) to the non-deterministic case, by modeling every step with a variational auto-encoder (VAE)[24].
Trajectron++ [44] is a state-of-the-art trajectory-modeling framework which also builds on topof RNNs and (conditional) VAEs [23].
Uncertainty is modeled as a Gaussian mixture model (GMM).
We adapt Trajectron++ to our data, making the encoding and decoding as similar to our setting aspossible (for fairness), while keeping the core of the framework intact.
We train two Transformer++versions, one with uniformly-sampled inputs and outputs (“Trajectron++ uni.”), and a second onewith non-uniform sampling, following the setup in our models (“Trajectron++”).
We also ablate ourmodel, and report results with and without training with re-encoded segments.
[Figure]
Input: 8 frames, 1 second
[Figure] [Figure]
[Figure]
[Figure] Tasks and metric We evaluate our framework on three different tasks: future prediction, pastprediction, and interpolation between two segments.
Future prediction consists in predicting pointsfrom a future segment given a past segment.
Past prediction is deﬁned symmetrically.
In theinterpolation task, we input two separated segments (past and future) from a trajectory, and predictthe segment in between them.
In our model, we do so by decoding from the latent-space intersectionof the two input segments.
Baselines (which are autoregressive) are not capable of doing thiscombination, so we only use the past segment as input.
Baselines are also not capable of directlyperforming past prediction, so we predict the future of the reversed trajectory instead.
As a metric,we use the average of the l 2 -norm distance across joints, which is used by all methods during training,and report the best out of M = 10 samples, to account for multiple modes and uncertainty in theprediction.
Note that our model has never been explicitly trained to perform any of the previous tasks.
We show results in Tables 1a and 1b, for long and short trajectories respectively.
Our modeloutperforms baselines in all the tasks, which proves its value and ﬂexibility.
We also show howcreating more interesting negatives with the re-encoding of decoded trajectories results in moreaccurate prediction results.
However, our method performs well even without re-encoding.
3.3 Qualitative Experiments
We show examples of our model’s inputs and outputs in Fig. 4.
Speciﬁcally, we show future, past andinterpolation predictions.
The results reﬂect that the model learns to predict sequences up to fourtimes longer than the input.
They also show the large temporal resolution of our model: the modelpredictions evolve smoothly and sensibly for time-steps separated by a few hundredths of a second(Figs.
4a and 4b).
When not all joints are present in the input (ﬁrst frame in Fig. 4a), our model is stillcapable of reconstructing the full spatial extent of the position.
Finally, note how the model can takeas inputs irregularly sampled time-steps (Fig. 4b), which makes it adaptable to temporal occlusions.
Baselines are only capable of predicting the future, and are restricted to predicting uniform time-steps.
The segments are directly tied to the temporal span they represent.
For example, two segments withthe exact same coordinates and evolution across time, but starting at different times will result indifferent (albeit similar) representations.
The speed of a movement and the time in the trajectorywhere the movement is done are important attributes of that movement, and the representation shouldnot be invariant to them: they belong to different trajectories.
However, because these trajectories arevery similar, our model learns to represent them close in the latent space.
In Fig.
5, we show that themodel encodes different temporal variations.
Moving along speciﬁc directions in the latent spaceresults in progressively faster trajectories (Fig. 5a), or in trajectories with an increasing temporaloffset with respect to the original one (Fig. 5b).
See Appendix E for details.
Related work
Modeling trajectories Spatial trajectories are usually modeled in the literature in an autoregressive(AR) fashion [47, 33, 26, 2, 49, 20, 44, 30, 58, 52, 25, 19], where trajectories are deﬁned conditionedon previous time-steps.
Despite their success, AR models are incapable of dealing with some of thechallenges stated in Section 1, most notably they do not represent time as a continuous variable, theycannot model the full extent of a trajectory (simultaneously both past and future), and no learnedtrajectory-level metric can be obtained from them.
Some of them model the uncertainty in theprediction [52, 25, 19, 20, 44, 13, 49], and we use two representative ones [13, 44] as our baselines.
A different line of work is focused on representing segments of trajectories (not just points) as pointsin a latent space [61, 64, 59, 60, 28, 63, 9, 31].
However, they are not capable of modeling the fullextent of a trajectory outside the limits of the considered segment.
Additionally, the segment-levelmetric is either unstructured [61], or is explicitly given [64, 62, 63].
Continuous time Modeling time as a continuous signal has gained traction recently in ﬁelds such asgraphics [56, 53, 39, 46] or physics modeling [8, 11], because it accurately represents the underlying(continuous) world being modeled.
In the graphics neural implicit functions literature, time is used to [Figure] condition the prediction of the network.
We adopt the same approach in our decoder.
We encode theset of continuous times in a segment by using a Transformer network [54], which by constructionis permutation invariant, but allows temporal embeddings to be concatenated with the input, bothdiscrete [5, 4] and continuous [53, 51].
Self-supervised representation learning Finding self-supervised representations for temporal datahas been the subject of a large amount of work in domains such as trajectories [31], video [35, 16, 40],or audio processing [21, 42, 15].
Most methods, however, represent segments as simple points ina Euclidean space.
Structured representations for temporal data [50, 36] allow the latent space tofollow certain inductive biases, like our framework’s idea that segments compose trajectories.
Wemodel segments either as normal distributions or as box embeddings [55].
The latter have beenused to represent hierarchical relationships in domans such as text [38, 34], knowledge bases [1], orimages [41].
We use them to represent temporal information.
In recent work, Park et al.
[36] alsomodel segments using normal distributions, where trajectories are weighted sums of the segmentrepresentations.