Representing Spatial Trajectories as Distributions
Anonymous Author(s)
AffiliationAddress email
Abstract
We introduce a representation learning framework for spatial trajectories.
Werepresent partial observations of trajectories as probability distributions in a learnedlatent space, which characterize the uncertainty about unobserved parts of thetrajectory.
Our framework allows us to obtain samples from a trajectory forany continuous point in time—both interpolating and extrapolating.
Our flexibleapproach supports directly modifying specific attributes of a trajectory, such as itspace, as well as combining different partial observations into single representations.
Experiments show our method’s superiority over baselines in prediction tasks.
Introduction
The visual world is full of objects moving around in predictable ways.
Examples of these spatialtrajectories include human motion, such as people dancing or exercising; objects moving, such asa ball rolling; trajectories of cars and bicycles; or animal migration patterns.
Evidence suggeststhat the human perceptual system encodes motion into high-level neural codes that represent themotion holistically, going beyond the specific input observations [22].
Humans use this abstractrepresentation for downstream tasks like inferring intention [6].
Computer vision systems likely needsimilar mechanisms to encode trajectories and motions into global representations.
Representation learning has been transformative in other domains such as images and text for itsability to obtain high-level representations that reorganize the information in the input, and are betterat downstream tasks than the original signals.
A global representation of trajectories would allowus to evaluate a trajectory at any point in time, even ones not yet observed.
However, modelingtrajectories presents a series of challenges for representation learning.
First, in real-time scenarios,the future of the trajectory is never observed.
Second, temporal and spatial occlusions may impedeobserving part of a trajectory.
Third, trajectories are by nature continuous in time.
And finally, atrajectory-level metric is usually not well defined and application-dependent.
We propose a representation learning framework for trajectories that deals with all these challengesin a unified way.
Our key contribution is the representation of a partial observation of a trajectoryas a probability distribution in a learned latent space, that represents all the possible trajectories theobservation could have been sampled from.
Our framework’s simplicity and generality allows it to beflexible: it does not constrain the input-space metric, accepts observations of different lengths and at any (irregularly sampled) point in time, can be implemented using different families of latent space distributions, and is capable of performing inference-time tasks for which it has not been explicitlytrained.
Our experiments on human movement datasets show that our method can accurately predictthe past and future of a trajectory segment, as well as the interpolation between two different segments,outperforming autoregressive baselines.
Additionally, it can do so for any continuous point in time.
We also show how we can modify given trajectories by manipulating their representations.
[Figure]
Method2.1 Framework, Definitions and Notation
The input to our framework is a sequence of samples obtained from a (continuous in time andinfinite) spatial trajectory u , which we define as the continuous temporal evolution of a set of spatialcoordinates.
We call each sample a point x , which lives in the input space R K .
We call the sequenceof points, together with the times t at which they were sampled, a segment s , which can be understoodas a partial observation of u .
We define a distance metric δ between points x in the input space.
Our goal is to transform these measurements of motion s into a representation z that will be useful fordownstream tasks.
We define a latent space R N of trajectories z .
Each z in this space represents thefull extent of a trajectory, both in time and in space.
We use Q to represent probability distributionsover trajectories z , in the latent space.
We define a distance function D between distributions oftrajectories, which assumes an underlying distance function d between trajectories z .
We use an encoder Θ to encode every segment s to a probability distribution Q ( · ; s ) =
Θ( s )over trajectories, where Q ( z ; s ) represents the probability that s was sampled from the trajectoryrepresented by z .
Additionally, we can decode a trajectory z at a specific time t by using a decoder Φ ,obtaining a point x = Φ( z, t ) .
Φ takes any continuous t as input.
See Fig.
2 for a schematic.
2.2 Representation Learning
When observing a segment s , one may have some uncertainty about the specific trajectory it wassampled from.
For instance, a segment showing a person jumping may correspond to a trajectorythat continues with the person falling, or to a trajectory that proceeds with them doing a backflipand landing on their feet, but it will not belong to a trajectory of a person swimming.
Therefore,we represent the segment as a distribution over trajectories, where Q represents the likelihood of atrajectory given the segment.
During training, the goal is to learn this mapping from the input space(segments of trajectories) to the latent space (distributions over trajectories).
Concretely, given two segments s a , s b that have been obtained from the same underlying trajectory,we want some z to exist such that its likelihood under the distributions Q a and Q b representing eachof the segments is high.
To encourage this, we train the model to maximize the overlap betweenthe distributions Q a and Q b .
Similarly, we minimize the overlap between (the distribution representations of) segments sampled from different trajectories, under the assumption that no trajectoryz exists that contains both segments.
Specifically, we minimize a self-supervised triplet loss:
[Equation]
[Equation] where α is a margin hyperparameter, and T is a set of triplets: for every segment i in the dataset, wedefine several triplets by sampling pairs consisting of a positive segment k + (such that ( i, k + ) is apositive pair) and a negative segment k − (such that ( i, k − ) is a negative pair).
[Figure]
Latent [Figure] In addition to learning representations of trajectories, we also wish to be able decode them back toinput-space points.
To achieve this, we train a decoder Φ that allows us to obtain the specific valueof any trajectory at any continuous time t .
In order to train the decoder Φ , we sample trajectoriesz ∼ Q ( · ; s ) from each segment representation, and decode them at specific time-steps t which werecontained in s , obtaining a prediction ˆ x t =
Φ( z, t ) for which we have ground truth x t .
There is nouncertainty in this prediction, as x t was part of the segment s in the first place; the decoder is onlyexplicitly distance δ .
Note we never explicitly define a trajectory-level distance in the inputspace; it is implicitly learned by the model.
The reconstruction loss is mathematically defined as:
[Equation]
where N is the number of segments in the dataset, and T i is the number of points in segment s i .
Weminimize Eqs.
(1) and (2) jointly and end-to-end.
We implement the encoder Θ using a Transformer
Encoder architecture [54], and the decoder Φ using a ResNet [17].
See Appendix C for more details.
2.3 Creating Positive and Negative Pairs
In order to define positives and negative pairs for Eq.
(1), we use the following
• Input-space relationships The simplest way is to take segments from the same trajectory aspositives and segments from other random trajectories as negatives. The initial segments can have different relationships, such as precedence, containment, or overlap [3].
In our experiments, we sample three segments for every trajectory: a past segment ( P in Fig. 3), a future segment ( F )whose starting time comes right after the end of the past segment, and a combination segment ( C ),which contains both the past and the future segments.
• Intersection An intersection I of two distributions Q in the latent space will represent all the trajectories that have a high likelihood for both intersected segments.
Note that an intersection in the latent space is a union in the input space: the intersection constrains the possible trajectories to those that are consistent simultaneously for the two segments.
Similarly, an intersection in the input space (assuming an overlap between segments) is a union in the latent space. In the latent space, the intersection of the past and future segments should be equal to the representation of the
combination segment, and therefore the pair ( C , I ) is a positive one. • Re-encoding Given a trajectory z , we can decode it into any set of times t , obtaining a new segment. This segment can be (re-)encoded using Θ , and a representation Q can be obtained for
it, resulting in a new positive or negative for other segment representations.
For example, when
PF
PF PF
[Figure]
P
CP CP
C
C FP
FP
CF CF
O O
F F
Input Space
Latent Space
Legend
P
F
C
FP
CP
CF
O
Past
Future
Combination
Future given past Past given future
Combination given past
Combination given future
Other given the past we randomly sample a possible the future, the resulting segment ( FP - future givenpast ) will be different than the ground truth future, so the pair ( F , FP ) will be treated as a negative.
We exemplify a combination of these possibilities in Fig.
In order to determine which pairs ofsegments are positive, and which are negative, the rule is always the same: if they can belong to thesame trajectory they are positives, otherwise they are negatives.
For example, looking at Fig. 3 itis clear that, as discussed above, there is no trajectory that can contain both F and FP . We list allnegative and positive pairs in Appendix C.1.
2.4 Comparing Distributions
[Figure]
Eq.
(1) uses the distance function D to compare distributions of trajectories.
In this section, weintroduce two different ways of designing D , resulting in different intuitions about the latent space.
Symmetric Distance
If two segments can belong to the same trajectory, the distributions Qrepresenting each segment should be similar and close to each other (positives), and the (symmetric)distance D between them should be small.
For example, in Fig.
3, the representations of the past Pand future F segments belonging to the same trajectory are treated as positives.
Conditional Instead of computing a distance or a similarity, we compute the probability that asegment s a belongs to the same trajectory as another segment s b .
We model this as a conditionalprobability P ( Q a | Q b ) .
There are four possibilities:
P ( Q a | Q b ) = 1 , when s b includes s a , like the combination segment C including the past P .
2. 0 < P ( Q a | Q b ) < 1 , when Q a is possible but not necessary given by Q b , like P and F in Fig.
3. 0 < P ( Q b | Q a ) < 1 , defined in a similar manner.
P ( Q a | Q b ) = 0 , for unrelated segments, like C and O .
We treat the first three cases as positives, and the last case as a negative.
Because pairs belonging tothe first case have a stricter correspondence than those belonging to the second and third cases, wesample them more often during training.
Note that under this interpretation, a past P and a future Ffrom the same trajectory do not have a strong correspondence (first case), but a softer one (secondand third cases): one does not fully define the other.
This approach results in probability values thatwe either maximize (positives) or minimize (negatives), so we define D ( A, B ) = 1 − P ( A | B ) .
The previous approaches require a way of computing either a distance between the distributions Q , ora conditional probability between them.
In the next section, we show two families of distributions forwhich these can be defined.
2.5 Trajectory Segments as Distributions
In order to obtain Q , the encoder Θ predicts the parameters of a distribution family.
Conditions forthe distribution families are: 1) we can sample from it in a differentiable way, 2) we can parameterizeit, 3) we can compute, in closed form, an intersection that returns a distribution from the same family,and 4) we can compute either a similarity function or a conditional probability, or both (see Sec.2.4).
Next, we introduce two distribution families that meet the previous criteria.
Normal distributions We use uncorrelated multivariate normal distributions, and parameterizethem with a mean µ and a standard deviation σ .
We compute the intersection as the product of twonormal distributions, which remains normal when the dimensions are uncorrelated (see Appendix D).
We use the symmetrized Kullback-Leibler (KL) divergence between distributions as a distancefunction.
This distance is not a proper metric; alternatives are discussed in Appendix D. Normaldistributions assume an underlying Euclidean distance metric d between trajectories z .
Box embeddings Box embeddings [55] represent objects with high-dimensional products-ofintervals (or boxes), parameterized by their two extreme vertices z ∧ and z ∨ .
The intersection between box embeddings is well defined and results in another box embedding.
This makes them anatural choice to represent conditional probabilities, which can be computed as P ( A | B ) = Vol( A ∩ B ) /
Vol( B ) , where Vol( A ) = (cid:81) Ni max( z ∨ i − z ∧ i , 0) is the volume of the box, and ∩ represents theintersection operation.
These operations are straightforward to compute.
Boxes are not actual distributions, as they need not integrate to one.
However, they are easily normalized by dividing by theirvolume, and therefore they can be treated as distributions for all the practical purposes required in ourframework ( i.e. sampling, where we approximate the boxes with a uniform distribution).
Symmetricdistance functions can also be defined on box embeddings; we define a few in Appendix D.2.
In both cases, we use the reparameterization trick [24] in order to sample from the distributions whilekeeping gradient information.
We found the best-performing option was using box embeddings underthe conditional scenario; the values reported in Section 3.2 use this setting.
2.6 Inference
Once trained, our decoder Φ is able to decode a trajectory at any continuous time t , including timesthat were not part of the input.
For example, our framework can decode a future segment given aninput past segment, by sampling from its representation, and evaluating that sample at some futuretimes.
This future segment will not necessarily be equal to the ground truth future segment (in case itexists), because a single past can have multiple futures.
Overall, our framework is capable of doing 1) future and past prediction, by decoding a segment attimes outside of its range; 2) continuous reconstruction given a discrete input, by decoding at anycontinuous time t ; 3) interpolation between two segments, by decoding trajectories in their latentspace intersection; and 4) modifying existing trajectories, by manipulating the latent space.
All theprevious tasks are possible without explicitly training to do any of them.
We show examples in Sec. 3.
Experiments
3.1 Datasets
For our experiments, we selected data adhering to the following criteria.
First, there has to beuncertainty in the trajectory when given just a segment (for instance, the future is not fully specifiedgiven the past).
Second, the prediction should not require external contextual information.
Context can
[Table]
VRNN [13]
[Table]
Trajectron++ uni.
[Table]
Trajectron++ [44]
[Table]
(ours, ablation)
[Table]
+ re-encoding (ours)
[Table]
VRNN [13]
[Table]
Trajectron++ uni.
[Table]
Trajectron++ [44]
[Table]
[Table]
, ablation)
[Table]
be seamlessly added to our architecture, but it involves additional task-specific engineering decisions,and we want our evaluation to be orthogonal to them. Similarly, we avoid trajectories that requirehighly-engineered point-level distances δ .
Finally, we prefer our trajectories to be obtained from realworld data.
For all the previous reasons, we implement our framework on human movement datasets .
Specifically, we extract keypoints from human action datasets using OpenPose [10].
For every video,we keep the most salient human trajectories.
This results in sequences of dimension [ L, 25 , 2] , where
L is the number of frames in the trajectory, 25 is the number of joints in a human skeleton extractedby OpenPose, and 2 corresponds to the number of spatial coordinates for every joint.
We refer to thewhole skeleton at every time-step —the combination of all joints, resulting in a K = 50 -dimensionalvector— as a point .
As a distance function δ between points ( i.e. skeletons) we use l 2 -norm distanceper-joint, and average across all visible joints.
We extract human movement trajectories from the
FineGym [45], Diving48 [29] and FisV [57] datasets, which correspond to gymnastics, diving andfigure skating, respectively.
For each of the datasets, we experiment with short sequences (up to 10 time-steps, or slightly over onesecond) and long ones (up to 30 time-steps, representing slightly under four seconds of the trajectory),and report results for both.
We provide more details on the dataset creation in Appendix B.
3.2 Quantitative Experiments
Baselines and ablations As baselines, we select trajectory-modeling methods that are capable ofencoding uncertainty about the future.
Variational RNNs [13] extend recurrent neural networks(RNNs) to the non-deterministic case, by modeling every step with a variational auto-encoder (VAE)[24].
Trajectron++ [44] is a state-of-the-art trajectory-modeling framework which also builds on topof RNNs and (conditional) VAEs [23].
Uncertainty is modeled as a Gaussian mixture model (GMM).
We adapt Trajectron++ to our data, making the encoding and decoding as similar to our setting aspossible (for fairness), while keeping the core of the framework intact.
We train two Trajectron++versions, one with uniformly-sampled inputs and outputs (“Trajectron++ uni.”), and a second one [Figure]
Input
[Figure]
frames, 1 second
[Figure]
[Figure] [Figure]
with non-uniform sampling, following the setup in our models (“Trajectron++”). We also ablate ourmodel, and report results with and without training with re-encoded segments.
Tasks and metric We evaluate our framework on three different tasks: future prediction, pastprediction, and interpolation between two segments.
Future prediction consists in predicting pointsfrom a future segment given a past segment.
Past prediction is defined symmetrically.
In theinterpolation task, we input two separated segments (past and future) from a trajectory, and predictthe segment in between them.
In our model, we do so by decoding from the latent-space intersectionof the two input segments.
Baselines (which are autoregressive) are not capable of doing thiscombination, so we only use the past segment as input.
Baselines are also not capable of directlyperforming past prediction, so we predict the future of the reversed trajectory instead.
As a metric,we use the average of the l 2 -norm distance across joints, which is used by all methods during training,and report the best out of M = 10 samples, to account for multiple modes and uncertainty in theprediction.
Note that our model has never been explicitly trained to perform any of the previous tasks.
We show results in Tables 1a and 1b, for long and short trajectories respectively.
Our modeloutperforms baselines in all the tasks, which proves its value and flexibility.
We also show howcreating more interesting negatives with the re-encoding of decoded trajectories results in moreaccurate prediction results.
However, our method performs well even without re-encoding.
3.3 Qualitative Experiments
We show examples of our model’s inputs and outputs in Fig.
Specifically, we show future, past andinterpolation predictions.
The results reflect that the model learns to predict sequences up to fourtimes longer than the input.
They also show the large temporal resolution of our model: the modelpredictions evolve smoothly and sensibly for time-steps separated by a few hundredths of a second(Figs.
4a and 4b).
When not all joints are present in the input (first frame in Fig. 4a), our model is stillcapable of reconstructing the full spatial extent of the position.
Finally, note how the model can takeas inputs irregularly sampled time-steps (Fig. 4b), which makes it adaptable to temporal occlusions.
Baselines are only capable of predicting the future, and are restricted to predicting uniform time-steps.
The segments are directly tied to the temporal span they represent.
For example, two segments withthe exact same coordinates and evolution across time, but starting at different times will result indifferent (albeit similar) representations.
The speed of a movement and the time in the trajectorywhere the movement is done are important attributes of that movement, and the representation shouldnot be invariant to them: they belong to different trajectories.
However, because these trajectories arevery similar, our model learns to represent them close in the latent space.
In Fig.
5, we show that themodel encodes different temporal variations.
Moving along specific directions in the latent spaceresults in progressively faster trajectories (Fig. 5a), or in trajectories with an increasing temporaloffset with respect to the original one (Fig. 5b).
See Appendix E for details.
Related work
Modeling trajectories Spatial trajectories are usually modeled in the literature in an autoregressive(AR) fashion [47, 33, 26, 2, 49, 20, 44, 30, 58, 52, 25, 19], where trajectories are defined conditionedon previous time-steps.
Despite their success, AR models are incapable of dealing with some of thechallenges stated in Section 1, most notably they do not represent time as a continuous variable, theycannot model the full extent of a trajectory (simultaneously both past and future), and no learnedtrajectory-level metric can be obtained from them.
Some of them model the uncertainty in theprediction [52, 25, 19, 20, 44, 13, 49], and we use two representative ones [13, 44] as our baselines.
A different line of work is focused on representing segments of trajectories (not just points) as pointsin a latent space [62, 65, 60, 61, 28, 64, 9, 31].
However, they are not capable of modeling the fullextent of a trajectory outside the limits of the considered segment.
Additionally, the segment-levelmetric is either unstructured [62], or is explicitly given [65, 63, 64].
[Figure] Continuous time Modeling time as a continuous signal has gained traction recently in fields such asgraphics [56, 53, 39, 46] or physics modeling [8, 11], because it accurately represents the underlying(continuous) world being modeled.
In the graphics neural implicit functions literature, time is used tocondition the prediction of the network.
We adopt the same approach in our decoder.
We encode theset of continuous times in a segment by using a Transformer network [54], which by constructionis permutation invariant, but allows temporal embeddings to be concatenated with the input, bothdiscrete [5, 4] and continuous [53, 51].
Self-supervised representation learning Finding self-supervised representations for temporal datahas been the subject of a large amount of work in domains such as trajectories [31], video [35, 16, 40],or audio processing [21, 42, 15].
Most methods, however, represent segments as simple points in a Euclidean space.
Structured representations for temporal data [50, 36] allow the latent space to followcertain inductive biases, like our framework’s idea that segments compose trajectories.
We modelsegments as either normal distributions or box embeddings [55].
The latter have been used to representhierarchical relationships in domans such as text [38, 34], knowledge bases [1], or images [41].
Weuse them to represent temporal information.
In recent work, Park et al.
[36] also model segmentsusing normal distributions, where trajectories are weighted sums of the segment representations. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] , see Sections 2 and 3.
(b) Did you describe the limitations of your work?
[Yes] , we discuss them in Appendix A.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] , wediscuss them in Appendix A.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplementary material or as a URL)?
[Yes] , they are inthe supplementary materials.(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] , see Appendix C.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] , see Table 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] , see Appendix C.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] , we cite the datasetsbeing used.
(b) Did you mention the license of the assets?
[No] , the licenses are mentioned in the
respective dataset and library papers, which we cite.
(c) Did you include any new assets either in the supplementary material or as a URL?
[Yes] , data and code are provided.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]