[RE] It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
The following paper is a reproducibility report for It Is Not the Journey but the Destination: Endpoint Conditioned
Trajectory Prediction [3].
The basic code was made available by the author <link>.
To reproduce the rest of theablation studies mentioned in the paper, we had to modify the model structure accordingly.
The well-commented versionof the code containing all ablation studies performed derived from the original code is available at <link> with properinstructions to execute experiments in ReadMe.
Scope of Reproducibility
We have veriﬁed all claims made by the paper and results from different experiments mentioned in the paper to supportthe claims.
The central claim of PECNet was to improve state-of-the-art performance on the Stanford Drone trajectoryprediction benchmark by 20.9 percent and on the ETH/UCY benchmark by 40.8 percent.
Methodology
The PECNet model was trained on the drone dataset with social pooling at different conditioned points and on the ETH/UCY datasets without social pooling.
Results that were obtained matched with those claimed in the paper.
Furthermore, the trained model was evaluated on the drone dataset (with social pooling) at different values of evaluatedsamples (referenced as ’k’ in the paper).
For the latter, GitHub was used as a reference with author-given code.
Results
Overall, we were able to reproduce all the results mentioned in the paper within 5% error compared to what wasmentioned in the paper.
What was easy
Veriﬁcation of the claims against the ETH/UCY benchmarks and Stanford drone benchmark trajectory prediction withthe PECNet models was an easy task.
What was difﬁcult
For the datasets of ZARA1 and ZARA2, there were gaps in the sequence of frames, and thus interpolation wasdone to ensure the continuity of way-points.
This caused the ADE and FDE errors to increase.
Also, to maintaincommon frequency for all the datasets, they were down-sampled accordingly.
For the conditioned way-point positioningexperiment (with and without ORACLE) experiment, ADE had to be calculated from 11 predicted positions to not alterthe structure of the model, and FDE was also calculated from the 11th point.
However, due to it, some ADE ﬂuctuationsafter the sixth way-point (and later) were larger than the claimed results.
Similar ﬂuctuations were observed for FDE aswell, but the relative trends support the paper’s claim.
Communication with original authors
We have not contacted any of the original authors as all the results were reproduced satisfactorily
Introduction
The paper reproduced in this report aims to tackle multiple pedestrian trajectory predictions using rich multi-modalpredictions for the use of autonomous vehicles, social robots, etc.
Earlier approaches to this problem have beenauto-regressive in nature, i.e., using n points (or analogically, data from the last t seconds) from the dataset to producethe immediately next point, and then this process is recurred.
In this paper, the end-point distribution conditioned on the past trajectory and the past trajectory features are modeledseparately for each pedestrian.
The future trajectory points are predicted based on the past and features from otherpedestrians via social pooling.
An assumption in this model is the absence of passive pedestrians or the fact that eachpedestrian has an actual preconceived end-point or destination and is motivated to reach it.
To formulate this report, we have experimented on the author’s code by adding/removing social pooling layers, usingtruncation tricks, visualisation tools, and changing between CVAE and VAE architectures to verify all the claims madeby the author described in detail below.
We also performed some experiments such as shifting origin to the currentpoint, using different architecture for encoder and decoder networks with the hope of improving the results, which arealso described in detail at the end.
Scope of reproducibility
The paper revolves around the claim that an important component of predicting the trajectory is the destination in multitrajectory forecasting.
If the destination for the pedestrian is clear, then the trajectory can be easily resolved usinga separate network that takes the past trajectory and the destination as input taking into account social interactionsamong fellow pedestrians.
Hence the central idea and claim of the paper is to use Conditional Variational Auto Encoder(CVAE) to get the latent variable encoding conditioned on the destination from the ground truth, use the latent variableto infer the predicted destination, and use it for predicting the rest of the future trajectory.
We take k samples of thelatent variable for testing purposes to predict k different admissible trajectories as output for different destinationsderived from the latent encoding.
The overall reduction in the value of best ADE and FDE values for the Stanford Drone, ETH/UCY datasets by using the CVAE network is the central claim of the paper.
To support the argument that indeed given the destination, the rest of the predicted trajectory contributes much lesserror than the previous state of the art methods such as SGAN, which directly predict the future trajectory, the paperperforms an ablation study where they give the ground truth of a way-point which they call as oracle instead of thebest one from taking k samples of the latent variable to get the decoupled error of predicting the trajectory.
The resultsstrongly support the argument.
Further, they also experimented with different values of k to show that FDE tends to 0 as k increases and ADE tends toa certain value, which also shows the decoupled error in predicting the rest of the trajectory.
This paper also introduces a non-local social pooling layer and a “truncation-trick,” which improves diversity andmulti-modal trajectory prediction performance.
Hence the claims can be summarized as follows:-
Conditioning the destination on the past trajectory using CVAE helps in explicit decoupling of the destination prediction and path prediction errors.
It hence helps reduce the destination prediction error and the subsequent path prediction error.
Using the social pooling layer helps reduce the error in predicting the path given the history and the destination.
Using truncation trick i.e., truncating the distribution for fewer values of k from which samples are taken helpsreduce the destination prediction error.
Also, taking a higher sigma value for larger values of k reduces theerror.
Methodology
We used the GitHub repository provided by the author as the base.
However, it only contained the base model for resultson the drone data set.
In order to reproduce the rest of the experiments, we had to make changes accordingly.
3.1 Model descriptions
The model used in the paper consists of 2 parts:
First, the CVAE or Conditional Variational AutoEncoder to get the representation of the latent variable conditioned ondestination and given the past trajectory.
Second, the predictor network consists of social pooling layers and an MLP network to get the future trajectory.
A representative diagram of the network is given in ﬁgure 1 and the architecture parameters are shown in table 1.
[Equation] [Equation]
3.2 Datasets
We used Stanford Drone [5] and ETH [4] / UCY [2] data sets.
The Stanford drone data set was given in the author’scode, but ETH/UCY was not given, so we took the data set from an open source.
Network Architecture
E way 2 -> 8 -> 16 ->
E past 16 -> 512 -> 256 ->
E latent 32 -> 8 -> 50 ->
D latent 32 -> 1024 -> 512 -> 1024 - [Table] 3.3 Hyperparameters
We used Hyperparameters given in the paper. We occasionally changed them accordingly to perform the ablationstudies described below.
3.4 Experimental setup
We ran code in google colab with GPU (NVIDIA-SMI 450.36.06 Driver Version: 418.67 CUDA Version: 10.1 {+).3.
+}
Computational requirements
Typically, it took less than an hour to train the model both for the drone and ETH/UCY data sets.
Results
The following experiments/ablation studies support the claims made earlier.
A detailed description of the experimentsand their results to support the claim are listed below:-
4.1 Experiment on drone data set (with and without social pooling, truncation trick)
Stanford drone data set: We did it with social pooling and got results within 95% accuracy from claim results.
Thepreprocessed data set for train and test were given on GitHub (by author).
We used them to verify the results.
We didtwo experiments with n-samples 5 and another with n-samples 20 as required for reproducing the results in the ﬁrsttable of the paper.
[Table]
data set consists of 5 scenes eth hotel, univ zara1, zara2 extracted from another source <link because, in the paper, the source was not mentioned.
We Followed the conventional leave-one-out approach, i.e., trainedon 4 sets and tested on the last set to get the results.
We veriﬁed results within 98% accuracy from claimed results.
Thedata set was further downsampled by 6 to get a 0.4 second gap between consecutive frames as demanded by the paper.
The result is shown below in the table.
With these 2 experiments, the reduction in error with respect to the previousresults by using CVAE and subsequent reduction by using social pooling layer and truncation trick can be demonstrated.
[Table]
4.3 Change in the structure of CVAE
In this experiment during training, the ground truth Eend ( G k ) was used to predict the future T f instead of the oneobtained from the latent variable.
We did it on the Stanford drone dataset with social pooling and got results within 95%accuracy from the claim results.
ADE : 10.87 / 10.
FDE : 17.03 / 16.
4.4 Effect of Number of samples (K)
We did this experiment on the Stanford drone dataset with social pooling.
We trained the PECNet model with defaultsigma values and test on different k-sample value with and without truncation.
Without truncation for k-sample<=3 weused σ with variance 1 and for k-sample > 3 we used σ with variance 1.3.
With truncation for k-sample > 3 we used σwith variance 1 and for k-sample<=3 we used σ with variance c * √ k − 1 .
In this experiment we got results within 95accuracy from the claim results.
[Table]
4.5 Conditioned Way-point positions Oracles
In this experiment, we conditioned on future trajectory points other than the last observed point, which we refer to asway-points.
This was not clear in the paper about how to calculate FDE error because we can not predict last observedpoint in the model so we calculated FDE from the 11th point of the predicted trajectory.
It was done in two parts:1.
With oracle : During prediction of future trajectory (at time of testing and validation), we gave ground-truthvalue of conditioned point instead of the best guessed one from sampling to predict trajectory from the model.
The Stanford drone data set with social pooling and truncation trick was used to match with the results onpaper.2.
Without oracle : The same thing was done here except during prediction of the future trajectory the bestguess for the conditioned point(predicted by model) was taken (at time of testing and validation).
Way-point Prediction Error was calculated as difference between ground truth of conditioned point and the one predictedby the model.
4.6 Reference shift <link>[1] (Extra experiment)
We took the reference of the trajectory for each pedestrian as the current point instead of the ﬁrst point of the pasttrajectory.
This helped the CVAE network to get a better representation of the destination point as all input past
ADE 18.
FDE 35.
Way-point error 4.
Oracle ADE 18.
Oracle FDE 35.
trajectories have a common last point, which makes it easier for the encoder and decoder network to function; also, thepredictor and social pooling network gets more easily trained.
This showed about 8% further decrease in ADE and FDEmetrics for drone dataset as follows:-
ADE : 8.
FDE : 14.644.7 Using encoder and decoder LSTM network <
link>[1] (Extra experiment)
We used encoder LSTM instead of MLP to form the encoding of the past trajectory to accommodate variable length ofpast trajectory and form a better representation as to the input temporal data.
Also, we used the decoder LSTM networkto predict the rest of the trajectory given the destination.
However, the FDE error reduced by about 5 %, but the ADE issurprisingly more, demonstrating that decoder LSTM does not perform well given the destination point.
ADE : 26.
FDE : 14.
Discussion
From each of the experiments, the claims made by the paper as described above can be strongly supported andempirically proved.
Strong correspondence between destination and rest of the path is observed as evident from theresults in comparison to previous experiments.
Also, use of social pooling layer and truncation trick, reduce the errorto a great extent as demonstrated from the ablation studies described above.
In order to further study the choice ofstructure of the network, 2 other experiments were performed described above and they strongly support the choice of MLP architecture used for past encoding, future prediction instead of LSTM/GRU RNN structures.