[RE] It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
The following paper is a reproducibility report for It Is Not the Journey but the Destination: Endpoint Conditioned
Trajectory Prediction [5].
The basic code was made available by the author at this https url.
To reproduce the rest ofthe ablation studies mentioned in the paper, we had to modify the model structure accordingly.
The well-commentedversion of the code containing all ablation studies performed derived from the original code will be made availablewith proper instructions to execute experiments in ReadMe
Scope of Reproducibility
We have veriﬁed all claims made by the paper and results from different experiments mentioned in the paper to supportthe claims.
The central claim of PECNet was to improve state-of-the-art performance on the Stanford Drone trajectoryprediction benchmark by 20.9% and on the ETH/UCY benchmark by 40.8%.
Methodology
The PECNet model was trained on the drone dataset with social pooling at different conditioned points and on the
ETH/UCY datasets without social pooling.
Furthermore, the trained model was evaluated on the drone dataset atdifferent values of evaluated samples.
For the latter, GitHub was used as a reference with author-given code.
Results
Overall, we were able to reproduce all the results mentioned in the paper within 5% error compared to what wasmentioned in the paper. 5% error is quite acceptable for this application, and this variation could be caused by settingthe initial random seed before training
What was easy
Veriﬁcation of the claims against the ETH/UCY benchmarks and Stanford drone benchmark trajectory prediction withthe PECNet models was an easy task.
What was difﬁcult
For the datasets of ZARA1 and ZARA2, there were gaps in the sequence of frames, and thus interpolation wasdone to ensure the continuity of way-points.
This caused the ADE and FDE errors to increase.
Also, to maintaincommon frequency for all the datasets, they were down-sampled accordingly.
For the conditioned way-point positioningexperiment (with and without ORACLE), ADE had to be calculated from 11 predicted positions to not alter the structureof the model, and FDE was also calculated from the 11th point.
However, due to it, some ADE ﬂuctuations after thesixth way-point (and later) were larger than the claimed results.
Similar ﬂuctuations were observed for FDE as well, butthe relative trends support the paper’s claim.
Communication with original authors
We have not contacted any of the original authors as all the results were reproduced satisfactorily
Introduction
The paper reproduced in this report aims to tackle multiple pedestrian trajectory predictions using rich multi-modalpredictions for the use of autonomous vehicles, social robots, etc.
Earlier approaches to this problem have beenauto-regressive in nature, i.e., using n points (or analogically, data from the last t seconds) from the dataset to producethe immediately next point, and then this process recurs.
In this paper, the endpoint distribution conditioned on the past trajectory and the past trajectory features are modeledseparately for each pedestrian.
The future trajectory points are predicted based on the past and features from otherpedestrians via social pooling.
An assumption in this model is the absence of passive pedestrians or the fact that eachpedestrian has an actual preconceived endpoint or destination and is motivated to reach there.
To formulate this report, we have experimented on the author’s code by adding/removing social pooling layers, usingtruncation tricks, visualization tools, and changing between CVAE and VAE architectures to verify all the claims madeby the author described in detail below.
We also performed some experiments such as shifting origin to the currentpoint, using different architecture for encoder and decoder networks with the hope of improving the results, which arealso described in detail at the end.
Scope of reproducibility
The paper revolves around the claim that an important component of predicting the trajectory is the destination in multitrajectory forecasting.
If the destination for the pedestrian is clear, then the trajectory can be easily resolved usinga separate network that takes the past trajectory and the destination as input taking into account social interactionsamong fellow pedestrians.
Hence the central idea and claim of the paper is to use Conditional Variational Auto Encoder(CVAE) to get the latent variable encoding conditioned on the destination from the ground truth, thus using the latentvariable to infer the predicted destination, and also using it for predicting the rest of the future trajectory.
We take ksamples of the latent variable for testing purposes to predict k different admissible trajectories as output for differentdestinations derived from the latent encoding.
The overall reduction in the value of best ADE (Average Displacement
Errors) and FDE (Final Displacement Error) values for the Stanford Drone, ETH/UCY datasets by using the CVAEnetwork is the central claim of the paper.
To support the argument that indeed given the destination, the rest of the predicted trajectory contributes much lesserror than the previous state of the art methods such as SGAN [3], which directly predict the future trajectory, the paperperforms an ablation study where they give the ground truth of a way-point which they call as oracle instead of thebest one from taking k samples of the latent variable to get the decoupled error of predicting the trajectory.
The resultsstrongly support the argument.
Further, they also experimented with different values of k to show that FDE tends to 0 as k increases and ADE tends toa certain value, which also shows the decoupled error in predicting the rest of the trajectory.
This paper also introduces a non-local social pooling layer and a "truncation-trick," which improves diversity andmulti-modal trajectory prediction performance.
Hence the claims can be summarized as follows:-
Conditioning the destination on the past trajectory using CVAE helps in explicit decoupling of the destination
prediction and path prediction errors.
It hence helps reduce the destination prediction error and the subsequent
path prediction error.
Using the social pooling layer helps reduce the error in predicting the path given the history and the destination.3.
Using truncation trick, i.e., truncating the distribution for fewer values of k from which samples are taken helps reduce the destination prediction error.
Also, taking a higher sigma value for larger values of k reducesthe error.
Methodology
We used the GitHub repository provided by the author as the base.
However, it only contained the base model for resultson the drone dataset.
In order to reproduce the rest of the experiments, we had to make changes accordingly.
3.1 Model descriptions
The base model used in the paper consists of 2 parts: [Equation]
Starting with the past trajectory, the CVAE or Conditional Variational Auto Encoder part is used to get the representationof the latent variable conditioned on destination. The past trajectory after ﬂattening is passed through an E past layer toget the past encoding.
During training, the ground truth ﬁnal destination is passed through the E end layer to get thedestination encoding.
The past and the destination encoding are concatenated and passed through the E latent layerto get the latent encoding distribution with dimension R n × 2 z dim (Where n is the no of vehicles in the batch and z dimis the hyperparameter denoting the size of the latent encoding) which characterize mean and variance of the latentencoding.
At this stage, a latent encoding is sampled from this distribution and passed through the D latent layer to getthe destination.
Second, the predictor network consists of social pooling layers and an MLP network to get the future trajectory.
Thepredicted destination is concatenated with the past encoding and the absolute current position of the pedestrian withrespect to a common global reference frame for all pedestrians.
This concatenated encoding is passed through a seriesof social pooling layers which contain g, ψ and θ networks masked by the social mask at each step to get the ﬁnal futureencoding.
This future encoding is passed through the P future network to get the future trajectory with t f time steps.
The social mask is represented as a binary matrix M ∈ R n × n where n is the no. of vehicles in the batch.
The value (i,j)is 1 in the matrix M if the i th and the j th vehicle come close to each other with a threshold distance d in at least one ofthe time frames from their past trajectories for the frames they are observed.
Refer to (3) where F ( . ) denotes the framenumber for that position. The loss function used to train the model is given in (4). It consists of 3 terms.
This ﬁrst term is the KL divergenceterm to bring the distribution of the latent variable close to the required one, which is N (0 , 1) .
The second term is thereconstruction loss from CVAE, called the Average Endpoint Loss (AEL), and the last term is the Average Trajectory
Loss (ATL), calculated as the sum of L2 losses between each of the predicted and ground truth future trajectory point.
The metric used for validation and testing is ADE and FDE.
ADE is the Average Displacement Error and is calculatedas the average of euclidean distances at all future time steps between predicted and ground-truth positions.
While FDEis the Final Displacement Error (FDE), and it is the euclidean distance between the ﬁnal predicted and ground truthpositions of the future trajectory.
Refer to eqn 1 and 2 for mathematical formulation of ADE and FDE.
Here ˆ u t refers topredicted trajectory position at time t, and u t is the ground truth trajectory position at time t.
A representative diagram of the network is given in Figure 1 and the architecture parameters for all the networks areshown in Table 1.
[Equation] [Table]
M [ i, j ] =  
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
otherwise (3)
[Equation]
3.2 Datasets
We used Stanford Drone [7] and ETH [6] / UCY [4] datasets.
The Stanford drone dataset was given in the author’scode, but ETH/UCY was not available.
We took the processed ETH/UCY dataset from this https url which is availablefor open source use.
3.3 Hyperparameters
We used Hyperparameters given in the paper.
We occasionally changed them accordingly, as mentioned in the paper, toperform the ablation studies described below.
Mainly, the hyperparameters are σ : The variance used for sampling latentvariable with mean 0, K: The no of guesses of ﬁnal destination to make, z dim : The size of latent encoding, and themodel hyperparameters as explained above in model descriptions are summarized in Table 1.
3.4 Experimental setup
We used PyTorch to ﬂuently conduct the aforementioned experiments. We extensively used Weights & Biases[1] forlogging the experiments. For proper reproducibility, even after changing the machines, we set 42 as a system-wide seedbefore running every experiment. This helps in [-google colab with GPU (NVIDIA-SMI 450.36.06 Driver Version: 418.67 CUDA Version: 10.1 ).3.
-]reproducing the exact results that are mentioned in the report.
3.5 Computational requirements
The proposed model can be trained on a single NVIDIA-K80 with 12 GB memory in less than an hour for boththe Stanford Drone and ETH/UCY datasets. We were able to execute the experiments smoothly on Google Colab. Speciﬁcations of the machine are as follows: NVIDIA-K80 GPU, Memory
[Equation] 12 GB, Memory Clock [Equation] .82 GHz, Driver
{+Version: 418.67, CUDA Version: 10.
  Results+} The following experiments/ablation studies support the claims made earlier.
The results are within 5% error from theones claimed in the paper.
We believe this much tolerance is acceptable in the context of this problem as results changeby this variance on changing the random initial seed.
A detailed description of the experiments and their results tosupport the claim are listed below:-
4.1 Experiment on the Stanford Drone Dataset (with and without social pooling, truncation trick)
The original model with hyperparameters, as mentioned in the paper, was trained and tested on the Standard Drone
Dataset (SDD) [7]. The train-val-test set split is the standard split as described in [2], which is to preserve some scenesfor test and validation and use others for training. We did it with social pooling and got results within 95% accuracyfrom claim results.
The preprocessed data set for train and test were given on GitHub (by author).
We used them toverify the results.
Also, the truncation trick here refers to that σ hyperparameter (Refer to hyperparameters section) isused as c √ K − 1 for K > 2 where c is a constant.
The resultant distribution is truncated with | z | < 1 for sampling,meaning the sampling is done from a conditional Normal distribution conditioned on | z | < 1 .
Hence, the resultingsampled z from this distribution will always have | z | < 1 . We did two experiments with hyperparameters, changingn-samples to 5 and another with n-samples to 20 as required for reproducing the results in the ﬁrst table of the paper.
[Table]
ETH/UCY dataset consists of 5 scenes ETH, Hotel, Univ, Zara1, Zara2 extracted coordinates.
We followed theconventional leave-one-out approach, i.e., trained on 4 sets and tested on the last set to get the results as was mentionedin the original paper.
We veriﬁed results within 98% accuracy from claimed results.
The occasional differences areunderstandable as the author did not mention the initial random seed, due to which there are small variations fromclaimed results which are understandable as they are within a 2% bound from the claimed results.
The dataset wasfurther down-sampled by 6 to get a 0.4 second gap between consecutive frames as demanded by the paper.
The result isshown below in Table 3.
With these two experiments, the reduction in error with respect to the previous results by using CVAE and subsequent reduction by using social pooling layer and truncation trick can be demonstrated.
[Table]
4.3 Change in the structure of CVAE
In this experiment during training, the ground truth destination ( G k ) was used to predict the future T f instead of theone obtained from the latent variable during training. Hence the changes inside the code were to pass the ground truth
G k and pass it to the social pooling layer during training.
Hence, in this experiment, the training of the CVAE and thepredictor networks are done separately decoupled from each other.
Results of this experiment, as shown in Table 4,demonstrate that training on the latent encoding helps in coupling both parts of the network and improves the results.
This newly trained network was tested on the Stanford drone dataset with social pooling, and we got results within 95%accuracy from the [-claim results. ADE : 10.87 / 10.
 FDE : 17.03 / 16.
-]claimed results; again, the variation though very small, is due to the initial random seed difference.
[Table] 4.4 Effect of Number of samples (K)
We did this experiment on the Stanford drone dataset with social pooling.
We trained the PECNet model with default σvalues of the CVAE and test on different k-sample values with and without truncation.
Speciﬁcally, experiments wereperformed with changes in the hyperparameter σ without truncation for k-sample <= 3, we used σ with variance 1 andfor k-sample > 3 we used σ with variance 1.3.
When using truncation trick for k-sample > 3, we used σ with variance 1and for k-sample < = 3 we used σ with variance c * √ k − 1 as mentioned in the paper.
In this experiment, we got resultsas shown in Table 5, within 95% accuracy from the claimed results with differences albeit small due to the differedrandom seed as it was not mentioned in the paper and with the same trend.
[Table]
[Table]
& Oracles
In this experiment, we conditioned on future trajectory points other than the last observed point, which we refer to asway-points.
This was not clear in the paper about how to calculate FDE error because we can not predict the destinationpoint according to the model architecture. We calculated FDE from the L2 difference between the last point of thepredicted trajectory, as the ﬁnal destination prediction is not available for this experiment. The observed result trendsmatch exactly with the proposed results in the paper.
It was done in two parts exactly as mentioned in the paper:1.
With oracle : During prediction of the future trajectory (at time of testing and validation), we gave ground-truthvalue of conditioned point instead of the best guessed one from sampling to predict trajectory from the model.
The Stanford drone data set with social pooling and truncation trick was used to match with the results onpaper.
With this experiment, it can be demonstrated that the errors in destination prediction and path prediction given the destination can be decoupled from each other.
Hence using CVAE for inference on the destination asthe ﬁrst part helps in improving the results.2. Without oracle : The same thing was done here, except during prediction of the future trajectory, the best guessfor the conditioned point (predicted by the model) was taken (at time of testing and validation).
Way-point Prediction Error was calculated as the difference between the ground truth of the conditioned point and the onepredicted by the model.
With this experiment, it can be empirically established that we get less inference errorby conditioning on the destination point rather than on any of the intermediate points on its future trajectory.
ADE 18.
FDE 35.
Way-point error 4.
Oracle ADE 18. Oracle FDE 35.
This is an extra experiment that we performed. The motivation behind this experiment is that the past trajectories passedas input are reference shifted with respect to the starting point of the past trajectory. We believe this would make itdifﬁcult for the social pooling layer to consider the social interaction impact on the future trajectory as their currentposition will be based on different reference frames for all the neighboring pedestrians on which the interaction wouldmostly depend.
Instead, we experimented with setting the reference frame for past trajectory as the current positioninstead. Hence, passing the global current position and the relative past trajectory with respect to that frame for allpedestrians will be easier to learn from for the social pooling layer as social forces as it will be easy to infer globalpositions of all pedestrians at past frames from this setup. We took the reference of the trajectory for each pedestrian as the current point instead of the ﬁrst point of the pasttrajectory. This helped the CVAE network to get a better representation of the destination point as all past inputtrajectories have a common last point, which makes it easier for the encoder-decoder network to function; also, thepredictor and social pooling network gets more easily trained.
This experiment showed about 10% further decrease in ADE and FDE metrics for drone dataset as [-follows:- ADE : 8.
-]shown in the Table 7.
4.7 Using encoder and decoder LSTM network (Extra experiment)
The motivation behind this experiment is that using MLP for encoding the past trajectory and predicting the futuretrajectory won’t implicitly leverage the advantage of the sequential nature of the past trajectory and future trajectory. Training using MLP is suitable for inputs with less and ﬁxed history time frames, but would be difﬁcult to tune for [Table]
larger history sizes. Hence, we experimented by using an LSTM network instead. This would also help to considervariable lengths of the past and future trajectory based on the requirement. We used encoder LSTM instead of MLP to form the encoding of the past trajectory to accommodate the variable lengthof past trajectory and form a better representation as to the input temporal data.
Also, we used the decoder LSTMnetwork to predict the rest of the trajectory given the destination.
However, the FDE error reduced by about 5%, but the ADE is surprisingly more, demonstrating that decoder LSTM does not perform well given the destination point.
[-ADE : 26.
 FDE : 14.
-][Table]
Discussion
From each of the experiments, the claims made by the paper as described above can be strongly supported andempirically proved.
The strong correspondence between destination and rest of the path is observed, as evident from theresults in comparison to previous experiments.
Also, the use of the social pooling layer and truncation trick reduces theerror to a great extent, as demonstrated from the ablation studies described above.
In order to further study the choice ofstructure of the network, two other experiments were performed described above, and they strongly support the choiceof MLP architecture used for past encoding future prediction instead of LSTM/GRU RNN structures.