Reproducibility report for It Is Not the Journey but the Destination
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
The following paper is a reproducibility report for t is Not the Journey but the Destination [3].
Thebasic code was made available by the author via the GitHub repository <link>.
To reproduce therest of the ablation studies mentioned in the paper, we had to modify the model structure accordingly.
The well-commented version of the code containing all ablation studies performed derived from theoriginal code is available at <link> with proper instructions to execute experiments in ReadMe.
Scope of Reproducibility
We have veriﬁed all claims made by the paper and results from different experiments mentionedinside the paper to support the claims.
The central claim of PECNet was to improve state-of-the-artperformance on the Stanford Drone trajectory prediction benchmark by 20.9 percent and on the
ETH/UCY benchmark by 40.8 percent, which has been veriﬁed to be true.
Methodology
The PECNet model was trained on the drone dataset with social pooling at different conditionedpoints and on the ETH/UCY datasets without social pooling.
Results that were obtained matchedwith those claimed in the paper.
Furthermore, the trained model was evaluated on the drone dataset(with social pooling) at different values of evaluated samples (referenced as ’k’ in the paper).
For thelatter, GitHub was used as a reference with author-given code.
Results
Overall, we were able to reproduce all the results mentioned in the paper within 5% error comparedto what was mentioned in the paper.
What was easy
Veriﬁcation of the claims against the ETH/UCY benchmarks and Stanford drone benchmark trajectoryprediction with the PECNet models was an easy task.
What was difﬁcult
For the datasets of ZARA1 and ZARA2, there were gaps in the sequence of frames, and thusinterpolation was done to ensure the continuity of way-points.
This caused the ADE and FDE errorsto increase.
Also, to maintain common frequency for all the datasets, they were down-sampledaccordingly.
For the conditioned way-point positioning experiment (with and without ORACLE)experiment, ADE had to be calculated from 11 predicted positions to not alter the structure of themodel, and FDE was also calculated from the 11th point.
However, due to it, some ADE ﬂuctuations after the sixth way-point (and later) were larger than the claimed results.
Similar ﬂuctuations were observed for FDE as well but the relative trends support the paper’s claim
Communication with original authors
We have contacted any of the original authors all the results were reproduced satisfactorily
Introduction
The paper reproduced in this report aims to tackle multiple pedestrian trajectory predictions using richmulti-modal predictions for the use of autonomous vehicles, social robots, etc.
Earlier approachesto this problem have been auto-regressive in nature, i.e., using n points (or analogically, data fromthe last t seconds) from the dataset to produce the immediately next point, and then this process isrecurred.
In this paper, the end-point distribution conditioned on the past trajectory and the past trajectoryfeatures are modeled separately for each pedestrian.
The future trajectory points are predicted basedon the past and features from other pedestrians via social pooling.
An assumption in this model is theabsence of passive pedestrians or the fact that each pedestrian has an actual preconceived end-pointor destination and is motivated to reach it.
To formulate this report, we have experimented on the author’s code by adding/removing socialpooling layers, using truncation tricks, visualisation tools, and changing between CVAE and VAEarchitectures to verify all the claims made by the author described in detail below.
We also performedsome experiments such as shifting origin to the current point, using different architecture for encoderand decoder networks with the hope of improving the results, which are also described in detail at theend.
Scope of reproducibility
The paper revolves around the claim that an important component of predicting the trajectory isthe destination in multi trajectory forecasting.
If the destination for the pedestrian is clear, thenthe trajectory can be easily resolved using a separate network that takes the past trajectory and thedestination as input taking into account social interactions among fellow pedestrians.
Hence thecentral idea and claim of the paper is to use Conditional Variational Auto Encoder (CVAE) to get thelatent variable encoding conditioned on the destination from the ground truth, use the latent variableto infer the predicted destination, and use it for predicting the rest of the future trajectory.
We takek samples of the latent variable for testing purposes to predict k different admissible trajectories asoutput for different destinations derived from the latent encoding.
The overall reduction in the valueof best ADE and FDE values for the Stanford Drone, ETH/UCY datasets by using the CVAE networkis the central claim of the paper.
To support the argument that indeed given the destination, the rest of the predicted trajectory contributes much less error than the previous state of the art methods such as SGAN, which directlypredict the future trajectory, the paper performs an ablation study where they give the ground truthof a way-point which they call as oracle instead of the best one from taking k samples of the latentvariable to get the decoupled error of predicting the trajectory.
The results strongly support theargument.
Further, they also experimented with different values of k to show that FDE tends to 0 as k increasesand ADE tends to a certain value, which also shows the decoupled error in predicting the rest of thetrajectory.
This paper also introduces a non-local social pooling layer and a “truncation-trick,” which improvesdiversity and multi-modal trajectory prediction performance.
Hence the claims can be summarized as follows:-
Conditioning the destination on the past trajectory using CVAE helps in explicit decouplingof the destination prediction and path prediction errors.
It hence helps reduce the destinationprediction error and the subsequent path prediction error.2.
Using the social pooling layer helps reduce the error in predicting the path given the historyand the destination.3.
Using truncation trick i.e., truncating the distribution for fewer values of k from whichsamples are taken helps reduce the destination prediction error.
Also, taking a higher sigmavalue for larger values of k reduces the error.
Methodology
We used the GitHub repository provided by the author as the base.
However, it only contained thebase model for results on the drone data set.
In order to reproduce the rest of the experiments, we hadto make changes accordingly.
3.1 Model descriptions
The model used in the paper consists of 2 parts:
First, the CVAE or Conditional Variational AutoEncoder to get the representation of the latent variableconditioned on destination and given the past trajectory.
Second, the predictor network consists of social pooling layers and an MLP network to get the futuretrajectory.
A representative diagram of the network is given in ﬁgure 1 and the architecture parameters are shownin table 1.
[Equation]
[Equation]
3.2 Datasets
We used Stanford Drone [5] and ETH [4] / UCY [2] data sets.
The Stanford drone data set was givenin the author’s code, but ETH/UCY was not given, so we took the data set from an open source.
3.3 Hyperparameters
We used Hyperparameters given in the paper.
We occasionally changed them accordingly to performthe ablation studies described below.
Network Architecture
E way 2 -> 8 -> 16 ->
E past 16 -> 512 -> 256 ->
E latent 32 -> 8 -> 50 -> D latent 32 -> 1024 -> 512 -> 1024 ->θ, Φ 32 -> 512 -> 64 ->g 32 -> 512 -> 64 -> P predict 32 -> 1024 -> 512 -> 256 -> 3.4 Experimental setup
We ran code in google colab with GPU (NVIDIA-SMI 450.36.06 Driver Version: 418.67 CUDA
Version: 10.1 ).
3.5 Computational requirements
Typically, it took less than an hour to train the model both for the drone and ETH/UCY data sets.
Results
The following experiments/ablation studies support the claims made earlier.
A detailed description ofthe experiments and their results to support the claim are listed below:-
4.1 Experiment on drone data set (with and without social pooling, truncation trick)
Stanford drone data set: We did it with social pooling and got results within 95% accuracy fromclaim results.
The preprocessed data set for train and test were given on GitHub (by author).
We usedthem to verify the results.
We did two experiments with n-samples 5 and another with n-samples 20as required for reproducing the results in the ﬁrst table of the paper.
[Table]
-S’ ‘-TT’ represents ablations of our method without social pooling truncation trick.
We report results for in pixels for both K = 5 20 and for several other values of K. The format for each cell is <claimed result> / <reproduced result> 4.2 Experiment on ETH/UCY data sets (with and without social pooling, truncation trick
ETH/UCY: ETH/UCY data set consists of 5 scenes eth, hotel, univ, zara1, zara2 extracted fromanother source <link> because, in the paper, the source was not mentioned.
We Followed theconventional leave-one-out approach, i.e., trained on 4 sets and tested on the last set to get the results.
We veriﬁed results within 98% accuracy from claimed results.
The data set was further downsampledby 6 to get a 0.4 second gap between consecutive frames as demanded by the paper.
The result isshown below in the table.
With these 2 experiments, the reduction in error with respect to the previousresults by using CVAE and subsequent reduction by using social pooling layer and truncation trickcan be demonstrated.
[Table]
4.3 Change in the structure of CVAE
In this experiment during training, the ground truth Eend ( G k ) was used to predict the future T finstead of the one obtained from the latent variable.
We did it on the Stanford drone dataset withsocial pooling and got results within 95% accuracy from the claim results.
ADE : 10.87 / 10.
FDE : 17.03 / 16.
4.4 Effect of Number of samples (K)
We did this experiment on the Stanford drone dataset with social pooling.
We trained the PECNetmodel with default sigma values and test on different k-sample value with and without truncation.
Without truncation for k-sample<=3 we used σ with variance 1 and for k-sample > 3 we used σ withvariance 1.3.
With truncation for k-sample > 3 we used σ with variance 1 and for k-sample<=3 weused σ with variance c * √ k − 1 .
In this experiment we got results within 95 accuracy from the claimresults. ADE 24.
FDE 51. Truncated-ADE 17. Truncated-FDE 35. 4.5 Conditioned Way-point positions Oracles
In this experiment, we conditioned on future trajectory points other than the last observed point,which we refer to as way-points.
This was not clear in the paper about how to calculate FDE errorbecause we can not predict last observed point in the model so we calculated FDE from the 11thpoint of the predicted trajectory.
It was done in two parts:1.
With oracle : During prediction of future trajectory (at time of testing and validation), wegave ground-truth value of conditioned point instead of the best guessed one from samplingto predict trajectory from the model.
The Stanford drone data set with social pooling andtruncation trick was used to match with the results on paper.2.
Without oracle : The same thing was done here except during prediction of the futuretrajectory the best guess for the conditioned point(predicted by model) was taken (at time oftesting and validation).
Way-point Prediction Error was calculated as difference betweenground truth of conditioned point and the one predicted by the model.
ADE 18.
FDE 35.
Way-point error 4.
Oracle ADE 18.
Oracle FDE 35.
[Table]
link [Table] 1] (Extra experiment)
We took the reference of the trajectory for each pedestrian as the current point instead of the ﬁrst pointof the past trajectory.
This helped the CVAE network to get a better representation of the destinationpoint as all input past trajectories have a common last point, which makes it easier for the encoder anddecoder network to function; also, the predictor and social pooling network gets more easily trained.
This showed about 8% further decrease in ADE and FDE metrics for drone dataset as follows:-
ADE : 8.
FDE : 14.644.7 Using encoder and decoder LSTM network <
link>[1] (Extra experiment)
We used encoder LSTM instead of MLP to form the encoding of the past trajectory to accommodatevariable length of past trajectory and form a better representation as to the input temporal data.
Also, we used the decoder LSTM network to predict the rest of the trajectory given the destination.
However, the FDE error reduced by about 5 %, but the ADE is surprisingly more, demonstrating thatdecoder LSTM does not perform well given the destination point.
ADE : 26.
FDE : 14.
Discussion
From each of the experiments, the claims made by the paper as described above can be stronglysupported and empirically proved.
Strong correspondence between destination and rest of the pathis observed as evident from the results in comparison to previous experiments.
Also, use of socialpooling layer and truncation trick, reduce the error to a great extent as demonstrated from theablation studies described above.
In order to further study the choice of structure of the network, 2other experiments were performed described above and they strongly support the choice of MLParchitecture used for past encoding, future prediction instead of LSTM/
GRU RNN structures