Original: M OTION F ORECASTING WITH U NLIKELIHOOD L EARNING
Revised: M OTION F ORECASTING WITH U NLIKELIHOOD T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Motion forecasting is essential for making safe and intelligent decisions in robotic applications such as autonomous driving.
Revised: Motion forecasting is essential for making safe and intelligent decisions in robotic applications such as autonomous driving.

Original: State-of-the-art methods formulate it as a sequence-to-sequence prediction problem, which is solved in an encoderdecoder framework with a maximum likelihood estimation objective.
Revised: State-of-the-art methods formulate it as a sequence-to-sequence prediction problem, which is solved in an encoderdecoder framework with a maximum likelihood estimation objective.

Original: In this paper, we show that the likelihood objective itself results in a model assigning too much probability to trajectories that are unlikely given the contextual information such as maps and states of surrounding agents.
Revised: In this paper, we show that the likelihood objective itself results in a model assigning too much probability to trajectories that are unlikely given the contextual information such as maps and states of surrounding agents.

Original: This is despite the fact that many state-of-the-art models do take contextual information as part of their input.
Revised: This is despite the fact that many state-of-the-art models do take contextual information as part of their input.

Original: We propose a new objective, unlikelihood training, which forces generated trajectories that conﬂicts with contextual information to be assigned a lower probability by our model.
Revised: We propose a new objective, unlikelihood training, which forces generated trajectories that conﬂicts with contextual information to be assigned a lower probability by our model.

Original: We demonstrate that our method can signiﬁcantly improve state-ofart models’ performance on challenging real-world trajectory forecasting datasets (nuScenes and Argoverse) by 8% and reduce the standard deviation by up to 50%.
Revised: We demonstrate that our method can signiﬁcantly improve state-ofart models’ performance on challenging real-world trajectory forecasting datasets (nuScenes and Argoverse) by 8% and reduce the standard deviation by up to 50%.

Original: The code will be released soon.
Revised: Code will be made available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For robotic applications deployed in the real world, the ability to foresee the future motions of agents in the surrounding environment plays an essential role for safe and intelligent decision making.
Revised: For robotic applications deployed in the real world, the ability to foresee the future motions of agents in the surrounding environment plays an essential role for safe and intelligent decision making.

Original: This is a very challenging task.
Revised: This is a very challenging task.

Original: For example, in the autonomous driving domain, to predict nearby agents’ future trajectories, an agent needs to consider contextual information such as their past trajectories, potential interactions, and maps.
Revised: For example, in the autonomous driving domain, to predict nearby agents’ future trajectories, an agent needs to consider contextual information such as their past trajectories, potential interactions, and maps.

Original: State of the art prediction models (Salzmann et al., 2020; Tang & Salakhutdinov, 2019; Rhinehart et al., 2019) directly take contextual information as part of their input and use techniques such as graph neural networks to extract high-level features for prediction.
Revised: State of the art prediction models (Salzmann et al., 2020; Tang & Salakhutdinov, 2019; Rhinehart et al., 2019) directly take contextual information as part of their input and use techniques such as graph neural networks to extract high-level features for prediction.

Original: They are typically trained with a maximum likelihood estimation (MLE) objective that maximizes the likelihood of ground truth trajectories in the predicted distribution.
Revised: They are typically trained with a maximum likelihood estimation (MLE) objective that maximizes the likelihood of ground truth trajectories in the predicted distribution.

Original: Although MLE loss encourages the prediction to be close to the ground truth geometrically, it does not focus on learning a good distribution that is plausible with respect to the contextual information.
Revised: Although MLE loss encourages the prediction to be close to the ground truth geometrically, it does not focus on learning a good distribution that is plausible with respect to the contextual information.

Original: These models predict trajectories that violate the contextual information (e.g., go to opposite driving direction or out of the driving area) but still closes to ground truth.
Revised: These models predict trajectories that violate the contextual information (e.g., go to opposite driving direction or out of the driving area) but still closes to ground truth.

Original: In contrast, humans can easily notice that these trajectories are unlikely in a speciﬁc context.
Revised: In contrast, humans can easily notice that these trajectories are unlikely in a speciﬁc context.

Original: This phenomenon suggests that simply applying MLE loss cannot fully exploit contextual information.
Revised: This phenomenon suggests that simply applying MLE loss cannot fully exploit contextual information.

Original: To address the problem, we propose a novel and simple method, unlikelihood training, that injects contextual information into the learning signal.
Revised: To address the problem, we propose a novel and simple method, unlikelihood training, that injects contextual information into the learning signal.

Original: Our loss penalizes the trajectories that violate the contextual information, called negative trajectories, by minimizing their likelihood in the predicted distribution.
Revised: Our loss penalizes the trajectories that violate the contextual information, called negative trajectories, by minimizing their likelihood in the predicted distribution.

Original: To generate negative trajectories, we ﬁrst draw a number of candidate trajectories from our model’s predicted distribution.
Revised: To generate negative trajectories, we ﬁrst draw a number of candidate trajectories from our model’s predicted distribution.

Original: Then, a context checker is used to cull out the trajectories that violate contextual information as negative trajectories.
Revised: Then, a context checker is used to cull out the trajectories that violate contextual information as negative trajectories.

Original: This context checker does not need to be differentiable.
Revised: This context checker does not need to be differentiable.

Original: By minimizing the likelihood of negative trajectories, the model is forced to use the contextual information to avoid predictions that violate context.
Revised: By minimizing the likelihood of negative trajectories, the model is forced to use the contextual information to avoid predictions that violate context.

Original: Therefore, the prediction quality is improved.
Revised: Therefore, the prediction quality is improved.

Original: Existing methods (Casas et al., 2020; Park et al., 2020) using contextual information as learning signals either introduce new learning parameters or using high-variance learning methods such as the REINFORCE algorithm (Casas et al., 2020).
Revised: Existing methods (Casas et al., 2020; Park et al., 2020) using contextual information as learning signals either introduce new learning parameters or using high-variance learning methods such as the REINFORCE algorithm (Casas et al., 2020).

Original: In contrast, our method injects rich contextual information into the training objective and keeps the training process simple.
Revised: In contrast, our method injects rich contextual information into the training objective and keeps the training process simple.

Original: Unlikelihood training (Welleck et al., 2019) has been applied to neural text generation.
Revised: Unlikelihood training (Welleck et al., 2019) has been applied to neural text generation.

Original: We are the ﬁrst to propose unlikelihood training for continuous space of trajectories.
Revised: We are the ﬁrst to propose unlikelihood training for continuous space of trajectories.

Original: For the discrete space of token sequences, repeating tokens or n-grams in the generated sequence are chosen as negative tokens.
Revised: For the discrete space of token sequences, repeating tokens or n-grams in the generated sequence are chosen as negative tokens.

Original: In contrast, we design a context checker to select negative trajectories sampled from the continuous distribution of model predictions.
Revised: In contrast, we design a context checker to select negative trajectories sampled from the continuous distribution of model predictions.

Original: Our method can be viewed as a simple add-on to any models that estimate the distribution of future trajectories.
Revised: Our method can be viewed as a simple add-on to any models that estimate the distribution of future trajectories.

Original: It improves their performance by encouraging models to focus more on contextual information without increasing the complexity of its original training process.
Revised: It improves their performance by encouraging models to focus more on contextual information without increasing the complexity of its original training process.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We propose a novel and simple method, unlikelihood training for motion forecasting in autonomous driving that encourages models to use contextual information by minimizing the likelihood of trajectories that violate contextual information.
Revised: • We propose a novel and simple method, unlikelihood training for motion forecasting in autonomous driving that encourages models to use contextual information by minimizing the likelihood of trajectories that violate contextual information.

Original: Our method can be easily incorporated into state-of-the-art models.
Revised: Our method can be easily incorporated into state-of-the-art models.

Original: • Our experimental results on challenging real-world trajectory forecasting datasets, nuScenes and Argoverse, shows that unlikelihood training can improve prediction performance by 8% and reduce the standard deviation by up to 50%.
Revised: • Our experimental results on challenging real-world trajectory forecasting datasets, nuScenes and Argoverse, shows that unlikelihood training can improve prediction performance by 8% and reduce the standard deviation by up to 50%.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we brieﬂy review the two most related topics.
Revised: In this section, we brieﬂy review the two most related topics.

Original: Trajectory Forecasting Trajectory forecasting of dynamic agents, a core problem for robotic applications such as autonomous driving and social robots, has been well studied in the literature.
Revised: Trajectory Forecasting Trajectory forecasting of dynamic agents, a core problem for robotic applications such as autonomous driving and social robots, has been well studied in the literature.

Original: State-of-the-art models solves it as a sequence-to-sequence multi-modal prediction problem (Lee et al., 2017; Cui et al., 2018;
Revised: State-of-the-art models solves it as a sequence-to-sequence multi-modal prediction problem (Lee et al., 2017; Cui et al., 2018;

Original: Chai et al., 2019; Rhinehart et al., 2019; Kosaraju et al., 2019; Tang & Salakhutdinov, 2019;
Revised: Chai et al., 2019; Rhinehart et al., 2019; Kosaraju et al., 2019; Tang & Salakhutdinov, 2019;

Original: Ridel et al., 2020; Salzmann et al., 2020; Huang et al., 2019).
Revised: Ridel et al., 2020; Salzmann et al., 2020; Huang et al., 2019).

Original: (Cui et al., 2018;
Revised: (Cui et al., 2018;

Original: Chai et al., 2019; Ridel et al., 2020) predicts multiple future trajectories without learning low dimensional latent agent behaviors.
Revised: Chai et al., 2019; Ridel et al., 2020) predicts multiple future trajectories without learning low dimensional latent agent behaviors.

Original: (Lee et al., 2017; Kosaraju et al., 2019; Rhinehart et al., 2019; Huang et al., 2019)
Revised: (Lee et al., 2017; Kosaraju et al., 2019; Rhinehart et al., 2019; Huang et al., 2019)

Original: encodes agent behaviors in continuous low dimensional latent space while (Tang & Salakhutdinov, 2019; Salzmann et al., 2020) uses discrete latent variables.
Revised: encodes agent behaviors in continuous low dimensional latent space while (Tang & Salakhutdinov, 2019; Salzmann et al., 2020) uses discrete latent variables.

Original: Discrete latent variables succinctly capture semantically meaningful modes such as turn left, turn right.
Revised: Discrete latent variables succinctly capture semantically meaningful modes such as turn left, turn right.

Original: (Tang & Salakhutdinov, 2019; Salzmann et al., 2020) learns discrete latent variables without explicit labels.
Revised: (Tang & Salakhutdinov, 2019; Salzmann et al., 2020) learns discrete latent variables without explicit labels.

Original: All of them use a maximum likelihood estimation (MLE) objective or its approximations (e.g.,
Revised: All of them use a maximum likelihood estimation (MLE) objective or its approximations (e.g.,

Original: VAE).
Revised: VAE).

Original: In this paper, we show that MLE loss can ignore contextual information such as maps and states of surrounding agents.
Revised: In this paper, we show that MLE loss can ignore contextual information such as maps and states of surrounding agents.

Original: As a result, models with such a loss can assign too much probability to unlikely trajectories.
Revised: As a result, models with such a loss can assign too much probability to unlikely trajectories.

Original: We propose an unlikelihood training objective to avoid such cases.
Revised: We propose an unlikelihood training objective to avoid such cases.

Original: All models with the maximum likelihood estimation objective can potentially beneﬁt from our methods.
Revised: All models with the maximum likelihood estimation objective can potentially beneﬁt from our methods.

Original: Contrastive learning and unlikelihood training To date, several studies have investigated the possibilities to beneﬁt from negative data.
Revised: Contrastive learning and unlikelihood training To date, several studies have investigated the possibilities to beneﬁt from negative data.

Original: One of the popular direction is contrastive learning.
Revised: One of the popular direction is contrastive learning.

Original: Contrastive learning has achieved signiﬁcant success in many ﬁelds (Oord et al., 2018;
Revised: Contrastive learning has achieved signiﬁcant success in many ﬁelds (Oord et al., 2018;

Original: Kipf et al., 2019;
Revised: Kipf et al., 2019;

Original: Ma & Collins, 2018; Abid & Zou, 2019; Welleck et al., 2019).
Revised: Ma & Collins, 2018; Abid & Zou, 2019; Welleck et al., 2019).

Original: NCE (Ma & Collins, 2018) CPC (Oord et al., 2018) maximizes the mutual information between data and latent representation by a novel contrastive loss to extract useful representation from data.
Revised: NCE (Ma & Collins, 2018) CPC (Oord et al., 2018) maximizes the mutual information between data and latent representation by a novel contrastive loss to extract useful representation from data.

Original: C-SWMs (Kipf et al., 2019) utilizes contrastive learning to learn a better world model for reinforcement learning tasks.
Revised: C-SWMs (Kipf et al., 2019) utilizes contrastive learning to learn a better world model for reinforcement learning tasks.

Original: Recently, unlikelihood training (Welleck et al., 2019) proposes a new method to utilize negative data.
Revised: Recently, unlikelihood training (Welleck et al., 2019) proposes a new method to utilize negative data.

Original: In addition, to maximize the likelihood of the ground truth token, it minimizes the likelihood of negative tokens for better text generation.
Revised: In addition, to maximize the likelihood of the ground truth token, it minimizes the likelihood of negative tokens for better text generation.

Original: Their method is on the discrete space of token sequences.
Revised: Their method is on the discrete space of token sequences.

Original: Repeating tokens or n-grams in the generated sequence is chosen as negative tokens.
Revised: Repeating tokens or n-grams in the generated sequence is chosen as negative tokens.

Original: In contrast, our proposed method works in the continuous space of trajectories.
Revised: In contrast, our proposed method works in the continuous space of trajectories.

Original: We design a novel method, context checker, to select negative trajectories.
Revised: We design a novel method, context checker, to select negative trajectories.

Original:  M ETHOD 3. P ROBLEM F ORMULATION
Revised:  M ETHOD 3. P ROBLEM F ORMULATION

Original: We are targeting at better predicting the future trajectory Y i of a vehicle i given input X i .
Revised: We are targeting at better predicting the future trajectory Y i of a vehicle i given input X i .

Original: X i can include any related information like rasterized maps or past positions of vehicle i and surrounding agents, depends on the design of the method.
Revised: X i can include any related information like rasterized maps or past positions of vehicle i and surrounding agents, depends on the design of the method.

Original: Here we skip the detailed choice of input and denote it as X i for conciseness.
Revised: Here we skip the detailed choice of input and denote it as X i for conciseness.

Original: Due to different driving strategies, driving intents, and the complex trafﬁc environment, there are usually multiple possible future trajectories given an input x
Revised: Due to different driving strategies, driving intents, and the complex trafﬁc environment, there are usually multiple possible future trajectories given an input x

Original: i (although there is only one ground truth future trajectory y i,gt in a dataset recorded in the real world).
Revised: i (although there is only one ground truth future trajectory y i,gt in a dataset recorded in the real world).

Original: To handle this situation, most state of the art methods (Salzmann et al., 2020) model a distribution of possible future trajectories
Revised: To handle this situation, most state of the art methods (Salzmann et al., 2020) model a distribution of possible future trajectories

Original: p θ
Revised: p θ

Original: ( Y i | X i ) to cover all the possibilities given the input
Revised: ( Y i | X i ) to cover all the possibilities given the input

Original: X i instead of predicting one trajectory.
Revised: X i instead of predicting one trajectory.

Original: θ denotes the learning parameters of the model.
Revised: θ denotes the learning parameters of the model.

Original: To train such methods, most stateof-the-art models usually use maximum likelihood estimation (MLE) to maximize the likelihood of ground truth trajectory Y i,gt in the predicted distribution.
Revised: To train such methods, most stateof-the-art models usually use maximum likelihood estimation (MLE) to maximize the likelihood of ground truth trajectory Y i,gt in the predicted distribution.

Original: For example, the loss of CVAE-based model Trajectron++ (Salzmann et al., 2020) is Eq.1.
Revised: For example, the loss of CVAE-based model Trajectron++ (Salzmann et al., 2020) is Eq.1.

Original: This loss is used to maximize the lower bound of ground truth’s likelihood when the coefﬁcient k = 1 .
Revised: This loss is used to maximize the lower bound of ground truth’s likelihood when the coefﬁcient k = 1 .

Original: [Equation]
Revised: [Equation]

Original: when
Revised: when

Original: [Equation]
Revised: [Equation]

Original: White
Revised: White

Original: - tion.
Revised: - tion.

Original: Some of the prediction go outside of the drivable region or go to the lane in opposite direction.
Revised: Some of the prediction go outside of the drivable region or go to the lane in opposite direction.

Original: Limitation of MLE on Motion Forecasting
Revised: Limitation of MLE on Motion Forecasting

Original: MLE encourages the model to predict a distribution that allocates reasonable probability mass to the region where Y i is located by minimizing the KL-divergence of predicted distribution and ground truth distribution.
Revised: MLE encourages the model to predict a distribution that allocates reasonable probability mass to the region where Y i is located by minimizing the KL-divergence of predicted distribution and ground truth distribution.

Original: Because the domain of trajectory distribution is over the geometric locations, MLE makes these 2 distributions ”close” to each other geometrically.
Revised: Because the domain of trajectory distribution is over the geometric locations, MLE makes these two distributions ”close” to each other geometrically.

Original: However, we argue that maintaining the geometrical nearness only is not good enough for motion forecasting task in autonomous driving.
Revised: However, we argue that maintaining the geometrical nearness only is not good enough for motion forecasting task in autonomous driving.

Original: In complex trafﬁc scenarios, there can be many potential trajectoriesthat are close enough to the ground truth geometrically but are very unlike to happen.
Revised: In complex trafﬁc scenarios, there can be many potential trajectories close enough to the ground truth geometrically but are very unlikely to happen.

Original: For example, if the ground truth trajectory Y i,gt is on the outermost lane, a trajectory that is close to Y i,gt but outside the drivable region is unlike to happen in  real world.
Revised: For example, if the ground truth trajectory Y i,gt is on the outermost lane, a trajectory that is close to Y i,gt but outside the drivable region is unlike to happen in the real world.

Original: But MLE loss will not impose big enough penalty on such a case to avoid such a prediction.
Revised: However, MLE loss will not impose a significant enough penalty on such a case to avoid such a prediction.

Original: Fig.1 demonstrates a prediction example from Trajectron++ (Salzmann et al., 2020) where part of the distribution is outside of the derivable region or on the lane with the wrong direction.
Revised: Fig.1 demonstrates a prediction example from Trajectron++ (Salzmann et al., 2020) where part of the distribution is outside of the derivable region or on the lane with the wrong direction.

Original:  MLE-based loss only offers learning signals that contains the geometric location information of the ground truth trajectories.
Revised: The MLE-based loss only offers learning signals that contain the geometric location information of the ground truth trajectories.

Original: All the other contextual information, like the drivable region and the lane direction, are missing in the learning signals.
Revised: All the other contextual information, like the drivable region and the lane direction, are missing in the learning signals.

Original: While the inputs to a model contain rich contextual information, the model cannot use it to avoid the prediction that is geometrically close to ground truth but violates context.
Revised: While the inputs to a model contain rich contextual information, the model cannot use it to avoid the prediction that is geometrically close to ground truth but violates context.

Original: In contrast, this isa quite  simple task for humans.
Revised: In contrast, this is quite a simple task for humans.

Original: 3. U NLIKELIHOOD L OSS
Revised: 3. U NLIKELIHOOD L OSS

Original: To mitigate this problem, we design a new loss term that encourages the model to take the contextual information into account.
Revised: To mitigate this problem, we design a new loss term that encourages the model to consider the contextual information.

Original: Inspired by contrastive learning and unlikelihood training, we additionally train our model to minimize the likelihood of trajectories that violate the contextual information given input X i .
Revised: Inspired by contrastive learning and unlikelihood training, we additionally train our model to minimize the likelihood of trajectories that violate the contextual information given input X i .

Original: We denote them as negative trajectories Y i,neg .
Revised: We denote them as negative trajectories Y i,neg .

Original: Let’s ﬁrst assume that we already have a distribution of negative trajectories
Revised: Let’s ﬁrst assume that we already have a distribution of negative trajectories

Original: p neg ( Y i |
Revised: p neg ( Y i |

Original: X i ) .
Revised: X i ) .

Original: One intuitive way is to directly minimize the log likelihood of Y i,neg in our predicted distribution, similar to MLE but in an opposite manner
Revised: One intuitive way is to directly minimize the log likelihood of Y i,neg in our predicted distribution, similar to MLE but in an opposite manner

Original: L unlike =
Revised: L unlike =

Original: E X
Revised: E X

Original: i , ∼ D , Y i,neg ∼ p neg ( Y i |
Revised: i , ∼ D , Y i,neg ∼ p neg ( Y i |

Original: [Equation]
Revised: [Equation]

Original: i ) [
Revised: i ) [

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: However, the gradient of log function tends to inﬁnity when the input tends to 0, which leads to unstable training since the model are optimized to minimize p θ ( Y i,neg | X i ) and p θ ( Y i,neg |
Revised: However, the gradient of log function tends to inﬁnity when the input tends to 0, which leads to unstable training since the model are optimized to minimize p θ ( Y i,neg | X i ) and p θ ( Y i,neg |

Original: X i ) ≥ 0 .
Revised: X i ) ≥ 0 .

Original: To avoid the inﬁnity gradient region of log function, we add a small constant (cid:15) to the likelihood.
Revised: To avoid the inﬁnity gradient region of log function, we add a small constant (cid:15) to the likelihood.

Original: The ﬁnal loss term we propose is
Revised: The ﬁnal loss term we propose is

Original: [Equation]
Revised: [Equation]

Original: We call it unlikelihood loss.
Revised: We call it unlikelihood loss.

Original: We use a coefﬁcient γ to balance L unlike .
Revised: We use a coefﬁcient γ to balance L unlike .

Original: The ﬁnal training objective in case we combine our method with Trajectron++ is
Revised: The ﬁnal training objective in case we combine our method with Trajectron++ is

Original: [Equation]
Revised: [Equation]

Original: Eq.4 is also easily adapted to combine with any other models.
Revised: Eq.4 is also easily adapted to combine with any other models.

Original: With the help of L unlike , we inject the contextual information into the learning signal, force the model to better extract and use contextual information in X i , and generate more reasonable predicted distribution to avoid high L unlike .
Revised: With the help of L unlike , we inject the contextual information into the learning signal, force the model to better extract and use contextual information in X i , and generate more reasonable predicted distribution to avoid high L unlike .

Original: 3. N EGATIVE T RAJECTORIES
Revised: 3. N EGATIVE T RAJECTORIES

Original: Our proposed loss term is highly dependent on the negative samples
Revised: Our proposed loss term is highly dependent on the negative samples

Original: Y i,neg from the distribution p neg ( Y i |
Revised: Y i,neg from the distribution p neg ( Y i |

Original: X i ) .
Revised: X i ) .

Original: However, there are not given in the dataset.
Revised: However, there are not given in the dataset.

Original: To solve this issue, we approximate the samples by directly drawing a set of trajectories from the predicted distribution and select the trajectories that violate the contextual information out by a context checker.
Revised: To solve this issue, we approximate the samples by directly drawing a set of trajectories from the predicted distribution and select the trajectories that violate the contextual information out by a context checker.

Original: Note that this checker does not need to be differentiable and it can be as complex and advanced as necessary.
Revised: Note that this checker does not need to be differentiable and it can be as complex and advanced as necessary.

Original: The type of unlike predictions the model learns to avoid by our method depends on the type of unlike trajectories the checker can detect.
Revised: The type of unlike predictions the model learns to avoid by our method depends on the type of unlike trajectories the checker can detect.

Original: Design of Our Checker
Revised: Design of Our Checker

Original: We implement a map-based checker to judge whether a given trajectory suits the context or not.
Revised: We implement a map-based checker to judge whether a given trajectory suits the context or not.

Original: In detail, The checker examines whether the trajectory goes into the lane with opposite direction or out of the road.
Revised: In detail, The checker examines whether the trajectory goes into the lane in the opposite direction or out of the road.

Original: We create a map that stores the lane direction at every location of lanes and the drivable region.
Revised: We create a map that stores the lane direction at every location of lanes and the drivable region.

Original: Two examples are shown in Fig.2.
Revised: Two examples are shown in Fig.2.

Original: We ﬁrst check whether all the locations of a given trajectory are in the drivable region.
Revised: We ﬁrst check whether all the locations of a given trajectory are in the drivable region.

Original: If so, we further calculate angles between velocity and the lane direction at each time step to see whether they are all inside  90 degree.
Revised: If so, we further calculate angles between velocity and the lane direction at each time step to see whether they are all inside a 90 degree.

Original: The velocities are approximated by differentiating the trajectory.
Revised: The velocities are approximated by differentiating the trajectory.

Original: The trajectories that fail to pass the exam are judged as negative trajectories Y i,neg .
Revised: The trajectories that fail to pass the exam are judged as negative trajectories Y i,neg .

Original: Note that the lane direction information originally comes from the dataset and is usually incomplete or invalid like in the intersection.
Revised: Note that the lane direction information originally comes from the dataset and is usually incomplete or invalid like in the intersection.

Original: In this case, we only use the drivable region information.
Revised: In this case, we only use the drivable region information.

Original: In addition, there are also a small part of ground truth trajectories in the dataset that violates the lane direction or drivable region.
Revised: In addition, there are also a small part of ground truth trajectories in the dataset that violates the lane direction or drivable region.

Original: In this case, our checker skips the checking and we train it without L unlike to allow similar prediction.
Revised: In this case, our checker skips the checking and we train it without L unlike to allow similar prediction.

Original: Note that this checker is not perfect due to the incomplete information and simple checking mechanism.
Revised: Note that this checker is not perfect due to the incomplete information and simple checking mechanism.

Original: A trajectory that passes the exam of the checker doesn’t mean it is 100% compatible with the context.
Revised: A trajectory that passes the exam of the checker doesn’t mean it is 100% compatible with the context.

Original: However, our method can still work properly, because our method only depends on the negative trajectories that do not pass the exam and has nothing to do with the passing ones.
Revised: However, our method can still work properly, because our method only depends on the negative trajectories that do not pass the exam and has nothing to do with the passing ones.

Original: Our checker design offers reasonable negative trajectories to support our approach.
Revised: Our checker design offers reasonable negative trajectories to support our approach.

Original: But of course, a more advanced checker helps the model to avoid more complex unlikely prediction.
Revised: But of course, a more advanced checker helps the model to avoid more complex unlikely prediction.

Original: We leave it open for future research.
Revised: We leave it open for future research.

Original: 3. A LGORITHM
Revised: 3. A LGORITHM

Original: The ﬁnial algorithm is shown in Alg.1.
Revised: The ﬁnial algorithm is shown in Alg.1.

Original: At each iteration, we ﬁrst run the forward pass of the model to get the output distribution p θ ( Y i | X i ) given the input X i .
Revised: At each iteration, we ﬁrst run the forward pass of the model to get the output distribution p θ ( Y i | X i ) given the input X i .

Original: Then, K negative candidate trajectories are drawn from this distribution and we select the negative trajectories out Y i,neg via
Revised: Then, K negative candidate trajectories are drawn from this distribution and we select the negative trajectories out Y i,neg via

Original: [Figure]
Revised: [Figure]

Original: our checker.
Revised: our checker.

Original: After that, ground truth trajectory and negative trajectories are used to calculate the loss function, update the model, and go to next iteration.
Revised: After that, ground truth trajectory and negative trajectories are used to calculate the loss function, update the model, and go to next iteration.

Original: Note that if there are no Y i,neg in the K negative candidates judged by our checker for data i , we don’t apply L unlike on i .
Revised: Note that if there are no Y i,neg in the K negative candidates judged by our checker for data i , we don’t apply L unlike on i .

Original: 3. G RADIENT A NALYSIS
Revised: 3. G RADIENT A NALYSIS

Original: Let’s assume a single-mode prediction case that the future position y i,gt,t at time step t of agent i is modeled by a simple Gaussian distribution
Revised: Let’s assume a single-mode prediction case that the future position y i,gt,t at time step t of agent i is modeled by a simple Gaussian distribution

Original: N ( y i,t ; ˆ µ i,t , ˆ σ i,t I ) .
Revised: N ( y i,t ; ˆ µ i,t , ˆ σ i,t I ) .

Original: ˆ µ and ˆ σ i,t are calculated by the model.
Revised: ˆ µ and ˆ σ i,t are calculated by the model.

Original: With a single negative position y i,neg,t we deﬁne a simple loss for step t
Revised: With a single negative position y i,neg,t we deﬁne a simple loss for step t

Original: L t = − log N ( y gt,t ; ˆ µ t , ˆ σ t I )
Revised: L t = − log N ( y gt,t ; ˆ µ t , ˆ σ t I )

Original: + log N ( y neg,t ; ˆ µ t , ˆ σ t I ) and omit the subscript i for brevity.
Revised: + log N ( y neg,t ; ˆ µ t , ˆ σ t I ) and omit the subscript i for brevity.

Original: The gradient of L t with respect to ˆ µ t and ˆ σ t in this case is (Derivation in Appx.
Revised: The gradient of L t with respect to ˆ µ t and ˆ σ t in this case is (Derivation in Appx.

Original: A):
Revised: A):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Eq.5 shows that the center of the predicted distribution ˆ µ t is pushed towards y gt,t and pushed away from y neg,t by this learning objective.
Revised: Eq.5 shows that the center of the predicted distribution ˆ µ t is pushed towards y gt,t and pushed away from y neg,t by this learning objective.

Original: In Eq.6, when y gt,t is closer to the center than y neg,t , ∂L∂ ˆ σ t is positive and ˆ σ t is decreased.
Revised: In Eq.6, when y gt,t is closer to the center than y neg,t , ∂L∂ ˆ σ t is positive and ˆ σ t is decreased.

Original: Note that y neg,t is selected out from samples of N (ˆ µ t , ˆ σ t I )) , this means when N (ˆ µ t , ˆ σ t I )) covers context-violated region and this region is farther than ground truth region, N (ˆ µ t , ˆ σ t I )) will shrink to exclude the negative region and become a better estimation to the true data distribution.
Revised: Note that y neg,t is selected out from samples of N (ˆ µ t , ˆ σ t I )) , this means when N (ˆ µ t , ˆ σ t I )) covers context-violated region and this region is farther than ground truth region, N (ˆ µ t , ˆ σ t I )) will shrink to exclude the negative region and become a better estimation to the true data distribution.

Original: When the prediction is not so accurate (e.g. at the beginning of training), our ground truth y gt,t may be farther than the negative location y neg,t .
Revised: When the prediction is not so accurate (e.g. at the beginning of training), our ground truth y gt,t may be farther than the negative location y neg,t .

Original: In this case, N (ˆ µ t , ˆ σ t I ) will expand to better cover the ground truth and make the prediction more uncertain.
Revised: In this case, N (ˆ µ t , ˆ σ t I ) will expand to better cover the ground truth and make the prediction more uncertain.

Original: A simple approach to alleviate this issue is turning off our unlikelihood loss L unlike in the ﬁrst few training epochs.
Revised: A simple approach to alleviate this issue is turning off our unlikelihood loss L unlike in the ﬁrst few training epochs.

Original: We implement this by making γ in Eq.4 as a sigmoid function centered at a speciﬁed epoch.
Revised: We implement this by making γ in Eq.4 as a sigmoid function centered at a speciﬁed epoch.

Original: By this way, we smoothly turn on L unlike during training.
Revised: By this way, we smoothly turn on L unlike during training.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we present the experimental results of our method to demonstrate our performance.
Revised: In this section, we present the experimental results of our method to demonstrate our performance.

Original: Our method is easily to applied on state of the art models that generate a future trajectory distribution and can further improve their performance.
Revised: Our method is easily to applied on state of the art models that generate a future trajectory distribution and can further improve their performance.

Original: In our experiments, we select Trajectron++ (Salzmann et al., 2020), one of the state-of-the-art methods on NuScenes dataset (Caesar et al., 2019b) with open-source implementation, as our base model.
Revised: In our experiments, we select Trajectron++ (Salzmann et al., 2020), one of the state-of-the-art methods on NuScenes dataset (Caesar et al., 2019b) with open-source implementation, as our base model.

Original: We extend the implementation to work on Argoverse dataset (Chang et al., 2019).
Revised: We extend the implementation to work on Argoverse dataset (Chang et al., 2019).

Original: We evaluate our approach on these two motion forecasting datasets.
Revised: We evaluate our approach on these two motion forecasting datasets.

Original: Initialize the model parameters θ ;
Revised: Initialize the model parameters θ ;

Original: Initialize learning rate α and coeifﬁcient γ ; while not converge do
Revised: Initialize learning rate α and coeifﬁcient γ ; while not converge do

Original: X i , Y i,gt ∼ D run forward pass to compute p θ ( Y i |
Revised: X i , Y i,gt ∼ D run forward pass to compute p θ ( Y i |

Original: X i ) draw K trajectotries
Revised: X i ) draw K trajectotries

Original: Y i,k ∼ p θ ( Y i |
Revised: Y i,k ∼ p θ ( Y i |

Original: X i ) select Y i,neg via checker
Revised: X i ) select Y i,neg via checker

Original: Compute L traj++ using Eq.
Revised: Compute L traj++ using Eq.

Original: Compute L unlike using Eq.
Revised: Compute L unlike using Eq.

Original: L = L traj++ + γL unlike θ = θ − α ∇ θ ( L ) end
Revised: L = L traj++ + γL unlike θ = θ − α ∇ θ ( L ) end

Original: Test Model Trajectron++ (Salzmann et al., 2020) is a CVAE-based (Sohn et al., 2015) model.
Revised: Test Model Trajectron++ (Salzmann et al., 2020) is a CVAE-based (Sohn et al., 2015) model.

Original: Its input X i contains positions, velocities, heading of the predicted and surrounding vehicles, and a map patch.
Revised: Its input X i contains positions, velocities, heading of the predicted and surrounding vehicles, and a map patch.

Original: The output distribution p θ ( Y i | X i ) =
Revised: The output distribution p θ ( Y i | X i ) =

Original: (cid:80) z p θ 1 ( z | X i ) p θ 2 ( Y i |
Revised: (cid:80) z p θ 1 ( z | X i ) p θ 2 ( Y i |

Original: X i , z ) is a Gaussian mixture model withcomponents and modeled by an encoder  p θ 1 ( z | X i ) and a decoder net p θ 1 ( Y i |
Revised: X i , z ) is a Gaussian mixture model withcomponents and modeled by an encoder net p θ 1 ( z | X i ) and a decoder net p θ 1 ( Y i |

Original: X i , z ) .
Revised: X i , z ) .

Original: In addition, it has another encoder net q θ 3 ( z |
Revised: In addition, it has another encoder net q θ 3 ( z |

Original: X i , Y i,gt ) used only in training.
Revised: X i , Y i,gt ) used only in training.

Original: The original learning objective is shown in Eq.1.
Revised: The original learning objective is shown in Eq.1.

Original: I q denotes the mutual information.
Revised: I q denotes the mutual information.

Original: Evaluation Metrics we use average l 2 displacement error (ADE) and ﬁnal l 2 displacement error (FDE) to evaluate the prediction performance.
Revised: Evaluation Metrics we use average l 2 displacement error (ADE) and ﬁnal l 2 displacement error (FDE) to evaluate the prediction performance.

Original: Each of them contains some sub-versions.
Revised: Each of them contains some sub-versions.

Original: FDE-1 is the FDE calculated using only 1 predicted trajectory.
Revised: FDE-1 is the FDE calculated using only 1 predicted trajectory.

Original: In both original Trajectron++ and our approach, this single trajectory is drawn from predicted distribution by greedy search step by step.
Revised: In both original Trajectron++ and our approach, this single trajectory is drawn from predicted distribution by greedy search step by step.

Original: ADE-Full/FDE-Full represents the quality of the whole output distribution.
Revised: ADE-Full/FDE-Full represents the quality of the whole output distribution.

Original: To compute ADEFull/FDE-Full, we randomly sample 200 trajectories and calculate the average performance as the reported scores.
Revised: To compute ADEFull/FDE-Full, we randomly sample 200 trajectories and calculate the average performance as the reported scores.

Original: 4. NU S CENES D ATASET
Revised: 4. NU S CENES D ATASET

Original: nuScenes dataset (Caesar et al., 2019b) contains 1000 city driving scenes from both left-hand (Singapore) and right-hand (Boston) trafﬁc regions.
Revised: nuScenes dataset (Caesar et al., 2019b) contains 1000 city driving scenes from both left-hand (Singapore) and right-hand (Boston) trafﬁc regions.

Original: Each scene is 20s long and recorded in 2Hz.
Revised: Each scene is 20s long and recorded in 2Hz.

Original: It is one of the biggest open-source motion forecasting datasets with detailed semantic maps.
Revised: It is one of the biggest open-source motion forecasting datasets with detailed semantic maps.

Original: Experiments The batch size is set to 1024.
Revised: Experiments The batch size is set to 1024.

Original: Models are trained for 35 epochs and we test the weights from the best epoch measured by average ADE on validation set.
Revised: Models are trained for 35 epochs and we test the weights from the best epoch measured by average ADE on validation set.

Original: The coefﬁcient γ in Alg.
Revised: The coefﬁcient γ in Alg.

Original: increases gradually from 0 to 1 as a sigmoid function centered at 24th epoch.
Revised: increases gradually from 0 to 1 as a sigmoid function centered at 24th epoch.

Original: Initial learning rate is 3e-3 and it decays exponentially by 0.9995 per iteration.
Revised: Initial learning rate is 3e-3 and it decays exponentially by 0.9995 per iteration.

Original: These hyperparameters except γ are optimized for Trajectron++ and lead to better performance than that in original paper.
Revised: These hyperparameters except γ are optimized for Trajectron++ and lead to better performance than that in original paper.

Original: In addition, we rotate the scenes randomly from 15 ◦ to 345 ◦ in the training set for data augmentation following the setting of original Trajectron++.
Revised: In addition, we rotate the scenes randomly from 15 ◦ to 345 ◦ in the training set for data augmentation following the setting of original Trajectron++.

Original: For each model, we run 5 experiments and report the mean and standard deviation of the measured metrics.
Revised: For each model, we run 5 experiments and report the mean and standard deviation of the measured metrics.

Original: The models are trained to predict 3 seconds into the future.
Revised: The models are trained to predict 3 seconds into the future.

Original: To evaluate on generalization beyond the training horizon, we test on both 3 second andsecond prediction horizons.
Revised: To evaluate on generalization beyond the training horizon, we test on both 3 second andsecond prediction horizons.

Original: Tab.1 shows the quantitative results 3s and 4s prediction measured by the FDE/ADE-Full metrics
Revised: Tab.1 shows the quantitative results 3s and 4s prediction measured by the FDE/ADE-Full metrics

Original: Our unlikelihood loss improves Trajectron++ by about 8% for both metrics in both 3 and 4 second prediction horizons.
Revised: Our unlikelihood loss improves Trajectron++ by about 8% for both metrics in both 3 and 4 second prediction horizons.

Original: Results indicate that our method helps improve the accuracy of the predicted distribution.
Revised: Results indicate that our method helps improve the accuracy of the predicted distribution.

Original: This is also demonstrated in the qualitative comparison in Fig.3.
Revised: This is also demonstrated in the qualitative comparison in Fig.3.

Original: The predicted distribution from our methods covers the not-drivable region less compared to original Trajectron++ without our proposed loss.
Revised: The predicted distribution from our methods covers the not-drivable region less compared to original Trajectron++ without our proposed loss.

Original: In contrast, original Trajectron++ tends to violate the contextual information when prediction horizon is long.
Revised: In contrast, original Trajectron++ tends to violate the contextual information when prediction horizon is long.

Original: This shows that our method encourages the model to be more sensitive to the road boundary and the lane direction.
Revised: This shows that our method encourages the model to be more sensitive to the road boundary and the lane direction.

Original: In addition, we observe amirror reduction of the performance variance measured by standard deviation in Tab1 , indicates that our method helps to stable the training process.
Revised: In addition, we observe a reduction of the performance variance measured by standard deviation in Tab1, indicates that our method helps to stable the training process.

Original: Comparison with other methods is shown in Tab.2.
Revised: Comparison with other methods is shown in Tab.2.

Original: The FDE for single predicted trajectory is also improved by our method in addition to the improved quality for of the whole predicted distribution.
Revised: The FDE for single predicted trajectory is also improved by our method.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. A RGOVERSE D ATASET
Revised: 4. A RGOVERSE D ATASET

Original: Argoverse dataset (Chang et al., 2019) contains 300,000 5-second tracked scenarios in 2 American cities Miami and Pittsburgh.
Revised: Argoverse dataset (Chang et al., 2019) contains 300,000 5-second tracked scenarios in 2 American cities Miami and Pittsburgh.

Original: The data is recorded in 10 Hz.
Revised: The data is recorded in 10 Hz.

Original: The ﬁrst 2 seconds are used as input to predict the next 3 seconds future.
Revised: The ﬁrst 2 seconds are used as input to predict the next 3 seconds future.

Original: It is also one of the biggest open-source motion forecasting datasets that offer semantic maps.
Revised: It is also one of the biggest open-source motion forecasting datasets that offer semantic maps.

Original: Experiments We downsample the data from 10Hz to 2Hz to make the setting similar to nuScenes following the setting of (Park et al., 2020).
Revised: Experiments We downsample the data from 10Hz to 2Hz to make the setting similar to nuScenes following the setting of (Park et al., 2020).

Original: Argoverse does not release the ground truth future trajectories for the test dataset.
Revised: Argoverse does not release the ground truth future trajectories for the test dataset.

Original: Therefore, we use the original validation set as our test set in this experiments and randomly split the original training set into our training set with 95% data and our validation set with 5% data.
Revised: Therefore, we use the original validation set as our test set in this experiments and randomly split the original training set into our training set with 95% data and our validation set with 5% data.

Original: Batch size is 256.
Revised: Batch size is 256.

Original: Initial learning rate is 3e-3 and it decays exponentially by 0.9995 per iteration.
Revised: Initial learning rate is 3e-3 and it decays exponentially by 0.9995 per iteration.

Original: The model is trained for maximal 60 epochs and we select the weights from the best epoch measured by average ADE in validation set.
Revised: The model is trained for maximal 60 epochs and we select the weights from the best epoch measured by average ADE in validation set.

Original: The coefﬁent γ in Alg.1 increases gradually from 0 to 1 as a sigmoid function centered at 18th epoch.
Revised: The coefﬁent γ in Alg.1 increases gradually from 0 to 1 as a sigmoid function centered at 18th epoch.

Original: Trajectron++ is not designed for
Revised: Trajectron++ is not designed for

Original: Argoverse.
Revised: Argoverse.

Original: To make the experiments runnable, we made some small modiﬁcations that are explained in Appx.
Revised: To make the experiments runnable, we made some small modiﬁcations that are explained in Appx.

Original: B. We execute 6 training instances for both original Trajectron++ and our method, report the average performance and the standard deviation.
Revised: B. We execute 6 training instances for both original Trajectron++ and our method, report the average performance and the standard deviation.

Original: The results are listed in Tab.3.
Revised: The results are listed in Tab.3.

Original: Compared to
Revised: Compared to

Original: Trajectron++ without our method, our model improves the accuracy of the prediction by about 8% with our simple loss and reduces the performance variance by about 50% measured by standard deviation.[Table] [Figure] [Table]
Revised: Trajectron++ without our method, our model improves the accuracy of the prediction by about 8% with our simple loss and reduces the performance variance by about 50% measured by standard deviation.

Original: [Table]

Original:  A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Prediction horizon for negative candidates Assume we have a predicted trajectory with a length six, and it is on the drivable region and obeys the lane direction.
Revised: Prediction horizon for negative candidates Assume we have a predicted trajectory with a length six, and it is on the drivable region and obeys the lane direction.

Original: However, the trajectory tends to hit the road boundary in the near future (e.g., in 1 second).
Revised: However, the trajectory tends to hit the road boundary in the near future (e.g., in 1 second).

Original: Such a trajectory can pass our checker’s exam, but it is still unlikely to happen in the real world.
Revised: Such a trajectory can pass our checker’s exam, but it is still unlikely to happen in the real world.

Original: We can easily select out such a trajectory by extending the prediction horizon for the candidate trajectories and examining our checker’s extended version.
Revised: We can easily select out such a trajectory by extending the prediction horizon for the candidate trajectories and examining our checker’s extended version.

Revised: To verify whether this helps us build a better checker, we extend the prediction horizon for

Original: To verify whether this helps us build a better checker, we extend the prediction horizon for negative candidates from 3 seconds to 4, 5, 6 seconds, respectively, and examine them by our original checker.
Revised: [Figure] [Table] [Table] negative candidates from 3 seconds to 4, 5, 6 seconds, respectively, and examine them by our original checker.

Original: The selected negative trajectories are truncated back to 3 seconds for computing L unlike .
Revised: The selected negative trajectories are truncated back to 3 seconds for computing L unlike .

Original: We can see in table 4 that the model beneﬁts from an adequately extended prediction horizon.
Revised: We can see in table 4 that the model beneﬁts from an adequately extended prediction horizon.

Original: The prediction horizon for ground truth trajectory is 3s.
Revised: The prediction horizon for ground truth trajectory is 3s.

Original: By extending the negative trajectories 1 second more, we improve the prediction accuracy.
Revised: By extending the negative trajectories 1 second more, we improve the prediction accuracy.

Original: Numbers are averaged over two training instances.
Revised: Numbers are averaged over two training instances.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present unlikelihood training, an approach that minimizes the probability of unlikely trajectories.
Revised: We present unlikelihood guided trajectory prediction method, that minimizes the probability of unlikely trajectories.

Original: During training, our context checker detects predicted unlikely trajectories and their probabilities are reduced through an unlikelihood loss.
Revised: During training, our context checker detects predicted unlikely trajectories and their probabilities are reduced through an unlikelihood loss.

Original: Our method can be incorporated intoany state-of-the-art models with a maximum likelihood estimation objective.
Revised: Our method can be incorporated into state-of-the-art models with a maximum likelihood estimation objective.

Original: Experimental results demonstrate that our method signiﬁcantly improves state-of-the-art trajectory prediction models.
Revised: Our experimental results demonstrate that our method signiﬁcantly improves state-of-the-art trajectory prediction models.

Original: For future work, we will explore the design of a more powerful checker capable of identifying more negative trajectories.
Revised: We hope that our work may encourage future work on exploring better unlikelihood methods for trajectory prediction and improved context checker models.

Original: A D ERIVATION OF G RADIENT
Revised: A D ERIVATION OF G RADIENT

Original: Here we show how to obtain Eq.5 and Eq.
Revised: Here we show how to obtain Eq.5 and Eq.

Original: [Equation]
Revised: [Equation]

Original: B M ODIFICATION ON T RAJECTRON ++ F OR A RGOVERSE E VALUATION
Revised: B M ODIFICATION ON T RAJECTRON ++ F OR A RGOVERSE E VALUATION

Original: The original Trajectron++ (Salzmann et al., 2020) is evaluated on nuScenes dataset (Caesar et al., 2019b) but not on Argoverse (Chang et al., 2019).
Revised: The original Trajectron++ (Salzmann et al., 2020) is evaluated on nuScenes dataset (Caesar et al., 2019b) but not on Argoverse (Chang et al., 2019).

Original: To make it runnable on Argoverse, we make some modiﬁcations on Trajectron++.
Revised: To make it runnable on Argoverse, we make some modiﬁcations on Trajectron++.

Original: The input states of the full version of Trajectron++ consist of the positions, velocities, accelerations, heading angles and heading angular velocities.
Revised: The input states of the full version of Trajectron++ consist of the positions, velocities, accelerations, heading angles and heading angular velocities.

Original: However,
Revised: However,

Original: Argoverse doesn’t offer the data for heading angles and heading angular velocities.
Revised: Argoverse doesn’t offer the data for heading angles and heading angular velocities.

Original: Therefore, we remove them from the input states.
Revised: Therefore, we remove them from the input states.

Original: In addition, Trajectron++ has a unicycle physics model to convert the predicted velocity from neural networks to locations.
Revised: In addition, Trajectron++ has a unicycle physics model to convert the predicted velocity from neural networks to locations.

Original: The unicycle physics model requires also the angular velocities information.
Revised: The unicycle physics model requires also the angular velocities information.

Original: Therefore, we replace the unicycle model by the single integrator model that requires only velocities and initial positions.
Revised: Therefore, we replace the unicycle model by the single integrator model that requires only velocities and initial positions.

Original: The maps offered by nuScens and Argoverse are different, too.
Revised: The maps offered by nuScens and Argoverse are different, too.

Original: We stack the drivable region and region of interest offered by Argoverse and upsample them to the same resolution as nuScenes as the input map.
Revised: We stack the drivable region and region of interest offered by Argoverse and upsample them to the same resolution as nuScenes as the input map.

Original: Please refer to Argoverse for the details.
Revised: Please refer to Argoverse for the details.

Original: Lastly, Trajectron++ use separate modules to process surrounding vehicles and surrounding pedestrians in nuScenes.
Revised: Lastly, Trajectron++ use separate modules to process surrounding vehicles and surrounding pedestrians in nuScenes.

Original: However, Argoverse doesn’t offer the labels for the surrouding agents and just simply denote them as ”others”.
Revised: However, Argoverse doesn’t offer the labels for the surrouding agents and just simply denote them as ”others”.

Original: Therefore, we remove the pedestrian modules in Trajectron++ and use the vehicle modules to process both surrounding pedestrians and vehicles.
Revised: Therefore, we remove the pedestrian modules in Trajectron++ and use the vehicle modules to process both surrounding pedestrians and vehicles.

Original: C C OMPARISON WITH OTHER METHODS ON A RGOVERSE
Revised: C C OMPARISON WITH OTHER METHODS ON A RGOVERSE

Original: Our experiment setting on Argoverse follows AttGlobal-CAM-Nf (Park et al., 2020).
Revised: Our experiment setting on Argoverse follows AttGlobal-CAM-Nf (Park et al., 2020).

Original: Here we list the detailed comparison with their numbers using their evaluation metrics minADE-12 and minFDE12, which are the minimal ADE and FDE over 12 prediction candidates, in Tab.5.
Revised: Here we list the detailed comparison with their numbers using their evaluation metrics minADE-12 and minFDE12, which are the minimal ADE and FDE over 12 prediction candidates, in Tab.5.

Original: Both Trajectron++ and our method outperform their numbers.
Revised: Both Trajectron++ and our method outperform their numbers.

Original: In addition, our unlikelihood loss helps improve the performance of Trajectron++.
Revised: In addition, our unlikelihood loss helps improve the performance of Trajectron++.

Original: Our scores and the standard deviation are calculated over 4 training instances.
Revised: Our scores and the standard deviation are calculated over 4 training instances.

Original: [Table]
Revised: [Table]

Original: D Q UALITATIVE R ESULTS
Revised: D Q UALITATIVE R ESULTS

Original: Here, we demonstrate our method’s qualitative results compared with Trajectron++ for 3 seconds prediction.
Revised: Here, we demonstrate our method’s qualitative results compared with Trajectron++ for 3 seconds prediction.

Original: We randomly sample 50 trajectories from the predicted prediction, use kernel density estimation (KDE) to approximate the total output distribution from the samples, and print it out in
Revised: We randomly sample 50 trajectories from the predicted prediction, use kernel density estimation (KDE) to approximate the total output distribution from the samples, and print it out in

Original: Fig.4.
Revised: Fig.4.

Original: White points represent the ground truth trajectories.
Revised: White points represent the ground truth trajectories.

Original: Compared to Trajectron++, our method suits the contextual information more and therefore is more accurate and plausible.
Revised: Compared to Trajectron++, our method suits the contextual information more and therefore is more accurate and plausible.

Original: [Figure]
Revised: [Figure]