Original: SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving
Revised: SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving

Original: Anonymous Author(s)

Original: AffiliationAddress email Abstract: Self-supervised learning (SSL) is an emerging technique that has beensuccessfully employed to train convolutional neural networks (CNNs) and graphneural networks (GNNs) for more transferable, generalizable, and robust representation learning.
Revised: Abstract: Self-supervised learning (SSL) is an emerging technique that has been successfully employed to train convolutional neural networks (CNNs) and graph neural networks (GNNs) for more transferable, generalizable, and robust representation learning.

Original: However its potential in motion forecasting for autonomousdriving has rarely been explored.
Revised: However its potential in motion forecasting for autonomous driving has rarely been explored.

Original: In this study, we report the first systematic exploration and assessment of incorporating self-supervision into motion forecasting.
Revised: In this study, we report the ﬁrst systematic exploration and assessment of incorporating self-supervision into motion forecasting.

Original: We first propose to investigate four novel self-supervised learning tasks for motionforecasting with theoretical rationale and quantitative and qualitative comparisonson the challenging large-scale Argoverse dataset.
Revised: We ﬁrst propose to investigate four novel self-supervised learning tasks for motion forecasting with theoretical rationale and quantitative and qualitative comparisons on the challenging large-scale Argoverse dataset.

Original: Secondly, we point out thatour auxiliary SSL-based learning setup not only outperforms forecasting methodswhich use transformers, complicated fusion mechanisms and sophisticated onlinedense goal candidate optimization algorithms in terms of performance accuracy,but also has low inference time and architectural complexity.
Revised: Secondly, we point out that our auxiliary SSL-based learning setup not only outperforms forecasting methods which use transformers, complicated fusion mechanisms and sophisticated online dense goal candidate optimization algorithms in terms of performance accuracy, but also has low inference time and architectural complexity.

Original: Lastly, we conductseveral experiments to understand why SSL improves motion forecasting.
Revised: Lastly, we conduct several experiments to understand why SSL improves motion forecasting.

Original:  Introduction
Revised:  Introduction

Original: Motion forecasting in a real-world urban environment is an important task for autonomous robots.
Revised: Motion forecasting in a real-world urban environment is an important task for autonomous robots.

Original: Itinvolves predicting the future trajectories of traffic agents including vehicles and pedestrians.
Revised: It involves predicting the future trajectories of trafﬁc agents including vehicles and pedestrians.

Original: This isabsolutely crucial in the self-driving domain for safe, comfortable and efficient operation.
Revised: This is absolutely crucial in the self-driving domain for safe, comfortable and efﬁcient operation.

Original: However,this is a very challenging problem.
Revised: However, this is a very challenging problem.

Original: Difficulties include inherent stochasticity and multimodalityof driving behaviors and that future motion can involve complicated maneuvers such as yielding,nudging, lane-changing, turning and acceleration or deceleration.
Revised: Difﬁculties include inherent stochasticity and multimodality of driving behaviors, and that future motion can involve complicated maneuvers such as yielding, nudging, lane-changing, turning and acceleration or deceleration.

Original: The motion prediction task has traditionally been based on kinematic constraints and road map information with handcrafted rules.
Revised: The motion prediction task has traditionally been based on kinematic constraints and road map information with handcrafted rules.

Original: These approaches however fail to capture long-term behavior andinteractions with map structure and other traffic agents in complex scenarios.
Revised: These approaches however fail to capture long-term behavior and interactions with map structure and other trafﬁc agents in complex scenarios.

Original: Tremendous progresshas been made with data-driven methods in motion forecasting [3, 4, 5, 6, 7, 8, 9, 10].
Revised: Tremendous progress has been made with data-driven methods in motion forecasting [3, 4, 5, 6, 7, 8, 9, 10].

Original: Recentmethods use a vector representation for HD maps and agent trajectories, including approaches like
Revised: Recent methods use a vector representation for HD maps and agent trajectories, including approaches like

Original: Lane-GCN [2], Lane-RCNN [11], Vector-Net [12], TNT [5] and Dense-TNT [6].
Revised: Lane-GCN [2], Lane-RCNN [11], Vector-Net [12], TNT [5] and Dense-TNT [6].

Original: More recently, theenormous success of transformers [13] has been leveraged for forecasting in mm-Transformer [9],
Revised: More recently, the enormous success of transformers [13] has been leveraged for forecasting in mm-Transformer [9],

Original: Scene transformer [8], Multimodal transformer [14] and Latent Variable Sequential Transformers[15].
Revised: Scene transformer [8], Multimodal transformer [14] and Latent Variable Sequential Transformers [15].

Original: Most of these methods however are extremely complex in terms of architecture and have lowinference speeds, which makes them unsuitable for real-world settings.
Revised: Most of these methods however are extremely complex in terms of architecture and have low inference speeds, which makes them unsuitable for real-world settings.

Original: In this work, we extend ideas from self-supervised learning (SSL) to the motion forecasting task.
Revised: In this work, we extend ideas from self-supervised learning (SSL) to the motion forecasting task.

Original: Self-supervision has seen huge interest in both natural language processing and computer vision[16] to make use of freely available data without the need for annotations.
Revised: Self-supervision has seen huge interest in both natural language processing and computer vision [16] to make use of freely available data without the need for annotations.

Original: It aims to assist themodel to learn more transferable and generalized representation from pseudo-labels via pretext tasks.
Revised: It aims to assist the model to learn more transferable and generalized representation from pseudo-labels via pretext tasks.

Original: Given the recent success of self-supervision with CNNs, transformers, and GNNs, we are naturallymotivated to ask the question: Can self-supervised learningplay a similar role in motion forecasting [Figure] to improveits accuracy and generalizability, without sacrificing inference speed or architecturalsimplicity?
Revised: Given the recent success of self-supervision with CNNs, transformers, and GNNs, we are naturally motivated to ask the question: Can self-supervised learning improve accuracy and generalizability of motion forecasting, without sacriﬁcing inference speed or architectural simplicity?

Original: Contributions: Our work, SSL-Lanes, presents the first systematic study on how to incorporateself-supervision in a standard data-driven motion forecasting model.
Revised: Contributions: Our work, SSL-Lanes, presents the ﬁrst systematic study on how to incorporate self-supervision in a standard data-driven motion forecasting model.

Original: Our contributions are: (a) 
Revised: Our contributions are: (a) 6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand. [Figure]

Original: We demonstrate the effectiveness of incorporating self-supervised learning in motion forecasting.
Revised: We demonstrate the effectiveness of incorporating self-supervised learning in motion forecasting.

Original: Since this does not add extra parameters or compute during inference, SSL-Lanes achieves the bestaccuracy-simplicity-efficiency trade-off on the challenging large-scale Argoverse [1] benchmark.(b)
Revised: Since this does not add extra parameters or compute during inference, SSL-Lanes achieves the best accuracy-simplicity-efﬁciency trade-off on the challenging large-scale Argoverse [1] benchmark.

Original:  We propose four self-supervised tasks based on the nature of the motion forecasting problem.
Revised: (b) We propose four self-supervised tasks based on the nature of the motion forecasting problem.

Original: The key idea is to leverage easily accessible map/agent-level information to define domain-specificpretext tasks that encourage the standard model to capture more superior and generalizable representations for forecasting.
Revised: The key idea is to leverage easily accessible map/agent-level information to deﬁne domain-speciﬁc pretext tasks that encourage the standard model to capture more superior and generalizable representations for forecasting in comparison to pure supervised learning.

Original: (c) We further design experiments to explore why forecasting benefitsfrom SSL.
Revised: (c) We further design experiments to explore why forecasting beneﬁts from SSL.

Original: We provide extensive results to hypothesize that SSL-Lanes learns richer features fromthe SSL training as compared to a model trained with vanilla supervised learning.
Revised: We provide extensive results to hypothesize that SSL-Lanes learns richer features from the SSL training as compared to a model trained with vanilla supervised learning.

Original:  Related Work
Revised:  Related Work

Original: Motion Forecasting: Traditional methods for motion forecasting primarily use Kalman filtering[17] with a prior from HD-maps to predict future motion states [18, 19].
Revised: Motion Forecasting: Traditional methods for motion forecasting primarily use Kalman ﬁltering [17] with a prior from HD-maps to predict future motion states [18, 19].

Original: With the huge successof deep learning, recent works use data-driven approaches for motion forecasting.
Revised: With the huge success of deep learning, recent works use data-driven approaches for motion forecasting.

Original: These methodsexplore different architectures involving rasterized images and CNNs [3, 20, 21], vectorized representations and GNNs [12, 11, 22, 4, 7], point-cloud representations [23], transformers [8, 9, 15, 14]and sophisticated fusion mechanisms [2], to generate features that predict final output trajectories.
Revised: These methods explore different architectures involving rasterized images and CNNs [3, 20, 21], vectorized representations and GNNs [12, 11, 22, 4, 7], point-cloud representations [23], transformers [8, 9, 15, 14] and sophisticated fusion mechanisms [2], to generate features that predict ﬁnal output trajectories.

Original: While the focus of these works is to find more effective ways of feature extraction from HD-mapsand interacting agents, they need huge model capacity, heavy parameterization, and extensive augmentations or large amounts of data to converge to a general solution.
Revised: While the focus of these works is to ﬁnd more effective ways of feature extraction from HD-maps and interacting agents, they need huge model capacity, heavy parameterization, and extensive augmentations or large amounts of data to converge to a general solution.

Original: Other works [5, 10, 24, 25]build on them to incorporate prior knowledge in the form of predefined candidate trajectories obtained from sampling or clustering strategies from training data.
Revised: Other works [5, 10, 24, 25] build on them to incorporate prior knowledge in the form of predeﬁned candidate trajectories obtained from sampling or clustering strategies from training data.

Original: However the disadvantage of thesemethods is that their performance is highly related to the quality of the trajectory proposals, whichbecomes an extra dependency.
Revised: However the disadvantage of these methods is that their performance is highly related to the quality of the trajectory proposals, which becomes an extra dependency.

Original: End-to-end solutions for optimizing end-points of these candidatestrajectories are proposed by Dense-TNT [6] and HOME [26].
Revised: End-to-end solutions for optimizing end-points of these candidates trajectories are proposed by Dense-TNT [6] and HOME [26].

Original: Dense-TNT has state-of-the-art accuracy with a reasonable parameter budget, but its online dense goal candidate optimization strategy iscomputationally very expensive, which is unrealistic for real-time operations like autonomous driv- [Figure]
Revised: Dense-TNT has state-of-the-art accuracy with a reasonable parameter budget, but its online dense goal candidate optimization strategy is computationally very expensive, which is unrealistic for real-time operations like autonomous driving.

Original: ing. Lately, ensembling techniques like MultiPath++ [27] and DCMS [28] have been proposed andwhile they have high forecasting performance, a major disadvantage is their high memory cost fortraining and heavy computational cost at inference.
Revised: Lately, ensembling techniques like MultiPath++ [27] and DCMS [28] have been proposed and while they have high forecasting performance, a major disadvantage is their high memory cost for training and heavy computational cost at inference.

Original: Self-supervised Learning
Revised: Self-supervised Learning

Original: : SSL is a rapidly emerging learning framework that generates additionalsupervised signals to train deep learning models through carefully designed pretext tasks. 
Revised: : SSL is a rapidly emerging learning framework that generates additional supervised signals to train deep learning models through carefully designed pretext tasks. In the [Figure]

Original: In theimage domain, various self-supervised learning techniques have been developed for learning highlevel image representations, including predicting the relative locations of image patches [29], jigsawpuzzle [30], image rotation [31], image clustering [32], image inpainting [33], image colorization[34] and segmentation prediction [35].
Revised: [Figure] ForecastingModel (Sec. 3) [Figure] image domain, various self-supervised learning techniques have been developed for learning highlevel image representations, including predicting the relative locations of image patches [29], jigsaw puzzle [30], image rotation [31], image clustering [32], image inpainting [33], image colorization [34] and segmentation prediction [35].

Original: In the domain of graphs and graph neural networks, pretexttasks include graph partitioning, node clustering, context prediction and graph completion [36, 37,38, 39].
Revised: In the domain of graphs and graph neural networks, pretext tasks include graph partitioning, node clustering, context prediction and graph completion [36, 37, 38, 39].

Original: To the best of our knowledge, this is the first principled approach that explores motionforecasting for autonomous driving with self-supervision.
Revised: To the best of our knowledge, this is the ﬁrst principled approach that explores motion forecasting for autonomous driving with self-supervision.

Original:  Background
Revised:  Background

Original: Problem Formulation: We are given the past motion of N actors.
Revised: Problem Formulation: We are given the past motion of N actors.

Original: The i -th actor is denoted as aset of center locations over the past L time-steps.
Revised: The i -th actor is denoted as a set of center locations over the past L time-steps.

Original: We pre-process it to represent each trajectory asa sequence of displacements P i = { ∆ p − L +1 i , ..., ∆ p − 1 i , ∆ p 0 i } , where p li is the 2D displacementfrom time step l − 1 to l .
Revised: We pre-process it to represent each trajectory as a sequence of displacements P i = { ∆ p − L +1 i , ..., ∆ p − 1 i , ∆ p 0 i } , where p li is the 2D displacement from time step l − 1 to l .

Original: We are also given a high-definition (HD) map, which contains lanes andsemantic attributes.
Revised: We are also given a high-deﬁnition (HD) map, which contains lanes and semantic attributes.

Original: Each lane is composed of many consecutive lane nodes, with a total of M nodes.
Revised: Each lane is composed of many consecutive lane nodes, with a total of M nodes.

Original: X ∈ R M
Revised: X ∈ R M

Original: ×
Revised: ×

Original: F denotes the lane node feature matrix, where x j = X
Revised: F denotes the lane node feature matrix, where x j = X

Original: [ j, :] T is the F -dimensional lanenode vector.
Revised: [ j, :] T is the F -dimensional lane node vector.

Original: Following the connections between lane centerlines (i.e., predecessor, successor, leftneighbour and right neighbour), we represent the connectivity within the lane nodes with 4 adjacencymatrices { A f } f ∈{ pre,suc,left,right } , with A f ∈ R M × M .
Revised: Following the connections between lane centerlines (i.e., predecessor, successor, left neighbour and right neighbour), we represent the connectivity within the lane nodes with 4 adjacency matrices { A f } f ∈{ pre,suc,left,right } , with A f ∈ R M × M .

Original: This implies that if A f,gh = 1 , then node h isan f -type neighbor of node g .
Revised: This implies that if A f,gh = 1 , then node h is an f -type neighbor of node g .

Original: Our goal is to forecast the future motions of all actors in the scene
Revised: Our goal is to forecast the future motions of all actors in the scene

Original: O 1: T GT =
Revised: O 1: T GT =

Original: { ( x 1 i , y 1 i ) , ..., ( x Ti , y Ti ) | i = 1 , ..., N } , where T is our prediction horizon.
Revised: { ( x 1 i , y 1 i ) , ..., ( x Ti , y Ti ) | i = 1 , ..., N } , where T is our prediction horizon.

Original: Standard Motion Forecasting Model: We briefly introduce a standard data-driven motion forecasting framework, consisting of a feature encoder, interaction-modeler and prediction header.
Revised: Standard Motion Forecasting Model: We brieﬂy introduce a standard data-driven motion forecasting framework, consisting of a feature encoder, interaction-modeler and prediction header.

Original: Feature Encoding: We first encode the agent and map inputs similar to Lane-GCN [2].
Revised: Feature Encoding: We ﬁrst encode the agent and map inputs similar to Lane-GCN [2].

Original: The agentencoder includes a 1D convolution with a feature pyramid network, parameterized by g enc , as givenby Eq.
Revised: The agent encoder includes a 1D convolution with a feature pyramid network, parameterized by g enc , as given by Eq.

Original: For map-encoding, we adopt two Lane-Conv residual blocks, parameterized by Θ ={ W 0 , W left , W right , W pre ,k , W suc ,k } , where k ∈ { 1 , 2 , 4 , 8 , 16 , 32 } , as given by Eq.[Equation] [Equation]
Revised: For map-encoding, we adopt two Lane-Conv residual blocks, parameterized by Θ = { W 0 , W left , W right , W pre ,k , W suc ,k } , where k ∈ { 1 , 2 , 4 , 8 , 16 , 32 } , as given by Eq.

Original: [Equation]
Revised: [Equation]

Original: Modeling Interactions: Since the behavior of agents depends on map topology and social consistency, each encoded agent i subsequently aggregates context from the surrounding map features and
Revised: Modeling Interactions: Since the behavior of agents depends on map topology and social consistency, each encoded agent i subsequently aggregates context from the surrounding map features and

Original: [Table]
Revised: [Table]

Original: its neighboring agent features, via spatial attention [40] as given by Eq.
Revised: its neighboring agent features, via spatial attention [40] as given by Eq.

Original: [Equation]
Revised: [Equation]

Original: Here, y j is the feature of the j -th node, ˆ p i is the feature of the i -th agent, ϕ the composition of layernormalization and ReLU, and ∆ ij =
Revised: Here, y j is the feature of the j -th node, ˆ p i is the feature of the i -th agent, φ the composition of layer normalization and ReLU, and ∆ ij =

Original: MLP ( v j − v i ) , where v denotes the ( x, y ) 2-D BEV locationof the agent or the lane node.
Revised: MLP ( v j − v i ) , where v denotes the ( x, y ) 2-D bird’s-eye-view (BEV) location of the agent or the lane node.

Original: The parameters for map and agent feature aggregation is representedby Λ = { W M2A , W 1 , W 2 , W A2A , W 3 , W 4 } .
Revised: The parameters for map and agent feature aggregation is represented by Λ = { W M2A , W 1 , W 2 , W A2A , W 3 , W 4 } .

Original: Trajectory Prediction: Finally, we decode the future trajectories from the features ´ p i correspondingto the agents of interest as given by: O 1: T pred =
Revised: Trajectory Prediction: Finally, we decode the future trajectories from the features ´ p i corresponding to the agents of interest as given by: O 1: T pred =

Original: { g dec ( ´ p i ) | i = 1 , ..., N } , where g dec is the parameterized trajectory decoder.
Revised: { g dec ( ´ p i ) | i = 1 , ..., N } , where g dec is the parameterized trajectory decoder.

Original: The parameters for the motion forecasting model are learned by minimizingthe supervised loss ( L sup ) calculated between the predicted output and the ground-truth future trajectories ( O 1: T GT ), as given by Eq.
Revised: The parameters for the motion forecasting model are learned by minimizing the supervised loss ( L sup ) calculated between the predicted output and the ground-truth future trajectories ( O 1: T GT ), as given by Eq.

Original: [Equation]
Revised: [Equation]

Original:  [Figure] [Equation] [Figure]
Revised:  SSL-Lanes

Original: The goal of our proposed SSL-Lanes framework is to improve the performance of the primarymotion forecasting baseline by learning simultaneously with various self-supervised tasks.
Revised: The goal of our proposed SSL-Lanes framework is to improve the performance of the primary motion forecasting baseline by learning simultaneously with various self-supervised tasks.

Original: Fig. 2shows the pipeline of our proposed approach, and Tab. 1 summarizes the self-supervised tasks.
Revised: Fig.shows the pipeline of our proposed approach, and Tab. 1 summarizes the self-supervised tasks.

Original: Self-Supervision meets Motion Forecasting:
Revised: Self-Supervision meets Motion Forecasting:

Original: Considering our motion forecasting task and a selfsupervised task, the output and the training process can be formulated as:
Revised: Considering our motion forecasting task and a selfsupervised task, the output and the training process can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where, L ss ( · , · ) is the loss function of the self-supervised task, Θ ss parameterizes the correspondingtask-specific layers, and α 1 , α 2 ∈ R > 0 are the weights for the supervised and self-supervised losses.
Revised: where, L ss ( · , · ) is the loss function of the self-supervised task, Θ ss parameterizes the corresponding task-speciﬁc layers, and α 1 , α 2 ∈ R > 0 are the weights for the supervised and self-supervised losses.

Original: If the pretext task only focuses on the map encoder, then Ψ = { Θ } and Ω = { g enc , Λ , g dec } .
Revised: If the pretext task only focuses on the map encoder, then Ψ = { Θ } and Ω = { g enc , Λ , g dec } .

Original: Otherwise, Ψ = { g enc , Θ , Λ } and Ω = { g dec } .
Revised: Otherwise, Ψ = { g enc , Θ , Λ } and Ω = { g dec } .

Original: Henceforth, we also define the following representations.
Revised: Henceforth, we also deﬁne the following representations.

Original: We will represent the primary task encoder as function f Ψ , parameterized by Ψ .
Revised: We will represent the primary task encoder as function f Ψ , parameterized by Ψ .

Original: Furthermore, givena pretext task, which we will design in the next section, the pretext decoder p Θ ss is a function thatpredicts pseudo-labels and is parameterized by Θ ss .
Revised: Furthermore, given a pretext task, which we will design in the next section, the pretext decoder p Θ ss is a function that predicts pseudo-labels and is parameterized by Θ ss .

Original: 4.1 Pretext tasks for Motion Forecasting
Revised: 4.1 Pretext tasks for Motion Forecasting

Original: At the core of our SSL-Lanes approach is defining pretext tasks based upon self-supervised information from the underlying map structure and the overall temporal prediction problem itself (Tab. 1).
Revised: At the core of our SSL-Lanes approach is deﬁning pretext tasks based upon self-supervised information from the underlying map structure and the overall temporal prediction problem itself (Tab. 1).

Original: 4.1.1 Lane-Masking
Revised: 4.1.1 Lane-Masking

Original: The goal of the Lane-Masking pretext task is to encourage the map encoder Ψ
Revised: The goal of the Lane-Masking pretext task is to encourage the map encoder Ψ

Original: =
Revised: =

Original: { Θ } to learn localstructure information in addition to the forecasting task that is being optimized.
Revised: { Θ } to learn local structure information in addition to the forecasting task that is being optimized.

Original: Specifically, werandomly mask (i.e., set equal to zero) the features of m a percent of nodes per lane and then ask theself-supervised decoder to reconstruct these features.
Revised: Speciﬁcally, we randomly mask (i.e., set equal to zero) the features of m a percent of nodes per lane and then ask the self-supervised decoder to reconstruct these features

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Here, ˜ X is the node feature matrix corrupted with random masking, i.e., some rows of X corresponding to nodes v i are set to zero.
Revised: Here, ˜ X is the node feature matrix corrupted with random masking, i.e., some rows of X corresponding to nodes v i are set to zero.

Original: p Θ ss is a fully connected network that maps the representationsto the reconstructed features.
Revised: p Θ ss is a fully connected network that maps the representations to the reconstructed features.

Original: L mse is the mean squared error (MSE) loss function penalizing thedistance between the reconstructed map features p Θ ss ([ f Ψ ( ˜ X , A f )] v i ) for node v i and its actualfeatures X i .
Revised: L mse is the mean squared error (MSE) loss function penalizing the distance between the reconstructed map features p Θ ss ([ f Ψ ( ˜ X , A f )] v i ) for node v i and its actual features X i .

Original: 4.1.2 Distance to Intersection
Revised: 4.1.2 Distance to Intersection

Original: Distance-to-Intersection pretext task is proposed to guide the map-encoder, Ψ = { Θ } , to maintainglobal topology information by predicting the distance (in terms of shortest path length) from alllane nodes to intersection nodes.
Revised: Distance-to-Intersection pretext task is proposed to guide the map-encoder, Ψ = { Θ } , to maintain global topology information by predicting the distance (in terms of shortest path length) from all lane nodes to intersection nodes.

Original: We aim to regress the distances from each lane node to prelabeled intersection nodes annotated as part of the dataset.
Revised: We aim to regress the distances from each lane node to prelabeled intersection nodes annotated as part of the dataset.

Original: Given K labeled intersection nodes
Revised: Given K labeled intersection nodes

Original: V intersection = { v intersection ,k | k
Revised: V intersection = { v intersection ,k | k

Original: = 1 , ...K } , we first generate reliable pseudo labels using breadth-firstsearch (BFS).
Revised: = 1 , ...K } , we ﬁrst generate reliable pseudo labels using breadth-ﬁrst search (BFS).

Original: Specifically, BFS calculates the shortest distance d i ∈ R for every lane node v i fromthe given set V intersection .
Revised: Speciﬁcally, BFS calculates the shortest distance d i ∈ R for every lane node v i from the given set V intersection .

Original: The target of this task is to predict the pseudo-labeled distances using apretext decoder.
Revised: The target of this task is to predict the pseudo-labeled distances using a pretext decoder.

Original: If p Θ ss ([ f Ψ ( X , A f )] v i ) is the prediction of node v i , and L mse is the mean-squarederror loss function for regression, then the loss formulation for this SSL pretext task is as follows:
Revised: If p Θ ss ([ f Ψ ( X , A f )] v i ) is the prediction of node v i , and L mse is the mean-squared error loss function for regression, then the loss formulation for this SSL pretext task is as follows:

Original: [Equation]
Revised: [Equation]

Original: 4.1.3 Maneuver Classification
Revised: 4.1.3 Maneuver Classiﬁcation

Original: We propose Maneuver Classification , and we expect it to provide prior regularization to Ψ
Revised: We propose Maneuver Classiﬁcation , and we expect it to provide prior regularization to Ψ

Original: ={ g enc , Θ , Λ } , based on driving modes of agents.
Revised: = { g enc , Θ , Λ } , based on driving modes of agents.

Original: We aim to construct pseudo label to divide agents into different clusters according to their driving behavior and thus explore unsupervised clustering algorithms to acquire the maneuver for each agent.
Revised: We aim to construct pseudo label to divide agents into different clusters according to their driving behavior and thus explore unsupervised clustering algorithms to acquire the maneuver for each agent.

Original: We find that using naive k -Means (on agent end-points) or DBSCAN (on Hausdorff distance between entiretrajectories [41]) leads to noisy clustering.
Revised: We ﬁnd that using naive k -Means (on agent end-points) or DBSCAN (on Hausdorff distance between entire trajectories [41]) leads to noisy clustering.

Original: We find that constrained k -means [42] on agentend-points works best to divide trajectory samples into C clusters equally.
Revised: We ﬁnd that constrained k -means [42] on agent end-points works best to divide trajectory samples into C clusters equally.

Original: We define C ={ maintain-speed , accelerate , decelerate , turn-left , turn-right , lane-change } and the clustering function as ρ .
Revised: We deﬁne C = { maintain-speed , accelerate , decelerate , turn-left , turn-right , lane-change } and the clustering function as ρ .

Original: If p Θ ss ( f Ψ ( P i , X , A f )) is the prediction of agent i ’s intention and E i =
Revised: If p Θ ss ( f Ψ ( P i , X , A f )) is the prediction of agent i ’s intention and E i =

Original: ( x T i, GT , y Ti, GT )is its ground-truth end-point, then the learning objective is to classify each agent maneuver into itscorresponding cluster using cross-entropy loss L ce as:
Revised: ( x Ti, GT , y Ti, GT ) is its ground-truth end-point, then the learning objective is to classify each agent maneuver into its corresponding cluster using cross-entropy loss L ce as:

Original: [Equation]
Revised: [Equation]

Original: 4.1.4 Forecasting Success/Failure Classification
Revised: 4.1.4 Forecasting Success/Failure Classiﬁcation

Original: We propose a pretext task called Success/Failure Classification , which trains an agent specialized atachieving end-point goals and thus links directly to the forecasting task.
Revised: We propose a pretext task called Success/Failure Classiﬁcation , which trains an agent specialized at achieving end-point goals and thus links directly to the forecasting task.

Original: We expect this to constrainΨ = { g enc , Θ , Λ } to predict trajectories ϵ distance away from the correct final end-point.
Revised: We expect this to constrain Ψ = { g enc , Θ , Λ } to predict trajectories (cid:15) distance away from the correct ﬁnal end-point.

Original: Similar tomaneuver classification, we wish to create pseudo-labels for our data samples.
Revised: Similar to maneuver classiﬁcation, we wish to create pseudo-labels for our data samples.

Original: We label trajectorypredictions as successful ( c = 1 ) if the final prediction ( x Ti, pred , y Ti, pred ) is within ϵ < 2 m of thefinal end-point E i , and as failure ( c = 0 ) otherwise.
Revised: We label trajectory predictions as successful ( c = 1 ) if the ﬁnal prediction ( x Ti, pred , y Ti, pred ) is within (cid:15) < 2 m of the ﬁnal end-point E i , and as failure ( c = 0 ) otherwise.

Original: We choose 2 m as our ϵ threshold because it isalso used for miss-rate calculation (Sec. 5).
Revised: We choose 2 m as our (cid:15) threshold because it is also used for miss-rate calculation (Sec. 5).

Original: If the pretext decoder predicts agent i ’s final-endpointas p Θ ss ( f Ψ ( P i , X , A f )) and, given the ground-truth end-point E i , its success or failure label is c i ,
Revised: If the pretext decoder predicts agent i ’s ﬁnal-endpoint as p Θ ss ( f Ψ ( P i , X , A f )) and, given the ground-truth end-point E i , its success or failure label is c i ,

Original: [Table]
Revised: [Table]

Original: then the pretext loss can be formulated as:
Revised: then the pretext loss can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: 4.2 Learning
Revised: 4.2 Learning

Original: As all the modules are differentiable, we can train the model in an end-to-end way.
Revised: As all the modules are differentiable, we can train the model in an end-to-end way.

Original: We use the sumof classification, regression and self-supervised losses to train the model.
Revised: We use the sum of classiﬁcation, regression and self-supervised losses to train the model.

Original: Specifically, we use:
Revised: Speciﬁcally, we use:

Original: [Equation]
Revised: [Equation]

Original: For classification and regression loss design, we adopt the formulation proposed in [2].
Revised: For classiﬁcation and regression loss design, we adopt the formulation proposed in [2].

Original: L terminal =1 N (cid:80)
Revised: L terminal = 1 N (cid:80)

Original: Ni =
Revised: Ni =

Original: 1 L 2 (cid:16) ( x Ti, pred , y Ti, pred ) , ( x Ti, GT , y Ti, GT ) (cid:17) is a simple L2 loss that minimizes the distance betweenpredicted final-endpoints and the ground-truth.
Revised: 1 L 2 (cid:16) ( x Ti, pred , y Ti, pred ) , ( x Ti, GT , y Ti, GT ) (cid:17) is a simple L2 loss that minimizes the distance between predicted ﬁnal-endpoints and the ground-truth.

Original: This is because L reg is averaged across all timepoints 1 : T , and from a practical end user perspective, minimizing the endpoint loss is much moreimportant than weighting loss from all time-steps equally.
Revised: This is because L reg is averaged across all timepoints 1 : T , and from a practical end user perspective, minimizing the endpoint loss is much more important than weighting loss from all time-steps equally.

Original: Our proposed pretext tasks contribute to
Revised: Our proposed pretext tasks contribute to

Original: L ss .
Revised: L ss .

Original: During evaluation, we study each pretext task separately, and their corresponding loss formulations defined in Eq.
Revised: During evaluation, we study each pretext task separately, and their corresponding loss formulations deﬁned in Eq.

Original: (6), Eq.
Revised: (6), Eq.

Original: (7), Eq.
Revised: (7), Eq.

Original: (8), Eq.
Revised: (8), Eq.

Original: (9) are used for joint training.
Revised: (9) are used for joint training.

Original:  Experiments
Revised:  Experiments

Original: Dataset: Argoverse provides a large-scale dataset [1] for the purpose of training, validating andtesting models, where the task is to forecast 3 seconds of future motions, given 2 seconds of past observations.
Revised: Dataset: Argoverse provides a large-scale dataset, where the task is to forecast 3 seconds of future motions, given 2 seconds of past observations.

Original: This dataset has more than 30K real-world driving sequences collected in Miami (MIA)and Pittsburgh (PIT).
Revised: It has more than 300K real-world driving sequences collected in Miami (MIA) and Pittsburgh (PIT).

Original: Those sequences are further split into train, validation, and test sets, withoutany geographical overlap.
Revised: Those sequences are further split into train, validation, and test sets, without any geographical overlap.

Original: Each of them has 205,942, 39,472, and 78,143 sequences respectively.
Revised: Each of them has 205,942, 39,472, and 78,143 sequences respectively.

Original: In particular, each sequence contains the positions of all actors in a scene within the past 2 secondshistory, annotated at 10Hz.
Revised: In particular, each sequence contains the positions of all actors in a scene within the past 2 seconds history, annotated at 10Hz.

Original: It also specifies one actor of interest in the scene, with type ‘agent’,whose future 3 seconds of motion are used for the evaluation.
Revised: It also speciﬁes one actor of interest in the scene, with type ‘agent’, whose future 3 seconds of motion are used for the evaluation.

Original: Metrics: ADE is defined as the average displacement error between ground-truth trajectories andpredicted trajectories over all time steps. FDE is defined as displacement error between ground-truthtrajectories and predicted trajectories at  the final time step.
Revised: The train and validation splits additionally provide future locations of all actors within 3 second horizon labeled at 10Hz, while annotations for test sequences are withheld from the public and used for the leaderboard evaluation. HD map information is available for all sequences.

Original: We compute K likely trajectories foreach scenario with the ground truth label, where K = 1 and K = 6 are used. Therefore, minADEand minFDE are minimum ADE and FDE over the top K predictions, respectively.
Revised: Experimental Details: To normalize the data, we translate and rotate the coordinate system of each sequence so that the origin is at current position t = 0 of ‘agent’ actor and x-axis is aligned with its current direction, i.e., orientation from the agent location at t = − 1 to the agent location at t =is the positive x axis.

Original: Miss rate (MR)is defined as the percentage of the best-predicted trajectories whose FDE is within a threshold (2 m).
Revised: We use all actors and lanes whose distance from the agent is smaller than 100 meters as the input.

Original: Brier-minFDE is the minFDE plus (1 − p ) 2 , where p is the corresponding trajectory probability. Experimental Details: To normalize the data, we translate and rotate the coordinate system of eachsequence so that the origin is  at current position t = 0 of ‘agent’ actor and x-axis is aligned with its
Revised: We train the model on 4 TITAN-X GPUs using a batch size of 128 with the Adam [43] optimizer with an initial learning rate of 1 × 10 − 3 , which is decayed to 1 × 10 − 4 at 100,000 steps. The training process ﬁnishes at 128,000 steps and takes about 10 hours to complete. We provide more implementation details in the supplementary.

Original:      0LVV5DWH#PD      P L Q  ) ' (   P  0XOWL3DWK 'HQVH71 71 /DQH*&1 /DQH5&1 PP7UDQVIRUPHU 2XUV    
Revised:      0LVV5DWH#PD      P L Q  ) ' (   P  0XOWL3DWK 'HQVH71 71 /DQH*&1 /DQH5&1 PP7UDQVIRUPHU 2XUV    

Original: 0RGHO3DUDPV0E     P L Q  ) ' (   P  :D\3RLQW 'HQVH71 6FHQH7UDQVIRUPHU /DQH*& 2XUV
Revised: 0RGHO3DUDPV0E     P L Q  ) ' (   P  :D\3RLQW 'HQVH71 6FHQH7UDQVIRUPHU /DQH*& 2XUV

Original:    ,QIHUHQFHWLPHPVF     P L Q  ) ' (   P  'HQVH71 6FHQH7UDQVIRUPHU /DQH*& 2XUVcurrent direction, i.e., orientation from the agent location at t = − 1 to the agent location at t = 0is
Revised:    ,QIHUHQFHWLPHPVF     P L Q  ) ' (   P  'HQVH71 6FHQH7UDQVIRUPHU /DQH*& 2XUV

Original:  the positive x axis.
Revised:  Results 6.1 Ablation Studies

Original: We use all actors and lanes whose distance from the agent is smaller than100 meters as the input.
Revised: We ﬁrst examine the effect of incorporating our proposed pretext tasks (Sec. 4) with the standard data-driven motion forecasting baseline (Sec. 3).

Original: We train the model on 4 TITAN-X GPUs using a batch size of 128 withthe Adam [43] optimizer with an initial learning rate of 1 × 10 − 3 , which is decayed to 1 × 10 − 4 at100,000 steps.
Revised: While evaluating the importance of our proposed pretext tasks, we wish to underline that motion prediction for autonomous driving is a safety-critical task, especially at intersections where most of our data is collected, and most accidents also happen.

Original: The training process finishes at 128,000 steps and takes about 10 hours to complete.
Revised: We thus posit that in this situation, even a small error in predicting ﬁnal locations (FDE) for a given agent can lead to dangerous potential collision scenarios.

Original: All our results are based on the same model, whose architecture is described in Sec.
Revised: Results in Tab.

Original: We providemore implementation and hyper-parameter details in the supplementary.
Revised: 2 show that all proposed pretext tasks improve motion forecasting performance for Argoverse.

Original: Results
Revised: Speciﬁcally, the Lane Mask pretext task improves min-FDE by 8.9% and MR@2m by 20.3%.

Original: 6.1 Ablation Studies
Revised: Distance to Intersection improves min-FDE by 7.1% and 19.3%.

Original: We first examine the effect of incorporating our proposed pretext tasks (Sec. 4) with the standarddata-driven motion forecasting baseline (Sec. 3).
Revised: Maneuver classiﬁcation improves min-FDE by 6.3% and MR@2m by 15.4%.

Original: Results in Tab.
Revised: We expect that improving the quality of clustering for maneuvers and thus creating better pseudo-labels will improve this further.

Original: 2 show that all proposed pretexttasks improve motion forecasting performance for Argoverse.
Revised: Finally, Success/Failure classiﬁcation improves min-FDE by 9.8% and, perhaps expectedly, MR@2m by 22.4%.

Original: Specifically, the Lane Mask pretexttask improves min-FDE by 8.9% and MR@2m by 20.3%.
Revised: Moreover, since pretext tasks are not used for inference and only for training, they also do not add any extra parameters or FLOPs to the baseline, thereby increasing accuracy but at no cost to computational efﬁciency or architectural complexity.

Original: Distance to Intersection improves minFDE by 7.1% and 19.3%.
Revised: 6.2 Comparison with State-of-the-Art

Original: Maneuver classification improves min-FDE by 6.3% and MR@2m by15.4%.
Revised: Performance:

Original: We expect that improving the quality of clustering for maneuvers and thus creating betterpseudo-labels will improve this further.
Revised: We compare our approach with top entries on Argoverse [1] in Tab. 3.

Original: Finally, Success/Failure classification improvesmin-FDEby 9.8% and, perhaps expectedly, MR@2m by 22.4%. Moreover, since pretext tasks are not used forinference and only for training, they also do not add any extra parameters or FLOPs to the baseline,thereby increasing accuracy but at no cost to computational efficiency or architectural complexity. We present some multi-modal prediction trajectories on several hard cases in Fig. 6.2 Comparison with State-of-the-Art
Revised: SSL-Lanes improves the metrics for K = 1 convincingly and outperforms existing approaches w.r.t.

Original: Performance:

Original: We compare our approach with top entries on Argoverse [1] in Tab. 3.

Original: SSL-Lanesimproves the metrics for K = 1 convincingly and outperforms existing approaches w.r.t.

Original: min-ADE 1 ,min-FDE 1 and MR 1 .We are strongly competitive w.r.t.
Revised: minADE 1 , min-FDE 1 and MR 1 .

Original: min-ADE 6 , min-FDE 6 and MR 6 against topapproaches, with a relatively simple architecture.
Revised: We are strongly competitive w.r.t.

Revised: min-ADE 6 , min-FDE 6 and MR 6 .

Revised: with a relatively simple architecture.

Original: Trade-off between min-FDE and Miss-Rate: min-FDE 6 and MR 6 are both important for autonomous robots to optimize.
Revised: Trade-off between min-FDE and Miss-Rate: min-FDE 6 and MR 6 are both important for autonomous robots to optimize.

Original: Ideally we wish for both of these metrics to be low.
Revised: Ideally we wish for both of these metrics to be low.

Original: However, thereexists a frequent trade-off between them.
Revised: However, there exists a frequent trade-off between them.

Original: We compare this trade-off in Fig. 3(a) w.r.t 6 other popular motion forecasting models (in terms of citations and GitHub stars), namely: Lane-GCN [2],
Revised: We compare this trade-off in Fig. 3(a) w.r.t 6 other popular motion forecasting models (in terms of citations and GitHub stars), namely: Lane-GCN [2],

Original: Lane-RCNN [2], MultiPath [3], mm-Transformer [9], TNT [5] and Dense-TNT [6] on the Argoverse Validation Set.
Revised: Lane-RCNN [2], MultiPath [3], mm-Transformer [9], TNT [5] and Dense-TNT [6] on the Argoverse Validation Set.

Original: We are on the lowest-left of Fig. 3(a), meaning we optimize both min-FDE 6 and
Revised: We are on the lowest-left of Fig. 3(a), meaning we optimize both min-FDE 6 and

Original: MR 6 successfully in comparison to other top models.
Revised: MR 6 successfully in comparison to other top models.

Original: Trade-off between accuracy, efficiency and complexity: We are the first to point out a trade-offthat exists for current state-of-the-art motion forecasting models between forecasting performance, 
Revised: Trade-off between accuracy, efﬁciency and complexity: We are the ﬁrst to point out a trade-off that exists for current state-of-the-art motion forecasting models between forecasting performance, architectural complexity and inference speed, in this work.

Original: [Table]

Original: architectural complexity and inference speed, in this work.
Revised: This is illustrated in Fig.

Original: This is illustrated in Fig.
Revised: 3(b, c).

Original: 3(b, c).
Revised: In contrast to the popular models, our approach has high accuracy ( min-FDE 6 : 1.25m, MR 6 : 13.3%), while also having low architectural complexity (1.84M parameters) and high inference speed (3. ms).

Original: Incontrast to the popular models, our approach has high accuracy while also having low architectural complexity (1.84M parameters) and high inference speed (3.3ms).
Revised: Thus it provides a great balance for application to real-time safety-critical autonomous robots.

Original: Thus it provides a great balance for application to real-time safety-critical autonomous robots.
Revised: [Table]

Original:  When 
Revised: 6.3 When does SSL help Motion Forecasting?

Original: does
Revised: We design 6 different training and testing setups as shown in Tab. 4. We use Success/Failure classiﬁcation as the pretext task

Original: SSL help Motion Forecasting?
Revised: [Table]

Original: We design 6 different training and testing setups as shown in Tab. 4.
Revised: and all models are trained for 50,000 steps.

Original: We use Success/Failure classification as the  pretext task
Revised: We initialize the map-encoder with the parameters from a model trained with the lane-masking pretext task.

Original: [Table]
Revised: We hypothesize that training with SSL pretext tasks helps motion forecasting in the following ways: (a) Topology-based context prediction assumes feature similarity or smoothness in small neighborhoods of maps, and the resulting feature representation may improve prediction performance.

Original:  and all models are trained for 500,000 steps.
Revised: This is mainly expected to help in the ﬁrst and second settings, which requires generalizing to new topologies.

Original: We initialize the map-encoderwith the parameters from a model trained with the lane-masking pretext task.
Revised: (b) Clustering and classiﬁcation assumes that feature similarity implies target-label similarity and can group distant nodes with similar features together, leading to better generalization.

Original: We hypothesize that training with SSL pretext tasks helps motion forecasting in the following ways:(a
Revised: This is mainly expected to help with dataset imbalance and performance on difﬁcult maneuvers, which requires generalizing to hard cases.

Original: Topology-based context prediction assumes feature similarity or smoothness insmall neighborhoods of maps. Such a context-based feature representation may improve prediction performance.This is mainly expected to help in the first and second settings, which requires generalizing to newtopologies.
Revised: (c) Supervised learning with imbalanced datasets sees signiﬁcant degradation in performance.

Original: (b) Clustering and classification assumes that feature similarity implies target-label similarity and can group distant nodes with similar features together, leading to better generalization.
Revised: Although most of the data samples in Argoverse are at an intersection, a signiﬁcantly large number involve driving straight while maintaining speed.

Original: This is mainly expected to help with dataset imbalance and performance on difficult maneuvers,which requires generalizing to hard cases.
Revised: Recent studies [44] have shown that SSL tends to learn richer features from more frequent classes, which also allows it to generalize to other classes better.

Original: (c) Supervised learning with imbalanced datasets seessignificant degradation in performance.
Revised: We expect this to help with imbalanced data, limited training data and noisy data.

Original: Although most of the data samples in Argoverse are at anintersection, a significantly large number involve driving straight while maintaining speed.
Revised: SSL leads to better generalization compared to pure supervised learning: To provide evidence for our hypotheses, we design 6 different training and testing setups as shown in Tab. 4. We use

Original: Recentstudies [44] have shown that SSL tends to learn richer features from more frequent classes, whichalso allows it to generalize to other classes better. We expect this to help with imbalanced data,limited training data and noisy data. There is strong evidence from our experiments that SSL-basedtasks provide better generalization and can thus be more effective than pure supervised training.
Revised: Success/Failure classiﬁcation as the pretext task, and all models are trained for 500,000 steps.

Original: Conclusion
Revised: We initialize the map-encoder with the parameters from a model trained with the lane mask pretext task.

Original: In this paper, we propose SSL-Lanes to leverage supervisory signals generated from data for freein the form of pseudo-labels and integrate it with a standard motion forecasting model.
Revised: There is strong evidence from our experiments that SSL-based tasks provide better generalization and can thus be more effective than pure supervised training.

Original:  We validateour proposed approach by achieving competitive results on the challenging large-scale Argoversebenchmark.
Revised:  Conclusion

Original: The main advantage of SSL-Lanes is that ithas high accuracy combined with lowarchitectural complexity and high inference speed.
Revised: We propose SSL-Lanes to leverage supervisory signals generated from data for free in the form of pseudo-labels and integrate it with a standard motion forecasting model.

Original: We further demonstrate that each proposed SSLpretext task improves upon the vanilla motion forecasting baseline, especially in difficult cases likeleft/right turns and acceleration/deceleration.
Revised: We validate our proposed approach by achieving competitive results on the challenging large-scale Argoverse benchmark.