Original: CCIL: C ONTEXT - CONDITIONED IMITATION LEARNING FOR URBAN DRIVING
Revised: CCIL: C ONTEXT - CONDITIONED IMITATION LEARNING FOR URBAN DRIVING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Imitation learning is a promising solution to the challenging autonomous urban driving task as experienced human drivers can effortlessly tackle highly complex driving scenarios.
Revised: Imitation learning is a promising solution to the challenging autonomous urban driving task as experienced human drivers can effortlessly tackle highly complex driving scenarios.

Original: Behavior cloning is the most widely applied imitation learning approach in autonomous driving due to its exemption from potentially risky online interactions, but it suffers from the covariate shift issue.
Revised: Behavior cloning is the most widely applied imitation learning approach in autonomous driving due to its exemption from potentially risky online interactions, but it suffers from the covariate shift issue.

Original: To mitigate this problem, we propose a context-conditioned imitation learning approach that learns a policy to map the context state into the ego vehicle’s state instead of the typical formulation from both ego and context state to the ego action.
Revised: To mitigate this problem, we propose a context-conditioned imitation learning approach that learns a policy to map the context state into the ego vehicle’s state instead of the typical formulation from both ego and context state to the ego action.

Original: Besides, to make full use of the spatial and temporal relations in the context to infer the ego future states, we design a novel policy network based on the Transformer, whose attention mechanism has demonstrated excellent performance in capturing relations.
Revised: Besides, to make full use of the spatial and temporal relations in the context to infer the ego future states, we design a novel policy network based on the Transformer, whose attention mechanism has demonstrated excellent performance in capturing relations.

Original: Finally, during evaluation, a linear quadratic controller is employed to produce smooth planning based on the predicted states from the policy network.
Revised: Finally, during evaluation, a linear quadratic controller is employed to produce smooth planning based on the predicted states from the policy network.

Original: Experiments on the real-world large-scale Lyft and nuPlan datasets demonstrate that our method can surpass the state-of-the-art method significantly.
Revised: Experiments on the real-world large-scale Lyft and nuPlan datasets demonstrate that our method can surpass the state-of-the-art method significantly.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Planning a safe, comfortable, and efficient trajectory in a complex urban environment for a selfdriving vehicle (SDV) is an important and challenging task in autonomous driving (Yurtsever et al., 2020).
Revised: Planning a safe, comfortable, and efficient trajectory in a complex urban environment for a selfdriving vehicle (SDV) is an important and challenging task in autonomous driving (Yurtsever et al., 2020).

Original: Unlike highway driving (Henaff et al., 2019), urban driving requires handling more varied road geometry such as roundabouts and intersections while interacting with traffic lights, pedestrians, and other vehicles.
Revised: Unlike highway driving (Henaff et al., 2019), urban driving requires handling more varied road geometry such as roundabouts and intersections while interacting with traffic lights, pedestrians, and other vehicles.

Original: Classic manually-designed rule-based approaches (Fan et al., 2018) have achieved some success in industry but demand tedious human engineering to struggle with diverse real-world cases.
Revised: Classic manually-designed rule-based approaches (Fan et al., 2018) have achieved some success in industry but demand tedious human engineering to struggle with diverse real-world cases.

Original: Meanwhile, the rapid development of deep learning techniques motivates researchers (Bojarski et al., 2016; Pan et al., 2020) to employ a deep neural network to model the complicated driving policy.
Revised: Meanwhile, the rapid development of deep learning techniques motivates researchers (Bojarski et al., 2016; Pan et al., 2020) to employ a deep neural network to model the complicated driving policy.

Original: To learn such a policy, imitation learning (IL) from human drivers’ demonstrations is a promising solution as experienced drivers can tackle even extremely challenging situations, and their driving data can be collected at scale.
Revised: To learn such a policy, imitation learning (IL) from human drivers’ demonstrations is a promising solution as experienced drivers can tackle even extremely challenging situations, and their driving data can be collected at scale.

Original: The simplest IL algorithm is the behavior cloning (BC) method, which has wide applications in autonomous driving (Pomerleau, 1988; Bojarski et al., 2016; Codevilla et al., 2018b) due to its exemption from potentially dangerous online interactions.
Revised: The simplest IL algorithm is the behavior cloning (BC) method, which has wide applications in autonomous driving (Pomerleau, 1988; Bojarski et al., 2016; Codevilla et al., 2018b) due to its exemption from potentially dangerous online interactions.

Original: It learns a policy in a supervised fashion by minimizing the difference between the learner’s action and the expert’s action in the expert state distribution.
Revised: It learns a policy in a supervised fashion by minimizing the difference between the learner’s action and the expert’s action in the expert state distribution.

Original: However, the BC method suffers from the covariate shift issue (Ross et al., 2011), i.e . the state induced by the learner’s policy cumulatively deviates from the expert’s distribution.
Revised: However, the BC method suffers from the covariate shift issue (Ross et al., 2011), i.e . the state induced by the learner’s policy cumulatively deviates from the expert’s distribution.

Original: To overcome the covariate shift obstacle, existing methods such as DAgger (Ross et al., 2011) and DART (Laskey et al., 2017) query supervisor corrections at the learner’s states or perturbed expert’s states.
Revised: To overcome the covariate shift obstacle, existing methods such as DAgger (Ross et al., 2011) and DART (Laskey et al., 2017) query supervisor corrections at the learner’s states or perturbed expert’s states.

Original: Since human supervision is hard to collect, recent works like GAIL (Ho & Ermon, 2016) seek to provide feedback from a neural network-based discriminator to recover from out-of-distribution states generated by the learner’s policy.
Revised: Since human supervision is hard to collect, recent works like GAIL (Ho & Ermon, 2016) seek to provide feedback from a neural network-based discriminator to recover from out-ofdistribution states generated by the learner’s policy.

Original: However, these data augmentation methods need either expert supervision or rolling out their policy in the real world or a realistic simulator, which are impractical in autonomous driving.
Revised: However, these data augmentation methods need either expert supervision or rolling out their policy in the real world or a realistic simulator, which are impractical in autonomous driving.

Original: On the contrary, some researchers attempt to constrain the learned policy formulation to ensure its robustness to the policy error by incorporating control theoretic prior knowledge.
Revised: Instead, some researchers attempt to constrain the learned policy formulation to ensure its robustness to the policy error by incorporating control theoretic prior knowledge.

Original: For example, Palan et al.
Revised: For example, Palan et al.

Original: ; Havens & Hu (2021) pose Kalman or linear matrix inequality constraints on the learned linear policy to guarantee the closed-loop stability in a linear time-invariant (LTI) system.
Revised: ; Havens & Hu (2021) pose Kalman or linear matrix inequality constraints on the learned linear policy to guarantee its closed-loop stability in a linear time-invariant (LTI) system.

Original: Yin et al.
Revised: Yin et al.

Original: (2021) relaxes the  policy formulation into a simple feedforward neural network  and East (2022) extends the method in Havens & Hu (2021) to polynomial system and policy.
Revised: (2021) relaxes the linear policy formulation into a simple feed-forward neural network representation and East (2022) extends the method in Havens & Hu (2021) to polynomial policy and dynamical system.

Original: However, the urban driving task is too complex to be handled by these naive policy formulations.
Revised: However, the urban driving task is too complex to be handled by these naive policy formulations.

Original: To learn a stable and generalpolicy for urban driving  by imitating only the offline human demonstrations, we propose a context-conditioned imitation learning (CCIL) method, wherewe try to learn a policy  to predict SDV future states using only its past local context, different from classicdriving policy(Codevilla et al., 2019; Bansal et al., 2019) taking both SDV state and context as input to generate its action.
Revised: To learn a stable and general urban driving policy by imitating only the offline human demonstrations, we propose a context-conditioned imitation learning (CCIL) method, where a policy network is learned to predict the SDV’s future states using only its observed context, different from classic policy taking both ego state and context as input to generate its action (Codevilla et al., 2019; Bansal et al., 2019).

Original: The motivation is that the SDV state can be easily influenced by the policy’s error, thus leading to the catastrophic distribution shift while map objects remain unchanged andhuman drivers in the  context  will try to recover from SDV’s perturbation.
Revised: The motivation is that the ego state can be easily influenced by the policy’s error, thus leading to the catastrophic distribution shift. On the contrary, the static context elements such as lanes or crosswalks will not be influenced by the SDV and the dynamic context element like human drivers will try to recover from its perturbation.

Original: Based on the stability assumption of the traffic system, we  prove theoretically that our policy formulation can achieve closed-loop stability, thus addressing the distribution shift issue.
Revised: Thus, based on the stability assumption of the traffic system, we can prove theoretically that our policy formulation can achieve closed-loop stability, thus addressing the distribution shift issue.

Original: However, without the SDV state input, it becomes challenging to predict its actions accurately.
Revised: In practice, as it becomes challenging to accurately plan a trajectory for a SDV without its historical trajectory, we construct our policy network based on

Original: Therefore, we learn to infer SDV future states based on Transformer (Vaswani et al., 2017) which is capable of exploiting spatial and temporal  information in highly interactive and constrained urban driving scenarios.
Revised: Transformer (Vaswani et al., 2017) to exploit spatial and temporal relation information in highly interactive and constrained urban driving scenarios.

Original: Furthermore, during evaluation, we employ a linear-quadratic regulator (LQR) ( ˚Astr¨om & Murray, 2021) to yield the SDV’s action based on its currentstate and predicted states.
Revised: Furthermore, during evaluation, we employ a linear-quadratic regulator (LQR) ( ˚Astr ¨ om & Murray, 2021) to yield a smooth action based on the SDV’s current and predicted states.

Original: The main contributions of this paper can be summarized as follows:
Revised: The main contributions of this paper can be summarized as follows:

Original: To overcome the  issue of distribution shift, we propose a novel offline imitation learning method, wherewe learn a policy  to output the ego state using only its context as input.
Revised: To address the covariate shift issue in offline imitation learning, we propose a novel contextconditioned imitation learning method, where a policy is learned to output the ego state using only its context as input.

Original: A robustness assurance is provided for our policy formulation based on an assumption of the context’s stability.
Revised: A robustness assurance is provided for our policy formulation based on an assumption of the context’s stability.

Original: To apply our method to urban driving, we remove the explicit ego state information input and propose a new ego-perturbed goal-oriented coordinate system to reduce the implicit ego information in the coordinate system.
Revised: To apply our method to urban driving, we remove the explicit ego state information input and propose a new ego-perturbed goal-oriented coordinate system to reduce the implicit ego information in the coordinate system.

Original: Besides, we design a Transformer-based  network to make full use of the spatial and temporal information in the context.
Revised: Besides, we design a Transformer-based planning network to make full use of the spatial and temporal information in the context.

Original: To verify the effectiveness of our approach, we benchmark the real-world large-scale urban driving Lyft (Houston et al., 2020) and nuPlan (Caesar et al., 2021) datasets with state-of-the-art performance.
Revised: To verify the effectiveness of our approach, we benchmark the real-world large-scale urban driving Lyft (Houston et al., 2020) and nuPlan (Caesar et al., 2021) datasets with state-of-the-art performance.

Original: The video and code can be found at https://sites.google.com/view/ contextconditionedil .
Revised: The video and code can be found at https://sites.google.com/view/ contextconditionedil .

Original:  R ELATED W ORK 2. I MITATION LEARNING FOR AUTONOMOUS DRIVING
Revised:  R ELATED W ORK 2. I MITATION LEARNING FOR AUTONOMOUS DRIVING

Original: The objective of applying IL in autonomous driving is to learna driving behavior mimicking human drivers.
Revised: The objective of applying IL in autonomous driving is to learn driving behavior mimicking human drivers.

Original: The most straightforward IL method is BC which minimizes the difference between the learner’saction and the expert’s action on the expert state without demanding extra manually labeled data and online interaction.
Revised: The most straightforward solution is BC which minimizes the difference between the learner’s and the expert’s action on the expert state without demanding extra manually labeled data and online interaction.

Original: Early BC applications in autonomous driving such as ALVINN (Pomerleau, 1988) and PilotNet (Bojarski et al., 2016) learn an end-to-end policy that directly maps sensor inputs to vehicle control commands using a large amount of human driving experience.
Revised: Early BC applications in autonomous driving such as ALVINN (Pomerleau, 1988) and PilotNet (Bojarski et al., 2016) learn an end-to-end policy that directly maps sensor inputs to vehicle control commands using a large amount of human driving experience.

Original: Recently, ChauffeurNet (Bansal et al., 2019)instead provides intermediate planning usinginformation from a perception system to improve generalization and transparency.
Revised: Recently, ChauffeurNet (Bansal et al., 2019) provides intermediate planning using perception results to improve generalization and transparency.

Original: However, the BC approach typically leads to the covariate shift(Ross et al., 2011) between the training distribution and deploying distribution, asthe minor errors in the policy can lead to deviating from the expert state and then larger errors.
Revised: However, the BC approach typically leads to the covariate shift between the training distribution and deploying distribution, as minor errors in the policy can lead to deviating from the expert state and then larger errors.

Original: Methods to overcome the distribution shift obstacle can be categorized into online  and offline  methods.
Revised: IL methods to address the distribution shift challenge can be categorized into online methods and offline model-free and model-based methods.

Original: Online methods try to directly match the expert state-action distribution instead of matching the expert state conditioned action distribution in BC.
Revised: Online methods try to directly match the expert stateaction distribution instead of matching the expert state conditioned action distribution like BC.

Original: For instance, the method in Zhang & Cho (2017) based on DAgger (Ross et al., 2011) queries supervisor actions at the state the learner visits and then adds the new data into the dataset, thus adjusting the expert state distribution to match the learner’s state distribution.
Revised: For instance, the method in Zhang & Cho (2017) based on DAgger (Ross et al., 2011) queries supervisor actions at the state the learner visits and then adds the new data into the dataset, thus adjusting the expert state distribution to match the learner’s state distribution.

Original: To get rid of the requirement of interactive expert, several methods like Wang et al.
Revised: To get rid of the requirement of interactive expert, several methods like Wang et al.

Original: (2021) based on adversarial imitation learning (Ho & Ermon, 2016) use a discriminator to measure the difference between learner and expert’s state-action distribution, which is used to calculate the reward of reinforcement learning (RL).
Revised: based on adversarial imitation learning (Ho & Ermon, 2016) utilize a discriminator to measure the difference between learner and expert’s state- action distribution and then compute the reward in reinforcement learning.

Original: By raising the policy’s return using RL, the policy’s state distribution will get close to the expert distribution.
Revised: By raising the policy’s accumulated reward, the policy’s state distribution will get close to the expert distribution.

Original: However, the reinforcement loopin the method needs online interaction with the environment to generate the policy’s state distribution, which is hard to apply inthe safe-critical task like autonomous driving.
Revised: However, the reinforcement loop needs online interaction with the environment to generate the policy’s state distribution, which is hard to deploy in safe-critical tasks like autonomous driving.

Original: Instead, we aim to match the contextconditioned ego state distribution that can be learned by supervised learning without interaction with the environment or access toinformation from the expert policy during training.
Revised: On the contrary, our method matches ego state distribution conditioned on context, which can be learned by supervised learning without interaction with the environment or access to expert supervision.

Original: Several offline methods have been applied to alleviate the covariate shift issue.

Original: The most widely used model-free  methods are based on DART (Laskey et al., 2017)  by providing  examples of how the expert policy recovers fromerror, which synthesizes the deviated state and recovering action by perturbing the data following some rules.
Revised: The most popular model-free IL methods are based on DART (Laskey et al., 2017) which avoids the compounding error by providing synthetic examples of how to recover from the deviated state.

Original: In Codevilla et al.
Revised: In Codevilla et al.

Original: (2018b), they inject temporally correlated noise into the trajectory to simulate gradual drift away from the desired trajectory.
Revised: (2018b), they inject temporally correlated noise into the trajectory to simulate gradual drift away from the desired trajectory.

Original: Alternatively, ChauffeurNet (Bansal et al., 2019) adds a uniform perturbation to the current SDV location and fits a new smooth trajectory that brings the SDV back to the original target location.
Revised: Alternatively, ChauffeurNet (Bansal et al., 2019) adds a uniform perturbation to the current SDV state and fits a new smooth trajectory that brings the SDV back to the original target location.

Original: However, these rule-based trajectory augmentation methods are hard to cover the real motion distribution induced by the learner’s policy and the policy is very likely to learn a propensity for perturbed driving.
Revised: However, these rule-based trajectory augmentation methods are hard to cover the real motion distribution induced by the learner’s policy and the policy is very likely to learn a propensity for perturbed driving.

Original: Our method also applies perturbation to the SDV’s current position but it isused to mask the ego vehicle’s information instead of data augmentation.
Revised: Our method also applies perturbation to the SDV’s current position but its role is to blur the ego position information instead of data augmentation.

Original: Therefore, our method does not have the trajectory smoothing process during training.
Revised: Therefore, our method does not have the trajectory smoothing process during training.

Original: Rather than providing examples of how the expert policy recovers from error, we seek to endow a learned policy withthe robustness properties directly.
Revised: Our method is also model-free but we seek to endow the policy with robustness properties by constraining its formulation without resorting to recovery examples.

Original: The model-based IL methods aim at matching thesimulated trajectory distribution with the expert trajectory distribution. They train a policy network to minimize a differentiable cost over trajectories rolling out in a differentiable learned(Henaff et al., 2019) or data-driven simulator (Scheel et al., 2022). In Amos et al.
Revised: Model-based IL methods address the distribution shift by minimizing the difference between a trajectory rolling out in a differentiable learned or data-driven model with the expert trajectory.

Original: (2018), a dynamics model based Variational Auto-encoders (Kingma & Welling, 2013)is first learned from observational data. Then, they train the policy network to output actions that lead to a similar trajectory as the expert trajectory.
Revised: PPUU (Amos et al., 2018) first learn a dynamics model based on variational autoencoder (Kingma & Welling, 2013) from data and then train the policy network to output actions that lead to a similar trajectory as the expert trajectory.

Original: Since the dynamics model is differentiable, the actions can receive gradients from multiple time steps ahead which can penalize actions that will lead to large diverges in the future even if the instantaneous divergence is small.
Revised: Since the dynamics model is differentiable, the actions can receive gradients from multiple time steps ahead which can penalize actions that will lead to large diverges in the future even if the instantaneous divergence is small.

Original: Instead of learning adifferentiable model, UrbanDriver (Scheel et al., 2022) constructs a differentiable simulator usingvectorized representations of recorded perception data and High Definition (HD) maps, where newdriving observations are calculated by a coordinate transformation based on the pose of controlled SDV and collected data.
Revised: Instead of learning a model, UrbanDriver (Scheel et al., 2022) constructs a differentiable data-driven model using recorded perception data and High Definition (HD) maps, where new observations are calculated by a coordinate transformation based on the pose of controlled SDV and collected data.

Original: Then, they employ policy gradients to train a closed-loop policy by rolling it out in the simulator.

Original: However, the performance of the model-based approach is limited by the accuracy of the simulator’s dynamics.
Revised: However, the performance of the model-based approach is limited by the model’s accuracy.

Original: 2. I MITATION LEARNING WITH ROBUSTNESS
Revised: 2. I MITATION LEARNING WITH ROBUSTNESS

Original: Imitation learning is different from supervised learning by deploying the policy under dynamics, whose robustness is considered to be theability of the learned policy to recover from policy errors.
Revised: IL is different from supervised learning by deploying the policy under dynamics, whose robustness is considered to be the learned policy’s ability to recover from policy errors.

Original: Some researchers have attempted to learn a policy with a stability guarantee using control-theoretic methods to apply policy constraints.
Revised: Some researchers have attempted to learn a policy with a stability guarantee using control-theoretic methods by constraining policy and system dynamics.

Original: In Taylor Series IL (Pfrommer et al., 2022), it is shown that the trajectory induced by the learner and expert will be close if the derivatives of the learner and expert policy match atthe expert state. However, the high-order derivatives of the expertand learner policy are hard to compute without sufficient data.
Revised: Taylor Series IL (Pfrommer et al., 2022) proves that the trajectory induced by the learner and expert will be close if their derivative difference at expert states is small but computing the high-order derivatives of the expert policy are difficult without sufficient data.

Original: Others learn a robust linear (Palan et al., 2020; Havens & Hu, 2021) or simple feed-forward neural network (Yin et al., 2021) control policy for a linear dynamical system by posing constraints on the policy.
Revised: Others learn a robust linear (Palan et al., 2020; Havens & Hu, 2021) or simple feed-forward neural network (Yin et al., 2021) control policy for a linear dynamical system by posing constraints on the policy.

Original: Even though the authors in East (2022) extend the robustness guarantee to the polynomial systems and policy, obtaining guarantees on close-loop stability for the nonlinear autonomous driving system remains a challenge.
Revised: Even though the authors in East (2022) extend the robustness guarantee to the polynomial system and policy, obtaining guarantees on close-loop stability for the nonlinear autonomous driving system remains a challenge.

Original: Recently, the CMILe method (Tu et al., 2022) is capable of training nonlinear policies with the same safety guarantees as the expert but requires online expert access as DAgger.
Revised: Recently, the CMILe method (Tu et al., 2022) is capable of training nonlinear policies with the same safety guarantees as the expert but requires online expert access as DAgger.

Original: However, our method only constrains our policy in the formulation of receiving context state and producing ego state with a norm-constrained loss. Under a  stability assumption of the environment subsystem, the stability of our policy is guaranteed.
Revised: However, our method only constrains our policy in the formulation of receiving context state and producing ego state, whose stability is guaranteed under a mild input-to-state stability assumption on the environment dynamics.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: We first introduce the notations and definitions used in this paper.
Revised: We first introduce the notations and definitions used in this paper.

Original: For any vector x ∈ R n , ∥ x ∥ p stands for its L p norm and ∥ x ∥ for L 2 norm.
Revised: For any vector x ∈ R n , ∥ x ∥ p stands for its L p norm and ∥ x ∥ for L 2 norm.

Original: For any square matrix A ∈ R n × n , ∥ A ∥ denotes its induced L 2 norm and ρ ( A ) is its spectral radius (the maximum of the absolute values of its eigenvalues).
Revised: For any square matrix A ∈ R n × n , ∥ A ∥ denotes its induced L 2 norm and ρ ( A ) is its spectral radius (the maximum of the absolute values of its eigenvalues).

Original: For every induced matrix norm,we have ρ ( A ) ⩽ ∥ A ∥ .
Revised: For every induced matrix norm,

Original: Definition 1 (Comparison Functions)
Revised: Definition 1 (Comparison Functions)

Original: A function γ : R ⩾ 

Revised: A function γ : R ⩾

Original: → R ⩾ 0 is class K if it is continuous, strictly increasing and satisfies γ (0)
Revised: → R ⩾ 0 is class K if it is continuous, strictly increasing and satisfies γ (0)

Original: = 0 .
Revised: = 0 .

Original: A function β ( x, t ) : R ⩾ 

Revised: A function β ( x, t ) : R ⩾

Original: × R ⩾ 

Revised: × R ⩾

Original: → R ⩾ 0 is class KL if it is continuous, β ( · , t ) is class K for each t and β ( x, · ) is decreasing for each x .
Revised: → R ⩾ 0 is class KL if it is continuous, β ( · , t ) is class K for each t and β ( x, · ) is decreasing for each x .

Original: Definition 2 (input-to-state stability (ISS) (Sontag, 2008))
Revised: Definition 2 (input-to-state stability (ISS) (Sontag, 2008))

Original: A discrete-time system x t + = f
Revised: A discrete-time system x t + = f

Original: ( x t , u t ) , x 0 = ξ is input-to-state stable if there exists a class KL function β and a class K function γ such that, for each bounded input u and initial condition ξ and t ∈ Z + , it holds that:
Revised: ( x t , u t ) , x 0 = ξ is input-to-state stable if there exists a class KL function β and a class K function γ such that, for each bounded input u and initial condition ξ and t ∈ Z + , it holds that:

Original: [Equation]
Revised: [Equation]

Original: where ∥ u ∥ ∞ = sup t ∈ Z + ( u t ) is the sup norm of the input.
Revised: where ∥ u ∥ ∞ = sup t ∈ Z + ( u t ) is the sup norm of the input.

Original: If a system satisfies the ISS property, the distance between any two trajectories must eventually be bounded by its input signal and independent of initial conditions.
Revised: If a system satisfies the ISS property, the distance between any two trajectories must eventually be bounded by its input signal and independent of initial conditions.

Original: We consider performing imitation learning in the following nonlinear discrete-time system:
Revised: We consider performing imitation learning in a nonlinear discrete-time system:

Original: [Equation]
Revised: [Equation]

Original: where x t ∈ R n is the system state at time t ∈ N and u t ∈ R u is the control input.
Revised: where x t ∈ R n is the system state at time t ∈ N and u t ∈ R u is the control input.

Original: For an autonomous driving system, its state can be separated into two parts x t =
Revised: For an autonomous driving system, its state can be separated into two parts x t =

Original: ( s t , c t ) : an SDV state s t ∈ R m controlled by the control input u and context state c t ∈ R n − m which is influenced by the
Revised: ( s t , c t ) : an SDV state s t ∈ R m controlled by the control input u and context state c t ∈ R n − m which is influenced by the

Original: SDV state.
Revised: SDV state.

Original: By viewing the policy error as a disturbance to the system, we apply linearization to the nonlinear system at the expert’s state x ∗ to obtain the following linear subsystem:
Revised: By viewing the policy error as a disturbance to the system, we apply linearization to the nonlinear system at the expert’s state x ∗ to obtain the following linear subsystem:

Original: [Equation]
Revised: [Equation]

Original: with solution:
Revised: with solution:

Original: [Equation]
Revised: [Equation]

Original: Here by an abuse of notation, we use the same symbol c t , s t for the context deviation c t − c ∗ t and
Revised: Here with abuse of notation, we use the same symbol c t , s t for the context deviation c t − c ∗ t and

Original: SDV state deviation s t − s ∗ t .
Revised: SDV state deviation s t − s ∗ t .

Original: Because the expert traffic system  c t +1 = Ac twithout the SDV state deviation is stable, we have ρ ( A ) < 1 .
Revised: Because the real-world traffic system without the SDV state deviation c t +1 = Ac t is stable, we have ρ ( A ) < 1 .

Original: For such a Schur matrix, there are constants c > 0 and 0 ⩽ σ < 1 such that ∥ A t ∥ ⩽ cσ t (Jiang & Wang, 2001).
Revised: For such a Schur matrix, there are constants c > 0 and 0 ⩽ σ < 1 such that ∥ A t ∥ ⩽ cσ t (Jiang & Wang, 2001).

Original: Besides, the context in traffic systems generally includes static map elements that are not influenced by SDV vehicles and intelligent human drivers who can quickly recover from the other vehicle’s perturbation.
Revised: Besides, the context in traffic systems generally includes static map elements that are not influenced by SDV vehicles and intelligent human drivers who can quickly recover from the other vehicle’s perturbation.

Original: It is natural to assume the subsystem in equationto be input-to-state stable (ISS) with β ( r, t ) <
Revised: It is natural to assume the subsystem in equationto be input-to-state stable (ISS) with β ( r, t ) <

Original: σ t − 1 r and γ ( r ) ⩽ ϵr , where ϵ ⩾ 0 .
Revised: σ t − 1 r and γ ( r ) ⩽ ϵr , where ϵ ⩾ 0 .

Original: Combining equation 1 and equation 4, we need
Revised: Combining equation 1 and equation 4, we have:

Original: [Equation]
Revised: [Equation]

Original: which implies that c < 1 σ , and
Revised: which implies that c < 1 σ , and

Original: ∥ B ∥ ⩽ ϵ (1 − σ ) c .
Revised: ∥ B ∥ ⩽ ϵ (1 − σ ) c .

Original: Next, we study how theformulation of our policy  can be stable under this ISS system.
Revised: Next, we study how the policy in our formulation can be stable under this ISS system.

Original: We learn a context state feedback control s t = u t = f θ ( c t ) which directly maps the context state to the
Revised: We learn a context state feedback control s t = u t = f θ ( c t ) which directly maps the context state to the

Original: SDV state.
Revised: SDV state.

Original: For simplicity, we consider a linear policy u t =
Revised: For simplicity, we consider a linear policy u t =

Original: Kc t .
Revised: Kc t .

Original: Then, the linear subsystem in equation 3 can be simplified as c t +

Revised: Then, the linear subsystem in equation 3 can be simplified as c t +

Original: =
Revised: =

Original: ( A + BK ) c t whose stable condition is ρ ( A + BK ) < 1 .
Revised: ( A + BK ) c t whose stable condition is ρ ( A + BK ) < 1 .

Original: Thus, we have if ∥ K ∥ <
Revised: Thus, we have if ∥ K ∥ <

Original: c
Revised: c

Original: (1 − cσ )
Revised: (1 − cσ )

Original: ϵ (1 − σ ) , then
Revised: ϵ (1 − σ ) , then

Original: [Equation]
Revised: [Equation]

Original: This result shows that the stability of closed-loop autonomous driving can be guaranteed as long as the context system is stable enough and the norm of our policy is sufficiently small.
Revised: This result shows that the stability of closed-loop autonomous driving can be guaranteed as long as the context system is stable enough and the norm of our policy is sufficiently small.

Original: In practice, we will add a L 2 norm regularization into the training loss of our neural network policy to internalize this stability prior.
Revised: In practice, we will add a L 2 norm regularization into the training loss of our neural network policy to internalize this stability prior.

Original: Finally, we show how the naive BC policy formulation which takes both the SDV state and context state and generates the displacement from the current state can fail with even a fixed context c t = 0 .
Revised: Finally, we show how the naive BC policy formulation which takes both the SDV state and context state and generates the displacement from the current state can fail with even a fixed context c t = 0 .

Original: We consider a linear policy: u t =s t + K bc s t and update the state s t
Revised: We consider a linear policy: u t = K bc s t and update the state s t

Original: +1 =
Revised: +1 =

Original: s t + u t .
Revised: s t + u t .

Original: Then, the dynamics is s t +

Revised: Then, the dynamics is s t +

Original: = ( I + K bc ) s t .
Revised: = ( I + K bc ) s t .

Original: Because ∥ I ∥ = 1 , we cannot obtain the stable condition ρ ( I + K bc ) < 1 by limiting the ∥ K bc ∥ as our policy formulation.
Revised: Because ∥ I ∥ = 1 , we cannot obtain the stable condition ρ ( I + K bc ) <by limiting the ∥ K bc ∥ as our policy formulation.

Original: Thus, the system’s stability cannot be guaranteed even with a fixed context.
Revised: Thus, the system’s stability cannot be guaranteed.

Original: This simple analysis supports our choice to overlook the SDV history and current state in the learning to avoid causal confounding and distribution shift.
Revised: This simple analysis supports our choice to overlook the SDV history and current state in the learning to avoid causal confounding and distribution shift.

Original: Its advantage will also be validated in our experiments showing that this can significantly decrease the off-road rate.
Revised: Its advantage will also be validated in our experiments showing that this can significantly decrease the off-road rate.

Original:  M ETHOD
Revised:  M ETHOD

Original: As shown in Figure 1,  our architecture is composed oftwo modules: a neural planner which takes as input the environment state around the SDV to predict its future trajectories, and  a LQR controllerusing the coarse planning to generatea smooth  trajectory during testing.
Revised: As shown in Figure 1, we apply our context-conditioned imitation learning method to learn a policy for urban driving. The policy network consists of a spatial encoder and a temporal encoder. The policy network is trained to minimize the L 1 distance between its predictions and the ground truth trajectories. During the evaluation, we employ a LQR controller to generate smooth planning based on the predicted trajectory from the policy network.

Original: 4. I NPUT REPRESENTATION
Revised: 4. I NPUT REPRESENTATION

Original: As discussed above, we will overlook the SDV state toavoid distribution shift, keeping only thecontext as input on which our policy has limited influence.
Revised: Following Bansal et al. (2019), the intermediate representations such as oriented bounding boxes from a perception system and road networks from HD maps rather than raw sensor data (such as camera images or lidar points clouds) are harnessed to improve generalization and interpretability. In detail, there are two types of inputs to the policy network: map context and traffic participants.

Original: As shown in Figure 1, we consider four types of inputs to our neural planner: 1) a goal position: the mission goal that the SDV intends to reach; 2) n a agents: each agent’s d a features composed of its types ( e.g . car, bicycle, and cyclist), sizes, centroids and orientations of its bounding boxes at a few past time steps; 3) n c crosswalks: each crosswalk polygon approximated by a sequence ofv c vectors  as Gao et al. 
Revised: For the map context, we consider two types of map elements, which are crosswalks and lanes. Each map element is presented in a polyline which is a sequence of vectors with different associated features as Gao et al. The features of crosswalk vectors are their initial and terminal point’s position and sequence orders, and lane vectors have additional features such as their traffic light state and lane width. For the traffic participants, we divide them into ego vehicle and other agents.

Original: (2020), where each vector’s d c features consist of its starting and ending point coordinate, sequence orders; 4) n l lanes: each lane polyline also approximated by v l vectors whose d l features additionally include its traffic light state, lane width, travel distances to left, right lane and goal from its starting point. The intermediate representations such as oriented bounding boxes from a perception system and road networks from HD maps rather than raw sensor data are harnessed to improve generalization and interpretability as Bansal et al.
Revised: For the ego vehicle, as discussed above, we should overlook the features which can be influenced by the learned policy to overcome the distribution shift obstacle. Therefore, we only input its mission goal position while removing its history. For each other agent, its features are composed of its type (e.g. car, bicycle, and cyclist), sizes, centroids, and orientations of bounding boxes at several past time steps.

Original: Even though there isalready no explicit SDV state  input, the observations are typically translated to the SDV coordinate system using its rear axle’s midpoint as origin and orientation as x-axis direction in previous works (Bojarski et al., 2016; Scheel et al., 2022).
Revised: Even though there is no explicit SDV state in the network input, the observations are typically translated to the SDV coordinate system using its rear axle’s midpoint as origin and orientation as x-axis direction in previous works (Bojarski et al., 2016; Scheel et al., 2022).

Original: This implicit incorporation of the SDV state can still cause a distribution shift of inputs.
Revised: This implicit incorporation of the SDV state can still cause a distribution shift of inputs.

Original: To reduce the implicit influence and ensure local observability, we adopt an ego-perturbed goal-oriented coordinate system whose origin is the SDV position adding a 2D Gaussian perturbation, and x -axis direction is toward its goal position.
Revised: To reduce the implicit influence and ensure local observability, we adopt an ego-perturbed goal-oriented coordinate system whose origin is the SDV position adding a zero mean Gaussian perturbation, and x -axis direction is toward its goal position.

Original: Note that the perturbation is only required during training and the goal position can be replaced with any fixed point.
Revised: Note that the perturbation is only required during training and the goal position can be replaced with any point that is not influenced by the driving policy.

Original: 4.N EURAL P LANNER
Revised: 4. P OLICY N ETWORK

Original:  Since it is more challenging to planthe SDV trajectory without its state, we need to make full use of its interactions with and constraints from the  context to infer its future states.
Revised: The policy network is designed to directly plan a trajectory for the SDV using only its context. Since it is more challenging to plan without the historical trajectory, we need to make full use of the spatial-temporal interactions between the SDV and context to infer its future states.

Revised: Thus, we first embed the observation at each time step into an observation feature by a spatial encoder, and then the observation features in current and previous time steps are embedded to generate the predictions by a temporal encoder.

Original: In practice, we construct our neural planner based on the Transformer(Vaswani et al., 2017) whose multi-head attention mechanism has demonstrated excellent performance in capturing relations.
Revised: We construct our policy network based on the Transformer whose multi-head attention mechanism has demonstrated excellent performance in capturing relations.

Original: As depicted in Figure 1, there are four Transformer encoders in our network. The two local Transformer encoders are employed to embed the map vectors and capture the  relations between vectors belonging to the same polyline.
Revised: Spatial Encoder: To capture the spatial relations between vectors of the same polyline such as a lane or crosswalk, we employ a local Transformer encoder as the vector-vector interaction encoder.

Original: Subsequently, we aggregate the features of vectors belonging to the same polyline using max-pooling to obtain polyline-level features.
Revised: Subsequently, we aggregate the features of vectors belonging to the same polyline by max-pooling to obtain polyline-level features.

Original: Besides, two multi-layer perceptions (MLP) are used to embed agents and the goal respectively.
Revised: To obtain the agent and goal features, two multi-layer perceptions (MLP) are applied.

Original: Then, to model the high-order interactions between the goal, agents, and map elements, we apply a global Transformer encoder whose goal embedding is output as observation features o t at the observation time step t .
Revised: To model the high-order interactions between the goal, agents, and map elements, we utilize a global Transformer encoder whose goal embedding is output as observation features. Temporal Encoder:

Original: To take the temporal relation into account, a Transformer encoder with a causal self-attention mask is employed to embedall context features at past H time steps with  interval I .
Revised: To take the temporal information and interaction into account, a Transformer encoder with a causal self-attention mask is employed to embed H historical observation features with a step interval I .

Original: Finally, the causal Transformer’s hidden state at each past time step h is embedded by a full-connected linear layer to output a predictionˆ s h,t of the SDV’s states t − h ateach futuretime step t = 1 , . . . , T .
Revised: Then, each hidden state of the Transformer is decoded by a full-connected linear layer to generate a prediction of the SDV future state at future T time steps.

Original: For simplicity, we represent the SDV state with its rear axle’s midpoint coordinate and orientation s t =
Revised: For simplicity, we represent the SDV state with its rear axle’s midpoint coordinate and orientation.

Original: ( x t , y t , θ t ) .

Original: 4. T RAINING P ROCESS
Revised: 4. T RAINING P ROCESS

Original: We leverage supervised learning  to train the neural planner to predict the expert future trajectories by minimizing the L 1 loss, as it is more correlated to driving performance compared with the mean square error (Codevilla et al., 2018a).
Revised: We leverage supervised learning like BC to train the policy network by minimizing the L 1 distance between the predictions and ground truth trajectories, as L 1 is more correlated to driving performance compared with the mean square error (Codevilla et al., 2018a).

Original: To help the network converge and generalize, we  introduce an auxiliary task of minimizing the L 1 error at all historical time steps of the Casual Transformer. In addition, we apply a squared L 2 norm regularization to the network parameters θ according to equation 6.
Revised: To help the network converge and generalize, we additionally introduce an auxiliary task of minimizing the L 1 error at all previous time steps and apply a squared L 2 norm regularization to the network parameters θ inspired by equation 6.

Original: The final loss is:
Revised: The final loss is:

Original: [Equation]
Revised: [Equation]

Original: where s t is the ground-truth SDV state at time step t , µ is a small hyperparameter and λ is a regularization factor.
Revised: where s t is the ground-truth SDV state at time step t and ˆ s thI is the state prediction at the historical time step hI for its future t time step. µ is an auxiliary hyperparameter and λ is a regularization factor.

Original: 4. E VALUATION P ROCESS
Revised: 4. E VALUATION P ROCESS

Original: Even though the prediction without SDV state input is more stable, it has difficulty ensuring the smoothness of the planning from the current state.
Revised: Even though the prediction without SDV state input is more stable, it has difficulty ensuring the smoothness of the predicted trajectory from the current state.

Original: Prior works (Vitelli et al., 2022; Amos et al., 2018) usually add a differentiable kinematic layer into the policy network to generate physically feasible planning.
Revised: Prior works such as Vitelli et al. ; Amos et al. (2018) usually add a differentiable kinematic layer into the policy network to generate physically feasible planning.

Original: However, the differentiable kinematic layer will bring in the current SDV state which is undesirable for learning a policy.
Revised: However, the differentiable kinematic layer will bring in the current SDV state which is undesirable for learning a policy.

Original: Therefore, we choose to obtain a smooth trajectory during evaluation by a linear-quadratic regulator (LQR) ( ˚Astr¨om & Murray, 2021) which can efficiently minimize the total commutative quadratic cost of a linear dynamic system.
Revised: Therefore, we choose to obtain a smooth trajectory during evaluation by a linear-quadratic regulator (LQR) ( ˚Astr ¨ om & Murray, 2021) which can efficiently minimize the total commutative quadratic cost of a linear dynamic system.

Original: We consider a finitehorizon, discrete-time linear system  described by:
Revised: We consider a finite-horizon, discrete-time linear system whose dynamics is described by:

Original: [Equation]
Revised: [Equation]

Revised: 

Revised: I

Revised: D

Revised: D

Revised: I

Revised: D

Revised: I   (cid:34) p t ˙ p t ¨ p t (cid:35) +  

Original:  where D is a diagonal matrix with the time interval of each step as diagonal entries, ˙ p
Revised: D D I   u t , (8) where D is a diagonal matrix with the time interval of each step as diagonal entries, ˙ p

Original: = ( v t , w t ) , ¨ p t = ( a t , α t ) , u t = ( u pt , u at ) denote the positional and angular velocity, acceleration, and controlinput respectively, subject to a quadratic cost function:[Equation]
Revised: = ( v t , w t ) , ¨ p t = ( a t , α t ) , u t = ( u pt , u αt ) denote the positional and angular velocity, acceleration, and control respectively, subject to a quadratic cost function:

Original: [Equation]
Revised: [Equation]

Original: where p t is regarded as the planning trajectory and the predicted states ˆ s 0 ,t at the current step from the neural planner are regarded as target poses; w w , w a , w α and w u are the weights to balance the safety and the smoothness of the planned trajectory. 
Revised: where p t is regarded as the planning trajectory and the predicted states ˆ s 0 ,t at the current step from the policy network are regarded as target poses; w w , w a , w α and w u are the weights to balance the positional accuracy and the smoothness of the planned trajectory. At the end of the optimization, the LQR will output a smooth trajectory for the SDV to follow.

Original:  E XPERIMENTS 5. D ATASET
Revised:  E XPERIMENTS 5. D ATASET

Original: To benchmark our method’s performance, we conduct experiments on two real-world large-scale datasets: Lyft Level 5 Prediction Dataset (Houston et al., 2020): contains about 1000h urban driving demonstrations in Palo Alto, which have been separated into independent scenes of nearly 25s at 10Hz.
Revised: To benchmark our method’s performance, we conduct experiments on two real-world large-scale datasets: Lyft Level 5 Prediction Dataset (Houston et al., 2020): contains about 1000h urban driving demonstrations in Palo Alto, which have been separated into independent scenes of nearly 25s at 10Hz.

Original: We train our network on the provided 100h (16265 scenes) subset as Scheel et al.
Revised: We train our network on the provided 100h (16265 scenes) subset as Scheel et al.

Original: (2022) and test with all 16220 validation scenes.
Revised: (2022) and test with all 16220 validation scenes.

Original: nuPlan Dataset (Caesar et al., 2021) provides an urban driving dataset with 1312h of human driving data from 4 cities (Boston, Pittsburgh, Las
Revised: nuPlan Dataset (Caesar et al., 2021) provides an urban driving dataset with 1312h of human driving data from 4 cities (Boston, Pittsburgh, Las

Original: Vegas, and Singapore).
Revised: Vegas, and Singapore).

Original: Due to the huge difference in traffic rules and patterns in different cities, we extract driving data in Las Vegas as Phan-Minh et al.
Revised: Due to the huge difference in traffic rules and patterns in different cities, we extract driving data in Las Vegas as Phan-Minh et al.

Original: Then, we separate the data into independent scenes of 25s at 10Hz like Lyft, and filter out the scene without a mission goal.
Revised: Then, we separate the data into independent scenes of 25s at 10Hz like Lyft and filter out the scene without a mission goal.

Original: After filtering, we obtain 63181 training, 4774 validation, and 6386 testing scenes.
Revised: After filtering, we obtain 63181 training, 4774 validation, and 6386 testing scenes.

Original: More details about the data prepossessing and corresponding model details are presented in the appendix.
Revised: More details about the data prepossessing and corresponding model details are presented in the appendix.

Original: 5. C LOSED -L OOP E VALUATION
Revised: 5. C LOSED -L OOP E VALUATION

Original: To evaluate the closed-loop performance of our method, we use a log-replay simulator as prior work (Scheel et al., 2022; Huang et al., 2022).
Revised: To evaluate the closed-loop performance of our method, we use a log-replay simulator as prior work (Scheel et al., 2022; Huang et al., 2022).

Original: At each step in the log-replay simulator, the SDV vehicle updates its state according to the planned trajectory, while the other agents are assumed to follow their recorded trajectories in the dataset.
Revised: At each step in the log-replay simulator, the SDV vehicle updates its state according to the planned trajectory, while the other agents are assumed to follow their recorded trajectories in the dataset.

Original: For both datasets, we evaluate our method for 25s at 10Hz with the following metrics:
Revised: For both datasets, we evaluate our method for 25s at 10Hz with the following metrics:

Original: Collision Rate : If the SDV collides with the other agents at any time step in a scene, the scene is deemed as a collision scene.
Revised: Collision Rate : If the SDV collides with the other agents at any time step in a scene, the scene is deemed as a collision scene.

Original: The collision rate is the ratio of the collision scenes in all scenes.
Revised: The collision rate is the ratio of the collision scenes in all scenes.

Original: Off-road Rate :
Revised: Off-road Rate :

Original: In nuPlan, we use the official drivable area compliance metric, i.e . the distance of a corner of SDV’s bounding box from the drivable area is more than 0. 3m.
Revised: In nuPlan, we use the official drivable area compliance metric, i.e . the distance of a corner of SDV’s bounding box from the drivable area is more than 0.3m.

Original: But in Lyft, without access to the drivable area, if the SDV deviates laterally from the human driver’s ground truth more than 2m like in a scene, the scene will be deemed off-road as Scheel et al.
Revised: But in Lyft, without access to the drivable area, if the SDV deviates laterally from the human driver’s ground truth more than 2m like in a scene, the scene will be deemed off-road as Scheel et al.

Original: Discomfort : we count the rate of the absolute value of the acceleration more than 3 m / s 2 over time steps to quantify the comfort and feasibility of the planned trajectory.
Revised: Discomfort : we count the rate of the absolute value of the acceleration more than 3 m / s 2 over time steps to quantify the comfort and feasibility of the planned trajectory.

Original: L we use the average L2 position errors between the roll-out trajectory and the human driver’s ground truth to quantify the human driving similarity.
Revised: L2 : we use the average L2 position errors between the roll-out trajectory and the human driver’s ground truth to quantify the human driving similarity.

Original: 5. P ERFORMANCE E VALUATION
Revised: 5. P ERFORMANCE E VALUATION

Original: On the Lyft dataset, we compare our methods against three state-of-the-art methods to demonstrate the advantage of the proposed framework:
Revised: On the Lyft dataset, we compare our methods against three state-of-the-art methods to demonstrate the advantage of the proposed framework:

Original: Raster-perturb : an official BC planning
Revised: Raster-perturb : an official BC planning

Original: baseline, which based on ResNet50 receives a Bird-EyeView (BEV) representation of the scene surrounding the SDV and produces a trajectory of position and yaw displacements
Revised: baseline (Houston et al., 2020), which based on ResNet receives a Bird-Eye-View (BEV) representation of the scene surrounding the SDV and produces a trajectory of position and yaw displacements.

Original: To augment data, a perturbation is applied to the current position and then a new kinematically feasible trajectory to reach the original endpoint is generated as ChauffeurNet (Bansal et al
Revised: To augment data, a perturbation is applied to the current SDV position, and then a new kinematically feasible trajectory to reach the original endpoint is generated as ChauffeurNet. [Table]

Original: BC-perturb : abaseline BC model provided by Scheel et al. 
Revised: BC- perturb : a BC model provided by Scheel et al. with the

Original: with the same trajectory perturbation and output as Raster-perturb butits inputs are presented in vector formulation, which are processed by a VectorNet (Gao et al., 2020) and Transformer.
Revised: same trajectory perturbation and output as Raster-perturb but

Revised: its

Revised: inputs are

Revised: presented in

Revised: vector formulation,

Revised: which are

Revised: processed

Original:  UrbanDriver : an offline policy gradient method proposed by Scheel et al.(2022) to imitate the expert’s policy exploiting a differentiable data-driven simulator with the same data and model structure as BC-perturb .
Revised: by a VectorNet (Gao et al., 2020) and Transformer UrbanDriver : an offline policy gradient method proposed by Scheel et al. to imitate the ex

Revised: - pert’s

Revised: policy exploiting a differentiable data-driven simulator with the same data and

Revised: model structure

Original:  On  nuPlan dataset, we compare our methods against three official baselines learned by BC due to a lack of other prior work:
Revised: as BC-perturb On the nuPlan dataset, we compare our methods against two official baselines learned by BC due to a lack of other prior works:

Original: Raster : a raster-based model that uses the ResNet-50 backbone to encode SDV, agent, and map information as raster layers to plan the SDV’s trajectory
Revised: Raster : a raster-based model that uses the ResNet-50 backbone to encode SDV, agent, and map information as raster layers to plan the SDV’s trajectory.

Original: [Table] Vector-perturb : a simple vector-based model that consumes SDV, agent, and lane signals through a series of MLPs using perturbed trajectory input as ChauffeurNet (Bansal et al., 2019). LaneGCN-perturb : a vector-based model that uses a series of MLPs to encode SDV and agent signals and one LaneGCN (Liang et al., 2020) to encode vector-map elements and a fusion network to capture lane and agent intra/inter-interactions through attention layers.
Revised: LaneGCN-perturb : a vector-based model that uses a series of MLPs to encode SDV and agent signals and one LaneGCN (Liang et al., 2020) to encode vector-map elements and a fusion network to capture lane and agent intra/inter-interactions through attention layers.

Original: To augment data, the SDV trajectories are perturbed and other agents are randomly dropped out.
Revised: To augment data, the SDV trajectories are perturbed and other agents are randomly dropped out.

Revised: In addition to these models provided by prior works, we also learn our model by a representative method in behavior cloning and offline reinforcement learning on both datasets:

Revised: TD3+BC (Fujimoto & Gu, 2021): it adds behavior cloning term to the policy updating of Twin

Revised: Delayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al., 2018) for implicit policy constraint.

Original: Our performance is shown in Table 1.
Revised: We construct the offline RL dataset by applying the trajectory perturbation augmentation and consider the collision and comfort for the reward design. Vector-Chauffeur : our model learned using the same data augmentation method as ChauffeurNet including trajectory perturbation and ego past motion dropout. And we represent the data in the same coordinate system which uses the ego location as the origin and its heading perturbed by a uniform noise as the orientation. The performance is shown in Table 1.

Original: It demonstrates that our method can outperform previous work significantly on both datasets.
Revised: It demonstrates that our method can outperform previous work significantly on both datasets.

Original: The collision rate on the nuPlan dataset is higher than the Lyft dataset because there are much more road agents in its scene.
Revised: The collision rate on the nuPlan dataset is higher than the Lyft dataset because nuPlan’s scenarios are more complex with more road agents.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: The following ablation experiments on the Lyft dataset are used to expose the significance of different components of our model:
Revised: The following ablation experiments on the Lyft dataset are used to expose the significance of different components of our model, whose results are shown in Table 2:

Original: BC :
Revised: [Table]

Original: Firstly, we consider training a model by BC using the SDV state without perturbation.
Revised: Network input : We first study the importance of removing explicit ego information from the network input.

Original: In detail, we concatenate the SDV state including SDV’s current position, velocity, and acceleration with the goal position and represent the observation in the classical SDV vehicle coordinate system.
Revised: We consider two ways to bring the explicit ego input back. The first one is to directly input ego past positions and then process it similarly to the goal position.

Original: The result in Table 2 shows our CCIL outperforms the naive BC method by a large margin.
Revised: The other one is to additionally introduce a dropout of 50% at the ego input during training as Bansal et al.

Original: SDV input : To show the influence of explicit SDV state input , we concatenate the SDV state with the goal position, keeping other parts of our method unchanged. Then, we consider the influence of  implicit SDV state input in the coordinate system, where our  coordinate systemis replaced with the classical coordinate system. Both ego information input ways will deteriorate the performance significantly (see Table 2).
Revised: We can observe that in both explicit ego information input ways, there is a steep drop in the collision and off-road rate and L2 distance due to the covariate shift issue. Coordinate system : To analyze the impact of the implicit ego information in the coordinate system, we first consider replacing our ego-perturbed goal-oriented coordinate system with the ego-centric coordinate system in ChauffeurNet using orientation uniformly within the heading.

Original: Network structure : To demonstrate the effectiveness of the causal Transformer in capturing the temporal interactions, we replace it with a MLP.
Revised: We observe that the ChauffeurNet coordinate system leads to inferior closed-loop performance. In our coordinate system, we add a Gaussian noise with zero mean to the ego current position to obtain the origin.

Original: As is shown in Table 2, the temporal relations embedded by the causal Transformer can  improve the  performance.
Revised: Thus, we can increase the Gaussian noise’s standard deviation to diminish the implicit ego information. We can observe that with the increasing of the standard deviation, the discomfort increases but the off-road rate and closed-loop L2 decreases, which implies that the implicit ego information may improve the instantaneous prediction accuracy but deteriorate the closed-loop performance.

Original: Loss : To demonstrate the  effectiveness of the additional term, we remove the auxiliary and regularization loss term separately.
Revised: Architecture : To demonstrate the importance of the temporal information and the effectiveness of the causal Transformer in capturing the temporal interactions, we replace it with a MLP.

Original: The result in Table 2 shows that they are both beneficial to improving performance.
Revised: We find that the causal Transformer can achieve better performance.

Original: Perturbation : To investigate how the implicit ego information affects the closed-loop performance, we adjust the Gaussian perturbation’s standard deviation (std). We can see from Table 2 that with the increasing of the std and then less ego information, the closed-loop L2 decreases but the discomfort increases, which demonstrates that ego information can improve the instantaneous prediction accuracy but deteriorate the closed-loop performance. 5. T OY E XPERIMENT We design a toy example tovividly showour method’s ability to avoid the compounding error.
Revised: Besides, to show the importance of the

Original: The training data are trajectories on a circle of a random radius ranging from 10 to 100 with speed 1. [Table] The inputs of BC are composed of ego states (positions and orientations) and context (nearest circle point’s coordinate) represented in the ego coordinate system in the past 10 steps. However, our
Revised: LQR module in enhancing comfort, we ablate it.

Original: CCIL method only receives the context in the perturbed-ego origin-oriented coordinate system.
Revised: Removing the LQR mechanism leads to a huge drop in comfort.

Original: We train both models with 100 random initial weights and evaluate their closed-loop performance with a random starting point on a circle of radius 50. The closed-loop trajectories depicted in Figuredemonstrate that the BC method deviates from the expert state distribution but our method does not.
Revised: Loss : To demonstrate the effectiveness of the additional term, we remove the auxiliary and regularization loss term separately. The result in Table 2 shows that they are both beneficial to improving performance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed a new offline imitation learning method to mitigate the distribution shift of behavior cloning, where we learn a neural network to predict the SDV’s future positions and orientations without SDV information by making full use of its constraints from and interaction with the context.
Revised: We have proposed a new offline imitation learning method to mitigate the distribution shift of behavior cloning, where we learn a neural network to predict the SDV’s future positions and orientations without SDV information by making full use of its constraints from and interaction with the context.

Original: Firstly, we analyze theoretically the stability of our policy formulation.
Revised: Firstly, we analyze theoretically the stability of our policy formulation.

Original: In addition to removing the explicit SDV information input to the network, we present a new SDV-perturbed goal-oriented coordinate system for representing represent the observation input to remove the implicit SDV information and ensure local observability.
Revised: In addition to removing the explicit SDV information input to the network, we present a new SDV-perturbed goal-oriented coordinate system for representing represent the observation input to remove the implicit SDV information and ensure local observability.

Original: We design a Transformer-based network to make full use of the history information to handle the more challenging learning task.
Revised: We design a Transformer-based network to make full use of the history information to handle the more challenging learning task.

Original: Finally, we demonstrate the effectiveness of our approach in two real-world large-scale datasets with state-of-the-art performance.
Revised: Finally, we demonstrate the effectiveness of our approach in two real-world large-scale datasets with state-of-the-art performance

Original: A M AP PREPOSSESSING
Revised: A M AP PREPOSSESSING

Original: In Lyft, there are only two types of map elements: lane and crosswalk.
Revised: In Lyft, there are only two types of map elements: lane and crosswalk.

Original: In nuPlan, the map elements consist of  lane, lane connector, intersection, stop line, crosswalk, walkway and car park.
Revised: In nuPlan, the map elements consist of a lane, lane connector, intersection, stop line, crosswalk, walkway, and car park.

Original: We categorize them into polyline elements including lane and lane connector and polygon elements including stop line, crosswalk, intersection, walkway, and car park.
Revised: We categorize them into polyline elements including lanes and lane connectors, and polygon elements including stop lines, crosswalks, intersections, walkways, and car parks.

Original: For each polyline element, it is approximated by a sequence of vectors with an interval of 3 meters.
Revised: For each polyline element, it is approximated by a sequence of vectors with an interval of 3 meters.

Original: For both datasets, we prepossess the polyline map into a graph to take advantage of the topological information.
Revised: For both datasets, we prepossess the polyline map into a graph to take advantage of the topological information.

Original: We connect each vector with its nearest left, right, and next vector if exists.
Revised: We connect each vector with its nearest left, right, and next vector if exists.

Original: The nearest left or right vector means the nearest vector of its reachable left or right polyline.
Revised: The nearest left or right vector means the nearest vector of its reachable left or right polyline.

Original: And the distance between vectors is represented by the Euclidean distance between their starting points.
Revised: And the distance between vectors is represented by the Euclidean distance between their starting points.

Original: After building the vector graph, we can compute the travel distance between any two vectors using
Revised: After building the vector graph, we can compute the travel distance between any two vectors using

Original: Dijkstra’s algorithm.
Revised: Dijkstra’s algorithm.

Original: For both datasets, the polyline vector has shared features including coordinates of starting and ending point, distance to left and right vector, distance to the mission goal, traffic light states (red or green), and the sequence order.
Revised: For both datasets, the polyline vector has shared features including coordinates of starting and ending point, distance to left and right vectors, distance to the mission goal, traffic light states (red or green), and the sequence order.

Original: However, lane vectors in Lyft have a lane width feature because the vectors approximate computed middle lines of lanes, while the lane or lane connector vectors in nuPlan have a lane left and width feature because the vectors approximate an annotated baseline.
Revised: However, lane vectors in Lyft have a lane width feature because the vectors approximate computed middle lines of lanes, while the lane or lane connector vectors in nuPlan have a lane left and width feature because the vectors approximate an annotated baseline.

Original: In addition, the polyline vectors in nuPlan have  additional speed limit and type features.
Revised: In addition, the polyline vectors in nuPlan have an additional speed limit and type features.

Original: For each polygon element, we approximate it also with a sequence of vectors.
Revised: For each polygon element, we approximate it also with a sequence of vectors.

Original: In the Lyft dataset, the crosswalk vectors are directly constructed by connecting the original sequential annotation coordinates.
Revised: In the Lyft dataset, the crosswalk vectors are directly constructed by connecting the original sequential annotation coordinates.

Original: But we approximate each polygon with fixed 20 vectors because the annotation point number is too large for elements like intersection.
Revised: But we approximate each polygon with fixed 20 vectors because the annotation point number is too large for elements like intersection.

Original: For both datasets, each polygon vector has features including coordinates of starting and ending points, and its order sequence, while vectors in nuPlan have additional type features.
Revised: For both datasets, each polygon vector has features including coordinates of starting and ending points, and its order sequence, while vectors in nuPlan have additional type features.

Original: The inputs to our neural network are composed of two types of map elements: polyline and polygon, and two types of agent elements: other agents and ego goal.
Revised: The inputs to our neural network are composed of two types of map elements: polyline and polygon, and two types of agent elements: other agents and ego goal.

Original: The missing inputs are padded with zeros and masked out when calculating the attention.
Revised: The missing inputs are padded with zeros and masked out when calculating the attention.

Original: The origin is the perturbed SDV position.
Revised: The origin is the perturbed SDV position.

Original: Polyline :

Original:  topologically nearest polylines with vectors whose starting points are within 35m from the origin
Revised: Polyline : 30 topologically nearest polylines with vectors whose starting points are within 35m from the origin.

Original: The topological distance between the origin and a polyline is the minimum of topological distances between the origin and its vectors.
Revised: The topological distance between the origin and a polyline is the minimum of topological distances between the origin and its vectors.

Original: Polygon : 20 polygons whose boundaries are within 35m from the origin.
Revised: Polygons : 20 polygons whose boundaries are within 35m from the origin.

Original: If there are more than 20 polygons, they are selected according to the importance of their types: stop line, crosswalk, intersection, walkway, and car park
Revised: If there are more than 20 polygons, they are selected according to the importance of their types: stop line, crosswalk, intersection, walkway, and car park.

Original: Agents : the nearest 30 agents whose oriented boxes’ centroids are within 50m from the origin.
Revised: Agents : the nearest 30 agents whose oriented boxes’ centroids are within 50m from the origin.

Original: The agent features in Lyft include its centroid coordinates, yaws, shapes, types, and relative times in the past 2 and current steps.
Revised: The agent features in Lyft include its centroid coordinates, yaws, shapes, types, and relative times in the past 2 and current steps.

Original: The nuPlan agents additionally have velocity features as they are provided.
Revised: The nuPlan agents additionally have velocity features as they are provided.

Original: Goal : the x, y coordinates of the SDV’s mission goal of a scene.
Revised: Goal : the x, y coordinates of the SDV’s mission goal of a scene.

Original: In Lyft, the mission goal is not provided, so we regard the ending point of the lane where the SDV locates at the last time step of the scene as the mission goal.
Revised: In Lyft, the mission goal is not provided, so we regard the ending point of the lane where the SDV locates at the last time step of the scene as the mission goal.

Original: In nuPlan, we directly take the provided mission goal at the last time step of the scene as the scene mission goal.
Revised: In nuPlan, we directly take the provided mission goal at the last time step of the scene as the scene mission goal.

Original: B M ODEL
Revised: B M ODEL

Original: For both datasets.
Revised: For both datasets.

Original: the same model architecture is used, whose hyper-parameters are listed in Table 3.
Revised: the same model architecture is used, whose hyper-parameters are listed in Table 3.

Original: Our model is trained using the Adam optimizer with a learning rate of 0.0005, 10000 steps linear warm-up, β = (0 . 9 , 0 . 999) , and batch size 128.
Revised: Our model is trained using the Adam optimizer with a learning rate of 0.0005, 10000 steps linear warm-up, β = (0 . 9 , 0 . 999) , and batch size 128.

Revised: We stop training after 30 epochs and select the model with smallest validation collision rate for evaluation.

Revised: We train all models except the pretrained model in Lyft dataset independently for 3 times and then report the mean and std of their performances.

Revised: [Table]

Original: C E VALUATION
Revised: D E VALUATION

Original: For Lyft baselines, we directly evaluate the pretrained model provided by the Lyft dataset (Houston et al., 2020) and UrbanDriver (Scheel et al., 2022).
Revised: For Lyft baselines, we directly evaluate the pretrained model provided by the Lyft dataset and UrbanDriver. The Raster-perturb are model trained on train.zarr forepochs at https://github.com/woven-planet/l5kit/blob/master/examples/ planning/train.ipynb . The BC-perturb and UrbanDriver are from Open Loop and Urban Driver in https://github.com/woven-planet/l5kit/blob/master/examples/ urban_driver/train.ipynb .

Original: We evaluate all Lyft models from the second time step to compute current velocity using position information as input to the LQR with zero assumed initial acceleration.
Revised: We evaluate all Lyft models from the second time step to compute current velocity using position information as input to the LQR with zero assumed initial acceleration.

Original: For nuPlan baselines, we train the official models by ourselves using provided hyper-parameters and evaluate from the first time step because the velocity and acceleration information is provided. 
Revised: For nuPlan baselines, we train the official models by ourselves using provided hyper-parameters and evaluate from the first time step because the velocity and acceleration information is provided. E R UNTIME

Revised: We conduct runtime experiments using a single Nvidia GeForce GTX 1080 GPU and an Intel i78700@3.2GHz CPU on Lyft dataset.

Original: [Table]
Revised: We measure the runtime of each method its mean and std over all time-steps in an evaluation. The runtime results shown in Table 4 consider all components in each model including data-prepossessing, model inference and control. We observe that our architecture can achieve higher data processing efficiency and medium model inference efficiency compared with other methods. Our method takes longer total execution time due to the extra LQR control module which does not exist in prior works because they only focus on optimizing positional accuracy but not comfort.

Revised: However, our approach can still be executed in real-time on this hardware

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: F T OY E XPERIMENT

Revised: We design a toy experiment to vividly show our method’s ability to reduce compounding error.

Revised: In the experiment, we use synthetic data from a very ideal and simplified scenario where a SDV moves under ring road network with fixed 1 m/s at 1Hz, as shown in Figure 2.

Revised: During training, the radius of the ring is a variable with a range from 10m to 100m and the circular lane are represented as a series of fixed lane points with same interval of nearly 1m.

Revised: We compare our CCIL methods with several baseline methods introduced above including BC, BCperturb, UrbanDriver.

Revised: The inputs of these baseline methods are composed of two parts: SDV state (its position and orientation in the past 10 time step) and context (nearest 10 lane points) in the ego coordinate system, while our methods only takes the context in the past 10 time step as inputs in the ego-perturbed center-oriented coordinate system.

Revised: The ego-perturbed origin-oriented coordinate system means using the SDV position added a zero-mean one-std Gaussian perturbation as the origin and orientation to the center of the circular road as the x-axis direction.

Revised: The trajectory perturbation is applied to augment the data in the BC-perturb method.

Revised: For outputs, the BC and CCIL method generate the relative position and yaw at the next time step and the BC-perturb method produces the next 10 time steps.

Revised: In the UrbanDriver method, we unroll the policy for 32 time steps.

Revised: In each methods, we employ a two-layer MLP with hidden size of 128 as a policy network.

Revised: We train the neural networks using Adam optimizer of with a learning rate of 0.0001 with random initial weights for 100 times.

Revised: We stop training after 10000 steps and then unroll the policy from one random starting point on the circle of radius 50m for 100 time steps.

Revised: The 100 closed-loop trajectories for each method are depicted in Figure 3.

Revised: We can observe that some trajectories in the baseline methods deviate from the road due to the covariate shift issue but the trajectories in our method keep following the route.