Original: Learning an Explainable Planner for Autonomous Driving Anonymous Author(s)
Revised: PlanT: Explainable Planning Transformers via Object-Level Representations

Original: AfﬁliationAddress email

Original: Abstract: Planning an optimal route in a complex environment requires efﬁcientreasoning about the surrounding scene.
Revised: Abstract: Planning an optimal route in a complex environment requires efﬁcient reasoning about the surrounding scene.

Original: While human drivers prioritize importantobjects and ignore details not relevant to the decision, learning-based plannerstypically extract features from dense, high-dimensional grid representationsof thescene containing all vehicle and road context information.
Revised: While human drivers prioritize important objects and ignore details not relevant to the decision, learning-based planners typically extract features from dense, high-dimensional grid representations containing all vehicle and road context information.

Original: In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that usesa standard transformer architecture.
Revised: In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that uses a standard transformer architecture.

Original: PlanT is based on imitation learning witha compact object-level input representation.
Revised: PlanT is based on imitation learning with a compact object-level input representation.

Original: With this representation, we demonstrate that information regarding the ego vehicle’s route provides sufﬁcient contextregarding the road layout for planning. On thechallenging Longest6 benchmarkfor CARLA, PlanT outperforms all prior methods (matching the driving score ofthe expert) while being 5.3 × faster than equivalent pixel-based planning baselinesduring inference. 
Revised: On the Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3 × faster than equivalent pixel-based planning baselines during inference. Combining PlanT with an off-the-shelf perception module provides a sensorbased driving system that is more than 10 points better in terms of driving score than the existing state of the art.

Original: Furthermore, we propose an evaluation protocol to quantify theability of planners to identify relevant objects, providing insights regarding theirdecision making.
Revised: Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision-making.

Original: Our results indicate that PlanT canreliably focus on the mostrelevant object in the scene, even when this object is geometrically distant.
Revised: Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.

Original:  Introduction
Revised:  Introduction

Original: The ability to plan is an important aspect of human intelligence, allowing us to solve complex navigation tasks.
Revised: The ability to plan is an important aspect of human intelligence, allowing us to solve complex navigation tasks.

Original: For example, to change lanes on a busy highway, a driver must wait for sufﬁcientspace in the new lane and adjust the speed based on the expected behavior of the other vehicles.
Revised: For example, to change lanes on a busy highway, a driver must wait for sufﬁcient space in the new lane and adjust the speed based on the expected behavior of the other vehicles.

Original: Humans quickly learn this and can generalize to new scenarios, a trait we would also like autonomousagents to have.
Revised: Humans quickly learn this and can generalize to new scenarios, a trait we would also like autonomous agents to have.

Original: Due to the difﬁculty of the planning task, the ﬁeld of autonomous driving is shiftingaway from traditional rule-based algorithms [1, 2, 3, 4, 5, 6, 7, 8] towards learning-based solutions [9, 10, 11, 12, 13].
Revised: Due to the difﬁculty of the planning task, the ﬁeld of autonomous driving is shifting away from traditional rule-based algorithms [1, 2, 3, 4, 5, 6, 7, 8] towards learning-based solutions [9, 10, 11, 12, 13, 14].

Original: Learning-based planners directly map the environmental state representation (e.g., HD maps and object bounding boxes) to waypoints or vehicle controls.
Revised: Learning-based planners directly map the environmental state representation (e.g., HD maps and object bounding boxes) to waypoints or vehicle controls.

Original: They emerged asa scalable alternative to rule-based planners which require signiﬁcant manual effort to design.
Revised: They emerged as a scalable alternative to rule-based planners which require signiﬁcant manual effort to design.

Original: Interestingly, while humans reason about the world in terms of objects [14, 15, 16], most existinglearned planners [9, 11, 17] choose a high-dimensional pixel-level input representation by renderingbird’s eye view (BEV) images of detailed HD maps (Fig. 1 left).
Revised: Interestingly, while humans reason about the world in terms of objects [15, 16, 17], most existing learned planners [9, 12, 18] choose a high-dimensional pixel-level input representation by rendering bird’s eye view (BEV) images of detailed HD maps (Fig. 1 left).

Original: It is widely believed that this kindof accurate scene understanding is key for robust self-driving vehicles, leading to signiﬁcant interestin recovering pixel-level BEV information from sensor inputs [18, 19, 20, 21, 22, 23].
Revised: It is widely believed that this kind of accurate scene understanding is key for robust self-driving vehicles, leading to signiﬁcant interest in recovering pixel-level BEV information from sensor inputs [19, 20, 21, 22, 23, 24].

Original: In this paper,we investigate whether such detailed representations are actually necessary to achieve convincingplanning performance.
Revised: In this paper, we investigate whether such detailed representations are actually necessary to achieve convincing planning performance.

Original: We propose PlanT, a learning-based planner that leverages an object-levelrepresentation (Fig. 1 right) as an input to a transformer encoder [24].
Revised: We propose PlanT, a learning-based planner that leverages an object-level representation (Fig. 1 right) as an input to a transformer encoder [25].

Original: We represent a scene as aset of features corresponding to (1) nearby vehicles and (2) the route the planner must follow.
Revised: We represent a scene as a set of features corresponding to (1) nearby vehicles and (2) the route the planner must follow.

Original: Eachfeature is low-dimensional, e.g., position, extent, and orientation (details in Section 3).

Original: We showthat despite the low feature dimensionality, our model achieves state-of-the-art results using thisrepresentation.
Revised: We show that despite the low feature dimensionality, our model achieves state-of-the-art results.

Original: We then propose a novel evaluation scheme and metric to analyze explainabilitywhich is generally applicable to any learning-based planner.
Revised: We then propose a novel evaluation scheme and metric to analyze explainability which is generally applicable to any learning-based planner.

Original: Speciﬁcally, we test the ability of aplanner to identify the objects that are the most relevant to account for to plan a collision-free route.
Revised: Speciﬁcally, we test the ability of a planner to identify the objects that are the most relevant to account for to plan a collision-free route.

Original: We perform a detailed empirical analysis of learning-based planning on the Longest6 benchmark [25] of the CARLA simulator [26].
Revised: We perform a detailed empirical analysis of learning-based planning on the Longest6 benchmark [26] of the CARLA simulator [27].

Original: We ﬁrst identify the key missing elements in the design ofexisting learned planners such as their incomplete ﬁeld of view and sub-optimal dataset and modelsizes.
Revised: We ﬁrst identify the key missing elements in the design of existing learned planners such as their incomplete ﬁeld of view and sub-optimal dataset and model sizes.

Original: Addressing these issues leads to expert-level driving scores for planning with convolutionalneural networks (CNNs), which is the dominant paradigm of this ﬁeld, signiﬁcantly advancing thestate of the art.

Original: We then show theadded advantages of our proposed transformer architecture, includingfurther improvements in performance and signiﬁcantly faster inference times.
Revised: We then show the advantages of our proposed transformer architecture, including improvements in performance and signiﬁcantly faster inference times.

Original: Finally, weshow that the attention weights of the transformer, which are readily accessible, can be used to represent object relevance.
Revised: Finally, we show that the attention weights of the transformer, which are readily accessible, can be used to represent object relevance.

Original: Our qualitative and quantitative results on explainability conﬁrm that PlanTattends to the objects that match our intuition for the relevance of objects for safe driving.
Revised: Our qualitative and quantitative results on explainability conﬁrm that PlanT attends to the objects that match our intuition for the relevance of objects for safe driving.

Original: Contributions.
Revised: Contributions.

Original: (1) We demonstrate that a simple object-level representation is sufﬁcient to encodeall the information relevant for planning in urban driving environments. (2) With this representation,we signiﬁcantly improve upon the previous state of the art  on CARLAwhen we use a standard CNNarchitecture, and obtain further gains in performance and runtime via PlanT, our novel transformerbased approach. 
Revised: (1) Using a simple object-level representation, we signiﬁcantly improve upon the previous state of the art for planning on CARLA via PlanT, our novel transformer-based approach. (2) Through a comprehensive experimental study, we identify that the ego vehicle’s route, a full 360 ° ﬁeld of view, and information about vehicle speeds are critical elements of a planner’s input representation.

Original: (3) We propose a protocol and metric for evaluating a planner’s prioritization ofobstacles in a scene, and show that PlanT is more explainable than CNN-based methods, i.e., theattention weights of the transformer identify the most relevant objects more reliably.
Revised: (3) We propose a protocol and metric for evaluating a planner’s prioritization of obstacles in a scene and show that PlanT is more explainable than CNN-based methods, i.e., the attention weights of the transformer identify the most relevant objects more reliably.

Original:  Related Work
Revised:  Related Work

Original: Intermediate Representations for Driving.
Revised: Intermediate Representations for Driving.

Original: Early work on decoupling end-to-end driving into twostages predicts a set of low-dimensional affordances from sensor inputs with CNNs which are theninput to a rule-based planner [27].
Revised: Early work on decoupling end-to-end driving into two stages predicts a set of low-dimensional affordances from sensor inputs with CNNs which are then input to a rule-based planner [28].

Original: These affordances are scene-descriptive attributes (e.g. emergency brake, red light, center-line distance, angle) that are compact, yet comprehensive enough toenable simple driving tasks, such as urban driving on the initial version of CARLA [26].
Revised: These affordances are scene-descriptive attributes (e.g. emergency brake, red light, center-line distance, angle) that are compact, yet comprehensive enough to enable simple driving tasks, such as urban driving on the initial version of CARLA [27].

Original: Unfortunately, methods based on affordances perform poorly on subsequent benchmarks in CARLA whichinvolve higher task complexity [28].
Revised: Unfortunately, methods based on affordances perform poorly on subsequent benchmarks in CARLA which involve higher task complexity [29].

Original: Most state-of-the-art driving models instead rely heavily onannotated 2D data either as intermediate representations or auxiliary training objectives [25, 29].
Revised: Most state-of-the-art driving models instead rely heavily on annotated 2D data either as intermediate representations or auxiliary training objectives [26, 30].

Original: Several  studies show that using semantic segmentation as an intermediate representation helps fornavigational tasks [30, 31, 32, 33].
Revised: Several subsequent studies show that using semantic segmentation as an intermediate representation helps for navigational tasks [31, 32, 33, 34].

Original: More recently, there has been a rapid growth in interest on using BEV semantic segmentation maps as the input representation to planners [9, 11, 29, 17].
Revised: More recently, there has been a rapid growth in interest in using BEV semantic segmentation maps as the input representation to planners [9, 12, 30, 18].

Original: To reducethe immense labeling cost of such segmentation methods, Behl et al.
Revised: To reduce the immense labeling cost of such segmentation methods, Behl et al.

Original: [34] propose visual abstractions, which are label-efﬁcient alternatives to dense 2D semantic segmentation maps.
Revised: [35] propose visual abstractions, which are label-efﬁcient alternatives to dense 2D semantic segmentation maps.

Original: They showthat reduced class counts and the use of bounding boxes instead of pixel-accurate masks for certainclasses is sufﬁcient.
Revised: They show that reduced class counts and the use of bounding boxes instead of pixel-accurate masks for certain classes are sufﬁcient.

Original: Wang et al.
Revised: Wang et al.

Original: [35] explicitly extract objects and render them into a BEV image forplanning.
Revised: [36] explore the use of object-centric representations for planning by explicitly extracting objects and rendering them into a BEV input for a planner.

Revised: However, so far, the literature lacks a systematic analysis of whether object-centric representations are better or worse than BEV context techniques for planning in dense trafﬁc, which we address in this work.

Original: Instead of rendering objects onto a 2D image, we keep our representation low-dimensionaland compact by directly considering the set of objects as inputs to our model.
Revised: We keep our representation simple and compact by directly considering the set of objects as inputs to our models.

Original: In addition, instead ofusing a CNN-based architecture to process the rendered images, we show that using a transformerleads to improved performance, efﬁciency, and explainability.
Revised: In addition to baselines using CNNs to process the object-centric representation, we show that using a transformer leads to improved performance, efﬁciency, and explainability.

Original: Transformers for Control.
Revised: Transformers for Forecasting.

Original: Transformers obtain impressive results in several research areas [24,36, 37, 38]. To make use of this architecture, the input needs to be represented as a sequence orset.
Revised: Transformers obtain impressive results in several research areas [25, 37, 38, 39], including simple interactive environments such as Atari games [40, 41, 42, 43, 44].

Revised: While the end objective differs, one application domain that involves similar challenges to planning is motion forecasting.

Original: Using transformers by describing the environment state as a sequence has been successful for Atari games [39, 40, 41, 42, 43].
Revised: Most existing motion forecasting methods use a rasterized input in combination with a CNN-based network architecture [45, 46, 47, 48, 49, 50].

Original: In the ﬁeld of motion forecasting for driving, Gao et al.
Revised: Gao et al.

Original: [44] showthe advantages of representing an HD map as a set of vectors instead of a BEV image.
Revised: [51] show the advantages of object-level representations for motion forecasting via Graph Neural Networks (GNN).

Original: Severalfollow-up works use this representation in combination with transformer-based architectures [45,46].
Revised: Several follow-ups to this work use object-level representations in combination with Transformer-based architectures [52, 53, 54].

Original: However, they only tackle motion forecasting and do not address the downstream planning taskconsidered by our work. Moreover, while all of these methods rely on an HD map with detailedinformation regarding lane markings and road boundaries as inputs, we show that the ego vehicle’sroute is a more compact yet sufﬁcient representation.
Revised: Our key distinctions when compared to these methods are the architectural simplicity of PlanT (our use of simple self-attention transformer blocks and the proposed route representation) as well as our closed-loop evaluation protocol (we evaluate the driving performance in simulation and report online driving metrics).

Original: Explainability.
Revised: Explainability.

Original: Explaining the decisions of neural networks is a rapidly evolving researchﬁeld [47, 48, 49, 50, 51, 52, 53].
Revised: Explaining the decisions of neural networks is a rapidly evolving research ﬁeld [55, 56, 57, 58, 59, 60, 61].

Original: Prior work on explaining policies focuses on reward decomposition [54], identifying the salient areas in an image using perturbations [55], approximating a deepneural network with a more interpretable model [56], or using interpretable attention mechanismsfor predicting the correct action [57, 58].

Original: In the context of self-driving cars, existing work usestext [59] or heatmaps [60] to explain decisions.
Revised: In the context of self-driving cars, existing work uses text [62] or heatmaps [63] to explain decisions.

Original: In contrast, we can directly obtain post hoc explanations for decisions of our learning-based PlanT architecture by considering its learned attention.
Revised: In our work, we can directly obtain post hoc explanations for decisions of our learning-based PlanT architecture by considering its learned attention.

Original:  Furthermore, we introduce a simple approach to measure the quality of explanations for a planner byconsidering the change in driving performance when only considering the object that the explanationdeemed most relevant in a scene for the planner’s prediction.
Revised: While the concurrent work CAPO [64] uses a similar strategy, it only considers pedestrian-ego interactions on an empty route, while we consider the full planning task in an urban environment with dense trafﬁc. Furthermore, we introduce a simple metric to measure the quality of explanations for a planner.

Original:  Planning Transformers
Revised:  Planning Transformers

Original: In this section, we provide details about our task setup, novel scene representation, simple but effective architecture, and training strategy resulting in state-of-the-art performance.
Revised: In this section, we provide details about our task setup, novel scene representation, simple but effective architecture, and training strategy resulting in state-of-the-art performance.

Revised: A PyTorch-style pseudo-code snippet outlining PlanT and its training is provided in the supplementary material.

Original: Task.
Revised: Task.

Original: We consider the task of point-to-point navigation in an urban setting where the goal is to drivefrom a start to a goal location while reacting to other dynamic agents and following trafﬁc rules.
Revised: We consider the task of point-to-point navigation in an urban setting where the goal is to drive from a start to a goal location while reacting to other dynamic agents and following trafﬁc rules.

Original: We use Imitation Learning (IL) to train the driving agent.
Revised: We use Imitation Learning (IL) to train the driving agent.

Original: The goal of IL is to learn a policy π thatimitates the behavior of an expert π ∗ .
Revised: The goal of IL is to learn a policy π that imitates the behavior of an expert π ∗ (the expert implementation is described in Section 4).

Original: In our setup, the policy is a mapping π : X −→ W from ournovel object-level input representation X to the future trajectory W of an expert driver.
Revised: In our setup, the policy is a mapping π : X −→ W from our novel object-level input representation X to the future trajectory W of an expert driver.

Original: We describethe expert and data collection procedure in Section 4.

Original: For following trafﬁc rules, we assume accessto thetrafﬁc light statel ∈ { green , red } of the next trafﬁc light relevant to the ego vehicle.
Revised: For following trafﬁc rules, we assume access to the state of the next trafﬁc light relevant to the ego vehicle l ∈ { green , red } .

Original: Object-level Representation.
Revised: Tokenization.

Original: In order to drive safely, the input representation needs to encodeall the task-speciﬁc information about the scene. To this end, we represent the scene as a set ofobjects, with vehicles and segments of the route each being assigned an oriented bounding box in BEV space (Fig. 1 right).
Revised: To encode the task-speciﬁc information required from the scene, we represent it using a set of objects, with vehicles and segments of the route each being assigned an oriented bounding box in BEV space (Fig. 1 right).

Original: Let X ∈ R ( V + R ) ×
Revised: Let X t =

Original:  A be a set ofV vehicles and R route segments  with Aattributes each.
Revised: V t ∪S t , where V t ∈ R V t × A and S t ∈ R S t × A represent the set of vehicles and the set of route segments at time-step t with A = 6 attributes each.

Original: These attributes  include  the position of the bounding box ( x i , y i ) relative to the egovehicle, the extent ( w i , h i ) , the orientation ϕ i ∈ [0 , 2 π ]  and a type-speciﬁc attribute v i , giving A = 6dimensions.
Revised: Speciﬁcally, if o i,t ∈ X t represents a particular object, the attributes of o i,t include an object type-speciﬁc attribute z i,t (described below), the position of the bounding box ( x i,t , y i,t ) relative to the ego vehicle, the orientation ϕ i,t ∈ [0 , 2 π ] , and the extent ( w i,t , h i,t ) .

Original: Each object  can be described as a vector o i = 
Revised: Thus, each object o i,t can be described as a vector o i,t = { z i,t , x i,t , y i,t , ϕ i,t , w i,t , h i,t } , or concisely as { o i,t,a } 6 a =1 . For the vehicles V t , we extract the attributes directly from the simulator in our main experiments and use an off-the-shelf perception module based on CenterNet [65] (described in the supplementary material) for experiments involving a full driving system. We consider only vehicles up to a distance

Original: [ x i , y i , w i , h i , ϕ i , v i ] . For a vehicle, weuse v i to represent the speed.
Revised: D max from the ego vehicle, and use o i,t, 1 (i.e., z i,t ) to represent the speed.

Original: For segments  of  the route, v i denotes the ordering, starting from 0 forthe segment closest to the ego vehicle.
Revised: To obtain the route segments S t , we ﬁrst sample a dense set of N t points U t ∈ R N t × 2 along the route ahead of the ego vehicle at time-step t .

Original: To obtain these route segments we subsample the dense set ofpoints along the route provided by CARLA  using the Ramer-Douglas-Peucker algorithm [61, 62].
Revised: We directly use the ground-truth points from CARLA as U t in our main experiments and predict them with a perception module for the PlanT with perception experiments in Section 4.2. The points are subsampled using the Ramer-Douglas-Peucker algorithm [66, 67] to select a subset ˆ U t .

Original: One segment spans the area between twoof these subsampled route points, with the length equal tothe distance between the points and the width equal to the lane width.
Revised: One segment spans the area between two points subsampled from the route, u i,t , u i +1 ,t ∈ ˆ U t .

Original: In addition, we restrict themaximum length of any single segment  to L max and always input a ﬁxed number of segments N sto our policy.
Revised: Speciﬁcally, o i,t, 1 (i.e., z i,t ) denotes the ordering for the current time-step t , starting from 0 for the segment closest to the ego vehicle. We set the segment length

Original: Since the primary focus of our work is not perception but planning, we extract theobject attributes directly from the simulator. We consider objects up to a distance D max from theego vehicle without any further ﬁltering.
Revised: o i,t, 6 =

Original: Note that our representation contains similar informationas that in the widely used BEV map representation (Fig. 1 left), but only encodes the route relevantto the ego vehicle instead of the whole map. In practice, we ﬁnd that this is sufﬁcient for current CARLA benchmarks (Table 2a). More details regarding the object extraction and visualizations ofthe route representation are provided in the supplementary material.
Revised: ||

Original: Output.
Revised: u

Original: We predict the future trajectory W of the ego vehicle, centered at the coordinate frameof the current time-step t . The trajectory is represented by a sequence of 2D waypoints in BEVspace, { w t = ( x t , y t ) } Tt =1 for T = 4 future time-steps.
Revised: i,t − u i +1 ,t || 2 , and the width, o i,t, 5 , equal to the lane width. In addition, we clip o i,t, 6 ≤ L max , ∀ i, t ; and always input a ﬁxed number of segments N s to our policy.

Original: These waypoints are provided to lateral andlongitudinal PID controllers to output actions. Details are provided in the supplementary material.
Revised: More details and visualizations of the route representation are provided in the supplementary material.

Original: Planning Transformer (PlanT).
Revised: Token Embeddings.

Original: Our model is illustrated in Fig.
Revised: Our model is illustrated in Fig.

Original: The main building block forthe IL policy π is astandard transformer encoder, based on the BERT architecture [36]. Whileour architecture does not deviate from the original transformer implementation [24], adapting it to self-driving requires the tokenization of our proposed input representation. For this, we linearlyproject the attributes of each object to one token embedding.
Revised: As a ﬁrst step, applying a transformer backbone requires the generation of embeddings for each input token, for which we deﬁne a linear projection ρ : R 6 → R H (where H is the desired hidden dimensionality).

Original: We then add the token embeddingsto learnable object type embeddings indicating to which type the token belongs: [CLS] , vehicleor route. The [CLS] token (based on [36, 38]) is a learnable token prepended to the input.
Revised: To obtain token embeddings e i,t , we add the projected input tokens o i,t to a learnable object type embedding vector e v ∈ R H or e s ∈ R H , indicating to which type the token belongs (vehicle or route segment). [Equation]

Original: Thistoken’s processing involves an attention-based aggregation of the features from all other tokens, andis used for generating the waypoint predictions.
Revised: Main Task: Waypoint Prediction.

Original: At the ﬁnal layer of the transformer, we extractthe feature vector from the [CLS] token and reduce its dimensionality via a linear layer.{+standard transformer encoder, τ : R V t + S t +
+}
Revised: The main building block for the IL policy π is a {+standard transformer encoder, τ : R V t + S t +
+}

Original: As inprevious work [17], we concatenate the binary trafﬁc light ﬂag and the feature vector before passingit to an auto-regressive waypoint decoder that makes use of GRUs [63].
Revised: × H → R V t + S t +1 × H , based on the BERT architecture [37]. Specifically, we deﬁne a learnable [CLS] token, c ∈ R H (based on [37, 39]) and stack this with other token embeddings to obtain the transformer input [ c , e 1 ,t , ..., e V t + S t ,t ] .

Original: We input the current egoposition and goal location to the GRU, which predicts T = 4 differential waypoints { δ w t } Tt =1 ,where t is  the future time-step. With these differential waypoints we obtain the future waypoints as{ w t = w t − 1 + δ w t } Tt =1 .
Revised: The [CLS] token’s processing through τ involves an attention-based aggregation of the features from all other tokens, after which it is used for generating the waypoint predictions via an auto-regressive waypoint decoder, γ : R ( H +1) → R W × 2 .

Original: For a detailed description of the waypoint decoder, see [17, 25]. Training.
Revised: For a detailed description of the waypoint decoder architecture, see [18, 26].

Original: Following recent driving models [9, 29, 25], we leverage the L 1 loss to ground truthfuture waypoints  w gt as our main training objective.
Revised: We concatenate the binary trafﬁc light ﬂag, l t to the transformer output as the initial hidden state to the decoder which makes use of GRUs [68] to predict the future trajectory W t of the ego vehicle, centered at the coordinate frame of the current time-step t . The trajectory is represented by a sequence of 2D waypoints in BEV space, { w w = ( x w , y w ) } t + W w = t +1 for W = 4 future time-steps: W t = γ ( l t , h t [0]) , where h t = τ ([ c , e 1 ,t , ..., e V t + S t ,t ]) .

Original: Besides this, we propose the auxiliary taskof predicting the future attributes of other vehicles.
Revised: Auxiliary Task: Vehicle Future Prediction. In addition to the primary waypoint prediction task, we propose the auxiliary task of predicting the future attributes of other vehicles.

Original: This is aligned with the overall driving goalin two ways.
Revised: This is aligned with the overall driving goal in two ways.

Original: (1) The ability to reason about the future of other vehicles is important in an urbanenvironment as it heavily inﬂuences the ego vehicle’s own future.
Revised: (1) The ability to reason about the future of other vehicles is important in an urban environment as it heavily inﬂuences the ego vehicle’s own future.

Original: (2) Our main task is to predict theego vehicle’s future trajectory, which means the output feature of the transformer needs to encode allthe information necessary to predict the future.
Revised: (2) Our main task is to predict the ego vehicle’s future trajectory, which means the output feature of the transformer needs to encode all the information necessary to predict the future.

Original: Supervising the outputs of all vehicles on a similartask (i.e., predicting the location, pose and velocity of a future time-step) exploits synergies betweenthe task of the ego vehicle and the other vehicles [64, 29].
Revised: Supervising the outputs of all vehicles on a similar task (i.e., predicting vehicle poses at a future time-step) exploits synergies between the task of the ego vehicle and the other vehicles [69, 30].

Original: We input the  vehicle tokens for a giventime-step T in and predict class probabilities ˆ p  for every attribute of each vehicle of a future timestep T in + δt using a linear layer per attribute type on top of the corresponding output embedding.{+{ ψ a : R H → R Z a } 
 a =1 : p i,t +1 ,a = Softmax ( ψ a ( h i,t )) , where a = 1 , ..., 6 .+}
Revised: Speciﬁcally, using the output embeddings { h i,t } V t i =1 corresponding to all vehicle tokens o i,t ∈ V t , we predict class probabilities {{ p i,t +1 ,a } 6 a =1 } V t i =1 for the speed, position, orientation, and extent attributes from the next time-step { o i,t +1 ,a } 6 a =1 using a linear layer per attribute type {+{ ψ a : R H → R Z a } 
 a =1 : p i,t +1 ,a = Softmax ( ψ a ( h i,t )) , where a = 1 , ..., 6 .+}

Original: We choose to discretize the output attributes to allow  uncertainty in the predictions since the futureis non-deterministic.
Revised: We choose to discretize each attribute into Z a bins to allow for uncertainty in the predictions since the future is multi-modal.

Original: This is also better aligned with how humans drive without predicting exactlocations and velocities, where a rough estimate is sufﬁcient to make a safe decision.
Revised: This is also better aligned with how humans drive without predicting exact locations and velocities, where a rough estimate is sufﬁcient to make a safe decision.

Revised: Loss Functions.

Revised: Following recent driving models [9, 11, 30, 26], we leverage the L 1 loss to the ground truth future waypoints w gt as our main training objective.

Original: We calculatethe cross-entropy loss L CE using the one-hot encoded representation  of the attribute p gt as theground truth class label.
Revised: For the auxiliary task, we calculate the cross-entropy loss L CE using a one-hot encoded representation p gti,t +1 of the ground truth future vehicle attributes o gti,t +1 .

Original: We train the model in a multi-task setting using a weighted combination ofthese loss functions with a weighting factor λ :
Revised: We train the model in a multi-task setting using a weighted combination of these losses with a weighting factor λ :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  Experiments
Revised:  Experiments

Original: In this section, we describe our experimental setup, evaluate the driving performance of our approach, analyze the explainability of its driving decisions, and ﬁnally discuss limitations.
Revised: In this section, we describe our experimental setup, evaluate the driving performance of our approach, analyze the explainability of its driving decisions, and ﬁnally discuss limitations.

Original: Dataset and Benchmark.
Revised: Dataset and Benchmark.

Original: We use the expert, dataset and evaluation benchmark Longest 6 proposedby [25].
Revised: We use the expert, dataset and evaluation benchmark Longest 6 proposed by [26].

Original: The expert policy is a rule-based algorithm with access to ground truth locations of thevehicles as well as privileged information that is not available to PlanT such as their actions anddynamics.
Revised: The expert policy is a rule-based algorithm with access to ground truth locations of the vehicles as well as privileged information that is not available to PlanT such as their actions and dynamics.

Original: Using this information, the expert determines the future position of all vehicles andestimates intersections between its own future position and those of the other vehicles to preventmost collisions.
Revised: Using this information, the expert determines the future position of all vehicles and estimates intersections between its own future position and those of the other vehicles to prevent most collisions.

Original: The dataset collected with this expert contains 228k frames.
Revised: The dataset collected with this expert contains 228k frames.

Original: We use this as ourreference point denoted by 1 × .
Revised: We use this as our reference point denoted by 1 × .

Original: For our analysis, we also generate additional data following [25],but use different seeds to obtain different initializations of the trafﬁc.
Revised: For our analysis, we also generate additional data following [26] but with different initializations of the trafﬁc.

Original: The data quantities we use arealways relative to the original dataset (i.e. 2 × contains double the data, 3 × contains triple).
Revised: The data quantities we use are always relative to the original dataset (i.e., 2 × contains double the data, 3 × contains triple).

Original: We referthe reader to [25] for a detailed description of the expert algorithm and dataset collection.
Revised: We refer the reader to [26] for a detailed description of the expert algorithm and dataset collection.

Original: Metrics.
Revised: Metrics.

Original: We report the established metrics of the CARLA leaderboard [65]: Route Completion(RC) , Infraction Score (IS) , and Driving Score (DS) , which is the weighted average of the RC and
Revised: We report the established metrics of the CARLA leaderboard [70]: Route Completion (RC) , Infraction Score (IS) , and Driving Score (DS) , which is the weighted average of the RC and

Original: IS.
Revised: IS.

Original: In addition, weshow Collisions with Vehicles per kilometer (CV) and Inference Time (IT) forone forward pass of the model, measured in milliseconds on a single RTX 3080 GPU. A detaileddescription of the metrics is included in the supplementary material.
Revised: In addition, we

Original: Baselines.
Revised: Baselines.

Original: To highlight the advantages of learning-based planning, we include a rule-based planning baseline that uses the same inputs as PlanT. It follows the same high-level algorithm as theexpert but estimates the future of other vehicles using a constant speed assumption since it does nothave access to their actions.
Revised: To highlight the advantages of learning-based planning, we include a rule-based planning baseline that uses the same inputs as PlanT. It follows the same high-level algorithm as the expert but estimates the future of other vehicles using a constant speed assumption since it does not have access to their actions.

Original: AIM-BEV [17] is a recent privileged agent trained using IL.
Revised: AIM-BEV [18] is a recent privileged agent trained using IL.

Original: It uses a
Revised: It uses a

Original: BEV semantic map input with channels for the road, lane markings, vehicles, and pedestrians, anda GRU identical to PlanT to predict a trajectory for the ego vehicle which is executed using lateraland longitudinal PID controllers.
Revised: BEV semantic map input with channels for the road, lane markings, vehicles, and pedestrians, and a GRU identical to PlanT to predict a trajectory for the ego vehicle which is executed using lateral and longitudinal PID controllers.

Original: Roach [11] is a Reinforcement Learning (RL) based agent witha similar input representation as AIM-BEV that directly outputs driving actions.
Revised: Roach [12] is a Reinforcement Learning (RL) based agent with a similar input representation as AIM-BEV that directly outputs driving actions.

Original: Roach and AIMBEV are the closest existing methods to PlanT.
Revised: Roach and AIMBEV are the closest existing methods to PlanT.

Original: However, they use a different input ﬁeld of view intheir representation leading to sub-optimal performance.
Revised: However, they use a different input ﬁeld of view in their representation leading to sub-optimal performance.

Original: We additionally build PlanCNN , a morecompetitive CNN-based approach for planning with the same training data and input information as
Revised: We additionally build PlanCNN , a more competitive CNN-based approach for planning with the same training data and input information as

Original: PlanT, which is adapted from AIM-BEV to input a rasterized version of our object-level representation.
Revised: PlanT, which is adapted from AIM-BEV to input a rasterized version of our object-level representation.

Original: We render the oriented vehicle bounding boxes in one channel, represent the speed of eachpixel in a second channel, and render the oriented bounding boxes of the route in the third channel.
Revised: We render the oriented vehicle bounding boxes in one channel, represent the speed of each pixel in a second channel, and render the oriented bounding boxes of the route in the third channel.

Original: We provide detailed descriptions of the baselines in the supplementary material.
Revised: We provide detailed descriptions of the baselines in the supplementary material.

Original: Implementation.
Revised: Implementation.

Original: Our analysis includes three BERT encoder variants taken from [66]: MINI , SMALL ,and MEDIUM with 11.2M, 28.8M and 41.4M parameters respectively.
Revised: Our analysis includes three BERT encoder variants taken from [71]: MINI , SMALL , and MEDIUM with 11.2M, 28.8M and 41.4M parameters respectively.

Original: For PlanCNN, we experimentwith two backbones: ResNet-18 and ResNet-34.
Revised: For PlanCNN, we experiment with two backbones: ResNet-18 and ResNet-34.

Original: We choose these architectures to maintain an ITwhich enables real-time execution.
Revised: We choose these architectures to maintain an IT which enables real-time execution.

Original: We train the models from scratch on 4 RTX 2080Ti GPUs with atotal batch size of 128.
Revised: We train the models from scratch on 4 RTX 2080Ti GPUs with a total batch size of 128.

Original: Optimization is done with AdamW [67] for 47 epochs with an initial learningrate of 10 − 4 which we decay by 0.1 after 45 epochs.
Revised: Optimization is done with AdamW [72] for 47 epochs with an initial learning rate of 10 − 4 which we decay by 0.1 after 45 epochs.

Original: Training takes approximately 2.8, 3.4, and 4hours for the three BERT variants on the 1 × dataset.
Revised: Training takes approximately 2.8, 3.4, andhours for the three BERT variants on the 1 × dataset.

Original: We set the weight decay to 0.1 and clip thegradient norm at 1.0.
Revised: We set the weight decay to 0.1 and clip the gradient norm at 1.0.

Original: For the auxiliary objective we use quantization precisions  of 0 . 5 m for theposition and extent, 3 . 75 km / h for the speed and 11 . 25 ° for the orientation of the vehicles.
Revised: For the auxiliary objective, we use quantization precisions Z a of 128 bins for the position, 4 bins for the speed and 32 bins for the orientation of the vehicles.

Original: We use T in = 0 and δt = 1 for auxiliary supervision.
Revised: We use T in = 0 and δt = 1 for auxiliary supervision.

Original: The loss weight λ is set to 0.2.
Revised: The loss weight λ is set to 0.2.

Original: By default, we use D max =30 m , N s = 2 , and L max =30 m .
Revised: By default, we use D max =30 m , N s = 2 , and L max =10 m .

Original:  Additional details regarding the model conﬁgurations aswell as detailed ablation studies on the multi-task training and input representation hyperparametersare provided in the supplementary material. 
Revised: For our experiment with a full driving stack, we use a perception module based on TransFuser [26] to obtain the object-level input representation for PlanT. Additional details regarding this perception module as well as detailed ablation studies on the multi-task training and input representation hyperparameters are provided in the supplementary material. [Table] available checkpoints at the time of submission for LAV and Roach. See supplementary material for results with checkpoints released later.

Original: 4.1 Obtaining Expert-Level Driving Performance
Revised: 4.1 Obtaining Expert-Level Driving Performance

Original: In the following, we discuss the key ﬁndings of our study which enable expert-level driving withlearned planners.
Revised: In the following, we discuss the key ﬁndings of our study which enable expert-level driving with learned planners.

Revised: We begin with a discussion of the privileged methods and analyze the sensorbased methods in Section 4.2.

Original: Unless otherwise speciﬁed, the experiments consider the largest version of ourdataset (3 × ) and models ( MEDIUM for PlanT, ResNet-34 for PlanCNN).
Revised: Unless otherwise speciﬁed, the experiments consider the largest version of our dataset (3 × ) and models ( MEDIUM for PlanT, ResNet-34 for PlanCNN).

Original: We ﬁrst analyze LAV [29] and TransFuser [25], which are recent state-of-the-art sensor-based methods on CARLA. Since PlanT accesses additional input information, it is not directly comparable tothese baselines. However, we include them to put our results into perspective and to demonstrate the [Table]
Revised: Input Representation.

Original: potential gain for future sensor-based models. The results are shown in Table 1.
Revised: Table 1 compares the performance on the Longest6 benchmark.

Original: Interestingly, bothsensor-based methods outperform the rule-based planning algorithm despite their lack of groundtruth object and route information, showing the limitations of rule-based systems in complex environments. In contrast, PlanT is over 30 points better than TransFuser in terms of DS ( 81 . 36 vs.47 . 30 ), combining the high IS of the rule-based approach and high RC of TransFuser.
Revised: The rulebased system acts cautiously and gets blocked often.

Original: Importantly,this improvement is achieved using the same expert policy and IL-based training objective as TransFuser. Additionally, TransFuser incorporates manually designed heuristics in its controller to creepforward if the policy gets stuck without cause [25], which are unnecessary for PlanT. Input Representation.
Revised: Among the learning-based methods, both

Original: In Table 1, we observe that both PlanCNN and PlanT signiﬁcantly outperform AIM-BEV [17] and Roach [11].
Revised: PlanCNN and PlanT signiﬁcantly outperform AIM-BEV [18] and Roach [12].

Original: We systematically break down the factors leading to this in Table 2a by studying the following: (1) the representation used for the road layout, (2) the horizontal ﬁeld of view, (3) whether objects behind the ego vehicle are part of the representation, and (4)whether the input representation incorporates speed.
Revised: We systematically break down the factors leading to this in Table 2a by studying the following: (1) the representation used for the road layout, (2) the horizontal ﬁeld of view, (3) whether objects behind the ego vehicle are part of the representation, and (4) whether the input representation incorporates speed.

Original: Roach uses the same maximum distance to the sidesof the vehicle as AIM-BEV but additionallyincludes 8 m to the back and multiple input frames to reason about speed.
Revised: Roach uses the same view to the sides as AIM-BEV but additionally includes 8 m to the back and multiple input frames to reason about speed.

Original: We see in Table 2a thattraining PlanCNN in a conﬁguration close to Roach (with the key differences being the removal ofdetails from the map and a 0 m back view) results in a higher DS ( 59 . 97 vs. 55 . 27 ), showing thatit is possible to replace the lane marking and road boundary information in the HD map with ourcompact representation.
Revised: We see in Table 2a that training PlanCNN in a conﬁguration close to Roach (with the key differences being the removal of details from the map and a 0 m back view) results in a higher DS ( 59 . 97 vs. 55 . 27 ), demonstrating the importance of the route representation for urban driving. While additional information might be important when moving to more complex environments, our results suggest that the route is particularly important.

Original: Increasing the side view from 19 . 2 to 30 m improves PlanCNN from 59 .
Revised: Increasing the side view from 19 . 2 to 30 m improves PlanCNN from 59 .

Original: 97to 70 .
Revised: 97 to 70 .

Original: 72 (Table 2a).

Original: This indicates that reasoning about distant vehicles is important.

Original: Furthermore,including vehicles to the rear further boosts PlanCNN’s DS to 77 .
Revised: Including vehicles to the rear further boosts PlanCNN’s DS to 77 .

Original: This also holds true for PlanT,where adding the rear improves the DS from 72 .
Revised: and improves PlanT’s DS from 72 .

Original: 86 to 81 .
Revised: 86 to 81 .

Original: These results show that a full 360 °ﬁeld of view is helpful to handle certain situations encountered during our evaluation (e.g. changinglanes).
Revised: These results show that a full 360 ° ﬁeld of view is helpful to handle certain situations encountered during our evaluation (e.g. changing lanes).

Original: Finally, removing the vehicle speed input signiﬁcantly reduces the DS for both PlanCNNand PlanT (Table 2a), showing the importance of cues regarding motion.
Revised: Finally, removing the vehicle speed input signiﬁcantly reduces the DS for both PlanCNN and PlanT (Table 2a), showing the importance of cues regarding motion.

Original: PlanT vs. PlanCNN.
Revised: Scaling.

Original: In Table 2b, we show the impact of scaling the dataset size and the modelsize for PlanT and PlanCNN.
Revised: In Table 2b, we show the impact of scaling the dataset size and the model size for PlanT and PlanCNN.

Original: The circle size indicates the inference time (IT) needed for that model.
Revised: The circle size indicates the inference time (IT).

Original: First, we observe that PlanT demonstrates better data efﬁciency than PlanCNN, e.g., using the 1 ×data setting is sufﬁcient to reach the same performance as PlanCNN with 2 × .
Revised: First, we observe that PlanT demonstrates better data efﬁciency than PlanCNN, e.g., using the 1 × data setting is sufﬁcient to reach the same performance as PlanCNN with 2 × .

Original: Interestingly, scalingthe data from 1 × to 3 × leads to expert-level performance with all the models other than PlanCNNwith ResNet-18, showing the effectiveness of scaling.
Revised: Interestingly, scaling the data from 1 × to 3 × leads to expert-level performance, showing the effectiveness of scaling.

Original: In fact, PlanT MEDIUM ( 81 . 36 ) outperforms theexpert ( 76 . 91 ) in some evaluation runs.
Revised: In fact, PlanT MEDIUM ( 81 . 36 ) outperforms the expert ( 76 . 91 ) in some evaluation runs.

Original: We visualize one consistent failure mode of the expert thatleads to this discrepancy in Fig. 3a.
Revised: We visualize one consistent failure mode of the expert that leads to this discrepancy in Fig. 3a.

Original: We observe that the expert sometimes stops once it has alreadyentered an intersection if it anticipates a collision, which then leads to collisions or blocked trafﬁc.
Revised: We observe that the expert sometimes stops once it has already entered an intersection if it anticipates a collision, which then leads to collisions or blocked trafﬁc.

Original: On the other hand, PlanT learns to wait further outside an intersection before entering which is asmoother function than the discrete rule-based expert, and subsequently avoids these infractions.
Revised: On the other hand, PlanT learns to wait further outside an intersection before entering which is a smoother function than the discrete rule-based expert and subsequently avoids these infractions.

Original: Importantly, in our ﬁnal setting, PlanT MEDIUM is around 3 × as fast as PlanCNN while being 4 pointsbetter in terms of the DS, and PlanT MINI is 5.3 × as fast (IT= 5 .
Revised: Importantly, in our ﬁnal setting, PlanT MEDIUM is around 3 × as fast as PlanCNN while being 4 points better in terms of the DS and PlanT MINI is 5.3 × as fast (IT= 5 .

Original: 46 ms ) while reaching the same DS as PlanCNN.
Revised: 46 ms ) while reaching the same DS as PlanCNN.

Original: This shows that PlanT is suitable for systems where fast inference time is a requirement.
Revised: This shows that PlanT is suitable for systems where fast inference time is a requirement.

Original: To demonstrate the reproducibility of our results with different initializations, we report drivingscores of PlanCNN and PlanT with multiple training seeds in the supplementary material.
Revised: We report results with multiple training seeds in the supplementary material.

Original: Loss.
Revised: Loss.

Original: A detailed study of the training strategy for PlanT can be found in the supplementary material,where we show that the auxiliary loss proposed in Eq.
Revised: A detailed study of the training strategy for PlanT can be found in the supplementary material, where we show that the auxiliary loss proposed in Eq.

Original: (1) is crucial to its performance.
Revised: (4) is crucial to its performance.

Original: However,since this is a self-supervised objective, it can be incorporated without additional annotation costs.
Revised: However, since this is a self-supervised objective, it can be incorporated without additional annotation costs.

Revised: This is in line with recent ﬁndings on training transformers that show the effectiveness of supervising multiple output tokens instead of just a single [CLS] token [73].

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: 4.2 Combining an Off-the-Shelf Perception Module with PlanT

Revised: Next, we discuss the results of the sensor-based methods in Table 1.

Revised: We compare the proposed approach to LAV [30] and TransFuser [26], which are recent state-of-the-art sensor-based methods.

Revised: Our perception module is based on TransFuser, enabling a fair comparison to this approach.

Revised: Therefore, PlanT with perception only detects vehicles to its front and has a limited view to the sides (16m instead of 30m).

Revised: Our approach outperforms TransFuser [26] by 10.36 points and LAV by 24.92 points.

Revised: While TransFuser uses an ensemble and manually designed heuristics to creep forward if stuck [26], these are unnecessary for PlanT with perception.

Revised: Since we do not use an ensemble, we observe a 2.7 × speedup ( 101 .

Revised: 24 ms vs. 37 .

Revised: 61 ms ) in IT compared to TransFuser.

Revised: We refer to the supplementary material for a more detailed analysis.

Original: 4.2 Explainability: Identiﬁcation of Most Relevant Objects
Revised: 4.3 Explainability: Identiﬁcation of Most Relevant Objects

Original: Next, we investigate the explainability of PlanT and PlanCNN by analyzing the objects in the scenethat are relevant and crucial for the agent’s decision.
Revised: Finally, we investigate the explainability of PlanT and PlanCNN by analyzing the objects in the scene that are relevant and crucial for the agent’s decision.

Original: In particular, we measure the relevance of anobject in terms of the learned attention for PlanT and by considering the impact that the removalof each object has on the output predictions for PlanCNN.
Revised: In particular, we measure the relevance of an object in terms of the learned attention for PlanT and by considering the impact that the removal of each object has on the output predictions for PlanCNN.

Original: To quantify the ability to reason aboutthe most relevant objects, we propose a novel evaluation scheme together with the RelativeFiltered
Revised: To quantify the ability to reason about the most relevant objects, we propose a novel evaluation scheme together with the Relative

Original:  Driving Score (RFDS) .
Revised: Filtered Driving Score (RFDS) .

Original: For the rule-based expert algorithm, collision avoidance depends on a singlevehicle which it identiﬁes as the reason for braking.
Revised: For the rule-based expert algorithm, collision avoidance depends on a single vehicle which it identiﬁes as the reason for braking.

Original: To measure the RFDS of a learned planner, werun one forward pass of the planner (without executing the actions) to obtain a scalar relevance scorefor each vehicle in the scene.
Revised: To measure the RFDS of a learned planner, we run one forward pass of the planner (without executing the actions) to obtain a scalar relevance score for each vehicle in the scene.

Original: We then execute the expert algorithm while restricting its observationsto the (single) vehicle with the highest relevance score.
Revised: We then execute the expert algorithm while restricting its observations to the (single) vehicle with the highest relevance score.

Original: The RFDS is deﬁned as the relative DSof this restricted version of the expert compared to the default version which checks for collisionsagainst all vehicles.
Revised: The RFDS is deﬁned as the relative DS of this restricted version of the expert compared to the default version which checks for collisions against all vehicles.

Original: We describe the extraction of the relevance score for PlanT and PlanCNN in thefollowing.
Revised: We describe the extraction of the relevance score for PlanT and PlanCNN in the following.

Original: Our protocol leads to a fair comparison of different agents as the RFDS does not dependon the ability to drive itself but only on the obtained ranking of object relevance. 
Revised: Our protocol leads to a fair comparison of different agents as the RFDS does not depend on the ability to drive itself but only on the obtained ranking of object relevance. [Table]

Original: Baselines.
Revised: Baselines.

Original: As a naïve baseline, we consider the inverse distance to the ego vehicle as a vehicle’srelevance score, such that the expert only sees the closest vehicle.
Revised: As a naïve baseline, we consider the inverse distance to the ego vehicle as a vehicle’s relevance score, such that the expert only sees the closest vehicle.

Original: For PlanT, we extract the relevancescore by adding the attention weights of all layers and heads for the [CLS] token.
Revised: For PlanT, we extract the relevance score by adding the attention weights of all layers and heads for the [CLS] token.

Original: This only requiresa single forward pass of PlanT. Since PlanCNN does not use attention, we choose a masking methodto ﬁnd the most salient region in the image, using the same principle as [51, 50, 49].
Revised: This only requires a single forward pass of PlanT. Since PlanCNN does not use attention, we choose a masking method to ﬁnd the most salient region in the image, using the same principle as [59, 58, 57].

Original: We removeone object at a time from the input image and compute the L 1 distance to the predicted waypointsfor the full image.
Revised: We remove one object at a time from the input image and compute the L 1 distance to the predicted waypoints for the full image.

Original: The objects are then ranked based on how much their absence affects the totaldistance.
Revised: The objects are then ranked based on how much their absence affects the L 1 distance.

Original: In the supplementary material, we provide additional results using GradCAM [47] adaptedfor PlanCNN, which we found to not work as well as the proposed masking approach.

Original: [Table]

Original: [Table]
Revised: [Figure]

Original: the most relevant vehicle according to the respective planner.

Original: Results.
Revised: Results.

Original: We provide results for the reasoning of the plannersabout relevant objects in Table 3.
Revised: We provide results for the reasoning about relevant objects in Table 3.

Original: Both planners signiﬁcantlyoutperform the distance-based baseline, with PlanT obtaininga mean RFDS of 96 [-. 
 compared-] to 82 .
Revised: Both planners signiﬁcantly outperform the distance-based baseline, with PlanT obtaining a mean RFDS of 96 [-. 
 compared-].compared to 82 .

Original: 83 for PlanCNN.
Revised: 83 for PlanCNN.

Original: Weshow qualitative examples in Figure 3b where we highlight thevehicle with the highest relevance score using a red boundingrectangle.
Revised: We show qualitative examples in Figure 3b where we highlight the vehicle with the highest relevance score using a red bounding rectangle.

Original: Both planners correctly identify the most importantobject in simple scenarios when only a few vehicles are in theﬁeld of view or the relevant vehicle is the only one close to theego vehicle’s path.
Revised: Both planners correctly identify the most important object in simple scenarios.

Original: However, PlanT is also able to correctly[Figure] identify the most important object in complex scenes.
Revised: However, PlanT is also able to correctly identify the most important object in complex scenes.

Original: When merging into a lane (examples 1 &2 from ) it correctly looks at the moving vehicles coming from the rear to avoid collisions.
Revised: When merging into a lane (examples 1 & 2 from the left) it correctly looks at the moving vehicles coming from the rear to avoid collisions.

Original: Example 3 shows advanced reasoning about dynamics.
Revised: Example 3 shows advanced reasoning about dynamics.

Original: The two vehicles closer to the ego vehicleare moving awayfrom the intended route at a high speed and are therefore not as relevant forplanning.
Revised: The two vehicles closer to the ego vehicle are moving away at a high speed and are therefore not as relevant.

Original: PlanT already pays attention to the more distant vehicle behind them as this is the onethat it would collide with if it does not brake.
Revised: PlanT already pays attention to the more distant vehicle behind them as this is the one that it would collide with if it does not brake.

Original: One of the failures of PlanT we observe is thatit sometimes allocates the highest attention to a very close vehicle behind itself (example 4) andmisses the relevant object.
Revised: One of the failures of PlanT we observe is that it sometimes allocates the highest attention to a very close vehicle behind itself (example 4) and misses the relevant object.

Original: PlanCNN has more prominent errors when there are a large number ofvehicles in the scene, but nothing directly ahead, such as problems attending to the relevant objectwhen merging lanes (examples 2 & 3).
Revised: PlanCNN has more prominent errors when there are a large number of vehicles in the scene or when merging lanes (examples 2 & 3).

Original: To better assess the driving performance and relevance scoreswe provide additional results in the supplementary video.
Revised: To better assess the driving performance and relevance scores we provide additional results in the supplementary video.

Original:  4.3 Limitations
Revised:  Conclusion

Original: [Figure]
Revised: In this work, we take a step towards efﬁcient, high-performance, explainable planning for autonomous driving with a novel object-level representation and transformer-based architecture called

Original:  Our study has several limitations.
Revised: PlanT. Our experiments highlight the importance of correctly encoding the ego vehicle’s route for planning. We show that incorporating a 360 ° ﬁeld of view, information about vehicle speeds, and scaling up both the architecture and dataset size of a learned planner are essential to achieve stateof-the-art results.

Original: Firstly, since planning is a challenging task in itself and the capabilities of learning-based planners are not fully understood, we restrict the scope of our experimentsto exclude the task of perception from sensor inputs. Integrating PlanT with a state-of-the-art perception system for object detection and tracking is a promising next step.
Revised: Additionally, PlanT signiﬁcantly outperforms state-of-the-art end-to-end sensorbased models even with a noisy and incomplete input representation obtained via a perception module. Finally, we demonstrate that PlanT can reliably identify the most relevant object in the scene via a new metric and evaluation protocol that measure explainability.

Original: Second, the expert driver usedin our IL-based training strategy does not achieve a perfect scorein our evaluation setting (Table 1)and has certain consistent failure modes (Fig. 3a, more examplescan be found in the supplementarymaterial).
Revised: Limitations. Firstly, the expert driver used in our IL-based training strategy does not achieve a perfect score (Table 1) and has certain consistent failure modes (Fig. 3a, more examples in the supplementary material).

Original: While a human driver may be able to achieve higher driving scores, human data collectionwould be time-consuming (the 3 × dataset used in our experiments contains around 95 hours of driving).
Revised: Human data collection to address this would be time-consuming (the 3 × dataset used in our experiments contains around 95 hours of driving).

Original: Instead, we use the best publicly available algorithmic expert, which is a standard practice for

Original: IL on CARLA [69, 70, 34, 18].

Original: Finally, all our experiments are conducted in simulation.
Revised: Second, all our experiments are conducted in simulation.

Original: Real-worldscenarios are more diverse and challenging.
Revised: Real-world scenarios are more diverse and challenging.

Original: However, CARLA is a high-ﬁdelity simulator, and previous ﬁndings demonstrate that systems developed in simulators like CARLA can be transferred tothe real world [30, 71, 72].
Revised: However, CARLA is a high-ﬁdelity simulator actively used by many researchers for autonomous driving, and previous ﬁndings demonstrate that systems developed in simulators like CARLA can be transferred to the real world [31, 74, 75].

Original: Conclusion
Revised: Finally, our experiment with perception (Section 4.2) uses a single offthe-shelf perception module that was not speciﬁcally optimized for PlanT, leading to sub-optimal performance. It is a well-known limitation of modular systems that downstream modules cannot recover easily from errors made by earlier modules.

Original: In this work, we take a step towards efﬁcient, high-performance, explainable planning for autonomous driving with a novel object-level representation and transformer-based architecture called
Revised: A thorough analysis of perception robustness and uncertainty encoding are important research directions beyond the scope of this work.

Original: PlanT. Based on our experiments, we ﬁnd that it is possible to replace dense, high-dimensionalfeature grids including HD map information with a simple representation of the ego vehicle routewithout a drop in performance.
Revised: Acknowledgments

Original: We show that incorporating a 360 ° ﬁeld of view, information aboutvehicle speeds, and scaling up both the architecture and dataset size of a learned planner are essentialto achieve state-of-the-art results on the CARLA simulator.
Revised: This work was supported by the BMWi (KI Delta Learning, project number: 19A19013O), the BMBF (Tübingen AI Center, FKZ: 01IS18039A), the DFG (SFB 1233, TP 17, project number: 276693517), by the ERC (853489 - DEXIM) and by EXC (number 2064/1 – project number 390727645).

Original: Additionally, we show that PlanT canreliably identify the most relevant object in the scene via a new metric and evaluation protocol thatfocus on explainability.
Revised: We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting K. Renz, K. Chitta and O.-B. Mercea. The authors also thank Niklas Hanselmann and Markus Flicke for proofreading and Bernhard Jaeger for helpful discussions.