Original: Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula
Revised: Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula

Original: Anonymous Author(s) AfﬁliationAddress email Abstract: ML-based motion planning is a promising approach to produce agentsthat exhibit complex behaviors, and automatically adapt to novel environments.
Revised: Abstract: ML-based motion planning is a promising approach to produce agents that exhibit complex behaviors, and automatically adapt to novel environments.

Original: In the context of autonomous driving, it is common to treat all available trainingdata equally.
Revised: In the context of autonomous driving, it is common to treat all available training data equally.

Original: However, this approach produces agents that do not perform robustlyin safety-critical settings, an issue that can’t be addressed by simply adding moredata to the training set – we show that an agent trained using only a 10% subsetof the data performs just as well as an agent trained on the entire dataset.
Revised: However, this approach produces agents that do not perform robustly in safety-critical settings, an issue that can’t be addressed by simply adding more data to the training set – we show that an agent trained using only a 10% subset of the data performs just as well as an agent trained on the entire dataset.

Original: Wepresent a method to predict the inherent difﬁculty of a driving situation givendata collected from a ﬂeet of autonomous vehicles deployed on public roads.
Revised: We present a method to predict the inherent difﬁculty of a driving situation given data collected from a ﬂeet of autonomous vehicles deployed on public roads.

Original: Wethen demonstrate that this difﬁculty score can be used in a zero-shot transfer togenerate curricula for an imitation-learning based planning agent.
Revised: We then demonstrate that this difﬁculty score can be used in a zero-shot transfer to generate curricula for an imitation-learning based planning agent.

Original: Compared totraining on the entire unbiased training dataset, we show that prioritizing difﬁcultdriving scenarios both reduces collisions by 15% and increases route adherenceby 14% in closed-loop evaluation, while using only 10% of the training data.
Revised: Compared to training on the entire unbiased training dataset, we show that prioritizing difﬁcult driving scenarios both reduces collisions by 15% and increases route adherence by 14% in closed-loop evaluation, while using only 10% of the training data.

Original:  Introduction
Revised:  Introduction

Original: Autonomous vehicles (AV) typically rely on optimization-based motion planning and control methods.
Revised: Autonomous vehicles (AV) typically rely on optimization-based motion planning and control methods.

Original: These techniques involve bespoke components speciﬁc to the deployment region and AV hardware, and require copious hand-tuning to adapt to new environments.
Revised: These techniques involve bespoke components speciﬁc to the deployment region and AV hardware, and require copious hand-tuning to adapt to new environments.

Original: An alternative approach is toapply machine learning (ML) to the lifetimes of experience that AV ﬂeets can collect within days orweeks.
Revised: An alternative approach is to apply machine learning (ML) to the lifetimes of experience that AV ﬂeets can collect within days or weeks.

Original: A paradigm shift to ML-based planning could automate the adaptation of behaviors to newareas, improve planning latency, and increase the impact of hardware acceleration.
Revised: A paradigm shift to ML-based planning could automate the adaptation of behaviors to new areas, improve planning latency, and increase the impact of hardware acceleration.

Original: For example, imitation learning (IL) can utilize the large tranches of expert demonstrations collectedby the regular operations of AV ﬂeets to produce policies that perform well in common scenarios,without the need to specify a reward function.
Revised: For example, imitation learning (IL) can utilize the large tranches of expert demonstrations collected by the regular operations of AV ﬂeets to produce policies that perform well in common scenarios, without the need to specify a reward function.

Original: However, both the distribution from which experiences are sampled, and the policy used to generate the demonstrations can critically affect the ILpolicy’s performance [1].
Revised: However, both the distribution from which experiences are sampled, and the policy used to generate the demonstrations can critically affect the IL policy’s performance [1].

Original: The training data distribution is especially important when learning methods are applied to problems characterized by long tail examples ( c.f.
Revised: The training data distribution is especially important when learning methods are applied to problems characterized by long tail examples ( c.f.

Original: In the caseof autonomous driving, the vast majority of observed scenarios are simple enough to be navigatedwithout any negative safety outcomes.
Revised: In the case of autonomous driving, the vast majority of observed scenarios are simple enough to be navigated without any negative safety outcomes.

Original: A visual inspection of a random subset of our data suggeststhat half of it consists of scenarios with the AV as the only road user in motion, while another quartercontains other moving road users, but not necessarily close enough to the AV to affect its behavior.
Revised: A visual inspection of a random subset of our data suggests that half of it consists of scenarios with the AV as the only road user in motion, while another quarter contains other moving road users, but not necessarily close enough to the AV to affect its behavior.

Original: As a result, IL policies may not be robust in safety-critical and long tail situations.
Revised: As a result, IL policies may not be robust in safety-critical and long tail situations.

Original: Reinforcement learning (RL) can be used to explicitly penalize poor behavior, but due to the complex nature of driving [7], it is difﬁcult to design a reward function for AVs to align with humanexpectations.
Revised: Reinforcement learning (RL) can be used to explicitly penalize poor behavior, but due to the complex nature of driving [7], it is difﬁcult to design a reward function for AVs to align with human expectations.

Original: Even if reward signals were provided for safety-critical events or trafﬁc law violations(e.g., collisions, running a red light), they would be extremely sparse since such events are quite rare.
Revised: Even if reward signals were provided for safety-critical events or trafﬁc law violations (e.g., collisions, running a red light), they would be extremely sparse since such events are quite rare.

Original: Furthermore, exploration to collect more long tail data is challenging due to safety concerns [8].
Revised: Furthermore, exploration to collect more long tail data is challenging due to safety concerns [8].

Original: Despite these issues, most learning-based robotics and AV applications use the naive strategy of creating training datasets from all available demonstrations.
Revised: Despite these issues, most learning-based robotics and AV applications use the naive strategy of creating training datasets from all available demonstrations.

Original: In the context of AVs, since most drivingsituations are simple, this strategy is both inefﬁcient and unlikely to generate a policy that is robustto difﬁcult scenarios.
Revised: In the context of AVs, since most driving situations are simple, this strategy is both inefﬁcient and unlikely to generate a policy that is robust to difﬁcult scenarios.

Original: Naturally, a common solution is to upsample challenging examples, either byincreasing their sampling probability by a predetermined factorbefore training [9] or with curricu- [Figure] V
Revised: A common solution is to upsample challenging examples, either by increasing their sampling probability by a predetermined factor [9] or with curriculum learning [10], i.e., dynamically updating the sampling probability during training based on the agent’s performance.

Original: [Figure]
Revised: [Figure]

Original: i c l e C
Revised: i c l e C

Original: [Figure]
Revised: [Figure]

Original: n fi g u r a t
Revised: n fi g u r a t

Original: io
Revised: i

Original: [Figure]
Revised: [Figure]

Original:  Learning
Revised: Fleet Learning

Original: [Figure]
Revised: [Figure]

Original:  Run segments are selected lum learning [10], i.e., dynamically updating the sampling probability during training based on theagent’s performance.
Revised: D Run segments are selected for counterfactual simulations and human triage; the outcome of this process is a labeled set of difﬁculty scores is to to

Original: However, both of these approaches include signiﬁcant hurdles.
Revised: However, both of these approaches include signiﬁcant hurdles.

Original: Upsampling requires that we knowwhich examples are part of the long tail a priori , as in standard classiﬁcation problems where theclass-label imbalance can inform a sampling strategy.
Revised: Upsampling requires that we know which examples are part of the long tail a priori , as in standard classiﬁcation problems where the class-label imbalance can inform a sampling strategy.

Original: In IL and RL, no such labels are available.
Revised: In IL and RL, no such labels are available.

Original: Assuch, curriculum learning is more suitable since it uses the agent’s current performance to identifyhard examples.
Revised: As such, curriculum learning is more suitable since it uses the agent’s current performance to identify hard examples.

Original: However, standard approaches to curriculum learning are speciﬁc to the agent beingtrained; they do not, for example, utilize data collected by deployed AVs running other planners,which can provide more general, policy-agnostic insights into the long tail of driving.
Revised: However, standard approaches to curriculum learning are speciﬁc to the agent being trained; they do not, for example, utilize data collected by deployed AVs running other planners, which can provide more general, policy-agnostic insights into the long tail of driving.

Original: In this paper, we propose an approach (summarized in Figure 1) that addresses the challenges ofupsampling and curriculum learning applied to an AV setting.
Revised: In this paper, we propose an approach (summarized in Figure 1) that addresses the challenges of upsampling and curriculum learning applied to an AV setting.

Original: Developing a road-ready AV generally involves both collecting real-world data with an expert - which can be a combination of humandrivers and thoroughly evaluated AV planners; and evaluating new development planners , which areregularly simulated on the data collected by the expert to identify potential failure modes, generatinga large counterfactual dataset.
Revised: Developing a road-ready AV generally involves both collecting real-world data with an expert , which can be a combination of human drivers and thoroughly evaluated AV planners; and evaluating new development planners , which are regularly simulated on the data collected by the expert to identify potential failure modes, generating a large counterfactual dataset.

Original: Our method uses this readily available data to train a difﬁculty modelthat scores the inherent difﬁculty of a given scenario by predicting the probability of collisions andnear-misses in simulation.
Revised: Our method uses this readily available data to train a difﬁculty model that scores the inherent difﬁculty of a given scenario by predicting the probability of collisions and near-misses in simulation.

Original: This difﬁculty model provides several key beneﬁts: 1) it is computationally less expensive to predict a driving situation’s difﬁculty than to simulate it for a policy beingtrained; 2) the model learns the inherent, policy-agnostic difﬁculty of a scenario because it is trainedon multiple development planners in different geographic regions; and 3) the model predicts a continuous score that can be used to identify scenarios within an arbitrary difﬁculty range, rather thanobtaining a few counterfactual failures.
Revised: This difﬁculty model provides several key beneﬁts: 1) it is computationally less expensive to predict a driving situation’s difﬁculty than to simulate it for a policy being trained; 2) the model learns the inherent, policy-agnostic difﬁculty of a scenario because it is trained on multiple development planners in different geographic regions; and 3) the model predicts a continuous score that can be used to identify scenarios within an arbitrary difﬁculty range, rather than obtaining a few counterfactual failures.

Original: We show that a zero-shot transfer of this model can identify long tail examples that are difﬁcult for anew IL-based planning agent, without any ﬁne tuning.
Revised: We show that a zero-shot transfer of this model can identify long tail examples that are difﬁcult for a new IL-based planning agent, without any ﬁne tuning.

Original: This allows us to upsample difﬁcult trainingexamples without expensive evaluation of the agent during training.
Revised: This allows us to upsample difﬁcult training examples without expensive evaluation of the agent during training.

Original: Though we train the planningagent using IL as a case study, our approach can be applied to any ML-based planning approach.
Revised: Though we train the planning agent using IL as a case study, our approach can be applied to any ML-based planning approach.

Original: The main contributions of this paper are:
Revised: The main contributions of this paper are:

Original: We train a model to predict which driving scenarios are difﬁcult for development planners and show that it can zero-shot transfer to the task of ﬁnding challenging scenarios on which to trainan ML-based planning agent.
Revised: We train a model to predict which driving scenarios are difﬁcult for development planners and show that it can zero-shot transfer to the task of ﬁnding challenging scenarios on which to train an ML-based planning agent.

Original: This generalization suggests that the model can predict the inherentdifﬁculty of a driving situation.2.
Revised: This generalization suggests that the model can predict the inherent difﬁculty of a driving situation.

Original: We show that training an ML-based planning agent on unbiased driving data leads to poor per- formance on difﬁcult examples since easy driving scenarios dominate rarer, harder cases.
Revised: We show that training an ML-based planning agent on unbiased driving data leads to poor performance on difﬁcult examples since easy driving scenarios dominate rarer, harder cases.

Original: We show that using our difﬁculty model to upsample more challenging scenarios reduces collisions by 15% and increases route adherence by 14% on the unbiased test set.
Revised: We show that using our difﬁculty model to upsample more challenging scenarios reduces collisions by 15% and increases route adherence by 14% on the unbiased test set.

Original: This suggests thatthere are signiﬁcant diminishing returns in adding common scenarios to the training dataset.
Revised: This suggests that there are signiﬁcant diminishing returns in adding common scenarios to the training dataset.

Original:  Related Work
Revised:  Related Work

Original: The application of RL to the task of autonomous driving has received signiﬁcant attention in recentyears [11]; proposed methods span the gamut of methodologies and the AV stack itself.
Revised: The application of RL to the task of autonomous driving has received signiﬁcant attention in recent years [11]; proposed methods span the gamut of methodologies and the AV stack itself.

Original: RL hasbeen used to address a variety of problems including end-to-end motion planning, behavior generation, reward design, and even behavior prediction.
Revised: RL has been used to address a variety of problems including end-to-end motion planning, behavior generation, reward design, and even behavior prediction.

Original: In this work, we focus on imitation learningtechniques [12], which avoid direct speciﬁcation of a reward function, and rely instead on expertdemonstrations.
Revised: In this work, we focus on imitation learning techniques [12], which avoid direct speciﬁcation of a reward function, and rely instead on expert demonstrations.

Original: As a result, they can capture subtle human preferences and demonstrate impressiveperformance on a variety of robotics tasks.
Revised: As a result, they can capture subtle human preferences and demonstrate impressive performance on a variety of robotics tasks.

Original: However, despite many attempts [13, 14, 15], IL and RLtechniques still struggle with the long tail present in the driving task [4].
Revised: However, despite many attempts [13, 14, 15], IL and RL techniques still struggle with the long tail present in the driving task [4].

Original: Like this work, Brys et al.
Revised: Like this work, Brys et al.

Original: [16] and Suay et al.
Revised: [16] and Suay et al.

Original: [17] consider how to leverage potentially suboptimal demonstrations to improve the efﬁciency and robustness of learning.
Revised: [17] consider how to leverage potentially suboptimal demonstrations to improve the efﬁciency and robustness of learning.

Original: Unlike these works,we use ofﬂine methods to learn a model of each scenario’s difﬁculty, and bias the distribution that IL is performed on.
Revised: Unlike these works, we use ofﬂine methods to learn a model of each scenario’s difﬁculty, and bias the distribution that IL is performed on.

Original: This approach is similar to baselines [18, 19] inspired by Peters and Schaal[9]; however, unlike these works, our setting does not provide a reward signal for the proposeddemonstrations.
Revised: This approach is similar to baselines [18, 19] inspired by Peters and Schaal [9]; however, unlike these works, our setting does not provide a reward signal for the proposed demonstrations.

Original: Instead, we use counterfactual simulations toinfer the task difﬁculty and utilize theunlabeled demonstrations to learn a foundation model with which we can efﬁciently approximate ascenario’s difﬁculty, which would have otherwise requiredan expensive counterfactual simulation.
Revised: Instead, we use ofﬂine, off-policy simulations to learn a foundation model with which we can efﬁciently approximate a scenario’s difﬁculty, which would have otherwise required expensive counterfactual simulations during training.

Original: This approach side steps the inefﬁciencies of performing rollouts of the learnt policy on the entiredataset since inference using the difﬁculty model is computationally much cheaper than simulation.
Revised: This approach side steps the inefﬁciencies of performing rollouts of the learnt policy on the entire dataset since inference using the difﬁculty model is computationally much cheaper than simulation.

Original: Similar approaches have also been proposed by Brown et al.
Revised: Similar approaches have also been proposed by Brown et al.

Original: [20]; however, they focus largely onsituations with severely suboptimal demonstrations where the reward is speciﬁed.
Revised: [20]; however, they focus largely on situations with severely suboptimal demonstrations where the reward is speciﬁed.

Original: Similar problemshave also been identiﬁed in ofﬂine RL [21].
Revised: Similar problems have also been identiﬁed in ofﬂine RL [21].

Original: Interestingly, Kumar et al.
Revised: Interestingly, Kumar et al.

Original: [22] identify the tight relationship between imitation learning and ofﬂine RL, noting the theoretical advantage of incorporatingreward information in settings like autonomous driving which must avoid rare catastrophic failures.
Revised: [22] identify the tight relationship between imitation learning and ofﬂine RL, noting the theoretical advantage of incorporating reward information in settings like autonomous driving which must avoid rare catastrophic failures.

Original: Our experiments provide empirical support for thistheoretical insight.
Revised: Our experiments provide empirical support for this insight.

Original: Curriculum learning (CL)
Revised: Curriculum learning (CL)

Original: [10] is also closely related to this work.
Revised: [10] is also closely related to this work.

Original: While not originally classiﬁed as such, methods like automatic domain randomization, prioritized experience replay [23], and
Revised: While not originally classiﬁed as such, methods like automatic domain randomization, prioritized experience replay [23], and

Original: AlphaGo’s self-play [24, 25] have led to superhuman game-playing agents and breakthroughs insim2real transfer [23, 26, 27].
Revised: AlphaGo’s self-play [24, 25] have led to superhuman game-playing agents and breakthroughs in sim2real transfer [23, 26, 27].

Original: CL methods solve for surrogate objectives rather than directly optimizing the ﬁnal performance of the learner.
Revised: CL methods solve for surrogate objectives rather than directly optimizing the ﬁnal performance of the learner.

Original: They control which transitions are sampled, the behaviorof other agents in an environment, the generation of initial states, or even the reward function.
Revised: They control which transitions are sampled, the behavior of other agents in an environment, the generation of initial states, or even the reward function.

Original: CLmethods are also characterized by whether they are used on- or off-policy.
Revised: CL methods are also characterized by whether they are used on- or off-policy.

Original: For example, Uesatoet al.
Revised: For example, Uesato et al.

Original: [28] exploit low quality policies to obtain failures in an on-policy RL setting.
Revised: [28] exploit low quality policies to obtain failures in an on-policy RL setting.

Original: Finding hardexamples in the training data using this approach requires repeatedly generating rollouts for eachexpert trajectory in the dataset.
Revised: Finding hard examples in the training data using this approach requires repeatedly generating rollouts for each expert trajectory in the dataset.

Original: Such an approach is computationally infeasible when operating atscale since the training datasets can have hundreds of thousands of real-world driving miles.
Revised: Such an approach is computationally infeasible when operating at scale since the training datasets can have hundreds of thousands of real-world driving miles.

Original: Similar approaches known as hard-negative mining have been used in supervised learning settings [29];however, like Uesato et al.
Revised: Similar approaches known as hard-negative mining have been used in supervised learning settings [29]; however, like Uesato et al.

Original: [28] they evaluate the difﬁculty of examples online.
Revised: [28] they evaluate the difﬁculty of examples online.

Original: Instead, we consider variants of CL that exploit off-policy data.
Revised: Instead, we consider variants of CL that exploit off-policy data.

Original: As in the on-policy case, the keyproblem is to determine which data is interesting.
Revised: As in the on-policy case, the key problem is to determine which data is interesting.

Original: Off-policy compatible methods are also generallysurrogate-based.
Revised: Off-policy compatible methods are also generally surrogate-based.

Original: They select for diversity [30], moderate difﬁculty [27], surprise [23], or learningprogress [31].
Revised: They select for diversity [30], moderate difﬁculty [27], surprise [23], or learning progress [31].

Original: Our approach is most similar to Akkaya et al.
Revised: Our approach is most similar to Akkaya et al.

Original: [27], but instead of performing expensive agent evaluation during training, we use off-policy data both to train a foundation model [32],which encodes experiences, and to classify the difﬁculty of an interaction.
Revised: [27], but instead of performing expensive agent evaluation during training, we use off-policy data both to train a foundation model [32], which encodes experiences, and to classify the difﬁculty of an interaction.

Original: We also utilize large-scalereal-world data and demonstrate that simpler curricula are effective.
Revised: We also utilize large-scale real-world data and demonstrate that simpler curricula are effective.

Original:  Background
Revised:  Background

Original: Model-based Generative Adversarial Imitation Learning: Behavior cloning (BC) [33, 13] is anaive imitation learning method that applies supervised learning to match the expert’s conditionalaction distribution: arg max ✓ E s,a ⇠ ⇡ E [log ⇡ ✓ ( a | s )] .
Revised: Model-based Generative Adversarial Imitation Learning: Behavior cloning (BC) [33, 13] is a naive imitation learning method that applies supervised learning to match the expert’s conditional action distribution: arg max ✓ E s,a ⇠ ⇡ E [log ⇡ ✓ ( a | s )] .

Original: BC policies may suffer from covariate shift,resulting in quadratic worst-case error with respect to the time horizon [34].
Revised: BC policies may suffer from covariate shift, resulting in quadratic worst-case error with respect to the time horizon [34].

Original: To address this issue, generative adversarial imitation learning (GAIL) [35] formulates IL as an adversarial game betweenthe policy ⇡ ✓ and the discriminator D ! .
Revised: To address this issue, generative adversarial imitation learning (GAIL) [35] formulates IL as an adversarial game between the policy ⇡ ✓ and the discriminator D ! .

Original: The discriminator is trained to classify whether a giventrajectory was sampled from ⇡ ✓ (labeled 0 ) or from the expert demonstration (labeled 1 ), and thepolicy is trained to generate trajectories that are indistinguishable from demonstrations:
Revised: The discriminator is trained to classify whether a given trajectory was sampled from ⇡ ✓ (labeled 0 ) or from the expert demonstration (labeled 1 ), and the policy is trained to generate trajectories that are indistinguishable from demonstrations:

Original: [Equation]
Revised: [Equation]

Original: GAIL minimizes the gap in the joint distributions of states and actions p ( s, a ) between the policyand the expert, resulting in linear error with respect to the time horizon [36].
Revised: GAIL minimizes the gap in the joint distributions of states and actions p ( s, a ) between the policy and the expert, resulting in linear error with respect to the time horizon [36].

Original: However, GAIL relieson high variance policy gradient estimates because it uses an unknown dynamics model, making itsobjective function non-differentiable.
Revised: However, GAIL relies on high variance policy gradient estimates because it uses an unknown dynamics model, making its objective function non-differentiable.

Original: In contrast, model-based GAIL (MGAIL) [37] uses differentiable dynamics in combination with the reparameterization trick [38] to reduce the variance of thepolicy gradient estimates.
Revised: In contrast, model-based GAIL (MGAIL) [37] uses differentiable dynamics in combination with the reparameterization trick [38] to reduce the variance of the policy gradient estimates.

Original:  Method
Revised:  Method

Original: A key challenge in commercial AV development is to design an AV planner that can safely andefﬁciently navigate real-world settings while aligning with human expectations.
Revised: A key challenge in commercial AV development is to design an AV planner that can safely and efﬁciently navigate real-world settings while aligning with human expectations.

Original: At any given time,there may exist multiple development planners under evaluation.
Revised: At any given time, there may exist multiple development planners under evaluation.

Original: Iteratively improving an AV planner typically involves the following three steps.
Revised: Iteratively improving an AV planner typically involves the following three steps.

Original: 1) Data Collection: Real-world data is collectedby a ﬂeet of vehicles in the operational area.
Revised: 1) Data Collection: Real-world data is collected by a ﬂeet of vehicles in the operational area.

Original: 2) Data-Driven Simulation: A development planneris tested in simulation by having it control the data-collecting ego vehicle in a run segment , or ashort snippet of recorded driving data.
Revised: 2) Data-Driven Simulation: A development planner is tested in simulation by having it control the data-collecting ego vehicle in a run segment , or a short snippet of recorded driving data.

Original: 3) Evaluation and Improvement: The development planner isevaluated on key metrics based on these simulations, with potential issues identiﬁed and addressed.
Revised: 3) Evaluation and Improvement: The development planner is evaluated on key metrics based on these simulations, with potential issues identiﬁed and addressed.

Original: As mentioned in Section 1, we consider an ML-based approach to developing an AV planner fromthe ground up.
Revised: As mentioned in Section 1, we consider an ML-based approach to developing an AV planner from the ground up.

Original: One option is to use imitation learning to train a planning agent .
Revised: One option is to use imitation learning to train a planning agent .

Original: Given an initialdataset of logged expert driving, a naive approach is to train the agent on the entire dataset.
Revised: Given an initial dataset of logged expert driving, a naive approach is to train the agent on the entire dataset.

Original: However,this means that challenging long tail segments are used only a few times during training, yielding aplanning agent that has difﬁculty negotiating similar situations [2, 5, 6].
Revised: However, this means that challenging long tail segments are used only a few times during training, yielding a planning agent that has difﬁculty negotiating similar situations [2, 5, 6].

Original: Thus, to improve our agent,we require a method to upsample these rare segments.
Revised: Thus, to improve our agent, we require a method to upsample these rare segments.

Original: 4.1 Difﬁculty Model
Revised: 4.1 Difﬁculty Model

Original: The key idea behind our method is to use the  run segments simulated with development planners tolearn a difﬁculty model that predicts the difﬁculty of a logged segment, i.e., whether a developmentplanner is likely to have a poor safety outcome in simulation.
Revised: The key idea behind our method is to use the real-world run segments replayed in simulation with development planners to learn a difﬁculty model that predicts the difﬁculty of a logged segment, i.e., whether a development planner is likely to have a poor safety outcome in simulation.

Original: We train the difﬁculty model onsimulations of multiple development planners in different geographic areas, so it can be seen asmarginalizing over a diverse distribution of development planners.
Revised: We train the difﬁculty model on simulations of multiple development planners in different geographic areas, so it can be seen as marginalizing over a diverse distribution of development planners.

Original: This makes the model morelikely to be able to identify the inherent difﬁculty of a segment.
Revised: This makes the model more likely to be able to identify the inherent difﬁculty of a segment.

Original: In turn, this facilitates the zero-shottransfer from training on data from development planners to inferring difﬁculty for a substantiallydifferent planning agent.
Revised: In turn, this facilitates the zero-shot transfer from training on data from development planners to inferring difﬁculty for a substantially different planning agent.

Original: Intuitively, segments that development planners ﬁnd difﬁcult are likelyto be difﬁcult for the planning agent as well.
Revised: Intuitively, segments that development planners ﬁnd difﬁcult are likely to be difﬁcult for the planning agent as well.

Original: Speciﬁcally, we use the difﬁculty model’s scores toinform our upsampling strategy for training the planning agent.
Revised: Speciﬁcally, we use the difﬁculty model’s scores to inform our upsampling strategy for training the planning agent.

Original: The evaluation process for development planners typically involves large-scale simulations, withpotentially problematic behaviors ﬂagged for engineers to address.
Revised: The evaluation process for development planners typically involves large-scale simulations, with potentially problematic behaviors ﬂagged for engineers to address.

Original: We train the difﬁculty model topredict collisions and near-misses attributable to the development planner, as oppposed to other roadusers.
Revised: We train the difﬁculty model to predict collisions and near-misses attributable to the development planner, as opposed to other road users.

Original: This data is generated in the normal course of the AV planner development cycle, so no newtraining data is needed.
Revised: This data is generated in the normal course of the AV planner development cycle, so no new training data is needed.

Original: Since we want to marginalize out the idiosyncracies of individual development planners, we model asimulation’s safety outcome y 2 { 0 , 1 } ( 1 if a collision or near-miss occurred, 0 otherwise) as a function of the logged run segment alone.
Revised: Since we want to marginalize out the idiosyncrasies of individual development planners, we model a simulation’s safety outcome y 2 { 0 , 1 } ( 1 if a collision or near-miss occurred, 0 otherwise) as a function of the logged run segment alone.

Original: The input to our model is a learned segment embedding from aseparately trained model.
Revised: The input to our model is a learned segment embedding from a separately trained model.

Original: Given a logged run segment, we collect static features (e.g., road/lane layouts, crosswalks, stop signs), dynamic features (e.g., positions and orientations of other road usersover time), and kinematic information about the data-collecting ego vehicle.
Revised: Given a logged run segment, we collect static features (e.g., road/lane layouts, crosswalks, stop signs), dynamic features (e.g., positions and orientations of other road users over time), and kinematic information about the data-collecting ego vehicle.

Original: We use these features togenerate two top-down images of the segment: one of the ego vehicle’s trajectory, and another of thestatic features and other road users’ trajectories.
Revised: We use these features to generate two top-down images of the segment: one of the ego vehicle’s trajectory, and another of the static features and other road users’ trajectories.

Original: We encode each image into a dense d -dimensionalembedding vector (as in [39]) using a CNN and contrastively train a classiﬁer (e.g., [40])  to determine if two images are from the same run segment (see Figure 1c).
Revised: We encode each image into a dense d -dimensional embedding vector (as in [39]) using a CNN and contrastively train a classiﬁer (e.g., [40]) with cross-entropy loss to determine if two images are from the same run segment (see Figure 1c).

Original: Our difﬁculty model is an MLP that learns a function f : R d !
Revised: Our difﬁculty model is an MLP that learns a function f : R d !

Original: [0 , 1] mapping the embedding to the simulated safety outcome y .
Revised: [0 , 1] mapping the embedding to the simulated safety outcome y .

Original: We trained this model  on a dataset of 5.6k positive and 80k negative examples.
Revised: We trained this model using cross-entropy loss on a dataset of 5.6k positive and 80k negative examples.

Original: The number of negative examples was downsampled by multiple orders of magnitude since the prevalence of simulatedcollisions and near-misses is extremely low.
Revised: The number of negative examples was downsampled by multiple orders of magnitude since the prevalence of simulated collisions and nearmisses is extremely low.

Original: The model produces uncalibrated scores by design astrying to calibrate it to the unbiased prevalence rate leads to numerical issues.
Revised: The model produces uncalibrated scores by design as trying to calibrate it to the unbiased prevalence rate leads to numerical issues.

Original: 4.2 Sampling Strategies
Revised: 4.2 Sampling Strategies

Original: Given the long tail nature of the difﬁculty scores (see Figure 2), it is natural to upsample difﬁcultsegments during training.
Revised: Given the long tail nature of the difﬁculty scores (see Figure 2), it is natural to upsample difﬁcult segments during training.

Original: A standard solution for upsampling in classiﬁcation problems is to createseparate datasets for each class, and then generate a batch by sampling a speciﬁed proportion fromeach dataset.
Revised: A standard solution for upsampling in classiﬁcation problems is to create separate datasets for each class, and then generate a batch by sampling a speciﬁed proportion from each dataset.

Original: Since this requires discretized classes, it cannot be applied to our real-valued difﬁcultyscores.
Revised: Since this requires discretized classes, it cannot be applied to our real-valued difﬁculty scores.

Original: Moreover, due to the large training dataset size it is not scalable to upsample individual segments: the entire dataset cannot ﬁt in memory and random access to individual examples from disk isincompatible with distributed ﬁle sharding of data.
Revised: Moreover, due to the large training dataset size it is not scalable to upsample individual segments: the entire dataset cannot ﬁt in memory and random access to individual examples from disk is incompatible with distributed ﬁle sharding of data.

Original: Instead, we partition the dataset into ten equallysized buckets, each corresponding to a decile of the data by difﬁculty scores, with up/downsamplingachieved by assigning different sampling probabilities to each bucket.
Revised: Instead, we partition the dataset into ten equally sized buckets, each corresponding to a decile of the data by difﬁculty scores, with up/downsampling achieved by assigning different sampling probabilities to each bucket.

Original: This enables us to efﬁcientlygenerate batches on the ﬂy (e.g., sampling a weighted batch of k run segments requires minimaloverhead over the k constant-time accesses to the head pointers of each bucket).
Revised: This enables us to efﬁciently generate batches on the ﬂy (e.g., sampling a weighted batch of k run segments requires minimal overhead over the k constant-time accesses to the head pointers of each bucket).

Original: This decile-basedbucketing also ensures that our method is agnostic to the model scores, which are uncalibrated.
Revised: This decile-based bucketing also ensures that our method is agnostic to the model scores, which are uncalibrated.

Original: We consider two training variants: 1) a ﬁxed weighting scheme for each bucket, held constantthroughout training, and 2) a schedule of weights for each bucket that changes as training progresses.
Revised: We consider two training variants: 1) a ﬁxed weighting scheme for each bucket, held constant throughout training, and 2) a schedule of weights for each bucket that changes as training progresses.

Original: Speciﬁcally, we use the following three sampling strategies: “Highest-10%” trains the agent only onthe highest scoring bucket (i.e., on the segments with the highest 10% difﬁculty scores).
Revised: Speciﬁcally, we use the following three sampling strategies. “Highest-10%” trains the agent only on the highest scoring bucket (i.e., on the segments with the highest 10% difﬁculty scores).

Original: “Uniform10%” upsamples difﬁcult segments by setting each bucket’s sampling weight to the range of difﬁculty scores in that bucket (in the limit of inﬁnite buckets, this approaches a uniform distribution overthe difﬁculty scores).
Revised: “Uniform10%” upsamples difﬁcult segments by setting each bucket’s sampling weight to the range of difﬁculty scores in that bucket (in the limit of inﬁnite buckets, this approaches a uniform distribution over the difﬁculty scores).

Original: “Geometric-schedule-10%” implements a geometric progression of weightswith each bucket weighted equally at the beginning of training and weighted proportional to its average difﬁculty score at the end of training (see Appendix 8.1 for further details).
Revised: “Geometric-schedule-10%” implements a geometric progression of weights with each bucket weighted equally at the beginning of training and weighted proportional to its average difﬁculty score at the end of training (see Appendix 8.2 for further details).

Original: In Section 5.4we compare the performance of these training variants against several baselines.
Revised: In Section 5. we compare the performance of these training variants against several baselines.

Original: Our variants aretrained on only a 10% sample of available data.
Revised: Our variants are trained on only a 10% sample of available data.

Original:  Experiments
Revised:  Experiments

Original: To prevent information leakage between the difﬁculty model and the planning agent, the former istrained on a dataset collected more than six months prior to the dataset for the latter.
Revised: To prevent information leakage between the difﬁculty model and the planning agent, the former is trained on a dataset collected more than six months prior to the dataset for the latter.

Original: The trainingdataset for the planning agent consists of over 14k hours of driving logged by a ﬂeet of vehicles.
Revised: The training dataset for the planning agent consists of over 14k hours of driving logged by a ﬂeet of vehicles.

Original: Wesplit the data into 10 second run segments, resulting in over 5 million training segments.
Revised: We split the data into 10 second run segments, resulting in over 5 million training segments.

Original: We alsocreate two test sets, chronologically separate from the training set to prevent train-test leakage.
Revised: We also create two test sets, chronologically separate from the training set to prevent train-test leakage.

Original: Theﬁrst unbiased test set is composed of 20k segments sampled uniformly from logged data.
Revised: The ﬁrst unbiased test set is composed of 20k segments sampled uniformly from logged data.

Original: The secondset consists of 10k segments with difﬁculty scores in the top one percentile of the training data’s scoredistribution.
Revised: The second set consists of 10k segments with difﬁculty scores in the top one percentile of the training data’s score distribution.

Original: The distributions of the difﬁculty model scores for the train set and unbiased test setboth have long tails (see Figure 2) – scores above 0.85 account for only around 0.5% of the dataset.
Revised: The distributions of the difﬁculty model scores for the train set and unbiased test set both have long tails (see Figure 2) – scores above 0.85 account for only around 0.5% of the dataset.

Original: As described in Section 4.2, we split the training dataset into 10 equal sized buckets based on thedifﬁculty score deciles.
Revised: As described in Section 4.2, we split the training dataset into 10 equal sized buckets based on the difﬁculty score deciles.

Original: 200 run segments are further split from each training bucket to obtain validation buckets for model selection.
Revised: 200 run segments are further split from each training bucket to obtain validation buckets for model selection.

Original: To highlight the effect of our training schemes on performanceon segments of varying difﬁculty, we use the same bucketing approach for the unbiased test set asfor the training set.
Revised: To highlight the effect of our training schemes on performance on segments of varying difﬁculty, we use the same bucketing approach for the unbiased test set as for the training set.

Original: We report the full, unbiased test set results by aggregating over all buckets.
Revised: We report the full, unbiased test set results by aggregating over all buckets.

Original: 5.1 Baselines
Revised: 5.1 Baselines

Original: We report three baselines for comparison, which differ in their training data: “Baseline-all” is trainedon the full dataset, “Baseline-10%” is trained on a uniformly randomly sampled 10% of the fulldataset, and “Baseline-lowest-10%” is trained only on the bucket with the lowest difﬁculty scores.
Revised: We report three baselines for comparison, which differ in their training data: “Baseline-all” is trained on the full dataset, “Baseline-10%” is trained on a uniformly randomly sampled 10% of the full dataset, and “Baseline-lowest-10%” is trained only on the bucket with the lowest difﬁculty scores.

Original: 5.2 Training Details
Revised: 5.2 Training Details

Revised: We use the planning agent described in Bronstein et al.

Original: Our planning agent uses a stochastic continuous action policy conditioned on a goal route and trainedusing a combination of MGAIL and BC.
Revised: [41], which employs a stochastic continuous action policy conditioned on a goal route and is trained using a combination of MGAIL and BC.

Original: See Appendix 8.2 for additional details.
Revised: See Appendix 8.7 for additional details.

Original: We train [-
-]
Revised: We train [-
-]10 random seeds of each agent variant and baseline

Original: [Figure]
Revised: [Figure]

Original: random seeds of each agent variant and baseline for 200k steps.
Revised: for 200k steps.

Original: After the initial 100k training steps,we evaluate each agent on the validation set at intervals of 10k steps.
Revised: After the initial 100k training steps, we evaluate each agent on the validation set at intervals of 10k steps.

Original: We select the agent checkpointwith the lowest sum of collision and off-road driving rates, and evaluate it on the held-out test set.
Revised: We select the agent checkpoint with the lowest sum of collision and off-road driving rates, and evaluate it on the held-out test set.

Original: Since the learnt policy is stochastic, we report the average performance of 16 independent rolloutsfor each test run segment.
Revised: Since the learnt policy is stochastic, we report the average performance of 16 independent rollouts for each test run segment.

Original: 5.3 Metrics
Revised: 5.3 Metrics

Original: We assess the planning agent’s performance using the following binary metrics (1 if the event ofinterest occurred in the segment, 0 otherwise):
Revised: We assess the planning agent’s performance using the following binary metrics (1 if the event of interest occurred in the segment, 0 otherwise):

Original: Route Failure : the agent deviates from the goal “road route” at the start of the segment, whichincludes all lanes in the road containing the goal lane-speciﬁc route.2.
Revised: Route Failure : the agent deviates from the goal “road route” at the start of the segment, which includes all lanes in the road containing the goal lane-speciﬁc route.

Original: Collision : the agent’s bounding box intersects with another road user’s bounding box.3.
Revised: Collision : the agent’s bounding box intersects with another road user’s bounding box.

Original: Off-road : the agent’s bounding box exits the drivable road area.4.
Revised: Off-road : the agent’s bounding box exits the drivable road area.

Original: Route Progress ratio : ratio of the distance traveled along the route by the agent and the expert.
Revised: Route Progress ratio : ratio of the distance traveled along the route by the agent and the expert.

Original: We also report the overall failure rate as the union of the ﬁrst three metrics; a segment is considereda failure if any of the binary metrics is nonzero.
Revised: We also report the overall failure rate as the union of the ﬁrst three metrics; a segment is considered a failure if any of the binary metrics is nonzero.

Revised: When comparing the performance of different agents, we prioritize this failure rate due to the safety-critical nature of driving, while also considering the route progress ratio to ensure the agents are making efﬁcient forward progress.

Original: 5.4 Results
Revised: 5.4 Results

Original: We present the performance of our training variants and baselines on the full, unbiased test set in
Revised: We present the performance of our training variants and baselines on the full, unbiased test set in

Original: Table 1.
Revised: Table 1.

Original: Each variant’s action policy is conditioned on the expert’s initial goal route, which is heldconstant throughout the segment.
Revised: Each variant’s action policy is conditioned on the expert’s initial goal route, which is held constant throughout the segment.

Original: We observe no signiﬁcant difference between the performance of Baseline-10% and Baseline-all,demonstrating that simply increasing the training dataset size does not necessarily lead to betterperformance.
Revised: We observe no signiﬁcant difference between the performance of Baseline-10% and Baseline-all, demonstrating that simply increasing the training dataset size does not necessarily lead to better performance.

Original: Also, Baseline-lowest-10% has the worst performance for the collision and off-roadmetrics.
Revised: Also, Baseline-lowest-10% has the worst performance for the collision and off-road metrics.

Original: This suggests that the easiest segments are not representative of the entire test set distribution and do not contain enough useful information to learn from.
Revised: This suggests that the easiest segments are not representative of the entire test set distribution and do not contain enough useful information to learn from.

Original: However, Baseline-lowest-10%does achieve the lowest route failure rate.
Revised: However, Baseline-lowest-10% does achieve the lowest route failure rate.

Original: We believe this is because the least difﬁcult training bucketis primarily composed of segments in which it is simple to follow the route, such as one-lane roadswith no other road users and minimal interaction.
Revised: We believe this is because the least difﬁcult training bucket is primarily composed of segments in which it is simple to follow the route, such as one-lane roads with no other road users and minimal interaction.

Original: This could cause the Baseline-lowest-10% agentto overﬁt to the route features and follow the route well at the expense of safety.
Revised: This could cause the Baseline-lowest-10% agent to overﬁt to the route features and follow the route well at the expense of safety.

Original: All three of our upsampling variants achieve signiﬁcantly lower collision rates, and comparableoff-road and route failure rates to the baselines (with the exception of Baseline-lowest-10%’s routefailure rate).
Revised: All three of our upsampling variants achieve signiﬁcantly lower collision rates, and comparable off-road and route failure rates to the baselines (with the exception of Baseline-lowest-10%’s route failure rate).

Original: This key result demonstrates that segments with high predicted difﬁculty contain themajority of useful information needed for good aggregate performance.
Revised: This key result demonstrates that segments with high predicted difﬁculty contain the majority of useful information needed for good aggregate performance.

Original: Geometric-schedule-10%has the largest improvement over the baselines, with a signiﬁcantly lower collision rate, comparableroute failure and off-road rates, and a minimal decrease in the route progress ratio.
Revised: Geometric-schedule-10% has the largest improvement over the baselines, with a signiﬁcantly lower collision rate, comparable route failure and off-road rates, and a minimal decrease in the route progress ratio.

Original: This highlightsthe advantage of observing the whole spectrum of data at the start of training, and progressivelyincreasing the proportion of difﬁcult segments to emphasize more useful demonstrations.
Revised: This highlights the advantage of observing the whole spectrum of data at the start of training, and progressively increasing the proportion of difﬁcult segments to emphasize more useful demonstrations.

Original: [Table]
Revised: [Table]

Original: To get a more nuanced view of each variant’s performance
Revised: To get a more nuanced view of each variant’s performance

Original: [Table]
Revised: [Table]

Original: we compare the variants to Baseline10% for each of the test buckets.
Revised: we compare the variants to Baseline10% for each of the test buckets.

Original: Figure 3 shows the performance for the lowest (0-10%), low/mid(30-40%), highest (90-100%)
Revised: Figure 3 shows the performance for the lowest (0-10%), low/mid (30-40%), highest (90-100%)

Original: [Table]
Revised: [Table]

Original: and long tail (99-100%) test buckets.
Revised: and long tail (99-100%) test buckets.

Original: See Figures 5, 6 in the appendixfor metrics for all the test buckets.
Revised: See Figures 5, 6 in the appendix for metrics for all the test buckets.

Original: Not only does each agent’s collision rate correlate with the difﬁculty score, but so do the route failureand off-road rates, with the exception of Highest-10%’s off-road rate.
Revised: Not only does each agent’s collision rate correlate with the difﬁculty score, but so do the route failure and off-road rates, with the exception of Highest-10%’s off-road rate.

Original: This shows that segments thatwere challenging for development planners are also likely to be challenging for our planning agent,which enables the zero-shot transfer of the difﬁculty model.
Revised: This shows that segments that were challenging for development planners are also likely to be challenging for our planning agent, which enables the zero-shot transfer of the difﬁculty model.

Original: It also demonstrates that although thedifﬁculty model was only trained to predict collisions and near-misses, its predicted score describesa broader notion of difﬁculty, as measured by other key planning metrics.
Revised: It also demonstrates that although the difﬁculty model was only trained to predict collisions and near-misses, its predicted score describes a broader notion of difﬁculty, as measured by other key planning metrics.

Original: On the highest and long tail buckets, Highest-10% and Uniform-10% achieve much lower collision rates and overall failure rates than Geometric-schedule-10% and the baseline.
Revised: On the highest and long tail buckets, Highest-10% and Uniform-10% achieve much lower collision rates and overall failure rates than Geometric-schedule-10% and the baseline.

Original: This shows thatupsampling difﬁcult segments results in better overall performance on those segments, not just onmetrics that are highly correlated with the difﬁculty label (i.e., collisions and near-misses).
Revised: This shows that upsampling difﬁcult segments results in better overall performance on those segments, not just on metrics that are highly correlated with the difﬁculty label (i.e., collisions and near-misses).

Original: This isencouraging, since it suggests that the training labels for the difﬁculty model do not need to fullydeﬁne expert driving behavior in order for the resulting planning agent to exhibit improved performance across multiple metrics.
Revised: This is encouraging, since it suggests that the training labels for the difﬁculty model do not need to fully deﬁne expert driving behavior in order for the resulting planning agent to exhibit improved performance across multiple metrics.

Original: However, Highest-10% and Uniform-10% perform comparable to,or worse than the baseline on the lowest and low/mid buckets across all metrics, with especiallypoor performance on the route failure and off-road metrics.
Revised: However, Highest-10% and Uniform-10% perform comparable to, or worse than the baseline on the lowest and low/mid buckets across all metrics, with especially poor performance on the route failure and off-road metrics.

Original: Thus, extreme upsampling of difﬁcultsegments sacriﬁces performance at the other end of the spectrum, since the easiest segments becometoo rare in the training data.
Revised: Thus, extreme upsampling of difﬁcult segments sacriﬁces performance at the other end of the spectrum, since the easiest segments become too rare in the training data.

Original: Geometric-schedule-10% addresses this issue by upsampling difﬁcultsegments while maintaining sufﬁciently broad coverage over the difﬁculty distribution.
Revised: Geometric-schedule-10% addresses this issue by upsampling difﬁcult segments while maintaining sufﬁciently broad coverage over the difﬁculty distribution.

Original: While itdoes not achieve equally low collision rates as the Highest-10% and Uniform-10% variants on thehighest and long tail buckets, it outperforms the baseline on collisions and performs well on thelowest and low/mid buckets, yielding the best overall performance.
Revised: While it does not achieve equally low collision rates as the Highest-10% and Uniform-10% variants on the highest and long tail buckets, it outperforms the baseline on collisions and performs well on the lowest and low/mid buckets, yielding the best overall performance.

Original:  Limitations
Revised:  Limitations

Original: The sampling strategies we explored, whether implemented as a static distribution or one that wasupdated throughout training, were predetermined, and did not adapt to the agent’s performance.

Original: Automatic curriculum learning approaches, which use the agent’s performance during training toinform the sampling distribution, may lead to better results.

Original: We conducted several experimentswith adaptive sampling strategies, but these did not signiﬁcantly outperform the baseline.

Original: Whileadditional hyperparameter tuning might yield better results, it is unclear how sensitive these adaptivemethods are to the choice of hyperparameters, and whether the additional cost of hyperparametersearch is warranted.

Original: We plan to further investigate this question in the future.

Original: In addition, while our difﬁculty model successfully identiﬁed challenging segments, it was onlytrained to predict collisions and near-misses, which are just one indication of difﬁculty.
Revised: While our difﬁculty model successfully identiﬁed challenging segments, it was only trained to predict collisions and near-misses, which are just one indication of difﬁculty.

Original: There areother labels that would be helpful for a more comprehensive difﬁculty model, such as trafﬁc lawviolations, route progress, and discomfort caused to both the ego vehicle’s passengers and other roadusers.
Revised: There are other labels that would be helpful for a more comprehensive difﬁculty model, such as trafﬁc law violations, route progress, and discomfort caused to both the ego vehicle’s passengers and other road users.

Original: Moreover, the difﬁculty model could be improved by incorporating the severity of the negativesafety outcome into the training labels.
Revised: Moreover, the difﬁculty model could be improved by incorporating the severity of the negative safety outcome into the training labels.

Original: Furthermore, as noted in Section 1, a large proportion of theavailable data consists of situations with very few other road users in the scene.
Revised: Furthermore, as noted in Section 1, a large proportion of the available data consists of situations with very few other road users in the scene.

Original: The difﬁculty modelcould be replaced with a heuristics-based approach of pruning such scenarios, though the viabilityof doing so is difﬁcult to gauge a priori .[Table]
Revised: The difﬁculty model could be replaced with a heuristics-based approach of pruning such scenarios, though the viability of doing so is difﬁcult to gauge a priori .

Original: [Figure] In terms of evaluation metrics, we focused primarily on safety metrics, since these are of paramountimportance for real-world deployment.
Revised: In terms of evaluation metrics, we focused primarily on safety metrics, since these are of paramount importance for real-world deployment.

Original: However, we have not considered other facets of drivinglike comfort and reliability, which can also signiﬁcant affect the viability of ML-based planners.
Revised: However, we have not considered other facets of driving like comfort and reliability, which can also signiﬁcantly affect the viability of ML-based planners.

Original: Finally, while we have demonstrated that our method of upsampling long tail segments leads to betterperformance, we have done so only for an agent trained using MGAIL.
Revised: Finally, while we have demonstrated that our method of upsampling long tail segments leads to better performance, we have done so only for an agent trained using MGAIL.

Original: Quantifying the performancegains with other learning methods remains a topic for future work.
Revised: Quantifying the performance gains with other learning methods remains a topic for future work.

Revised: [Table]

Revised: [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: We showed that the naive strategy of training on an unbiased  dataset is suboptimal due to the largefraction of data that does not provide additional useful experience.
Revised: We showed that the naive strategy of training on an unbiased driving dataset is suboptimal due to the large fraction of data that does not provide additional useful experience.

Original: By utilizing readily availabledata collected while evaluating development planners in simulation, we trained a model to identifydifﬁcult segments with poor safety outcomes.
Revised: By utilizing readily available data collected while evaluating development planners in simulation, we trained a model to identify difﬁcult segments with poor safety outcomes.

Original: We then applied this model in a zero-shot mannerto develop training curricula that upsample difﬁcult examples.
Revised: We then applied this model in a zeroshot manner to develop training curricula that upsample difﬁcult examples.

Original: Planning agents trained with thesecurricula outperform the naive strategy in aggregate and are more robust in challenging, long tailscenarios.
Revised: Planning agents trained with these curricula outperform the naive strategy in aggregate and are more robust in challenging, long tail scenarios.

Original: However, overly aggressive upsampling produces policies that do not handle simpler situations well.
Revised: However, overly aggressive upsampling produces policies that do not handle simpler situations well.

Original: We conclude that sampling strategies that prioritize difﬁcult segments but also includeeasier ones are likely to achieve the best overall performance.
Revised: We conclude that sampling strategies that prioritize difﬁcult segments but also include easier ones are likely to achieve the best overall performance.

Original: We have also showed  training on the full dataset doesn’t yield any signiﬁcant beneﬁt over trainingon only 10% of the data sampled uniformly at random, demonstrating that simply adding moreunbiased data to the training set does not necessarily improve performance.
Revised: We have also showed that training on the full dataset doesn’t yield any signiﬁcant beneﬁt over training on only 10% of the data sampled uniformly at random, demonstrating that simply adding more unbiased data to the training set does not necessarily improve performance.

Original: This suggests that wecan use our difﬁculty model to reduce the cost of AV system development in two areas: operatinga ﬂeet of vehicles for active data collection, and retaining large scale sensor logs.
Revised: This suggests that we can use our difﬁculty model to reduce the cost of AV system development in two areas: operating a ﬂeet of vehicles for active data collection, and retaining large-scale sensor logs.

Original: First, since thedifﬁculty model can predict which driving scenarios are likely to be challenging for new planningagents, we could identify geographic “hotspots” where these scenarios occur, and use these locationsto inform our data collection process.
Revised: First, since the difﬁculty model can predict which driving scenarios are likely to be challenging for new planning agents, we could identify geographic “hotspots” where these scenarios occur, and use these locations to inform our data collection process.

Original: Second, since biasing the planning agent’s training datasettoward difﬁcult segments leads to better results with only a fraction of the available data, we coulduse the difﬁculty model scores to reduce the amount of saved data without sacriﬁcing performance. 
Revised: Second, since biasing the planning agent’s training dataset toward difﬁcult segments leads to better results with only a fraction of the available data, we could use the difﬁculty model scores to reduce the amount of saved data without sacriﬁcing performance. Acknowledgments

Revised: We thank Ben Sapp, Eugene Ie, Jonathan Bingham and Ryan Polkowski for their helpful comments, and to Ury Zhilinsky for his support with experiments and infrastructure.