Original: H INDSIGHT IS 20/20: L EVERAGING P AST T RAVERSALS TO A ID 3D P ERCEPTION
Revised: H INDSIGHT IS 20/20: L EVERAGING P AST T RAVERSALS TO A ID 3D P ERCEPTION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-driving cars must detect vehicles, pedestrians, and other trafﬁc participants accurately to operate safely.
Revised: Self-driving cars must detect vehicles, pedestrians and other trafﬁc participants accurately to operate safely.

Original: Small, far-away, or highly occluded objects are particularly challenging because there is limited information in the LiDAR point clouds for detecting them.
Revised: Small, far-away or highly occluded objects are particularly challenging because there is limited information in the LiDAR point clouds for detecting them.

Original: To address this challenge, we leverage valuable information from the past: in particular, data collected in past traversals of the same scene.
Revised: To address this challenge, we leverage valuable information from the past: in particular, data collected in past traversals of the same scene.

Original: We posit that these past data, which are typically discarded, provide rich contextual information for disambiguating the above-mentioned challenging cases.
Revised: We posit that these past data, which are typically discarded, provide rich contextual information for disambiguating the above-mentioned challenging cases.

Original: To this end, we propose a novel, end-to-end trainable H INDSIGHT framework to extract this contextual information from past traversals and store it in an easy-to-query data structure, which can then be leveraged to aid future 3D object detection of the same scene.
Revised: To this end, we propose a novel end-to-end trainable H INDSIGHT framework to extract this contextual information from past traversals and store it in an easy-to-query data structure which can then be leveraged to aid future 3D object detection of the same scene.

Original: We show that this framework is compatible with most modern 3D detection architectures and can substantially improve their average precision on multiple autonomous driving datasets, most notably by more than 300% on the challenging cases.
Revised: We show that this framework is compatible with most modern 3D detection architectures and can substantially improve their average precision on multiple autonomous driving datasets, most notably by more than 300% on the challenging cases.

Original: We will open-source the code upon acceptance.
Revised: Our code is available at https://github.com/YurongYou/Hindsight

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: To drive safely, a (semi-)autonomous vehicle needs to accurately detect and localize other participants, such as cars, buses, cyclists, or pedestrians who might walk onto the road at any time.
Revised: To drive safely, a (semi-)autonomous vehicle needs to accurately detect and localize other participants, such as cars, buses, cyclists, or pedestrians who might walk onto the road at any time.

Original: Such an object detection task is an extremely challenging 3D perception problem, especially when it comes to small, highly occluded, or far-away objects.
Revised: Such an object detection task is an extremely challenging 3D perception problem, especially when it comes to small, highly occluded, or far-away objects.

Original: In spite of considerable progress on this task in the past years (Geiger et al., 2012; Grigorescu et al., 2020; Janai et al., 2020), we have arguably not yet reached the accuracy levels needed for safe operations in general (non-geo-fenced) settings.
Revised: In spite of considerable progress on this task in the past years (Geiger et al., 2012; Grigorescu et al., 2020; Janai et al., 2020), we have arguably not yet reached the accuracy levels needed for safe operations in general (non-geo-fenced) settings.

Original: One reason why 3D object detectors struggle is that often there simply is not enough information in the scene to make a good call.
Revised: One reason why 3D object detectors struggle is that often there simply is not enough information in the scene to make a good call.

Original: For example, the best LiDAR sensors may only yield a few tens of points on a small child if s/he is some distance away.
Revised: For example, the best LiDAR sensors may only yield a few tens of points on a small child if s/he is some distance away.

Original: From this very limited data, the 3D object detector must decide if this is a pedestrian who may rush onto the road at any moment, or if this is just a tree that can be ignored.
Revised: From this very limited data, the 3D object detector must decide if this is a pedestrian who may rush onto the road at any moment, or if this is just a tree that can be ignored.

Original: Sophisticated machine learning models may bring all sorts of inductive biases to bear (Lang et al., 2019;
Revised: Sophisticated machine learning models may bring all sorts of inductive biases to bear (Lang et al., 2019;

Original: Zhou & Tuzel, 2018; Yan et al., 2018; Shi et al., 2019; 2020a), but such few LiDAR points may fundamentally lack enough information to make an accurate decision (Kendall & Gal, 2017).
Revised: Zhou & Tuzel, 2018; Yan et al., 2018; Shi et al., 2019; 2020a), but such few LiDAR points may fundamentally lack enough information to make an accurate decision (Kendall & Gal, 2017).

Original: What we need is additional information about the scene.
Revised: What we need is additional information about the scene.

Original: Current 3D object detectors treat every scene as completely novel and unknown — ignoring potentially valuable information from previous traversals of the same route.
Revised: Current 3D object detectors treat every scene as completely novel and unknown — ignoring potentially valuable information from previous traversals of the same route.

Original: In fact, many of us drive through the same routes every day: to and from work, schools, shops, and friends.
Revised: In fact, many of us drive through the same routes every day: to and from work, schools, shops, and friends.

Original: Even when we embark on a completely new route, we are often following in the footsteps of other drivers who have gone through this very section of a route before.
Revised: Even when we embark on a completely new route, we are often following in the footsteps of other drivers who have gone through this very section of a route before.

Original: In this paper we explore the following question:
Revised: In this paper we explore the following question:

Original: If we collect and aggregate unlabeled LiDAR information over time, potentially across vehicles, can we utilize the past traversals of the same route to better detect cars, bicycles and pedestrians?
Revised: If we collect and aggregate unlabeled LiDAR information over time, potentially across vehicles, can we utilize the past traversals of the same route to better detect cars, bicycles and pedestrians?

Original: We provide an afﬁrmative answer to the above question.
Revised: We provide an afﬁrmative answer to the above question.

Original: Our intuition is as follows. While historical traversals through a route will be different, i.e., we will encounter different cars on the road and different pedestrians wanting to cross, taken together, these traversals yield a wealth of information.
Revised: Our intuition is as follows: while historical traversals through a route will be different, i.e., we will encounter different cars on the road and different pedestrians on the crosswalks, taken together, these traversals yield a wealth of information.

Original: For instance, they reveal where pedestrians, cars, and cyclists generally tend to be in the scene, and where a stop sign or some unknown background object is persistently present across traversals.
Revised: For instance, they reveal where pedestrians, cars, and cyclists generally tend to be in the scene, and where a stop sign or some unknown background object is persistently present across traversals.

Original: With this contextual information at hand, a detector can better recognize, say, a pedestrian heavily occluded by cars on the roadside, since pedestrians have come and gone through these regions before, and it is unlikely that they are persistent background objects.
Revised: With this contextual information at hand, a detector can better recognize, say, a pedestrian heavily occluded by cars on the roadside, since pedestrians have come and gone through these regions before, and it is unlikely that they are persistent background objects.

Original: Thus, leveraging such contextual information could substantially improve the detection accuracy in these safety-critical scenarios.
Revised: Thus, leveraging such contextual information could substantially improve the detection accuracy in these safety-critical scenarios.

Original: Concretely, we formalize this insight by proposing a simple and efﬁcient approach that uses past traversals of a scene to vastly improve perception performance.
Revised: Concretely, we formalize this insight by proposing a simple and efﬁcient approach that uses past traversals of a scene to vastly improve perception performance.

Original: Off-line, we use neural networks to digest past traversals of the scene into a sparse, compact, geo-referenced representation that we call SQuaSH (Spatial-Quantized Sparse History features).
Revised: Off-line, we use neural networks to digest past traversals of the scene into a sparse, compact, geo-referenced representation that we call SQuaSH (Spatial-Quantized Sparse History features).

Original: While in operation, a self-driving car can then query the local SQuaSH context of every LiDAR point in the current scene, thus enriching the information available to perform recognition.
Revised: While in operation, a self-driving car can then query the local SQuaSH context of every LiDAR point in the current scene, thus enriching the information available to perform recognition.

Original: This information can be added as features to any LiDAR-based 3D object detector, and both the detector and the SQuaSH representation can be trained jointly without any additional supervision .
Revised: This information can be added as features to any LiDAR-based 3D object detector, and both the detector and the SQuaSH representation can be trained jointly without any additional supervision .

Original: The resulting detector is substantially more accurate, and in fact can become more accurate over time without any retraining as more traversals of the scene are collected.
Revised: The resulting detector is substantially more accurate, and in fact can become more accurate over time without any retraining as more traversals of the scene are collected.

Original: This local development and improvement with unlabeled data is an important distinction compared to other approaches, such as using high deﬁnition maps which are static and require extensive data collections and labeling.
Revised: This local development and improvement with unlabeled data is an important distinction compared to other approaches, such as using high deﬁnition maps which are static and require extensive data collections and labeling.

Original: We validate our approach on two large-scale, real-world self-driving datasets, Lyft Level 5 Perception (Kesten et al., 2019) and the nuScenes Dataset (Caesar et al., 2020), with multiple representative modern object detection models (Lang et al., 2019; Yan et al., 2018; Zhou & Tuzel, 2018; Shi et al., 2019; 2020a) under various settings and show consistent performance gains.
Revised: We validate our approach on two large-scale, real-world self-driving datasets, Lyft Level 5 Perception (Kesten et al., 2019) and the nuScenes Dataset (Caesar et al., 2020), with multiple representative modern object detection models (Lang et al., 2019; Yan et al., 2018; Zhou & Tuzel, 2018; Shi et al., 2019; 2020a) under various settings and show consistent performance gains.

Original: Concretely, our contributions are three-fold:
Revised: Concretely, our contributions are three-fold:

Original: We identify a rich trove of information in the form of hindsight from past traversals that can substantially boost 3D perception of challenging objects at no extra cost.
Revised: We identify a rich trove of information in the form of hindsight from past traversals that can substantially boost 3D perception of challenging objects at no extra cost.

Original: We propose a simple and efﬁcient method to leverage such information without any extra labels, that can be incorporated into most modern 3D perception pipelines
Revised: We propose a simple and efﬁcient method to leverage such information without any extra labels, that can be incorporated into most modern 3D perception pipelines.

Original: We evaluate our method on 3D object detection exhaustively across two large real-world datasets, different object types, and multiple detection architectures and demonstrate remarkably consistent and signiﬁcant improvements, especially on the challenging cases — small, far-away objects — by over 300%.
Revised: We evaluate our method on 3D object detection exhaustively across two large real-world datasets, different object types, and multiple detection architectures and demonstrate remarkably consistent and signiﬁcant improvements, especially on the challenging cases — small, far-away objects — by over 300%.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: 3D object detection is one of the most important perception tasks for autonomous driving.
Revised: 3D object detection is one of the most important perception tasks for autonomous driving.

Original: Most existing algorithms take LiDAR sensory data as input, which provides accurate 3D point clouds of the surrounding environment.
Revised: Most existing algorithms take LiDAR sensory data as input, which provides accurate 3D point clouds of the surrounding environment.

Original: There are two popular branches of methods.
Revised: There are two popular branches of methods.

Original: The ﬁrst is to voxelize
Revised: The ﬁrst is to voxelize

Original: LiDAR point clouds and use the resulting 3D-cube representation as input to 2D or 3D convolutional neural networks to infer 3D bounding boxes (Zhou & Tuzel, 2018; Yan et al., 2018; Zhou et al., 2020;
Revised: LiDAR point clouds and use the resulting 3D-cube representation as input to 2D or 3D convolutional neural networks to infer 3D bounding boxes (Zhou & Tuzel, 2018; Yan et al., 2018; Zhou et al., 2020;

Original: Li, 2017; Engelcke et al., 2017; Lang et al., 2019; Ku et al., 2018; Yang et al., 2018b; Liang et al., 2018; Chen et al., 2017; Shi et al., 2020a).
Revised: Li, 2017; Engelcke et al., 2017; Lang et al., 2019; Ku et al., 2018; Yang et al., 2018b; Liang et al., 2018; Chen et al., 2017; Shi et al., 2020a).

Original: The second is to design neural network architectures explicitly for point cloud inputs (Qi et al., 2018; 2017a;b; Shi et al., 2019; 2020b; Yang et al., 2020).
Revised: The second is to design neural network architectures explicitly for point cloud inputs (Qi et al., 2018; 2017a;b; Shi et al., 2019; 2020b; Yang et al., 2020).

Original: Our approach augments the input point cloud with features queried from the history and is agnostic to speciﬁc object detection pipelines as long as they can take point clouds with per-point features.
Revised: Our approach augments the input point cloud with features queried from the history and is agnostic to speciﬁc object detection pipelines as long as they can take point clouds with per-point features.

Original: In this work, we experiment with four representative, high-performing 3D object detection models (subsection 4.1) and demonstrate signiﬁcant and consistent improvements when using H INDSIGHT .
Revised: In this work, we experiment with four representative, high-performing 3D object detection models (subsection 4.1) and demonstrate signiﬁcant and consistent improvements when using H INDSIGHT .

Original: 3D object detection in contexts.
Revised: 3D object detection in contexts.

Original: We propose to augment the raw point cloud captured at driving time by features queried from the previous traversals.
Revised: We propose to augment the raw point cloud captured at driving time by features queried from the previous traversals.

Original: This relates to a recent body of literature that augment point clouds with extra information, e.g., high-deﬁnition (HD) maps (Yang et al., 2018a; Ravi Kiran et al., 2018; Seif & Hu, 2016; Liang et al., 2020) or semantic information from synchronized images (Chen et al., 2017;
Revised: This relates to a recent body of literature that augment point clouds with extra information, e.g., high-deﬁnition (HD) maps (Yang et al., 2018a; Ravi Kiran et al., 2018; Seif & Hu, 2016; Liang et al., 2020a) or semantic information from synchronized images (Chen et al., 2017;

Original: Qi et al., 2018; Ku et al., 2018; Liang et al., 2018; Wang et al., 2018; You et al., 2020; Vora et al., 2020).
Revised: Qi et al., 2018; Ku et al., 2018; Liang et al., 2018; Wang et al., 2018; You et al., 2020; Vora et al., 2020).

Original: Our approach is orthogonal to – and compatible with – these methods, but also differs from them in key ways.
Revised: Our approach is orthogonal to – and compatible with – these methods, but also differs from them in key ways.

Original: In contrast to HD Maps, our history features are learned without additional labels, can be updated easily with newly collected traversals,and are not limited to certain pre-deﬁned information types. In contrast to images, querying our pre-computed history features adds inﬁnitesimal latency to the perception pipeline (Table 3); the
Revised: In contrast to HD Maps, our history features are learned without additional labels, can be updated easily with newly collected traversals,

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 3ElmJ2AvJpQkkWytw65U37MgvujureSf345rh6cT32rIx20R6qIQ+dogt0hRqoiSh6Ri/oDb07r86n8+38jEpLzrhnB01EaekXyH63FA==</latexit>
Revised: +dogt0hRqoiSh6Ri/oDb07r86n8+38jEpLzrhnB01EaekXyH63FA==</latexit>

Original: ( PPP c , f history ( PPP c ))  history features can disambiguate hard cases even if they are faraway from the ego car, whereas the corresponding image patches can be very small and hard to recognize.
Revised: ( PPP c , f history ( PPP c )) and are not limited to certain pre-deﬁned information types. In contrast to images, querying our pre-computed history features adds inﬁnitesimal latency to the perception pipeline (Table 3); the history features can disambiguate hard cases even if they are faraway from the ego car, whereas the corresponding image patches can be very small and hard to recognize.

Original: Perception with historical context.
Revised: Perception with historical context.

Revised: There are works utilizing temporal scans as short-term context for perception (Liang et al., 2020b; Huang et al., 2020; Yang et al., 2021).

Original: There is very limited work on using  historical context for perception.
Revised: However, there is very limited work on using long-term historical context for perception.

Original: A few works use videos of people interacting with a scene to discover the affordances in the scene, e.g., where people sit, stand, or walk (Delaitre et al., 2012; Fouhey et al., 2012; Sun et al., 2020).
Revised: A few works use videos of people interacting with a scene to discover the affordances in the scene, e.g., where people sit, stand, or walk (Delaitre et al., 2012; Fouhey et al., 2012; Sun et al., 2020).

Original: Closer in spirit to our work is the work by Barnes et al.
Revised: Closer in spirit to our work is the work by Barnes et al.

Original: (2018), which uses multiple traversals of a scene to map out regions that are ephemeral , namely, that represent transient objects.
Revised: (2018), which uses multiple traversals of a scene to map out regions that are ephemeral , namely, that represent transient objects.

Original: In these works, past observations are used to make speciﬁc, hand-crafted measurements.
Revised: In these works, past observations are used to make speciﬁc, hand-crafted measurements.

Original: In contrast, our key idea is to automatically learn what is relevant from the history by training the SQuaSH representation end-to-end with the detector.
Revised: In contrast, our key idea is to automatically learn what is relevant from the history by training the SQuaSH representation end-to

Original:  H INDSIGHT
Revised:  H INDSIGHT

Original: Problem setup.
Revised: Problem setup.

Original: We assume we are operating in a typical self-driving system: the autonomous vehicle is equipped with various synchronized sensors including LiDAR, which senses the surrounding scene by capturing a 3D point cloud, and GPS/INS, which provides accurate 6-Degree-of-Freedom (6-DoF, 3D displacement + 3D rotation) localization.
Revised: We assume we are operating in a typical self-driving system: the autonomous vehicle is equipped with various synchronized sensors including LiDAR, which senses the surrounding scene by capturing a 3D point cloud, and GPS/INS, which provides accurate 6-Degree-of-Freedom (6-DoF, 3D displacement + 3D rotation) localization.

Original: The self-driving system reads the sensor inputs and detects other participants (cars, pedestrians and cyclists) in the scene.
Revised: The self-driving system reads the sensor inputs and detects other participants (cars, pedestrians and cyclists) in the scene.

Original: Most top-performing detection algorithms heavily rely on LiDAR scans for accurate detection.
Revised: Most top-performing detection algorithms heavily rely on LiDAR scans for accurate detection.

Original: We denote the LiDAR point cloud captured at the current location during driving as P c = { ( x i , y i , z i ) } ki =1 ∈ R k × 3 , and denote the corresponding 6-DoF localization as G c .
Revised: We denote the LiDAR point cloud captured at the current location during driving as P c = { ( x i , y i , z i ) } ki =1 ∈ R k × 3 , and denote the corresponding 6-DoF localization as G c .

Original: We assume that transformed by G c , P c is in a ﬁxed global 3D coordinate system.
Revised: We assume that transformed by G c , P c is in a ﬁxed global 3D coordinate system.

Original: Atypical of this conventional setting, we assume we have access to unlabeled point cloud scans from past traversals through the same location.
Revised: Atypical of this conventional setting, we assume we have access to unlabeled point cloud scans from past traversals through the same location.

Original: Overview of our approach H INDSIGHT .
Revised: Overview of our approach H INDSIGHT .

Original: In this work, we propose a novel approach, which we term H INDSIGHT , to endow the current scan P c with information from past traversals to improve 3D detection (Figure 1).
Revised: In this work, we propose a novel approach, which we term H INDSIGHT , to endow the current scan P c with information from past traversals to improve 3D detection (Figure 1).

Original: Given the current spatial location G c , H INDSIGHT retrieves a set of point clouds recorded by past traversals of the same location, encodes them by a spatial featurizer, and aggregates them into a spatially-indexed feature grid (which we call SQuaSH, Figure 1 (a)).
Revised: Given the current spatial location G c , H INDSIGHT retrieves a set of point clouds recorded by past traversals of the same location, encodes them by a spatial featurizer, and aggregates them into a spatially-indexed feature grid (which we call SQuaSH, Figure 1 (a)).

Original: Then, for each point in the current scan P c , H INDSIGHT queries the local neighborhood of that point in the feature grid, and linearly combines the features in this neighborhood to produce a “history” feature vector (Figure 1 (b)).
Revised: Then, for each point in the current scan P c , H INDSIGHT queries the local neighborhood of that point in the feature grid, and linearly combines the features in this neighborhood to produce a “history” feature vector (Figure 1 (b)).

Original: This feature vector is concatenated as additional channels into the representation of each 3D point, producing an endowed point cloud.
Revised: This feature vector is concatenated as additional channels into the representation of each 3D point, producing an endowed point cloud.

Original: The endowed point cloud, enriched with the past information, is then fed into a 3D object detector (Figure 1 (c)).
Revised: The endowed point cloud, enriched with the past information, is then fed into a 3D object detector (Figure 1 (c)).

Original: The whole pipeline is trained end-to-end with detection labels annotated only on current scans.
Revised: The whole pipeline is trained end-to-end with detection labels annotated only on current scans.

Original: While this process seems time-consuming at ﬁrst glance, many of the computations, especially those related to the past scans, can indeed be carried out off-line.
Revised: While this process seems time-consuming at ﬁrst glance, many of the computations, especially those related to the past scans, can indeed be carried out off-line.

Original: Speciﬁcally, we deliberately separate the computation such that the feature grid SQuaSH from the past scans is pre-computed and stored in a geo-indexed manner (Figure 1 (d)).
Revised: Speciﬁcally, we deliberately separate the computation such that the feature grid SQuaSH from the past scans is pre-computed and stored in a geo-indexed manner (Figure 1 (d)).

Original: While in operation, the self-driving system only needs to query the pre-computed SQuaSH to endow the current point cloud with little latency.
Revised: While in operation, the self-driving system only needs to query the pre-computed SQuaSH to endow the current point cloud with little latency.

Original: Below, we describe each of these components in detail.
Revised: Below, we describe each of these components in detail.

Original: 3. D ENSE 3D D ATA FROM P AST T RAVERSALS
Revised: 3. D ENSE 3D D ATA FROM P AST T RAVERSALS

Original: We ﬁrst describe the data we get from past traversals.
Revised: We ﬁrst describe the data we get from past traversals.

Original: We assume access to T ≥ 1 past traversals; to ensure the information is up-to-date, we maintain the most recent T max traversals.
Revised: We assume access to T ≥ 1 past traversals; to ensure the information is up-to-date, we maintain the most recent T max traversals.

Original: A single traversal t consists of a sequence of point clouds { P tf } and the associated global localization { G tf } recorded as the car was driven; here f is the index of the frame.
Revised: A single traversal t consists of a sequence of point clouds { P tf } and the associated global localization { G tf } recorded as the car was driven; here f is the index of the frame.

Original: We transform each point cloud into a ﬁxed global coordinate system.
Revised: We transform each point cloud into a ﬁxed global coordinate system.

Original: Then, at a location l every s metres along the road, we combine the point clouds from a range [ − H s , H e ] to produce a dense point cloud S tl =
Revised: Then, at a location l every s metres along the road, we combine the point clouds from a range [ − H s , H e ] to produce a dense point cloud S tl =

Original: (cid:83) G t f ∈
Revised: (cid:83) G tf ∈

Original: [ l − H s ,l + H e ] { P tf } (where with a slight abuse of notation, we have used G tf for the 3D location where f was captured).
Revised: [ l − H s ,l + H e ] { P tf } (where with a slight abuse of notation, we have used G tf for the 3D location where f was captured).

Original: Combining multiple scans in this way can densify the scene, providing us with detailed and highresolution point clouds for regions faraway from the current position.
Revised: Combining multiple scans in this way can densify the scene, providing us with detailed and highresolution point clouds for regions faraway from the current position.

Original: A static bush in a distance can yield very sparse LiDAR points similar to that of a car, causing a false positive detection.
Revised: A static bush in a distance can yield very sparse LiDAR points similar to that of a car, causing a false positive detection.

Original: However, it can be disambiguated with multiple past scans that have dense points, recorded when the car was much closer to the bush in past traversals.
Revised: However, it can be disambiguated with multiple past scans that have dense points, recorded when the car was much closer to the bush in past traversals.

Original: These dense scans thus provide context for our method to “see” further into the distance or around occlusions.
Revised: These dense scans thus provide context for our method to “see” further into the distance or around occlusions.

Original: 3. S PATIAL -Q UANTIZED S PARSE H ISTORY FEATURES (SQ UA SH)
Revised: 3. S PATIAL -Q UANTIZED S PARSE H ISTORY FEATURES (SQ UA SH)

Original: Next we take each dense point cloud S tl from each traversal t and encode it using a spatial featurizer that yields a spatially-quantized feature tensor Q tl .
Revised: Next we take each dense point cloud S tl from each traversal t and encode it using a spatial featurizer that yields a spatially-quantized feature tensor Q tl .

Original: In dense form, Q tl can be viewed as a 4D tensor in R H
Revised: In dense form, Q tl can be viewed as a 4D tensor in R H

Original: ×
Revised: ×

Original: W × D × d history , where H , W , and D are the nominal 3D spatial dimensions and d history is the feature dimension.
Revised: W × D × d history , where H , W , and D are the nominal 3D spatial dimensions and d history is the feature dimension.

Original: The quantization step size is δ , so the point located at ( x, y, z ) will be encoded at ( (cid:98) x/δ (cid:99) , (cid:98) y/δ (cid:99) , (cid:98)
Revised: The quantization step size is δ , so the point located at ( x, y, z ) will be encoded at ( (cid:98) x/δ (cid:99) , (cid:98) y/δ (cid:99) , (cid:98)

Original: z/δ (cid:99) ) .
Revised: z/δ (cid:99) ) .

Original: Any number of architectures can be used to map point clouds to such voxelized tensors.
Revised: Any number of architectures can be used to map point clouds to such voxelized tensors.

Original: In this work, we adopt the Sparse Residual U-Net (SR-UNet)
Revised: In this work, we adopt the Sparse Residual U-Net (SR-UNet)

Original: (Choy et al., 2019) as the spatial featurizer due to its high efﬁciency and performance.
Revised: (Choy et al., 2019) as the spatial featurizer due to its high efﬁciency and performance.

Original: Additionally, because most parts of the scene are unoccupied, we represent Q tl as a sparse tensor.
Revised: Additionally, because most parts of the scene are unoccupied, we represent Q tl as a sparse tensor.

Original: For every location l we compute T such tensors, one for each traversal.
Revised: For every location l we compute T such tensors, one for each traversal.

Original: We aggregate them into a single tensor Q gl by applying a per-voxel aggregation function f agg :
Revised: We aggregate them into a single tensor Q gl by applying a per-voxel aggregation function f agg :

Original: [Equation]
Revised: [Equation]

Original: There are a number of available choices for the aggregation function, e.g., max-pooling, meanpooling, etc .
Revised: There are a number of available choices for the aggregation function, e.g., max-pooling, meanpooling, etc .

Original: We do not observe a signiﬁcant performance difference among them (possibly due to the small T max ) and thus use max-pooling due to its simplicity.
Revised: We do not observe a signiﬁcant performance difference among them (possibly due to the small T max ) and thus use max-pooling due to its simplicity.

Original: We term the aggregated tensor Q gl for location l as the Spatial-Quantized Sparse History tensor (SQuaSH tensor) for this location.
Revised: We term the aggregated tensor Q gl for location l as the Spatial-Quantized Sparse History tensor (SQuaSH tensor) for this location.

Original: 3. Q UERY FROM SQ UA SH
Revised: 3. Q UERY FROM SQ UA SH

Original: Suppose now that the self-driving car captures a new scan P c at location G c , and suppose that the SQuaSH tensor at this location is Q gl c .
Revised: Suppose now that the self-driving car captures a new scan P c at location G c , and suppose that the SQuaSH tensor at this location is Q gl c .

Original: We endow each of the points in P c by querying their surrounding features in Q gl c .
Revised: We endow each of the points in P c by querying their surrounding features in Q gl c .

Original: To do so, we convolve Q gl c with a K × K × K ﬁlter θ , and look up each point p
Revised: To do so, we convolve Q gl c with a K × K × K ﬁlter θ , and look up each point p

Original: = ( x, y, z ) from the current scan in the resulting tensor after appropriate quantization:
Revised: = ( x, y, z ) from the current scan in the resulting tensor after appropriate quantization:

Original: [Equation]
Revised: [Equation]

Original: where f history ( p ) is the corresponding history feature for point p .
Revised: where f history ( p ) is the corresponding history feature for point p .

Original: In practice, this entire operation can be easily implemented by a 3D sparse convolution layer in Minkowski Engine (Choy et al., 2019) and runs extremely fast on a GPU (Table 3).
Revised: In practice, this entire operation can be easily implemented by a 3D sparse convolution layer in Minkowski Engine (Choy et al., 2019) and runs extremely fast on a GPU (Table 3).

Original: 3. 3D D ETECTION WITH SQ UA SH
Revised: 3. 3D D ETECTION WITH SQ UA SH

Original: Most modern LiDAR-based detection algorithms are able to take this point cloud P c with per-point features, e.g., LiDAR intensity.
Revised: Most modern LiDAR-based detection algorithms are able to take this point cloud P c with per-point features, e.g., LiDAR intensity.

Original: To make use of these off-the-shelf, high-performing detection algorithms without heavy modiﬁcations to their frameworks, we adopt the simplest way to incorporate the queried history features: attaching f history ( P c ) as per-point features.
Revised: To make use of these off-the-shelf, high-performing detection algorithms without heavy modiﬁcations to their frameworks, we adopt the simplest way to incorporate the queried history features: attaching f history ( P c ) as per-point features.

Original: Even with this simple strategy, we observe a signiﬁcant and consistent performance gain.
Revised: Even with this simple strategy, we observe a signiﬁcant and consistent performance gain.

Original: Our entire pipeline is simply trained in an end-to-end manner, including the spatial featurizer, the query ﬁlter θ and the detection model, via minimizing the corresponding detection loss from the detection algorithms on a dataset.
Revised: Our entire pipeline is simply trained in an end-to-end manner, including the spatial featurizer, the query ﬁlter θ and the detection model, via minimizing the corresponding detection loss from the detection algorithms on a dataset.

Original: 3. D EPLOYMENT
Revised: 3. D EPLOYMENT

Original: For deployment purposes, the self-driving system generates the SQuaSH tensors Q l for all locations l in an off-line manner once the spatial featurizer is trained.
Revised: For deployment purposes, the self-driving system generates the SQuaSH tensors Q l for all locations l in an off-line manner once the spatial featurizer is trained.

Original: The SQuaSH tensors can then be retrieved at run-time using a simple geo-location query.
Revised: The SQuaSH tensors can then be retrieved at run-time using a simple geo-location query.

Original: We intentionally disentangle the computation between the past traversals and the current scan so that such pre-computation is possible.
Revised: We intentionally disentangle the computation between the past traversals and the current scan so that such pre-computation is possible.

Original: This allows us to use large and complex spatial featurizers to process the past traversals without introducing latency into the on-board autonomous driving system .
Revised: This allows us to use large and complex spatial featurizers to process the past traversals without introducing latency into the on-board autonomous driving system .

Original: Moreover, these features can be stored efﬁciently, since the SQuaSH tensors are sparse and can be produced only once every s =meters without losing performance (Appendix B).
Revised: Moreover, these features can be stored efﬁciently, since the SQuaSH tensors are sparse and can be produced only once every s =meters without losing performance (Appendix B).

Original: As mentioned before, obtaining history features from the retreived SQuaSH is extremely fast, adding little burden to on-board systems (Table 3).
Revised: As mentioned before, obtaining history features from the retreived SQuaSH is extremely fast, adding little burden to on-board systems (Table 3).

Original: 3. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: Localization.
Revised: Localization.

Original: With current localization technology, we can reliably achieve very high localization accuracy (e.g., 1-2 cm-level accuracy with RTK 1 , 10 cm-level with Monte Carlo Localization scheme (Chong et al., 2013) as adopted in (Caesar et al., 2020)).
Revised: With current localization technology, we can reliably achieve very high localization accuracy (e.g., 1-2 cm-level accuracy with RTK 1 , 10 cm-level with Monte Carlo Localization scheme (Chong et al., 2013) as adopted in (Caesar et al., 2020)).

Original: We assume an accurate localization within at least 30 cm-level, and show our method is robust to such localization error (Table 6).
Revised: We assume an accurate localization within at least 30 cm-level, and show our method is robust to such localization error (Table 6).

Original: Spatial featurizer.
Revised: Spatial featurizer.

Original: We use a SR-UNet inplemented in Minkowski Engine (Choy et al., 2019) as the spatial featurizer.
Revised: We use a SR-UNet inplemented in Minkowski Engine (Choy et al., 2019) as the spatial featurizer.

Original: It is a 14-layer U-Net (Ronneberger et al., 2015) following the conventional
Revised: It is a 14-layer U-Net (Ronneberger et al., 2015) following the conventional

Original: ResNet (He et al., 2016) basic block design with sparse 3D conv-/deconvoluitons.
Revised: ResNet (He et al., 2016) basic block design with sparse 3D conv-/deconvolutions.

Original: The SR-UNet was originally designed by Choy et al.
Revised: The SR-UNet was originally designed by Choy et al.

Original: and achieved a signiﬁcant improvement against previous methods on various challenging semantic segmentation benchmarks.
Revised: and achieved a signiﬁcant improvement against previous methods on various challenging semantic segmentation benchmarks.

Original: In this work, we adopt this architecture to process each past traversal, and aggregate them by max-pooling at each quantized coordinate index into a SQuaSH tensor.
Revised: In this work, we adopt this architecture to process each past traversal, and aggregate them by max-pooling at each quantized coordinate index into a SQuaSH tensor.

Original: In Table 4, we include an ablation study on the relation between the ﬁnal detection performance and the complexity of the spatial featurizer.
Revised: In Table 4, we include an ablation study on the relation between the ﬁnal detection performance and the complexity of the spatial featurizer.

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: We set [ − H s , H e ] , the range of scans that are combined to produce dense point clouds, to be [0 , 20] m since we experiment with frontal-view detection only, and we combine only one scan into the dense point cloud S tl every 5 m within this range.
Revised: We set [ − H s , H e ] , the range of scans that are combined to produce dense point clouds, to be [0 , 20] m since we experiment with frontal-view detection only, and we combine only one scan into the dense point cloud S tl every 5 m within this range.

Original: When performing inference, we use the geographically closest possible SQuaSH tensor: we observe no performance drop if the available SQuaSH is within 5 m (Appendix B).
Revised: When performing inference, we use the geographically closest possible SQuaSH tensor: we observe no performance drop if the available SQuaSH is within 5 m (Appendix B).

Original: We use default quantization size δ = 0 .
Revised: We use default quantization size δ = 0 .

Original: 3 m for all the experiments (except we use 0.4 m for PointPillars due to limited GPU memory in training), and we conduct an ablation study on the quantization sizes in Table 3.
Revised: 3 m for all the experiments (except we use 0.4 m for PointPillars due to limited GPU memory in training), and we conduct an ablation study on the quantization sizes in Table 3.

Original: Due to GPU memory limit in training, we use T max = 5 available past traversals to compute the SQuaSH.
Revised: Due to GPU memory limit in training, we use T max = 5 available past traversals to compute the SQuaSH.

Original: The ﬁlter kernel size
Revised: The ﬁlter kernel size

Original: K is 5.
Revised: K is 5.

Original: The dimensionality of the history features d history is 64.
Revised: The dimensionality of the history features d history is 64.

Original:  E XPERIMENTS 4. S ETUP
Revised:  E XPERIMENTS 4. S ETUP

Original: Dataset.
Revised: Dataset.

Original: We validate our approach, H INDSIGHT , on the Lyft Level 5 PerceptionDataset (Kesten et al., 2019)
Revised: We validate our approach, H INDSIGHT , on the Lyft [Table] Perception

Original: (Lyft) and the nuScenes Dataset (Caesar et al., 2020).
Revised: [Table]

Original: To the best of our knowledge, these two datasets are the only two publicly available, large-scale autonomous driving datasets that have both annotated object bounding-box labels and multiple traversals with accurate localization.
Revised: these two datasets are the only two publicly available, large-scale autonomous driving datasets that have both annotated object bounding-box labels and multiple traversals with accurate localization.

Original: Since they are not originally designed for evaluating the efﬁcacy of utilizing historical traversals,
Revised: Since they are not originally designed for evaluating the efﬁcacy of utilizing historical traversals

Original: [Table]
Revised: [Table]

Original: some samples (i
Revised: some samples (i

Original: [Table]
Revised: [Table]

Original: e., point clouds at speciﬁc timestamps) in the datasets do not have past traversals.
Revised: e., point clouds at speciﬁc timestamps) in the datasets do not have past traversals.

Original: Thus
Revised: Thus

Original: [Table]
Revised: [Table]

Original: we re-split the datasets so that each training and test sample has at least 2 past traversals; sequences in training and test set are geographically disjoint .
Revised: we re-split the datasets so that each training and test sample has at least 2 past traversals; sequences in training and test set are geographically disjoint .

Original: This results in 12,407/2,274 training/test samples in the Lyft dataset and 3,985/2,324 training/test samples in the nuScenes dataset.
Revised: This results in 12,407/2,274 training/test samples in the Lyft dataset and 3,985/2,324 training/test samples in the nuScenes dataset.

Original: To apply off-the-shelf 3D object detectors which are typically built to take KITTI data (Geiger et al., 2012) [Table] we convert the raw Lyft and nuScenes data into the KITTI format and only train and evaluate frontal-view, frame-by-frame, and LiDAR-only detections.
Revised: To apply off-the-shelf 3D object detectors which are typically built to take KITTI data (Geiger et al., 2012), we convert the raw Lyft and nuScenes data into the KITTI format and only train and evaluate frontal-view, frame-by-frame, and LiDAR-only detections.

Original: We use the roof LiDAR signal (40 or 64 beams in Lyft; 32 beams in nuScenes), and obtain the global 6-DoF localization and extrinsic transformations between LiDAR and GPS/IMU directly from the raw data.
Revised: We use the roof LiDAR signal (40 or 64 beams in Lyft; 32 beams in nuScenes), and obtain the global 6-DoF localization and extrinsic transformations between LiDAR and GPS/IMU directly from the raw data.

Original: Evaluation metric.
Revised: Evaluation metric.

Original: We follow KITTI (Geiger et al., 2012) and evaluate object detection both in the bird’s-eye view (BEV) and in 3D for Car, Pedestrian, and Cyclist on the Lyft dataset (Cyclist includes Bicycle + Motorcycle in the raw Lyft dataset), and Car and Pedestrian in the nuScenes dataset, since there are too few cyclist instances.
Revised: We follow KITTI (Geiger et al., 2012) and evaluate object detection both in the bird’s-eye view (BEV) and in 3D for Car, Pedestrian, and Cyclist on the Lyft dataset (Cyclist includes Bicycle + Motorcycle in the raw Lyft dataset), and Car and Pedestrian in the nuScenes dataset, since there are too few cyclist instances.

Original: We also follow the KITTI convention to report average precision (AP) with the intersection over union (IoU) thresholds at 0.7/0.5 for Car and 0.5/0.25 for Pedestrian and Cyclist.
Revised: We also follow the KITTI convention to report average precision (AP) with the intersection over union (IoU) thresholds at 0.7/0.5 for Car and 0.5/0.25 for Pedestrian and Cyclist.

Original: We denote AP for BEV and 3D by AP BEV and AP 3D .
Revised: We denote AP for BEV and 3D by AP BEV and AP 3D .

Original: Since the
Revised: Since the

Original: Lyft and nuScenes datasets do not provide the ofﬁcial deﬁnition of object difﬁculty as that in KITTI, we follow Wang et al.
Revised: Lyft and nuScenes datasets do not provide the ofﬁcial deﬁnition of object difﬁculty as that in KITTI, we follow Wang et al.

Original: (2020) to evaluate the AP at various depth ranges.
Revised: (2020) to evaluate the AP at various depth ranges.

Original: In Tables 1 to 5 of the main paper, due to the limited space, we only present results with IoU= 0 .
Revised: In Tables 1 to 5 of the main paper, due to the limited space, we only present results with IoU= 0 .

Original: 7 for car objects and IoU= 0 .for
Revised: 7 for car objects and IoU= 0 .for

Original: pedestrian and cyclist objects.
Revised: pedestrian and cyclist objects.

Original: Please refer to Appendix A for results on AP 3D and other IoU metrics, where we observe a similar trend.
Revised: Please refer to Appendix A for results on AP 3D and other IoU metrics, where we observe a similar trend.

Original: Base 3D object detectors.
Revised: Base 3D object detectors.

Original: We adopt four representative, high-performing 3D object detectors:
Revised: We adopt four representative, high-performing 3D object detectors:

Original: PointPillars (Lang et al., 2019), SECOND (VoxelNet) (Yan et al., 2018; Zhou & Tuzel, 2018),
Revised: PointPillars (Lang et al., 2019), SECOND (VoxelNet) (Yan et al., 2018; Zhou & Tuzel, 2018),

Original: PointRCNN (Shi et al., 2019), and PV-RCNN (Shi et al., 2020a), as the base detector models to augment (and compare) with our H INDSIGHT framework.
Revised: PointRCNN (Shi et al., 2019), and PV-RCNN (Shi et al., 2020a), as the base detector models to augment (and compare) with our H INDSIGHT framework.

Original: These models are all state-of-theart LiDAR-based detectors and are distinct in architecture: pillar-wise (PointPillars) vs .
Revised: These models are all state-of-theart LiDAR-based detectors and are distinct in architecture: pillar-wise (PointPillars) vs .

Original: voxelwise (SECOND, PV-RCNN) vs features, and
Revised: voxelwise (SECOND, PV-RCNN) vs .

Original: -stage (PointPillars
Revised: point-wise (PointRCNN) features, and single-stage (PointPillars,

Original: SECOND) vs .
Revised: SECOND) vs .

Original: multi-stage (PointRCNN, PV-RCNN).
Revised: multi-stage (PointRCNN, PV-RCNN).

Original: We use publicly available code from OpenPCDet (Team, 2020) for all four models.
Revised: We use publicly available code from OpenPCDet (OpenPCDet, 2020) for all four models.

Original: We use most of the default hyper-parameters tuned for KITTI, with the exception that we enlarge the perception range from 70m to 90m – since Lyft provides labels beyond 70m – and we reduce the number of training epochs by 1/4 on Lyft – since its train set is about three times of the size of KITTI.
Revised: We use most of the default hyper-parameters tuned for KITTI, with the exception that we enlarge the perception range from 70m to 90m – since Lyft provides labels beyond 70m – and we reduce the number of training epochs by 1/4 on Lyft – since its train set is about three times of the size of KITTI.

Original: We observe consistent results with such hyperparameters.
Revised: We observe consistent results with such hyperparameters.

Original: We use exactly the same hyper-parameters and detection model architectures for thebaseline and baseline + H INDSIGHT , except a change in the input dimension of the ﬁrst layer due to the endowed hindsight features. We experiment with all four models on Lyft, and PointPillars
Revised: We use exactly the same hyper-parameters and detection model architectures for the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3.4 ms 32.3 MB 70.7 / 46. + H-0.5 m2.4 ms 15.3 MB 68.0 / 44.
Revised: m

Original:  + H-1.0 m 1.8 ms 5.2 MB 61.3 / 36.
Revised: [Table] baseline and baseline + H INDSIGHT , except a change in the input dimension of the ﬁrst layer due to the endowed hindsight features.

Original:  and PointRCNN on nuScenes, as training diverged for the other two models with the above default hyperparameters.
Revised: We experiment with all four models on Lyft, and PointPillars and PointRCNN on nuScenes, as training diverged for the other two models with the above default hyperparameters.

Original: All models are trained with 4 NVIDIA 3090 GPUs.
Revised: All models are trained with 4 NVIDIA 3090 GPUs.

Original: 4. E MPIRICAL R ESULTS 4.2. 3D O BJECT D ETECTION R ESULTS
Revised: 4. E MPIRICAL R ESULTS 4.2. 3D O BJECT D ETECTION R ESULTS

Original: In Table 1 and Table 2 we show the detection performance of the four representative 3D object detection models with and without the aid of H INDSIGHT on two different datasets and various object types.
Revised: In Table 1 and Table 2 we show the detection performance of the four representative 3D object detection models with and without the aid of H INDSIGHT on two different datasets and various object types.

Original: (Please refer to Appendix A for corresponding AP 3D results and results under other
Revised: (Please refer to Appendix A for corresponding AP 3D results and results under other

Original: IoU metrics.)
Revised: IoU metrics.)

Original: We observe that H INDSIGHT consistently improves the detection performance for all detectors in almost all cases.
Revised: We observe that H INDSIGHT consistently improves the detection performance for all detectors in almost all cases.

Original: In particular, models with H INDSIGHT enjoy substantial improvement over baseline counterparts on mid- to far-range, small (pedestrian and cyclist) objects, frequently yielding accuracy gains of over 20 percentage points in AP .
Revised: In particular, models with H INDSIGHT enjoy substantial improvement over baseline counterparts on mid- to far-range, small (pedestrian and cyclist) objects, frequently yielding accuracy gains of over 20 percentage points in AP .

Original: As also shown in Figure 2 (more in Appendix C), compared to base detectors, H INDSIGHT maintains a signiﬁcantly higher precision under the same recall for detection objects.
Revised: As also shown in Figure 2 (more in Appendix C), compared to base detectors, H INDSIGHT maintains a signiﬁcantly higher precision under the same recall for detection objects.

Original: We note that these are typically considered the hardest parts in LiDAR-based object detection, not only because small objects have much fewer training examples in these datasets, but also because small physical shapes and farther distance from the ego car inherently result in much sparser LiDAR signals.
Revised: We note that these are typically considered the hardest parts in LiDAR-based object detection, not only because small objects have much fewer training examples in these datasets, but also because small physical shapes and farther distance from the ego car inherently result in much sparser LiDAR signals.

Original: It is thus crucial to incorporate the historical prior of the scenes (captured by H INDSIGHT ) for improved detection performance.
Revised: It is thus crucial to incorporate the historical prior of the scenes (captured by H INDSIGHT ) for improved detection performance.

Original: 4.2. L ATENCY AND S TORAGE
Revised: 4.2. L ATENCY AND S TORAGE

Original: Low latency is crucial for on-board perception models since they need to make critical decisions realtime.
Revised: Low latency is crucial for on-board perception models since they need to make critical decisions realtime.

Original: In Table 3, we examine the extra latency that H INDSIGHT introduces to the detection model.
Revised: In Table 3, we examine the extra latency that H INDSIGHT introduces to the detection model.

Original: We observe an exceptionally low latency from our query operation (average 3.4 ms for quantization size 0.3 m).
Revised: We observe an exceptionally low latency from our query operation (average 3.4 ms for quantization size 0.3 m).

Original: It adds little burden to the real-time pipeline whose perception loop typically takes 100 ms (10 Hz).
Revised: It adds little burden to the real-time pipeline whose perception loop typically takes 100 ms (10 Hz).

Original: In Table 3 we also show the average uncompressed storage of one SQuaSH for a scene.
Revised: In Table 3 we also show the average uncompressed storage of one SQuaSH for a scene.

Original:  The storage highly depends on the quantization size; it takes 32.3 MB for the default quantization size 0.3 m. Though it is already small enough in light of current cheaper and cheaper storage technology, in practice, the autonomous driving systemcan build the SQuaSH at least every 10 metres without losing performance (Appendix B) to have a storage-efﬁcient engineering implementation.
Revised: Each SQuaSH roughly covers 90 m * 80 m range in BEV. The storage highly depends on the quantization size; it takes 32.3 MB for the default quantization size 0.3 m. Though it is already small enough in light of current cheaper and cheaper storage technology, in practice, the autonomous driving system

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 8 + H ∆0 .
Revised: + H ∆0 .

Original: 1 m 70.6 / 46.8 44.0 / 31.8 52.9 / 45.
Revised: 1 m 70.6 / 46.8 44.0 / 31.8 52.9 / 45.

Original: + H ∆0 .
Revised: + H ∆0 .

Original: 2 m 70.3 / 43.6 42.7 / 30.4 52.5 / 43.
Revised: 2 m 70.3 / 43.6 42.7 / 30.4 52.5 / 43.

Original: + H ∆0 .
Revised: + H ∆0 .

Original: 3 m 70.1 / 44.2 37.9 / 26.0 49.4 / 41.
Revised: 3 m 70.1 / 44.2 37.9 / 26.0 49.4 / 41.

Original: + H ∆0 .
Revised: + H ∆0 .

Original: 5 m 67.2 / 40.8 31.0 / 20.8 44.5 / 36.
Revised: 5 m 67.2 / 40.8 31.0 / 20.8 44.5 / 36.

Original: + H ∆1 . 

Revised: + H ∆1 .

Original: m 60.9 / 33.8 16.7 / 12.3 30.1 / 24.
Revised: m 60.9 / 33.8 16.7 / 12.3 30.1 / 24.

Revised: [Figure]

Original: Recall 0 .0 .0 .0 .0 .1 .
Revised: [Figure] [Figure] [Figure] 0 Recall

Revised: [Figure]

Revised: [Figure]

Original: P r ec i s i o n
Revised: i s i [Figure] n

Original: PRCNN+Hindsight PRCNN
Revised: [Figure] can build the SQuaSH at least every 10 metres without losing performance (Appendix B) to have a storage-efﬁcient engineering implementation.

Original: 4.2. A BLATION S TUDY
Revised: 4.2. A BLATION S TUDY

Original: Impact of the H INDSIGHT spatial featurizer.
Revised: Impact of the H INDSIGHT spatial featurizer.

Original: In Table 4 we study the relation between the complexity of H INDSIGHT spatial featurizer and the ﬁnal detection performance.
Revised: In Table 4 we study the relation between the complexity of H INDSIGHT spatial featurizer and the ﬁnal detection performance.

Original: We compare three possible spatial featurizers with different complexities: Identity, which directly quantizes each historical traversal and uses nominal “1” as features in Q tl to indicate the non-emptiness; FCN, which is a simple two-layer, fully convolutional network; SR-UNet, the model we use as default.
Revised: We compare three possible spatial featurizers with different complexities: Identity, which directly quantizes each historical traversal and uses nominal “1” as features in Q tl to indicate the non-emptiness; FCN, which is a simple two-layer, fully convolutional network; SR-UNet, the model we use as default.

Original: It is interesting to observe that by simply querying from the history occupancy tensor (Identity), H INDSIGHT enjoys a reasonable performance boost; the detection model beneﬁts even more with more complex feature backbones (FCN and SR-UNet).
Revised: It is interesting to observe that by simply querying from the history occupancy tensor (Identity), H INDSIGHT enjoys a reasonable performance boost; the detection model beneﬁts even more with more complex feature backbones (FCN and SR-UNet).

Original: This suggests that even the simple presence or absence of objects in past historical traversals is a valuable signal.
Revised: This suggests that even the simple presence or absence of objects in past historical traversals is a valuable signal.

Original: More sophisticated processing of the past traversals may also characterize exactly what objects were observed in the past, which is richer contextual information.
Revised: More sophisticated processing of the past traversals may also characterize exactly what objects were observed in the past, which is richer contextual information.

Original: Impact of the amount of available past traversals.
Revised: Impact of the amount of available past traversals.

Original: In Table 5 we study the effect of using different numbers of past traversals during testing.
Revised: In Table 5 we study the effect of using different numbers of past traversals during testing.

Original: We observe that with just one past traversal, the detection model can already beneﬁt from the history features; with only two, the model can achieve performance close to using ﬁve traversals (our manually-selected maximum).
Revised: We observe that with just one past traversal, the detection model can already beneﬁt from the history features; with only two, the model can achieve performance close to using ﬁve traversals (our manually-selected maximum).

Original: This result indicates that H INDSIGHT does not need a large number of past traversals to extract relevant information.
Revised: This result indicates that H INDSIGHT does not need a large number of past traversals to extract relevant information.

Original: Impact of quantization sizes.
Revised: Impact of quantization sizes.

Original: The spatial quantization sizes of SQuaSH can affect the granularity and the receptive ﬁeld of H INDSIGHT .
Revised: The spatial quantization sizes of SQuaSH can affect the granularity and the receptive ﬁeld of H INDSIGHT .

Original: With a ﬁxed spatial featurizers architecture, the smaller the quantization size is, the higher the granularity but the smaller the receptive ﬁeld is.
Revised: With a ﬁxed spatial featurizers architecture, the smaller the quantization size is, the higher the granularity but the smaller the receptive ﬁeld is.

Original: In Table 3, we study how different quantization sizes affect the ﬁnal detection performance.
Revised: In Table 3, we study how different quantization sizes affect the ﬁnal detection performance.

Original: It can be seen that the performance gain is more pronounced with a proper quantization size (0.3m).
Revised: It can be seen that the performance gain is more pronounced with a proper quantization size (0.3m).

Original: When the quantization is too small, the model cannot capture sufﬁcient surrounding priors and is susceptible to random errors in localization; when the quantization size is too large, H INDSIGHT loses its ability to differentiate points within one voxel and it becomes an extra burden to the learning process.
Revised: When the quantization is too small, the model cannot capture sufﬁcient surrounding priors and is susceptible to random errors in localization; when the quantization size is too large, H INDSIGHT loses its ability to differentiate points within one voxel and it becomes an extra burden to the learning process.

Original: Impact of localization error.
Revised: Impact of localization error.

Original: H INDSIGHT relies on (accurate) localization to query the correct history features.
Revised: H INDSIGHT relies on (accurate) localization to query the correct history features.

Original: Thus, it is important to investigate the robustness of H INDSIGHT against potentiallocalization error.
Revised: Thus, it is important to investigate the robustness of H INDSIGHT against potential

Original: In Table 6, we take the model trained with ground-truth localization (directly obtained from the dataset, without correcting for already present, minor errors), and evaluate its

Original: [Figure]
Revised: [Figure]

Original:  performance with the presence of simulated random localization error.
Revised: localization error. In Table 6, we take the model trained with ground-truth localization (directly obtained from the dataset, without correcting for already present, minor errors), and evaluate its performance with the presence of simulated random localization error.

Original: We simulate the error by sampling a unit vector in 3D space , scale it by (cid:15) ∼ N (0 , σ 2 ) , where σ is the error magnitude, and apply it to the test scan.
Revised: We simulate the error by sampling a unit vector in 3D space , scale it by (cid:15) ∼ N (0 , σ 2 ) , where σ is the error magnitude, and apply it to the test scan.

Original: We observe that H INDSIGHT is robust to localization error, where it can still bring substantial improvement on detection performance for all noise less than σ = 0 .
Revised: We observe that H INDSIGHT is robust to localization error, where it can still bring substantial improvement on detection performance for all noise less than σ = 0 .

Original: 3 (m).
Revised: 3 (m).

Original: Such localization error is well within the reliable range of current localization technology, showing the superior real-world application of H INDSIGHT .
Revised: Such localization error is well within the reliable range of current localization technology, showing the superior real-world application of H INDSIGHT .

Revised: We also test the model under the presence of random bearing error (please refer to Appendix D), where we observe similar robustness.

Original: 4. Q UALITATIVE R ESULTS
Revised: 4. Q UALITATIVE R ESULTS

Original: Detection performance.
Revised: Detection performance.

Original: In Figure 3, we visualize the detection results of PointRCNN and PointRCNN+H INDSIGHT at two random scenes of the Lyft test set (please refer to Appendix D for more).
Revised: In Figure 3, we visualize the detection results of PointRCNN and PointRCNN+H INDSIGHT at two random scenes of the Lyft test set (please refer to Appendix F for more).

Original: We observe that the base detector tends to generate many false positives to ensure a good coverage of the true objects; our H INDSIGHT has signiﬁcantly fewer false positives while being able to detect the true objects exhaustively (also illustrated by Figure 2).
Revised: We observe that the base detector tends to generate many false positives to ensure a good coverage of the true objects; our H INDSIGHT has signiﬁcantly fewer false positives while being able to detect the true objects exhaustively (also illustrated by Figure 2).

Original: Speciﬁcally, for faraway objects (two distant pedestrians in the left scene) and small objects (the close cyclist in the left scene), the base detector either completely misses them or classiﬁes them incorrectly, while H INDSIGHT works like a charm.
Revised: Speciﬁcally, for faraway objects (two distant pedestrians in the left scene) and small objects (the close cyclist in the left scene), the base detector either completely misses them or misclassiﬁes them, while H INDSIGHT works like a charm.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Limitations.
Revised: Limitations.

Original: Our approach relies on the past traversals being informative of the current state of the scene.
Revised: Our approach relies on the past traversals being informative of the current state of the scene.

Original: Therefore, sudden changes to the current scene (e.g., a snow-storm or an accident) may pose problems to the detector, since the scene it observes will be inconsistent with the history.
Revised: Therefore, sudden changes to the current scene (e.g., a snow-storm or an accident) may pose problems to the detector, since the scene it observes will be inconsistent with the history.

Original: Note that such adverse conditions may impact traditional detectors as well.
Revised: Note that such adverse conditions may impact traditional detectors as well.

Original: Unfortunately, existing datasets are collected in relatively mild conditions (e.g., clear weather for Lyft and sunny, rainy or cloudy conditions for nuScenes).
Revised: Unfortunately, existing datasets are collected in relatively mild conditions (e.g., clear weather for Lyft and sunny, rainy, or cloudy conditions for nuScenes).

Original: As such they do not offer the opportunity to test our approach under adverse weather or other abnormal changes to road conditions.
Revised: As such, they do not offer the opportunity to test our approach under adverse weather or other abnormal changes to road conditions.

Original: We hope this work can facilitate future research on this topic and inspire the creation of more related diverse datasets.
Revised: We hope this work can facilitate future research on this topic and inspire the creation of more related diverse datasets.

Original: Conclusion.
Revised: Conclusion.

Original: In this work, we explore a novel yet highly practical scenario for self-driving, where unlabeled LiDAR scans from previous traversals of the same scene are available.
Revised: In this work, we explore a novel yet highly practical scenario for self-driving, where unlabeled LiDAR scans from previous traversals of the same scene are available.

Original: We propose H INDSIGHT , which extracts the contextual information from multiple past traversals and aggregates them into a compact and geo-indexed SQuaSH.
Revised: We propose H INDSIGHT , which extracts the contextual information from multiple past traversals and aggregates them into a compact and geo-indexed SQuaSH.

Original: While the self-driving system is in operation, SQuaSH can be used to enrich the current 3D scan and substantially improve 3D perception.
Revised: While the self-driving system is in operation, SQuaSH can be used to enrich the current 3D scan and substantially improve 3D perception.

Original: The whole pipeline can be trained end-to-end with detection labels only.
Revised: The whole pipeline can be trained end-to-end with detection labels only.

Original: Once trained, the SQuaSH can be precomputed off-line and queried on-line with little latency.
Revised: Once trained, the SQuaSH can be precomputed off-line and queried on-line with little latency.

Original: We show on two large-scale, real-world self-driving datasets that H INDSIGHT consistently and substantially improves the detection performance of four high-performing 3D detectors, taking us one step closer towards safe self-driving.
Revised: We show on two large-scale, real-world self-driving datasets that H INDSIGHT consistently and substantially improves the detection performance of four high-performing 3D detectors, taking us one step closer towards safe self-driving.

Original: With this work, we hope to inspire more research into exploring the rich trove of information from past traversals of the same scene.
Revised: With this work, we hope to inspire more research into exploring the rich trove of information from past traversals of the same scene.

Revised: A CKNOWLEDGEMENT

Revised: This research is supported by grants from the National Science Foundation NSF (III-1618134,

Revised: III-1526012, IIS-1149882, IIS-1724282, TRIPODS-1740822, IIS-2107077, OAC-2118240, OAC2112606 and IIS-2107161), the Ofﬁce of Naval Research DOD (N00014-17-1-2175), the DARPA

Revised: Learning with Less Labels program (HR001118S0044), the Bill and Melinda Gates Foundation, the Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR1719875), and SAP America.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This work uses data from past traversals of a scene to substantially improve 3D perception, especially for far-off or occluded objects.
Revised: This work uses data from past traversals of a scene to substantially improve 3D perception, especially for far-off or occluded objects.

Original: We believe that our approach will lead to safer self-driving cars.
Revised: We believe that our approach will lead to safer self-driving cars.

Original: Our approach will also allow for self-driving cars to be driven beyond tight geo-fenced areas where HD maps are available, thus making this technology more broadly accessible.
Revised: Our approach will also allow for self-driving cars to be driven beyond tight geo-fenced areas where HD maps are available, thus making this technology more broadly accessible.

Original: Because our approach stores data that is typically discarded, a potential concern is the privacy of pedestrians, cars and cyclists on the road.
Revised: Because our approach stores data that is typically discarded, a potential concern is the privacy of pedestrians, cars and cyclists on the road.

Original: However, we note that the data we store is heavily quantized ( δ = 0 .
Revised: However, we note that the data we store is heavily quantized ( δ = 0 .

Original: 3 m ), featurized and only based on the sparse LiDAR point cloud.
Revised: 3 m ), featurized and only based on the sparse LiDAR point cloud.

Original: Therefore the stored representation is likely to contain only aggregate information about the presence or absence of certain kinds of objects, and is unlikely to have any identifying information.
Revised: Therefore the stored representation is likely to contain only aggregate information about the presence or absence of certain kinds of objects, and is unlikely to have any identifying information.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: In subsection 3.6 and subsection 4.1, we exhaustively list the hyper-parameters for our approach, the dataset details, evaluation metrics and the hyper-parameters for training 3D object detectors.
Revised: In subsection 3.6 and subsection 4.1, we exhaustively list the hyper-parameters for our approach, the dataset details, evaluation metrics and the hyper-parameters for training 3D object detectors.

Original: We will open-source the code to reproduce the results upon acceptance.
Revised: Our code is available at https://github.com/YurongYou/Hindsight .

Original: As mentioned in subsection 4.1, since the Lyft and the nuScenes dataset are not originally designed for evaluating the efﬁcacy of utilizing historical traversals, we use a custom split to have a valid evaluation.
Revised: As mentioned in subsection 4.1, since the Lyft and the nuScenes dataset are not originally designed for evaluating the efﬁcacy of utilizing historical traversals, we use a custom split to have a valid evaluation.

Original: We will release the split such that future works can have a fair comparison. Appendix
Revised: We also release the split in the link for future comparison

Original: We provide details omitted in the main text.

Original: • Appendix A: detailed evaluation results on the Lyft and the nuScenes dataset.

Original: (subsubsection 4.2.1 of the main paper, same naming as in Table 1 and Table 2) • Appendix B: ablation on using pre-computed SQuaSH from a distance.

Original: • Appendix C: more precision-recall curves of detection results (as in Figure 2).

Original: • Appendix D: more qualitative visualization (as in subsection 4.3).

Original: A M ORE E VALUATION R ESULTS ON THE L YFT D ATASET AND THE NU S CENES D ATASET
Revised: A M ORE E VALUATION R ESULTS ON THE L YFT D ATASET AND THE NU S CENES D ATASET

Original: In Table 1 and Table 2 of the main paper
Revised: In Table 1 and Table 2 of the main paper

Original: [Table]
Revised: [Table]

Original: we present evaluation results across various depth ranges on AP BEV with IoU= 0 .
Revised: we present evaluation results across various depth ranges on AP BEV with IoU= 0 .

Original: 7 for car objects and IoU= 0 .
Revised: 7 for car objects and IoU= 0 .

Original: 5 for pedestrian and cyclist objects; , on both the
Revised: 5 for pedestrian and cyclist objects, on both the

Original: Lyft and the nuScenes dataset.
Revised: Lyft and the nuScenes dataset.

Original: In Tables 7 to
Revised: In Tables 7 to

Original: [Table]
Revised: [Table]

Original: we present the evaluation results on AP BEV and
Revised: we present the evaluation results on AP BEV and

Original: AP 3D with two sets of IoU metrics (IoU= 0 .
Revised: AP 3D with two sets of IoU metrics (IoU= 0 .

Original: 7 for car objects and IoU= 0 .
Revised: 7 for car objects and IoU= 0 .

Original: 5 for pedestrian and cyclist objects; IoU= 0 .
Revised: 5 for pedestrian and cyclist objects; IoU= 0 .

Original: 5 for car objects and IoU= 0 .
Revised: 5 for car objects and IoU= 0 .

Original: 25 for pedestrian and cyclist objects).
Revised: 25 for pedestrian and cyclist objects).

Original: We observe a similar trend throughout different evaluation metrics

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Figure]
Revised: [Table] [Table]

Original: r

Original: [Figure]
Revised: [Figure]

Original:  i s i o n
Revised: P r ec i s i o n

Original: [Figure]
Revised: [Figure]

Original: B SQ UA SH AT A D ISTANCE
Revised: B SQ UA SH AT A D ISTANCE

Original: For experiments in the main paper, we use the SQuaSH pre-computed at the closest possible location from the ego car position G c (i.e., the l c is picked as the closest possible one from the ego car).
Revised: For experiments in the main paper, we use the SQuaSH pre-computed at the closest possible location from the ego car position G c (i.e., the l c is picked as the closest possible one from the ego car).

Original: For practical use, the self-driving system should pre-compute the SQuaSH once every s m to have a storage-efﬁcient implementation.
Revised: For practical use, the self-driving system should pre-compute the SQuaSH once every s m to have a storage-efﬁcient implementation.

Original: In this case, the available SQuaSH can have s/ 2 m offset from the ego car localization.
Revised: In this case, the available SQuaSH can have s/ 2 m offset from the ego car localization.

Original: We evaluate this situation in Table 13, where we use the SQuaSHs that are s/ 2 m away (either backward or forward) from the ego car localization.
Revised: We evaluate this situation in Table 13, where we use the SQuaSHs that are s/ 2 m away (either backward or forward) from the ego car localization.

Original: We observe no signiﬁcant performance difference in s/ 2 = 

Revised: We observe no signiﬁcant performance difference in s/ 2 =

Original: − 5 m offset.
Revised: − 5 m offset.

Original: It validates that in practice the self-driving system can pre-compute the SQuaSH at least every 10 m without losing performance gains.
Revised: It validates that in practice the self-driving system can pre-compute the SQuaSH at least every 10 m without losing performance gains.

Original: C A DDITIONAL P RECISION -R ECALL C URVES
Revised: C A DDITIONAL P RECISION -R ECALL C URVES

Original: In Figure 4, we show the detection precision-recall curves of the base detector (PointRCNN) and the base detector with H INDSIGHT , on car, pedestrian and cyclist objects under various evaluation metrics.
Revised: In Figure 4, we show the detection precision-recall curves of the base detector (PointRCNN) and the base detector with H INDSIGHT , on car, pedestrian and cyclist objects under various evaluation metrics.

Original: We observe a similar trend as shown in Figure 2. 
Revised: We observe a similar trend as shown in Figure 2. [Table] [Table]

Original: D A DDITIONAL Q UALITATIVE R ESULTS
Revised: D A DDITIONAL B EARING E RROR E VALUATION

Revised: Typically, with current technology

Revised: [Table]

Revised: the error on bearing angle is much smaller than that on localization (as small as 0.08 degrees 2 ).

Revised: Nevertheless

Revised: [Table]

Revised: we evaluate the performance under a much harsher estimate of bearing rotation errors, and present detection evaluation results in Table 14.

Revised: It can be seen that even under situations with large (potentially overestimated) bearing angle errors, H INDSIGHT models still signiﬁcantly outperform the baseline model.

Revised: E A DDITIONAL E VALUATION R ESULTS WITH D IFFERENT Q UANTIZATION

Revised: We show the evaluation results of Table 3 under a different metric (car object: IoU 0.5, pedestrian & cyclist object: IoU 0.25) in Table 15.

Revised: F A DDITIONAL Q UALITATIVE R ESULTS

Revised: We show additional qualitative detection results in Figure 5.

Original: [Figure]
Revised: [Figure]