Original: P OLICY P RE - TRAINING FOR A UTONOMOUS D RIVING VIA S ELF - SUPERVISED G EOMETRIC M ODELING
Revised: P OLICY P RE - TRAINING FOR A UTONOMOUS D RIVING VIA S ELF - SUPERVISED G EOMETRIC M ODELING

Original: A BSTRACT
Revised: A BSTRACT

Original: Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving.
Revised: Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving.

Original: Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task.
Revised: Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task.

Original: To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving.
Revised: To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving.

Original: We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos.
Revised: We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos.

Original: The proposed PPGeo is performed in two stages to support effective self-supervised training.
Revised: The proposed PPGeo is performed in two stages to support effective self-supervised training.

Original: In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input.
Revised: In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input.

Original: In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.
Revised: In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.

Original: As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks.
Revised: As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks.

Original: As a side product, thepre- trained geometric modeling networks could
Revised: As a side product, the

Original:  bring further improvement to the depth and odometry estimation tasks.Extensive experiments covering a wide
Revised: pre-trained geometric modeling networks could bring further improvement to the depth and odometry estimation tasks.

Original: span of challenging scenarios have

Original: demonstrated

Original: the superiority of

Original: our proposed approach

Original: , where

Original:  improvements range from 2% to even over 100% with very limited data.
Revised: Extensive experiments covering a wide span of challenging scenarios have demonstrated the superiority of our proposed approach, where improvements range from 2% to even over 100% with very limited data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Policy learning refers to the learning process of an autonomous agent acquiring the decision-making policy to perform a certain task in a particular environment.
Revised: Policy learning refers to the learning process of an autonomous agent acquiring the decision-making policy to perform a certain task in a particular environment.

Original: Visuomotor policy learning (Mnih et al., 2015; Levine et al., 2016; Hessel et al., 2018; Laskin et al., 2020; Toromanoff et al., 2020) takes as input raw sensor observations and predicts the action, simultaneously cooperating and training the perception and control modules in an end-to-end fashion.
Revised: Visuomotor policy learning (Mnih et al., 2015; Levine et al., 2016; Hessel et al., 2018; Laskin et al., 2020; Toromanoff et al., 2020) takes as input raw sensor observations and predicts the action, simultaneously cooperating and training the perception and control modules in an end-to-end fashion.

Original: For visuomotor policy models, learning tabula rasa is difficult, where it usually requires a prohibitively large corpus of labeled data or environment interactions to achieve satisfactory performance (Espeholt et al., 2018;
Revised: For visuomotor policy models, learning tabula rasa is difficult, where it usually requires a prohibitively large corpus of labeled data or environment interactions to achieve satisfactory performance (Espeholt et al., 2018;

Original: Wijmans et al., 2019; Yarats et al., 2020).
Revised: Wijmans et al., 2019; Yarats et al., 2020).

Original: To mitigate the sample efficiency caveat in visuomotor policy learning, pre-training the visual perception network in advance is a promising solution.
Revised: To mitigate the sample efficiency caveat in visuomotor policy learning, pre-training the visual perception network in advance is a promising solution.

Original: Recent studies (Shah & Kumar, 2021; Parisi et al., 2022; Xiao et al., 2022; Radosavovic et al., 2022;Shah et al., 2022) have demonstrated that applying popular visual pre-training approaches, including ImageNet (Deng et al., 2009) classification, contrastive learning (He et al., 2020; Chen et al., 2020c), masked image modeling (MIM) 
Revised: Recent studies (Shah & Kumar, 2021; Parisi et al., 2022; Xiao et al., 2022; Radosavovic et al., 2022; et al., 2022) have demonstrated that applying popular visual pre-training approaches, including ImageNet (Deng et al., 2009) classification, contrastive learning (He et al., 2020; Chen et al., ), masked image modeling (MIM) (He et al., 2022; Xie et al.,

Original: (He et al., 2022; Xie et al., 2022), and language-vision pre-training
Revised: Xue et al., 2022), and language-vision pre-training

Original: Radford et al., 2021), could guarantee superior representation for robotic policy learning tasks, e.g. , dexterous manipulation, motor control skills and visual navigation.
Revised: (Radford et al., 2021), could guarantee superior representation for robotic policy learning tasks, e.g. , dexterous manipulation, motor control skills and visual navigation.

Original: However, for one crucial and challenging visuomotor task in particular, namely end-to-end autonomous driving 1 , the aforementioned predominant pre-training methods may not be the optimal choice (Yamada et al., 2022; Zhang et al., 2022b).
Revised: However, for one crucial and challenging visuomotor task in particular, namely end-to-end autonomous driving 1 , the aforementioned predominant pre-training methods may not be the optimal choice (Yamada et al., 2022; Zhang et al., 2022b).

Original: In this paper, we aim to investigate why ever-victorious pre-training approaches for general computer vision tasks and robotic control tasks are prone to fail in case of end-to-end autonomous driving.
Revised: In this paper, we aim to investigate why ever-victorious pre-training approaches for general computer vision tasks and robotic control tasks are prone to fail in case of end-to-end autonomous driving.

Original: For conventional pre-training methods in general vision tasks, e.g. , classification, segmentation and detection, they usually adopt a wide range of data augmentations to achieve translation and view invariance (Zhang et al., 2016; Wu et al., 2018).
Revised: For conventional pre-training methods in general vision tasks, e.g. , classification, segmentation and detection, they usually adopt a wide range of data augmentations to achieve translation and view invariance (Zhang et al., 2016; Wu et al., 2018).

Original: For robotic control tasks, the input sequence is generally of small resolution; the environment setting is simple and concentrated on objects (Parisi et al., 2022; Radosavovic et al., 2022).
Revised: For robotic control tasks, the input sequence is generally of small resolution; the environment setting is simple and concentrated on objects (Parisi et al., 2022; Radosavovic et al., 2022).

Original: We argue that the visuomotor driving investigated in this paper, is sensitive to geometric relationships and usually comprises complex scenarios.
Revised: We argue that the visuomotor driving investigated in this paper, is sensitive to geometric relationships and usually comprises complex scenarios.

Original: As described in Fig. 1(a), the input data often carry irrelevant information, such as background buildings, far-away moving vehicles, nearby static obstacles, etc. , which are deemed as noises for the decision making task.
Revised: As described in Fig. 1(a), the input data often carry irrelevant information, such as background buildings, far-away moving vehicles, nearby static obstacles, etc. , which are deemed as noises for the decision making task.

Original: To obtain a good driving policy, we argue that the desirable model should only concentrate on particular parts/patterns of the visual input.
Revised: To obtain a good driving policy, we argue that the desirable model should only concentrate on particular parts/patterns of the visual input.

Original: That is, taking heed of direct or deterministic relation to the decision making, e.g. , traffic signals in Fig. 1(b).
Revised: That is, taking heed of direct or deterministic relation to the decision making, e.g. , traffic signals in Fig. 1(b).

Original: However, concurrent pre-training approaches fail to fulfill such a requirement.
Revised: However, concurrent pre-training approaches fail to fulfill such a requirement.

Original: There comes a natural and necessary demand to formulate a pre-training scheme curated for end-to-end autonomous driving.
Revised: There comes a natural and necessary demand to formulate a pre-training scheme curated for end-to-end autonomous driving.

Original: We attempt to pre-train a visual encoder with a massive amount of driving data crawled freely from the web, such that given limited labeled data, downstream applications could generalize well and quickly adapt to various driving environments as depicted in Fig. 1(c).
Revised: We attempt to pre-train a visual encoder with a massive amount of driving data crawled freely from the web, such that given limited labeled data, downstream applications could generalize well and quickly adapt to various driving environments as depicted in Fig. 1(c).

Original: The pivotal question is how to introduce driving-decision awareness into the pre-training process to help the visual encoder concentrate on crucial visual cues for driving policy .
Revised: The pivotal question is how to introduce driving-decision awareness into the pre-training process to help the visual encoder concentrate on crucial visual cues for driving policy .

Original: One may resort to directly predicting ego-motion based on single frame sensor input, constraining the network on learning policy-related features.
Revised: One may resort to directly predicting ego-motion based on single frame sensor input, constraining the network on learning policy-related features.

Original: Previous literature tackles the supervision problem with pseudo labeling training on either an open dataset (Zhang et al., 2022b) or the target domain data (Zhang et al., 2022a).
Revised: Previous literature tackles the supervision problem with pseudo labeling training on either an open dataset (Zhang et al., 2022b) or the target domain data (Zhang et al., 2022a).

Original: However, pseudo labeling approaches suffer from noisy predictions from poorly calibrated models - this is true especially when there exists distinct domain gap such as geographical locations and traffic complexities (Rizve et al., 2020).
Revised: However, pseudo labeling approaches suffer from noisy predictions from poorly calibrated models - this is true especially when there exists distinct domain gap such as geographical locations and traffic complexities (Rizve et al., 2020).

Original: To address the bottleneck aforementioned, we propose PPGeo ( P olicy P re-training via Geo metric modeling), a fully self-supervised driving policy pre-training framework to learn from unlabeled and uncalibrated driving videos.
Revised: To address the bottleneck aforementioned, we propose PPGeo ( P olicy P re-training via Geo metric modeling), a fully self-supervised driving policy pre-training framework to learn from unlabeled and uncalibrated driving videos.

Original: It models the 3D geometric scene by jointly predicting ego-motion, depth, and camera intrinsics.
Revised: It models the 3D geometric scene by jointly predicting ego-motion, depth, and camera intrinsics.

Original: Since directly learning ego-motion based on single frame input along with depth and intrinsics training from scratch is too difficult, it is necessary to separate the visual encoder pre-training from depth and intrinsics learning in two stages.
Revised: Since directly learning ego-motion based on single frame input along with depth and intrinsics training from scratch is too difficult, it is necessary to separate the visual encoder pre-training from depth and intrinsics learning in two stages.

Original: In the first stage, the ego-motion is predicted based on consecutive frames as does in conventional depth estimation frameworks (Godard et al., 2017; 2019).

Original: In the second stage, thefuture ego-motionis estimated based on the single
Revised: In the first stage, the ego-motion

Original: [Figure]
Revised: [Figure]

Original: frame by avisual encoder, andcould beoptimized with thedepth and camera intrinsics network well-learned inthe first stage.
Revised: is predicted based on as does in depth estimation frameworks (Go dard et al., In the is estimated based by a and be with the network well-learned in

Original: As such, thevisual encoder iscapable of inferring future ego-motion based oncurrent input alone.
Revised: As such the is based on input alone.

Original: The pre-trained visual encoder could bewell adoptedfor downstream driving tasks since it captures drivingpolicy related information.
Revised: The trained could be adopted since it captures driving related information.

Original: Asa side product, the depth and pose networks could be utilized asnew initial weights for depth and odometry estimation tasks, bringing in an additional performance gain.
Revised: As and pose networks could be utilized as bringing in an additional performance gain.

Original: To sum up, our key contributions are three-fold:
Revised: To sum up, our key contributions are three-fold

Original:  • We propose a pre-training paradigm curated for various visuomotor driving tasks.
Revised: : • We propose a pre-training paradigm curated for various visuomotor driving tasks.

Original: To the best of our knowledge, this is the first attempt to achieve a fully self-supervised framework without any need of pseudo-labels 2 , leveraging the effect of pre-training by large-scale data to the full extent.
Revised: To the best of our knowledge, this is the first attempt to achieve a fully self-supervised framework without any need of pseudo-labels 2 , leveraging the effect of pre-training by large-scale data to the full extent.

Original: • We devise a visual encoder capable of predicting ego-motion based on single visual input, being able to extract feature representations closely related to driving policy.
Revised: • We devise a visual encoder capable of predicting ego-motion based on single visual input, being able to extract feature representations closely related to driving policy.

Original: Such a design of visual encoder is flexible to extend to various downstream applications.
Revised: Such a design of visual encoder is flexible to extend to various downstream applications.

Original: • We demonstrate the superiority of our approach on a set of end-to-end driving scenarios, covering different types and difficulty levels.
Revised: • We demonstrate the superiority of our approach on a set of end-to-end driving scenarios, covering different types and difficulty levels.

Original: The performance in terms of various metrics is improved from 2% to even over 100% in challenging cases with very limited data.
Revised: The performance in terms of various metrics is improved from 2% to even over 100% in challenging cases with very limited data.

Original:  M ETHODOLOGY 2. O VERVIEW
Revised:  M ETHODOLOGY 2. O VERVIEW

Original: The visuomotor policy learning for autonomous driving targets generating a policy π , such that it makes driving decisions, e.g. , control actions or planned trajectory, from visual observation x .
Revised: The visuomotor policy learning for autonomous driving targets generating a policy π , such that it makes driving decisions, e.g. , control actions or planned trajectory, from visual observation x .

Original: Our goal is to pre-train a visual encoder ϕ
Revised: Our goal is to pre-train a visual encoder ϕ

Original: ( x ) , which maps the raw image input to a compact representation containing important information for driving decision making.
Revised: ( x ) , which maps the raw image input to a compact representation containing important information for driving decision making.

Original: The representation is then utilized by the policy π ( ϕ ( x )) to perform driving tasks.
Revised: The representation is then utilized by the policy π ( ϕ ( x )) to perform driving tasks.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, our pre-training method pre-trains the visual encoder on unlabeled driving videos via two stages in a self-supervised manner.
Revised: 2, our pre-training method pre-trains the visual encoder on unlabeled driving videos via two stages in a self-supervised manner.

Original: 2. T WO - STAGE S ELF - SUPERVISED T RAINING
Revised: 2. T WO - STAGE S ELF - SUPERVISED T RAINING

Original: Stage One: Self-supervised Geometric Modeling.
Revised: Stage One: Self-supervised Geometric Modeling.

Original: During the first stage, given a target image I t and source images I t ′ in a sequence, we jointly estimate the depth of the target image, the intrinsics of the camera, and the 6-DoF ego-motion between these two frames.
Revised: During the first stage, given a target image I t and source images I t ′ in a sequence, we jointly estimate the depth of the target image, the intrinsics of the camera, and the 6-DoF ego-motion between these two frames.

Original: Given the estimations, we are able to model the 3D geometry of the scene, and reconstruct the target image by projecting pixels in the source images.
Revised: Given the estimations, we are able to model the 3D geometry of the scene, and reconstruct the target image by projecting pixels in the source images.

Original: Formally, the pixel-wise correspondence between I t and I t ′ is calculated as:
Revised: Formally, the pixel-wise correspondence between I t and I t ′ is calculated as:

Original: [Equation]
Revised: [Equation]

Original: where p t and p t ′ are the homogeneous coordinates of the pixel in I t and I t ′ respectively, K is the predicted camera intrinsic matrix, and D t ( p t ) represents the predicted depth value at pixel p i in
Revised: where p t and p t ′ are the homogeneous coordinates of the pixel in I t and I t ′ respectively, K is the predicted camera intrinsic matrix, and D t ( p t ) represents the predicted depth value at pixel p i in

Original: I t .
Revised: I t .

Original: With this relationship, the target image I t ′
Revised: With this relationship, the target image I t ′

Original: → t could be reconstructed with pixels in I t ′ , and be optimized by the photometric reconstruction error.
Revised: → t could be reconstructed with pixels in I t ′ , and be optimized by the photometric reconstruction error.

Original: Following Godard et al.
Revised: Following Godard et al.

Original: (2019), we choose two images adjacent to the current frame as the source images, i.e. , t ′ ∈ { t − 1 , t + 1 } .
Revised: (2019), we choose two images adjacent to the current frame as the source images, i.e. , t ′ ∈ { t − 1 , t + 1 } .

Original: The DepthNet consists of a common encoder-decoder structure (Godard et al., 2019) and estimates the depth map of the input image.
Revised: The DepthNet consists of a common encoder-decoder structure (Godard et al., 2019) and estimates the depth map of the input image.

Original: Two images are stacked together and fed into the encoder of the PoseNet, whose bottleneck feature is then utilized to predict the camera intrinsics and the egomotion via two separate MLP-based heads.
Revised: Two images are stacked together and fed into the encoder of the PoseNet, whose bottleneck feature is then utilized to predict the camera intrinsics and the egomotion via two separate MLP-based heads.

Original: For camera intrinsics estimation, optical center ( c x , c y ) and focal lengths f x , f y are regressed similarly as in Gordon et al.
Revised: For camera intrinsics estimation, optical center ( c x , c y ) and focal lengths f x , f y are regressed similarly as in Gordon et al.

Original: ; Chanduri et al.
Revised: ; Chanduri et al.

Original: Stage Two: Visuomotor Policy Pre-training.
Revised: Stage Two: Visuomotor Policy Pre-training.

Original: After the first stage of training, the DepthNet and
Revised: After the first stage of training, the DepthNet and

Original: PoseNet are well trained and fitted to the driving video data.
Revised: PoseNet are well trained and fitted to the driving video data.

Original: Then, in the second stage, we replace the PoseNet for ego-motion estimation with the visual encoder ϕ
Revised: Then, in the second stage, we replace the PoseNet for ego-motion estimation with the visual encoder ϕ

Original: ( x ) prepared for downstream driving policy learning tasks.
Revised: ( x ) prepared for downstream driving policy learning tasks.

Original: Now the visual encoder only takes a single frame image as input and predicts ego-motion between the current frame and subsequent frame.
Revised: Now the visual encoder only takes a single frame image as input and predicts ego-motion between the current frame and subsequent frame.

Original: Specifically, the visual encoder estimates the ego-motion T t → t +1 based on I t alone and T t → t −based on I t − 1 followed by an inverse operation, respectively.
Revised: Specifically, the visual encoder estimates the ego-motion T t → t +1 based on I t alone and T t → t −based on I t − 1 followed by an inverse operation, respectively.

Original: The visual encoder is optimized by the photometric reconstruction error similar to the first stage, aside from a modification where the DepthNet and the intrinsics estimation are frozen and not backpropagated.
Revised: The visual encoder is optimized by the photometric reconstruction error similar to the first stage, aside from a modification where the DepthNet and the intrinsics estimation are frozen and not backpropagated.

Original: This is empirically observed towards better performance.
Revised: This is empirically observed towards better performance.

Original: By doing so, the visual encoder is enforced to learn the actual driving policy, since the ego-motion between two consecutive frames is straightforwardly related to the driving decision or action taken at the current timestamp.
Revised: By doing so, the visual encoder is enforced to learn the actual driving policy, since the ego-motion between two consecutive frames is straightforwardly related to the driving decision or action taken at the current timestamp.

Original: One might argue that the PoseNet trained in the first stage could provide pseudo motion labels, with which the visual encoder could be directly supervised.
Revised: One might argue that the PoseNet trained in the first stage could provide pseudo motion labels, with which the visual encoder could be directly supervised.

Original: However, the ego-motion predicted from the PoseNet is too sparse compared with the geometric projection approach.
Revised: However, the ego-motion predicted from the PoseNet is too sparse compared with the geometric projection approach.

Original: In our pipeline, every pixel provides supervision for the visual encoder so that inaccurate depth estimation in some pixels could be mitigated by the accurate ones, i.e. , it constructs a “global” optimization.
Revised: In our pipeline, every pixel provides supervision for the visual encoder so that inaccurate depth estimation in some pixels could be mitigated by the accurate ones, i.e. , it constructs a “global” optimization.

Original: In contrast, direct supervision from the PoseNet would be greatly affected by the undesirable prediction inaccuracy and noise results.
Revised: In contrast, direct supervision from the PoseNet would be greatly affected by the undesirable prediction inaccuracy and noise results.

Original: Thus far, the backbone of visual encoder ϕ
Revised: Thus far, the backbone of visual encoder ϕ

Original: ( x ) has gained knowledge about the driving policy from the diverse driving videos.
Revised: ( x ) has gained knowledge about the driving policy from the diverse driving videos.

Original: It can then be applied to downstream visuomotor autonomous driving tasks as the initial weights.
Revised: It can then be applied to downstream visuomotor autonomous driving tasks as the initial weights.

Original: Besides, the DepthNet and PoseNet trained on this large corpus of uncalibrated video data could also be utilized in depth and odometry estimation tasks.
Revised: Besides, the DepthNet and PoseNet trained on this large corpus of uncalibrated video data could also be utilized in depth and odometry estimation tasks.

Original: 2. L OSS F UNCTION
Revised: 2. L OSS F UNCTION

Original: Following Godard et al.
Revised: Following Godard et al.

Original: (2019), the loss function is comprised of the photometric loss and the smoothness loss.
Revised: (2019), the loss function is comprised of the photometric loss and the smoothness loss.

Original: The photometric error is comprised of an ℓ 1 term and an SSIM (structural similarity index measure) term
Revised: The photometric error is comprised of an ℓ 1 term and an SSIM (structural similarity index measure) term

Original: (Wang et al., 2004):
Revised: (Wang et al., 2004):

Original: [Equation]
Revised: [Equation]

Original: where we set α = 0 . 85 following the practice (Godard et al., 2017; 2019).
Revised: where we set α = 0 . 85 following the practice (Godard et al., 2017; 2019).

Original: The smooth loss is:
Revised: The smooth loss is:

Original: [Equation]
Revised: [Equation]

Original: where d ∗ t is the mean-normalized inverse depth map.
Revised: where d ∗ t is the mean-normalized inverse depth map.

Original: We also adopt the minimum reprojection loss and auto-masking scheme (Godard et al., 2019) to improve self-supervised depth estimation.
Revised: We also adopt the minimum reprojection loss and auto-masking scheme (Godard et al., 2019) to improve self-supervised depth estimation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: All pre-training experiments are conducted on the hours-long unlabeled YouTube driving videos (Zhang et al., 2022b).
Revised: All pre-training experiments are conducted on the hours-long unlabeled YouTube driving videos (Zhang et al., 2022b).

Original: It covers different driving conditions e.g. , geographical locations and weather.
Revised: It covers different driving conditions e.g. , geographical locations and weather.

Original: We sample 0.8 million frames in total at 1 Hz for training.
Revised: We sample 0.8 million frames in total at 1 Hz for training.

Original: For the first stage in PPGeo pipeline, we train the model for 30 epochs by Adam (Kingma & Ba, 2015) optimizer with a learning rate of 10 − 4 which drops to 10 − 5 after 25 epochs.
Revised: For the first stage in PPGeo pipeline, we train the model for 30 epochs by Adam (Kingma & Ba, 2015) optimizer with a learning rate of 10 − 4 which drops to 10 − 5 after 25 epochs.

Original: For the second stage, the encoder is trained forepochs using the AdamW (Loshchilov & Hutter, 2017) optimizer.
Revised: For the second stage, the encoder is trained forepochs using the AdamW (Loshchilov & Hutter, 2017) optimizer.

Original: A cyclic learning rate scheduler is applied with the learning rate ranging from 10 − 6 to 10 − 4 .
Revised: A cyclic learning rate scheduler is applied with the learning rate ranging from 10 − 6 to 10 − 4 .

Original: The batch size for both stages is 128.
Revised: The batch size for both stages is 128.

Original: We use data augmentations including ColorJitter, RamdomGrayScale, and GaussianBlur.
Revised: We use data augmentations including ColorJitter, RamdomGrayScale, and GaussianBlur.

Original: 3. D ESCRIPTION ON C OMPARED B ASELINES
Revised: 3. D ESCRIPTION ON C OMPARED B ASELINES

Original: We use ResNet-34 (He et al., 2016) as the encoder and load different pre-trained weights for the initialization of downstream tasks.
Revised: We use ResNet-34 (He et al., 2016) as the encoder and load different pre-trained weights for the initialization of downstream tasks.

Original: We compare PPGeo with pre-training methods including:
Revised: We compare PPGeo with pre-training methods including:

Original: Random.
Revised: Random.

Original: We use the default Kaiming initialization (He et al., 2015) for convolution layers and constant initialization for batchnorms.
Revised: We use the default Kaiming initialization (He et al., 2015) for convolution layers and constant initialization for batchnorms.

Original: ImageNet.
Revised: ImageNet.

Original: We use the model weight provided by Torchvision (Marcel & Rodriguez, 2010), which is pre-trained with the classification task on ImageNet (Deng et al., 2009).
Revised: We use the model weight provided by Torchvision (Marcel & Rodriguez, 2010), which is pre-trained with the classification task on ImageNet (Deng et al., 2009).

Original: MIM.
Revised: MIM.

Original: The model is pre-trained with the masked image modeling method on the YouTube driving video, which tries to reconstruct images with random masked-out patches.
Revised: The model is pre-trained with the masked image modeling method on the YouTube driving video, which tries to reconstruct images with random masked-out patches.

Original: SimMIM (Xie et al., 2022) is adopted as it is suitable for convolutional networks.
Revised: SimMIM (Xie et al., 2022) is adopted as it is suitable for convolutional networks.

Original: MoCo.
Revised: MoCo.

Original: We pre-train the model using MoCo-v2 (Chen et al., 2020c) on the YouTube driving videos.
Revised: We pre-train the model using MoCo-v2 (Chen et al., 2020c) on the YouTube driving videos.

Original: We exclude RandomResizedCrop and RandomHorizontalFlip augmentations as they are not suitable for the driving task.
Revised: We exclude RandomResizedCrop and RandomHorizontalFlip augmentations as they are not suitable for the driving task.

Original: ACO.
Revised: ACO.

Original: Following Zhang et al.
Revised: Following Zhang et al.

Original: (2022b)
Revised: (2022b)

Original: , it is pre-trained using action-conditioned contrastive learning on the YouTube driving videos.
Revised: , it is pre-trained using action-conditioned contrastive learning on the YouTube driving videos.

Original: ACO trains an inverse dynamic model to generate pseudo steer labels for driving videos, based on which steer-based discrimination is added on top of MoCo-v2.
Revised: ACO trains an inverse dynamic model to generate pseudo steer labels for driving videos, based on which steer-based discrimination is added on top of MoCo-v2.

Original: SelfD. SelfD (Zhang et al., 2022a) is not a pre-training method strictly since it needs to train the whole policy model on the driving video for each task, while other pre-training methods aforementioned provide a general pre-training visual model for all tasks.
Revised: SelfD. SelfD (Zhang et al., 2022a) is not a pre-training method strictly since it needs to train the whole policy model on the driving video for each task, while other pre-training methods aforementioned provide a general pre-training visual model for all tasks.

Original: We still include it for comparison due to its close relationship to our target.
Revised: We still include it for comparison due to its close relationship to our target.

Original: Specifically, we follow Zhang et al.
Revised: Specifically, we follow Zhang et al.

Original: (2022a) to train the model for each task with the following pipeline: training on the task data → training on the YouTube data with pseudo-label → fine-tuning on the task data.
Revised: (2022a) to train the model for each task with the following pipeline: training on the task data → training on the YouTube data with pseudo-label → fine-tuning on the task data.

Original: 3. D ESCRIPTION ON D OWNSTREAM A UTONOMOUS D RIVING T ASKS
Revised: 3. D ESCRIPTION ON D OWNSTREAM A UTONOMOUS D RIVING T ASKS

Original: We carry out experiments under (1) three imitation learning based closed-loop driving tasks in
Revised: We carry out experiments under (1) three imitation learning based closed-loop driving tasks in

Original: CARLA (Dosovitskiy et al., 2017),
Revised: CARLA (Dosovitskiy et al., 2017),

Original: (2) one reinforcement learning based driving task in CARLA, and (3) an open-loop planning task on real-world autonomous driving dataset nuScenes (Caesar et al., 2020), to fully validate the effectiveness of PPGeo.
Revised: (2) one reinforcement learning based driving task in CARLA, and (3) an open-loop planning task on real-world autonomous driving dataset nuScenes (Caesar et al., 2020), to fully validate the effectiveness of PPGeo.

Original: We briefly describe each task below.
Revised: We briefly describe each task below.

Original: Navigation.
Revised: Navigation.

Original: It corresponds to the goal-conditioned navigation task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).
Revised: It corresponds to the goal-conditioned navigation task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).

Original: The agent is trained in Town01 and tested in Town02 with unseen weather, and there are no other traffic participants.
Revised: The agent is trained in Town01 and tested in Town02 with unseen weather, and there are no other traffic participants.

Original: We use different sizes of training data (from 4K to 40K) following Zhang et al.
Revised: We use different sizes of training data (from 4K to 40K) following Zhang et al.

Original: (2022b) to evaluate the generalization ability of pre-trained visual encoders when labeled data is limited and conduct the closed-loop evaluation.
Revised: (2022b) to evaluate the generalization ability of pre-trained visual encoders when labeled data is limited and conduct the closed-loop evaluation.

Original: The evaluation metric is success rate, denoting the portion of 50 pre-defined routes finished without any collision.
Revised: The evaluation metric is success rate, denoting the portion of 50 pre-defined routes finished without any collision.

Original: And traffic lights are ignored here.
Revised: And traffic lights are ignored here.

Original: CILRS (Codevilla et al., 2019), a classic image based end-to-end autonomous driving model, is adopted for training and evaluation.
Revised: CILRS (Codevilla et al., 2019), a classic image based end-to-end autonomous driving model, is adopted for training and evaluation.

Original: Navigation Dynamic.
Revised: Navigation Dynamic.

Original: This is the navigation dynamic task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).
Revised: This is the navigation dynamic task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).

Original: The setting differentiates from Navigation that there are other dynamic objects such as randomly generated vehicles, which substantially increases the difficulty of driving safety.
Revised: The setting differentiates from Navigation that there are other dynamic objects such as randomly generated vehicles, which substantially increases the difficulty of driving safety.

Original: Leaderboard Town05-long.
Revised: Leaderboard Town05-long.

Original: This challenging and realistic benchmark corresponds to the LeaderBoard benchmark (CARLA, 2022).
Revised: This challenging and realistic benchmark corresponds to the LeaderBoard benchmark (CARLA, 2022).

Original: We collect 40K training data in Town01, 03, 04, 06 and evaluate on 10 routes in the unseen Town05 (Prakash et al., 2021).
Revised: We collect 40K training data in Town01, 03, 04, 06 and evaluate on 10 routes in the unseen Town05 (Prakash et al., 2021).

Original: Due to the challenging scenarios in this
Revised: Due to the challenging scenarios in this

Original: [Table]
Revised: [Table]

Original: model TCP (Wu et al., 2022).
Revised: model TCP (Wu et al., 2022).

Original: The metrics of this task are Driving Score, Route Completion, and Infraction Score (all the higher the better), where the main metric Driving Score is the product of Route Completion
Revised: The metrics of this task are Driving Score, Route Completion, and Infraction Score (all the higher the better), where the main metric Driving Score is the product of Route Completion

Original: Infraction Score
Revised: Infraction Score

Original: Reinforcement Learning.
Revised: Reinforcement Learning.

Original: Proximal Policy Optimization (PPO) (Schulman et al., 2017) is used to train the CILRS (Codevilla et al., 2019) model initialized with different pre-trained weights in
Revised: Proximal Policy Optimization (PPO) (Schulman et al., 2017) is used to train the CILRS (Codevilla et al., 2019) model initialized with different pre-trained weights in

Original: CARLA Town01 environment.
Revised: CARLA Town01 environment.

Original: The reward shaping details follow Roach (Zhang et al., 2021).
Revised: The reward shaping details follow Roach (Zhang et al., 2021).

Original: We also conduct experiments to freeze the pre-trained visual encoder during training to further study the effectiveness of the pre-trained feature representations.
Revised: We also conduct experiments to freeze the pre-trained visual encoder during training to further study the effectiveness of the pre-trained feature representations.

Original: nuScenes Planning.
Revised: nuScenes Planning.

Original: This task involves trajectory planning in real-world dataset nuScenes (Caesar et al., 2020).
Revised: This task involves trajectory planning in real-world dataset nuScenes (Caesar et al., 2020).

Original: Given the current visual input, the model plans a 3-second trajectory (0.5 Hz), and the planned trajectory is compared with the ground truth log.
Revised: Given the current visual input, the model plans a 3-second trajectory (0.5 Hz), and the planned trajectory is compared with the ground truth log.

Original: We also calculate the collision rate, where a collision is defined as overlaps with future vehicles and pedestrians based on planned waypoints.
Revised: We also calculate the collision rate, where a collision is defined as overlaps with future vehicles and pedestrians based on planned waypoints.

Original: The planning model used here is comprised of a visual encoder and a GRU-based planner to predict each waypoint auto-regressively.
Revised: The planning model used here is comprised of a visual encoder and a GRU-based planner to predict each waypoint auto-regressively.

Original: We use the official train-val split for training and evaluation.
Revised: We use the official train-val split for training and evaluation.

Original: 3. N UMERIC C OMPARISON ON D OWNSTREAM T ASKS
Revised: 3. N UMERIC C OMPARISON ON D OWNSTREAM T ASKS

Original: For imitation learning based closed-loop driving tasks, the evaluation results are shown in Table 13.
Revised: For imitation learning based closed-loop driving tasks, the evaluation results are shown in Table 13.

Original: We present the plot between episode return and environment steps of each method in Fig.
Revised: We present the plot between episode return and environment steps of each method in Fig.

Original: 3 for the reinforcement learning experiments.
Revised: 3 for the reinforcement learning experiments.

Original: The open-loop nuScenes planning results are provided in
Revised: The open-loop nuScenes planning results are provided in

Original: Table 4.
Revised: Table 4.

Original: We could observe that PPGeo outperforms other baselines by a large margin in all tasks.
Revised: We could observe that PPGeo outperforms other baselines by a large margin in all tasks.

Original: Note that the model is tested under a different number of fine-tuning samples from 10% (4K) to full 40K in the Navigation and Navigation Dynamic tasks.
Revised: Note that the model is tested under a different number of fine-tuning samples from 10% (4K) to full 40K in the Navigation and Navigation Dynamic tasks.

Original: In the case of the particularly small size of training samples, PPGeo still demonstrates competitive performance and has a larger improvement gap of over 100%.
Revised: In the case of the particularly small size of training samples, PPGeo still demonstrates competitive performance and has a larger improvement gap of over 100%.

Original: This validates the generalization ability of the pre-trained visual encoder, which is important when adapting to a new environment with very limited labeled data.
Revised: This validates the generalization ability of the pre-trained visual encoder, which is important when adapting to a new environment with very limited labeled data.

Original: In the more challenging and real-world style Leaderboard Town05-long task in Table 3, the model pre-trained with our method achieves the highest driving score and infraction score.
Revised: In the more challenging and real-world style Leaderboard Town05-long task in Table 3, the model pre-trained with our method achieves the highest driving score and infraction score.

Original: PPGeo well handles cases where the agent needs to stop, leading to much fewer vehicle collisions and red light infractions.
Revised: PPGeo well handles cases where the agent needs to stop, leading to much fewer vehicle collisions and red light infractions.

Original: Since ACO considers steering angles only during pre-training, its performance degrades on more challenging scenarios where brake and throttles are also important.
Revised: Since ACO considers steering angles only during pre-training, its performance degrades on more challenging scenarios where brake and throttles are also important.

Original: SelfD performs slightly better than ACO in complex cases while it significantly degenerates when the task data is limited, as affected by the unsatisfying pseudo labeling model.
Revised: SelfD performs slightly better than ACO in complex cases while it significantly degenerates when the task data is limited, as affected by the unsatisfying pseudo labeling model.

Original: ImageNet pre-training also shows competitive performance, which might credit to its ability of finding salient objects in the scene when the input contains little irrelevant information (see examples in Sec. 3.5).
Revised: ImageNet pre-training also shows competitive performance, which might credit to its ability of finding salient objects in the scene when the input contains little irrelevant information (see examples in Sec. 3.5).

Original: 3. D EPTH AND O DOMETRY E STIMATION
Revised: 3. D EPTH AND O DOMETRY E STIMATION

Original: In this part, we explore whether the large-scale training on uncalibrated data could benefit the depth and odometry estimation models as well and validate the effectiveness of first-stage training.
Revised: In this part, we explore whether the large-scale training on uncalibrated data could benefit the depth and odometry estimation models as well and validate the effectiveness of first-stage training.

Original: Specifically, we employ the DepthNet and PoseNet trained after the first stage as initial weights for Monodepthv

Revised: Specifically, we employ the DepthNet and PoseNet trained after the first stage as initial weights for Monodepthv

Original: (Godard et al., 2019), and conduct experiments on KITTI (Geiger et al., 2012).
Revised: (Godard et al., 2019), and conduct experiments on KITTI (Geiger et al., 2012).

Original: Results in Table 5 indicate that pre-training on large-scale driving videos could bring performance improvement to both depth and odometry estimation tasks, which is an additional harvest of our pre-training framework.
Revised: Results in Table 5 indicate that pre-training on large-scale driving videos could bring performance improvement to both depth and odometry estimation tasks, which is an additional harvest of our pre-training framework.

Original: We refer readers to Godard et al.
Revised: We refer readers to Godard et al.

Original: for details about the metrics of these tasks.
Revised: for details about the metrics of these tasks.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:         6WHSV.  
Revised:         6WHSV.  

Original:      ( S L V RG H  5 H W X U Q 9LVXDO(QFRGHU)LQHWXQLQJ ,PDJH1HW 0R&R $&2 33*HR         6WHSV.    
Revised:      ( S L V RG H  5 H W X U Q 9LVXDO(QFRGHU)LQHWXQLQJ ,PDJH1HW 0R&R $&2 33*HR         6WHSV.    

Original: [Table]
Revised: [Table]

Original: H
Revised: H

Original: [Table]
Revised: [Table]

Original: 9LVXDO(QFRGHU)UR]HQ
Revised: 9LVXDO(QFRGHU)UR]HQ

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 3. V ISUALIZATION R ESULTS
Revised: 3. V ISUALIZATION R ESULTS

Original: Here we provide heatmaps of the feature representations learned by different pre-training methods using Eigen-Cam (Muhammad & Yeasin, 2020) to show the attended regions in Fig. 4.
Revised: Here we provide heatmaps of the feature representations learned by different pre-training methods using Eigen-Cam (Muhammad & Yeasin, 2020) to show the attended regions in Fig. 4.

Original: In many cases (Row 1&2), our model mainly concentrates on the lane in front of the ego vehicle, which is highly related to driving.
Revised: In many cases (Row 1&2), our model mainly concentrates on the lane in front of the ego vehicle, which is highly related to driving.

Original: And our model PPGeo well captures the specific cues causing the brake
Revised: And our model PPGeo well captures the specific cues causing the brake

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: action including front vehicles (Row 3&4) and traffic lights (Row 5).
Revised: action including front vehicles (Row 3&4) and traffic lights (Row 5).

Original: We also observe that the model pre-trained with ImageNet classification tends to capture salient objects in the image.
Revised: We also observe that the model pre-trained with ImageNet classification tends to capture salient objects in the image.

Original: This is helpful when the salient objects are straightforwardly related to the driving decision (Row 4); but it may focus on wrong objects when the input contains other irrelevant information (Row 2&3). 3. A BLATIVE S TUDY
Revised: This is helpful when the salient objects are straightforwardly related to the driving decision (Row 4); but it may focus on wrong objects when the input contains other irrelevant information (Row 2&3). 3. A BLATIVE S TUDY

Original: We conduct ablative study as to different designs of PPGeo on the Navigation task in Table 6.
Revised: We conduct ablative study as to different designs of PPGeo on the Navigation task in Table 6.

Original: Training the visual encoder and DepthNet in a single stage simultaneously (Row 1) leads to an inferior performance, indicating that it is quite challenging for the visual encoder to learn the correct egomotion if depth estimation is also trained from scratch.
Revised: Training the visual encoder and DepthNet in a single stage simultaneously (Row 1) leads to an inferior performance, indicating that it is quite challenging for the visual encoder to learn the correct egomotion if depth estimation is also trained from scratch.

Original: Moreover, jointly optimizing the DepthNet in the second stage (Row 2, not frozen) degrades the depth estimation quality and harms the performance.
Revised: Moreover, jointly optimizing the DepthNet in the second stage (Row 2, not frozen) degrades the depth estimation quality and harms the performance.

Original: In Row 3, we observe that utilizing the PoseNet obtained in the first stage to provide pseudo label supervision directly leads to inferior results, since an inaccurate pseudo label impairs the learning process to great extent.
Revised: In Row 3, we observe that utilizing the PoseNet obtained in the first stage to provide pseudo label supervision directly leads to inferior results, since an inaccurate pseudo label impairs the learning process to great extent.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Pre-training for NLP and General Vision.
Revised: Pre-training for NLP and General Vision.

Original: Pre-training or representation learning has proved to be an essential key to the success of artificial intelligence.
Revised: Pre-training or representation learning has proved to be an essential key to the success of artificial intelligence.

Original: In the field of Natural Language Processing (NLP), with the powerful capability of Transformer (Vaswani et al., 2017), pre-training on largescale datasets with large models then fine-tuning on downstream tasks has become the dominant paradigm (Kenton & Toutanova, 2019; Brown et al., 2020).
Revised: In the field of Natural Language Processing (NLP), with the powerful capability of Transformer (Vaswani et al., 2017), pre-training on largescale datasets with large models then fine-tuning on downstream tasks has become the dominant paradigm (Kenton & Toutanova, 2019; Brown et al., 2020).

Original: As for the field of Computer Vision, training specific downstream tasks with the supervised pre-trained weights of visual encoder via
Revised: As for the field of Computer Vision, training specific downstream tasks with the supervised pre-trained weights of visual encoder via

Original: ImageNet classification task is widely adopted.
Revised: ImageNet classification task is widely adopted.

Original: Recently, unsupervised and self-supervised learning methods such as contrastive learning (He et al., 2020; Chen et al., 2020c;b) and masked image modeling (Bao et al., 2021;
Revised: Recently, unsupervised and self-supervised learning methods such as contrastive learning (He et al., 2020; Chen et al., 2020c;b) and masked image modeling (Bao et al., 2021;

Original: He et al., 2022; Xie et al., 2022; Peng et al.,  2022) have gained impressive improvement over ImageNet pre-training on various vision benchmarks.
Revised: He et al., 2022; Xie et al., 2022; Peng et al., 2022; Xue et al., 2022) have gained impressive improvement over ImageNet pre-training on various vision benchmarks.

Original: Very recent vision-language co-training approaches (Radford et al., 2021;
Revised: Very recent vision-language co-training approaches (Radford et al., 2021;

Original: Wang et al., 2022) demonstrate their extraordinary potential in the domain of multi-modal learning and applications.
Revised: Wang et al., 2022) demonstrate their extraordinary potential in the domain of multi-modal learning and applications.

Original: Yet, these generic representation learning methods adopt various data augmentation techniques to achieve translation and view invariance, while visuomotor driving sets in a highly dynamic environment.
Revised: Yet, these generic representation learning methods adopt various data augmentation techniques to achieve translation and view invariance, while visuomotor driving sets in a highly dynamic environment.

Original: In this work, we show that the ever-victorious pre-training methods may not be the optimal choice, and introduce a curated paradigm for visuomotor driving policy learning.
Revised: In this work, we show that the ever-victorious pre-training methods may not be the optimal choice, and introduce a curated paradigm for visuomotor driving policy learning.

Original: Pre-training for Visuomotor Applications.
Revised: Pre-training for Visuomotor Applications.

Original: Learning a control policy directly from raw visual input is challenging since the model needs to reason about visual pixels and dynamic behaviors simultaneously.
Revised: Learning a control policy directly from raw visual input is challenging since the model needs to reason about visual pixels and dynamic behaviors simultaneously.

Original: Moreover, training visuomotor models from scratch usually requires tons of labeled data or environment interactions.
Revised: Moreover, training visuomotor models from scratch usually requires tons of labeled data or environment interactions.

Original: To this end, recently, Shah & Kumar (2021) shows that feature representations from ResNet (He et al., 2016) pre-trained on ImageNet classification is helpful for
Revised: To this end, recently, Shah & Kumar (2021) shows that feature representations from ResNet (He et al., 2016) pre-trained on ImageNet classification is helpful for

Original: RL-based dexterous manipulation tasks.
Revised: RL-based dexterous manipulation tasks.

Original: Parisi et al.
Revised: Parisi et al.

Original: (2022) conducts extensive experiments on applying “off-the-shelf” pre-trained vision models in diverse control domains and validates their benefits to train control policies.
Revised: (2022) conducts extensive experiments on applying “off-the-shelf” pre-trained vision models in diverse control domains and validates their benefits to train control policies.

Original: CLIP (Radford et al., 2021) is also adopted in some embodied
Revised: CLIP (Radford et al., 2021) is also adopted in some embodied

Original: AI and robot navigation problems (Shah et al., 2022).
Revised: AI and robot navigation problems (Shah et al., 2022).

Original: Besides borrowing pre-trained weights for visuomotor tasks, researchers in robotics now desire a paradigm learning policy representations from raw data directly.
Revised: Besides borrowing pre-trained weights for visuomotor tasks, researchers in robotics now desire a paradigm learning policy representations from raw data directly.

Original: Xiao et al.
Revised: Xiao et al.

Original: ; Radosavovic et al.
Revised: ; Radosavovic et al.

Original: ; Seo et al.
Revised: ; Seo et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: (2022) inherit the MIM spirit to realize visual pre-training for control tasks.
Revised: (2022) inherit the MIM spirit to realize visual pre-training for control tasks.

Original: Yang & Nachum (2021) investigates unsupervised representation learning objectives from D4RL environments (Fu et al., 2020), and Yamada et al.
Revised: Yang & Nachum (2021) investigates unsupervised representation learning objectives from D4RL environments (Fu et al., 2020), and Yamada et al.

Original: (2022) further adopts task-induced approaches to learn from prior tasks.
Revised: (2022) further adopts task-induced approaches to learn from prior tasks.

Original: However, compared with visuomotor driving, the visual inputs of such control tasks are less diverse which usually concentrate on objects and are much more compact.
Revised: However, compared with visuomotor driving, the visual inputs of such control tasks are less diverse which usually concentrate on objects and are much more compact.

Original: To our best knowledge, ACO (Zhang et al., 2022b) is the only pre-training method customized for autonomous driving.
Revised: To our best knowledge, ACO (Zhang et al., 2022b) is the only pre-training method customized for autonomous driving.

Original: By first training an inverse dynamic model on nuScenes (Caesar et al., 2020), they get pseudo steer labels of the driving videos and then construct the steer-conditioned discrimination for contrastive learning following MoCo.
Revised: By first training an inverse dynamic model on nuScenes (Caesar et al., 2020), they get pseudo steer labels of the driving videos and then construct the steer-conditioned discrimination for contrastive learning following MoCo.

Original: However, ACO ignores other crucial driving factors such as throttle and brakes, and its performance is largely limited by the inverse dynamic model.
Revised: However, ACO ignores other crucial driving factors such as throttle and brakes, and its performance is largely limited by the inverse dynamic model.

Original: SelfD (Zhang et al., 2022a) is not strictly designed for pre-training while it also makes use of vast amounts of videos to learn driving policies via semi-supervised learning.
Revised: SelfD (Zhang et al., 2022a) is not strictly designed for pre-training while it also makes use of vast amounts of videos to learn driving policies via semi-supervised learning.

Original: It acquires the pseudo labeling knowledge from the target domain.
Revised: It acquires the pseudo labeling knowledge from the target domain.

Original: These two methods both depend on the accuracy of pseudo labeling.
Revised: These two methods both depend on the accuracy of pseudo labeling.

Original: In contrast, we realize fully self-supervised learning through dense geometric reconstruction, evading the possible adverse effect.
Revised: In contrast, we realize fully self-supervised learning through dense geometric reconstruction, evading the possible adverse effect.

Original: Policy Learning for Autonomous Driving.
Revised: Policy Learning for Autonomous Driving.

Original: Visuomotor autonomous driving learns a driving policy directly from sensor inputs in an end-to-end manner (Codevilla et al., 2018; 2019;
Revised: Visuomotor autonomous driving learns a driving policy directly from sensor inputs in an end-to-end manner (Codevilla et al., 2018; 2019;

Original: Liang et al., 2018; Chen et al., 2020a; Prakash et al., 2021; Chen et al., 2021; Wu et al., 2022;Shao et al., 2022).
Revised: Liang et al., 2018; Chen et al., 2020a; Prakash et al., 2021; Chen et al., 2021; Wu et al., 2022;

Original: In essence, the inherent difficulty of the urban-style autonomous driving tasks makes such methods data-hungry.
Revised: Shao et al., 2022;

Original: Interfuser (Shao et al., 2022), the current top-1 method on the CARLA Leaderboard (CARLA, 2022), requires 3 million labeled data samples for imitation learning (behavior cloning specifically).
Revised: Hu et al., 2022; 2023).

Original: RL-based model MaRLn (Toromanoff et al., 2020) needs 20 million environment steps of interaction.
Revised: In essence, the inherent difficulty of the urban-style autonomous driving tasks makes such methods data-hungry.

Original:  The sample efficiency problem greatly impedes the real-world application of such approaches.
Revised: Interfuser (Shao et al., 2022), the current top-1 method on the CARLA Leaderboard (CARLA, 2022), requires 3 million labeled data samples for imitation learning (behavior cloning specifically). RL-based model MaRLn (Toromanoff et al., 2020) needs 20 million environment steps of interaction. The sample efficiency problem greatly impedes the real-world application of such approaches.

Original: In this work, we propose a self-supervised pre-training pipeline to learn driving policy related representations on unlabeled driving videos, and pave the way for these visuomotor autonomous driving models to further achieve satisfying performance.
Revised: In this work, we propose a self-supervised pre-training pipeline to learn driving policy related representations on unlabeled driving videos, and pave the way for these visuomotor autonomous driving models to further achieve satisfying performance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have proposed a fully self-supervised visuomotor driving policy pre-training paradigm PPGeo by modeling the 3D geometry of large-scale unlabeled driving videos.
Revised: In this work, we have proposed a fully self-supervised visuomotor driving policy pre-training paradigm PPGeo by modeling the 3D geometry of large-scale unlabeled driving videos.

Original: Taking a direct approach to infer the ego-motion and benefiting from the two-stage pre-training pipeline, we enable the visual encoder to learn driving policies based on single visual input.
Revised: Taking a direct approach to infer the ego-motion and benefiting from the two-stage pre-training pipeline, we enable the visual encoder to learn driving policies based on single visual input.

Original: Our method outperforms the peer pre-training approaches by a large margin on a series of visuomotor driving tasks.
Revised: Our method outperforms the peer pre-training approaches by a large margin on a series of visuomotor driving tasks.