Original: P OLICY P RE - TRAINING FOR A UTONOMOUS D RIVING VIA S ELF - SUPERVISED G EOMETRIC M ODELING
Revised: P OLICY P RE - TRAINING FOR A UTONOMOUS D RIVING VIA S ELF - SUPERVISED G EOMETRIC M ODELING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving.
Revised: Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving.

Original: Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task.
Revised: Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task.

Original: To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving.
Revised: To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving.

Original: We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos.
Revised: We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos.

Original: The proposed PPGeo is performed in two stages to support effective self-supervised training.
Revised: The proposed PPGeo is performed in two stages to support effective self-supervised training.

Original: In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input.
Revised: In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input.

Original: In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.
Revised: In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.

Original: As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks.
Revised: As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks.

Original: As a side product, the pre-trained geometric modeling networks could bring further improvement to the depth and odometry estimation tasks.
Revised: As a side product, the pre-trained geometric modeling networks could bring further improvement to the depth and odometry estimation tasks.

Original: Extensive experiments covering a wide span of challenging scenarios have demonstrated the superiority of our proposed approach, where improvements range from 2% to even over 100% with very limited data.
Revised: Extensive experiments covering a wide span of challenging scenarios have demonstrated the superiority of our proposed approach, where improvements range from 2% to even over 100% with very limited data.

Original: Code and models would be made public.
Revised: Code and models would be made public.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Policy learning refers to the learning process of an autonomous agent acquiring the decision-making policy to perform a certain task in a particular environment.
Revised: Policy learning refers to the learning process of an autonomous agent acquiring the decision-making policy to perform a certain task in a particular environment.

Original: Visuomotor policy learning (Mnih et al., 2015; Levine et al., 2016; Hessel et al., 2018; Laskin et al., 2020; Toromanoff et al., 2020) takes as input raw sensor observations and predicts the action, simultaneously cooperating and training the perception and control modules in an end-to-end fashion.
Revised: Visuomotor policy learning (Mnih et al., 2015; Levine et al., 2016; Hessel et al., 2018; Laskin et al., 2020; Toromanoff et al., 2020) takes as input raw sensor observations and predicts the action, simultaneously cooperating and training the perception and control modules in an end-to-end fashion.

Original: For visuomotor policy models, learning tabula rasa is difficult, where it usually requires a prohibitively large corpus of labeled data or environment interactions to achieve satisfactory performance (Espeholt et al., 2018;
Revised: For visuomotor policy models, learning tabula rasa is difficult, where it usually requires a prohibitively large corpus of labeled data or environment interactions to achieve satisfactory performance (Espeholt et al., 2018;

Original: Wijmans et al., 2019; Yarats et al., 2020).
Revised: Wijmans et al., 2019; Yarats et al., 2020).

Original: To mitigate the sample efficiency caveat in visuomotor policy learning, pre-training the visual perception network in advance is a promising solution.
Revised: To mitigate the sample efficiency caveat in visuomotor policy learning, pre-training the visual perception network in advance is a promising solution.

Original: Recent studies (Shah & Kumar, 2021; Parisi et al., 2022; Xiao et al., 2022; Radosavovic et al., 2022; Shah et al., 2022) have demonstrated that applying popular visual pre-training approaches, including ImageNet (Deng et al., 2009) classification, contrastive learning (He et al., 2020; Chen et al., 2020c), masked image modeling (MIM)
Revised: Recent studies (Shah & Kumar, 2021; Parisi et al., 2022; Xiao et al., 2022; Radosavovic et al., 2022; Shah et al., 2022) have demonstrated that applying popular visual pre-training approaches, including ImageNet (Deng et al., 2009) classification, contrastive learning (He et al., 2020; Chen et al., 2020c), masked image modeling (MIM)

Original: (He et al., 2022; Xie et al., 2022), and language-vision pre-training
Revised: (He et al., 2022; Xie et al., 2022), and language-vision pre-training

Original: Radford et al., 2021), could guarantee superior representation for robotic policy learning tasks, e.g. , dexterous manipulation, motor control skills and visual navigation.
Revised: Radford et al., 2021), could guarantee superior representation for robotic policy learning tasks, e.g. , dexterous manipulation, motor control skills and visual navigation.

Original: However, for one crucial and challenging visuomotor task in particular, namely end-to-end autonomous driving 1 , the aforementioned predominant pre-training methods may not be the optimal choice (Yamada et al., 2022; Zhang et al., 2022b).
Revised: However, for one crucial and challenging visuomotor task in particular, namely end-to-end autonomous driving 1 , the aforementioned predominant pre-training methods may not be the optimal choice (Yamada et al., 2022; Zhang et al., 2022b).

Original: In this paper, we aim to investigate why ever-victorious pre-training approaches for general computer vision tasks and robotic control tasks are prone to fail in case of end-to-end autonomous driving.
Revised: In this paper, we aim to investigate why ever-victorious pre-training approaches for general computer vision tasks and robotic control tasks are prone to fail in case of end-to-end autonomous driving.

Original: For conventional pre-training methods in general vision tasks, e.g. , classification, segmentation and detection, they usually adopt a wide range of data augmentations to achieve translation and view invariance (Zhang et al., 2016; Wu et al., 2018).
Revised: For conventional pre-training methods in general vision tasks, e.g. , classification, segmentation and detection, they usually adopt a wide range of data augmentations to achieve translation and view invariance (Zhang et al., 2016; Wu et al., 2018).

Original: For robotic control tasks, the input sequence is generally of small resolution; the environment setting is simple and concentrated on objects (Parisi et al., 2022; Radosavovic et al., 2022).
Revised: For robotic control tasks, the input sequence is generally of small resolution; the environment setting is simple and concentrated on objects (Parisi et al., 2022; Radosavovic et al., 2022).

Original: We argue that the visuomotor driving investigated in this paper, is sensitive to geometric relationships and usually comprises complex scenarios.
Revised: We argue that the visuomotor driving investigated in this paper, is sensitive to geometric relationships and usually comprises complex scenarios.

Original: As described in Fig. 1(a), the input data often carry irrelevant information, such as background buildings, far-away moving vehicles, nearby static obstacles, etc. , which are deemed as noises for the decision making task.
Revised: As described in Fig. 1(a), the input data often carry irrelevant information, such as background buildings, far-away moving vehicles, nearby static obstacles, etc. , which are deemed as noises for the decision making task.

Original: To obtain a good driving policy, we argue that the desirable model should only concentrate on particular parts/patterns of the visual input.
Revised: To obtain a good driving policy, we argue that the desirable model should only concentrate on particular parts/patterns of the visual input.

Original: That is, taking heed of direct or deterministic relation to the decision making, e.g. , traffic signals in Fig. 1(b).
Revised: That is, taking heed of direct or deterministic relation to the decision making, e.g. , traffic signals in Fig. 1(b).

Original: However, concurrent pre-training approaches fail to fulfill such a requirement.
Revised: However, concurrent pre-training approaches fail to fulfill such a requirement.

Original: There comes a natural and necessary demand to formulate a pre-training scheme curated for end-to-end autonomous driving.
Revised: There comes a natural and necessary demand to formulate a pre-training scheme curated for end-to-end autonomous driving.

Original: We attempt to pre-train a visual encoder with a massive amount of driving data crawled freely from the web, such that given limited labeled data, downstream applications could generalize well and quickly adapt to various driving environments as depicted in Fig. 1(c).
Revised: We attempt to pre-train a visual encoder with a massive amount of driving data crawled freely from the web, such that given limited labeled data, downstream applications could generalize well and quickly adapt to various driving environments as depicted in Fig. 1(c).

Original: The pivotal question is how to introduce driving-decision awareness into the pre-training process to help the visual encoder concentrate on crucial visual cues for driving policy .
Revised: The pivotal question is how to introduce driving-decision awareness into the pre-training process to help the visual encoder concentrate on crucial visual cues for driving policy .

Original: One may resort to directly predicting ego-motion based on single frame sensor input, constraining the network on learning policy-related features.
Revised: One may resort to directly predicting ego-motion based on single frame sensor input, constraining the network on learning policy-related features.

Original: Previous literature tackles the supervision problem with pseudo labeling training on either an open dataset (Zhang et al., 2022b) or the target domain data (Zhang et al., 2022a).
Revised: Previous literature tackles the supervision problem with pseudo labeling training on either an open dataset (Zhang et al., 2022b) or the target domain data (Zhang et al., 2022a).

Original: However, pseudo labeling approaches suffer from noisy predictions from poorly calibrated models - this is true especially when there exists distinct domain gap such as geographical locations and traffic complexities (Rizve et al., 2020).
Revised: However, pseudo labeling approaches suffer from noisy predictions from poorly calibrated models - this is true especially when there exists distinct domain gap such as geographical locations and traffic complexities (Rizve et al., 2020).

Original: To address the bottleneck aforementioned, we propose PPGeo ( P olicy P re-training via Geo metric modeling), a fully self-supervised driving policy pre-training framework to learn from unlabeled and uncalibrated driving videos.
Revised: To address the bottleneck aforementioned, we propose PPGeo ( P olicy P re-training via Geo metric modeling), a fully self-supervised driving policy pre-training framework to learn from unlabeled and uncalibrated driving videos.

Original: It models the 3D geometric scene by jointly predicting ego-motion, depth, and camera intrinsics.
Revised: It models the 3D geometric scene by jointly predicting ego-motion, depth, and camera intrinsics.

Original: Since directly learning ego-motion based on single frame input along with depth and intrinsics training from scratch is too difficult, it is necessary to separate the visual encoder pre-training from depth and intrinsics learning in two stages.
Revised: Since directly learning ego-motion based on single frame input along with depth and intrinsics training from scratch is too difficult, it is necessary to separate the visual encoder pre-training from depth and intrinsics learning in two stages.

Original: In the first stage, the ego-motion is predicted based on consecutive frames as does in conventional depth estimation frameworks (Godard et al., 2017; 2019).
Revised: In the first stage, the ego-motion is predicted based on consecutive frames as does in conventional depth estimation frameworks (Godard et al., 2017; 2019).

Original: In the second stage, the future ego-motion is estimated based on the single frame by a visual encoder, and could be optimized with the depth and camera intrinsics network well-learned in the first stage.
Revised: In the second stage, the future ego-motion is estimated based on the single frame by a visual encoder, and could be optimized with the depth and camera intrinsics network well-learned in the first stage.

Original: As such, the visual encoder is capable of inferring future ego-motion based on current input alone.
Revised: As such, the visual encoder is capable of inferring future ego-motion based on current input alone.

Original: The pre-trained visual encoder could be well adopted for downstream driving tasks since it captures driving policy related information.
Revised: The pre-trained visual encoder could be well adopted for downstream driving tasks since it captures driving policy related information.

Original: As a side product, the depth and
Revised: As a side product, the depth and

Original: [Figure]
Revised: [Figure]

Original: pose networks could be utilized as new preliminary weights for and odometry estimation tasks, bringing in an additional performance gain.
Revised: pose networks could be utilized as new preliminary weights for and odometry estimation tasks, bringing in an additional performance gain.

Original: To sum up, our key contributions are three-fold:
Revised: To sum up, our key contributions are three-fold:

Original: • We propose a pre-training paradigm curated for various visuomotor driving tasks.
Revised: • We propose a pre-training paradigm curated for various visuomotor driving tasks.

Original: To the best of our knowledge, this is the first attempt to achieve a fully self-supervised framework without any need of pseudo-labels, leveraging the effect of pre-training by large-scale data to the full extent.
Revised: To the best of our knowledge, this is the first attempt to achieve a fully self-supervised framework without any need of pseudo-labels 2 , leveraging the effect of pre-training by large-scale data to the full extent.

Original: • We devise a visual encoder capable of predicting ego-motion based on single visual input, being able to extract feature representations closely related to driving policy.
Revised: • We devise a visual encoder capable of predicting ego-motion based on single visual input, being able to extract feature representations closely related to driving policy.

Original: Such a design of visual encoder is flexible to extend to various downstream applications.
Revised: Such a design of visual encoder is flexible to extend to various downstream applications.

Original: • We demonstrate the superiority of our approach on aset
Revised: • We demonstrate the superiority of our approach on a

Original:  of end-to-end driving scenarios, covering different types and difficulty levels.
Revised: set of end-to-end driving scenarios, covering different types and difficulty levels.

Original: The performance in terms of various metrics is improved from 2% to even over 100% in challenging cases with very limited data.
Revised: The performance in terms of various metrics is improved from 2% to even over 100% in challenging cases with very limited data.

Original:  M ETHODOLOGY 2. O VERVIEW
Revised:  M ETHODOLOGY 2. O VERVIEW

Original: The visuomotor policy learning for autonomous driving targets generating a policy π , such that it makes driving decisions, e.g. , control actions or planned trajectory, from visual observation x .
Revised: The visuomotor policy learning for autonomous driving targets generating a policy π , such that it makes driving decisions, e.g. , control actions or planned trajectory, from visual observation x .

Original: Our goal is to pre-train a visual encoder ϕ
Revised: Our goal is to pre-train a visual encoder ϕ

Original: ( x ) , which maps the raw image input to a compact representation containing important information for driving decision making.
Revised: ( x ) , which maps the raw image input to a compact representation containing important information for driving decision making.

Original: The representation is then utilized by the policy π ( ϕ ( x )) to perform driving tasks.
Revised: The representation is then utilized by the policy π ( ϕ ( x )) to perform driving tasks.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, our pre-training method pre-trains the visual encoder on unlabeled driving videos via two stages in a self-supervised manner.
Revised: 2, our pre-training method pre-trains the visual encoder on unlabeled driving videos via two stages in a self-supervised manner.

Original: 2. T WO - STAGE S ELF - SUPERVISED T RAINING
Revised: 2. T WO - STAGE S ELF - SUPERVISED T RAINING

Original: Stage One: Self-supervised Geometric Modeling.
Revised: Stage One: Self-supervised Geometric Modeling.

Original: During the first stage, given a target image I t and source images I t ′ in a sequence, we jointly estimate the depth of the target image, the intrinsics of the camera, and the 6-DoF ego-motion between these two frames.
Revised: During the first stage, given a target image I t and source images I t ′ in a sequence, we jointly estimate the depth of the target image, the intrinsics of the camera, and the 6-DoF ego-motion between these two frames.

Original: Given the estimations, we are able to model the 3D geometry of the scene, and reconstruct the target image by projecting pixels in the source images.
Revised: Given the estimations, we are able to model the 3D geometry of the scene, and reconstruct the target image by projecting pixels in the source images.

Original: Formally, the pixel-wise correspondence between I t and I t ′ is calculated as:
Revised: Formally, the pixel-wise correspondence between I t and I t ′ is calculated as:

Original: [Equation]
Revised: [Equation]

Original: where p t and p t ′ are the homogeneous coordinates of the pixel in I t and I t ′ respectively, K is the predicted camera intrinsic matrix, and D t ( p t ) represents the predicted depth value at pixel p i in
Revised: where p t and p t ′ are the homogeneous coordinates of the pixel in I t and I t ′ respectively, K is the predicted camera intrinsic matrix, and D t ( p t ) represents the predicted depth value at pixel p i in

Original: I t .
Revised: I t .

Original: With this relationship, the target image I t ′
Revised: With this relationship, the target image I t ′

Original: → t could be reconstructed with pixels in I t ′ , and be optimized by the photometric reconstruction error.
Revised: → t could be reconstructed with pixels in I t ′ , and be optimized by the photometric reconstruction error.

Original: Following Godard et al.
Revised: Following Godard et al.

Original: (2019), we choose two images adjacent to the current frame as the source images, i.e. , t ′ ∈ { t − 1 , t + 1 } .
Revised: (2019), we choose two images adjacent to the current frame as the source images, i.e. , t ′ ∈ { t − 1 , t + 1 } .

Original: The DepthNet consists of a common encoder-decoder structure (Godard et al., 2019) and estimates the depth map of the input image.
Revised: The DepthNet consists of a common encoder-decoder structure (Godard et al., 2019) and estimates the depth map of the input image.

Original: Two images are stacked together and fed into the encoder of the PoseNet, whose bottleneck feature is then utilized to predict the camera intrinsics and the egomotion via two separate MLP-based heads.
Revised: Two images are stacked together and fed into the encoder of the PoseNet, whose bottleneck feature is then utilized to predict the camera intrinsics and the egomotion via two separate MLP-based heads.

Original: For camera intrinsics estimation, optical center ( c x , c y ) and focal lengths f x , f y are regressed similarly as in (Gordon et al., 2019; Chanduri et al., 2021).
Revised: For camera intrinsics estimation, optical center ( c x , c y ) and focal lengths f x , f y are regressed similarly as in (Gordon et al., 2019; Chanduri et al., 2021).

Original: Stage Two: Visuomotor Policy Pre-training.
Revised: Stage Two: Visuomotor Policy Pre-training.

Original: After the first stage of training, the DepthNet and
Revised: After the first stage of training, the DepthNet and

Original: PoseNet are well trained and fitted to the driving video data.
Revised: PoseNet are well trained and fitted to the driving video data.

Original: Then, in the second stage, we replace the PoseNet for ego-motion estimation with the visual encoder  prepared for downstream driving policy learning tasks.
Revised: Then, in the second stage, we replace the PoseNet for ego-motion estimation with the visual encoder ϕ ( x ) prepared for downstream driving policy learning tasks.

Original: Now the visual encoder only takes a single frame image as input and predicts ego-motion between the current frame and subsequent frame.
Revised: Now the visual encoder only takes a single frame image as input and predicts ego-motion between the current frame and subsequent frame.

Original: This single input setting aligns with downstream driving tasks.

Original: Besides, learning temporal policies might potentially lead to the casual confusion problem (De Haan et al., 2019).

Original: Specifically, the visual encoder estimates the ego-motion T t → t +1 based on I t alone and T t → t − 1 based on I t − 1 followed by an inverse operation, respectively.
Revised: Specifically, the visual encoder estimates the ego-motion T t → t +1 based on I t alone and T t → t −based on I t − 1 followed by an inverse operation, respectively.

Original: The visual encoder is optimized by the photometric reconstruction error similar to the first stage, aside from a modification where the DepthNet and the intrinsics estimation are frozen and not backpropagated.
Revised: The visual encoder is optimized by the photometric reconstruction error similar to the first stage, aside from a modification where the DepthNet and the intrinsics estimation are frozen and not backpropagated.

Original: This is empirically observed towards better performance.
Revised: This is empirically observed towards better performance.

Original: By doing so, the visual encoder is enforced to learn the actual driving policy, since the ego-motion between two consecutive frames is straightforwardly related to the driving decision or action taken at the current timestamp.
Revised: By doing so, the visual encoder is enforced to learn the actual driving policy, since the ego-motion between two consecutive frames is straightforwardly related to the driving decision or action taken at the current timestamp.

Original: One might argue that the PoseNet trained in the first stage could provide pseudo motion labels, with which the visual encoder could be directly supervised.
Revised: One might argue that the PoseNet trained in the first stage could provide pseudo motion labels, with which the visual encoder could be directly supervised.

Original: However, the ego-motion predicted from the PoseNet is too sparse compared with the geometric projection approach.
Revised: However, the ego-motion predicted from the PoseNet is too sparse compared with the geometric projection approach.

Original: In our pipeline, every pixel provides supervision for the visual encoder so that inaccurate depth estimation in some pixels could be mitigated by the accurate ones, i.e. , it constructs a “global” optimization.
Revised: In our pipeline, every pixel provides supervision for the visual encoder so that inaccurate depth estimation in some pixels could be mitigated by the accurate ones, i.e. , it constructs a “global” optimization.

Original: In contrast, direct supervision from the PoseNet would be greatly affected by the undesirable prediction inaccuracy and noise results.
Revised: In contrast, direct supervision from the PoseNet would be greatly affected by the undesirable prediction inaccuracy and noise results.

Original: This is especially true for diverse uncalibrated online videos (Zhang et al., 2022a).
Revised: This is especially true for diverse uncalibrated online videos (Zhang et al., 2022a).

Original: Thus far, the  visual encoder  has gained knowledge about the driving policy from the diverse driving videos.
Revised: Thus far, the backbone of visual encoder ϕ ( x ) has gained knowledge about the driving policy from the diverse driving videos.

Original: It can then be applied to downstream visuomotor autonomous driving tasks as the initial weights.
Revised: It can then be applied to downstream visuomotor autonomous driving tasks as the initial weights.

Original: Besides, the DepthNet and PoseNet trained on this large corpus of uncalibrated video data could also be utilized in depth and odometry estimation tasks.
Revised: Besides, the DepthNet and PoseNet trained on this large corpus of uncalibrated video data could also be utilized in depth and odometry estimation tasks.

Original: 2. L OSS F UNCTION
Revised: 2. L OSS F UNCTION

Original: Following Godard et al.
Revised: Following Godard et al.

Original: (2019), the loss function is comprised of the photometric loss and the smoothness loss.
Revised: (2019), the loss function is comprised of the photometric loss and the smoothness loss.

Original: The photometric error is comprised of an ℓ 1 term and an SSIM (structural similarity index measure) term
Revised: The photometric error is comprised of an ℓ 1 term and an SSIM (structural similarity index measure) term

Original: (Wang et al., 2004):
Revised: (Wang et al., 2004):

Original: [Equation]
Revised: [Equation]

Original: (2) where we set α = 0 .  following the practice (Godard et al., 2017; 2019).
Revised: where we set α = 0 . 85 following the practice (Godard et al., 2017; 2019).

Original: The smooth loss is:
Revised: The smooth loss is:

Original: [Equation]
Revised: [Equation]

Original: (3) where d ∗ t is the mean-normalized inverse depth map.
Revised: where d ∗ t is the mean-normalized inverse depth map.

Original: We also adopt the minimum reprojection loss and auto-masking scheme (Godard et al., 2019) to improve self-supervised depth estimation.
Revised: We also adopt the minimum reprojection loss and auto-masking scheme (Godard et al., 2019) to improve self-supervised depth estimation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: All pre-training experiments are conducted on the hours-long unlabeled YouTube driving videos (Zhang et al., 2022b).
Revised: All pre-training experiments are conducted on the hours-long unlabeled YouTube driving videos (Zhang et al., 2022b).

Original: It covers different driving conditions e.g. , geographical locations and weather.
Revised: It covers different driving conditions e.g. , geographical locations and weather.

Original: We sample 0.8 million frames in total at 1 Hz for training.
Revised: We sample 0.8 million frames in total at 1 Hz for training.

Original: For the first stage in PPGeo pipeline, we train the model for 30 epochs by Adam (Kingma & Ba, 2015) optimizer with a learning rate of 10 − 4 which drops to 10 − 5 after 25 epochs.
Revised: For the first stage in PPGeo pipeline, we train the model for 30 epochs by Adam (Kingma & Ba, 2015) optimizer with a learning rate of 10 − 4 which drops to 10 − 5 after 25 epochs.

Original: For the second stage, the encoder is trained forepochs using the AdamW (Loshchilov & Hutter, 2017) optimizer.
Revised: For the second stage, the encoder is trained forepochs using the AdamW (Loshchilov & Hutter, 2017) optimizer.

Original: A cyclic learning rate scheduler is applied with the learning rate ranging from 10 − 6 to 10 − 4 .
Revised: A cyclic learning rate scheduler is applied with the learning rate ranging from 10 − 6 to 10 − 4 .

Original: The batch size for both stages is 128.
Revised: The batch size for both stages is 128.

Original: We use data augmentations including ColorJitter, RamdomGrayScale, and GaussianBlur.
Revised: We use data augmentations including ColorJitter, RamdomGrayScale, and GaussianBlur.

Original: 3. D ESCRIPTION ON C OMPARED B ASELINES
Revised: 3. D ESCRIPTION ON C OMPARED B ASELINES

Original: We use ResNet-34 (He et al., 2016) as the encoder and load different pre-trained weights for the initialization of downstream tasks.
Revised: We use ResNet-34 (He et al., 2016) as the encoder and load different pre-trained weights for the initialization of downstream tasks.

Original: We compare PPGeo with pre-training methods including:
Revised: We compare PPGeo with pre-training methods including:

Original: Random.
Revised: Random.

Original: We use the default Kaiming initialization (He et al., 2015) for convolution layers and constant initialization for batchnorms.
Revised: We use the default Kaiming initialization (He et al., 2015) for convolution layers and constant initialization for batchnorms.

Original: ImageNet.
Revised: ImageNet.

Original: We use the model weight provided by Torchvision (Marcel & Rodriguez, 2010), which is pre-trained with the classification task on ImageNet (Deng et al., 2009).
Revised: We use the model weight provided by Torchvision (Marcel & Rodriguez, 2010), which is pre-trained with the classification task on ImageNet (Deng et al., 2009).

Original: MIM.
Revised: MIM.

Original: The model is pre-trained with the masked image modeling method on the YouTube driving video, which tries to reconstruct images with random masked-out patches.
Revised: The model is pre-trained with the masked image modeling method on the YouTube driving video, which tries to reconstruct images with random masked-out patches.

Original: SimMIM (Xie et al., 2022) is adopted as it is suitable for convolutional networks.
Revised: SimMIM (Xie et al., 2022) is adopted as it is suitable for convolutional networks.

Original: MoCo.
Revised: MoCo.

Original: We pre-train the model using MoCo-v2 (Chen et al., 2020c) on the YouTube driving videos.
Revised: We pre-train the model using MoCo-v2 (Chen et al., 2020c) on the YouTube driving videos.

Original: We exclude RandomResizedCrop and RandomHorizontalFlip augmentations as they are not suitable for the driving task.
Revised: We exclude RandomResizedCrop and RandomHorizontalFlip augmentations as they are not suitable for the driving task.

Original: ACO.
Revised: ACO.

Original: Following Zhang et al.
Revised: Following Zhang et al.

Original: (2022b)
Revised: (2022b)

Original: , it is pre-trained using action-conditioned contrastive learning on the YouTube driving videos.
Revised: , it is pre-trained using action-conditioned contrastive learning on the YouTube driving videos.

Original: ACO trains an inverse dynamic model to generate pseudo steer labels for driving videos, based on which steer-based discrimination is added on top of MoCo-v2.
Revised: ACO trains an inverse dynamic model to generate pseudo steer labels for driving videos, based on which steer-based discrimination is added on top of MoCo-v2.

Original: SelfD. SelfD (Zhang et al., 2022a) is not a pre-training method strictly since it needs to train the whole policy model on the driving video for each task, while other pre-training methods aforementioned provide a general pre-training visual model for all tasks.
Revised: SelfD. SelfD (Zhang et al., 2022a) is not a pre-training method strictly since it needs to train the whole policy model on the driving video for each task, while other pre-training methods aforementioned provide a general pre-training visual model for all tasks.

Original: We still include it for comparison due to its close relationship to our target.
Revised: We still include it for comparison due to its close relationship to our target.

Original: Specifically, we follow Zhang et al.
Revised: Specifically, we follow Zhang et al.

Original: (2022a) to train the model for each task with the following pipeline: training on the task data → training on the YouTube data with pseudo-label → fine-tuning on the task data.
Revised: (2022a) to train the model for each task with the following pipeline: training on the task data → training on the YouTube data with pseudo-label → fine-tuning on the task data.

Original: 3. D ESCRIPTION ON D OWNSTREAM A UTONOMOUS D RIVING T ASKS
Revised: 3. D ESCRIPTION ON D OWNSTREAM A UTONOMOUS D RIVING T ASKS

Original: We carry out experiments under (1) three imitation learning based closed-loop driving tasks in
Revised: We carry out experiments under (1) three imitation learning based closed-loop driving tasks in

Original: CARLA (Dosovitskiy et al., 2017),
Revised: CARLA (Dosovitskiy et al., 2017),

Original: (2) one reinforcement learning based driving task in CARLA, and (3) an open-loop planning task on real-world autonomous driving dataset nuScenes (Caesar et al., 2020), to fully validate the effectiveness of PPGeo.
Revised: (2) one reinforcement learning based driving task in CARLA, and (3) an open-loop planning task on real-world autonomous driving dataset nuScenes (Caesar et al., 2020), to fully validate the effectiveness of PPGeo.

Original: We briefly describe each task below.
Revised: We briefly describe each task below.

Original: Navigation.
Revised: Navigation.

Original: It corresponds to the goal-conditioned navigation task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).
Revised: It corresponds to the goal-conditioned navigation task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).

Original: The agent is trained in Town01 and tested in Town02 with unseen weather, and there are no other traffic participants.
Revised: The agent is trained in Town01 and tested in Town02 with unseen weather, and there are no other traffic participants.

Original: We use different sizes of training data (from 4K to 40K) following Zhang et al.
Revised: We use different sizes of training data (from 4K to 40K) following Zhang et al.

Original: (2022b) to evaluate the generalization ability of pre-trained visual encoders when labeled data is limited and conduct the closed-loop evaluation.
Revised: (2022b) to evaluate the generalization ability of pre-trained visual encoders when labeled data is limited and conduct the closed-loop evaluation.

Original: The evaluation metric is success rate, denoting the portion of 50 pre-defined routes finished without any collision.
Revised: The evaluation metric is success rate, denoting the portion of 50 pre-defined routes finished without any collision.

Original: And traffic lights are ignored here.
Revised: And traffic lights are ignored here.

Original: CILRS (Codevilla et al., 2019), a classic image based end-to-end autonomous driving model, is adopted for training and evaluation.
Revised: CILRS (Codevilla et al., 2019), a classic image based end-to-end autonomous driving model, is adopted for training and evaluation.

Original: Navigation Dynamic.
Revised: Navigation Dynamic.

Original: This is the navigation dynamic task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).
Revised: This is the navigation dynamic task in the CoRL2017 benchmark (Dosovitskiy et al., 2017).

Original: The setting differentiates from Navigation that there are other dynamic objects such as randomly generated vehicles, which substantially increases the difficulty of driving safety.
Revised: The setting differentiates from Navigation that there are other dynamic objects such as randomly generated vehicles, which substantially increases the difficulty of driving safety.

Original: Leaderboard Town05-long.
Revised: Leaderboard Town05-long.

Original: This challenging and realistic benchmark corresponds to the LeaderBoard benchmark (CARLA, 2022).
Revised: This challenging and realistic benchmark corresponds to the LeaderBoard benchmark (CARLA, 2022).

Original: We collect 40K training data in Town01, 03, 04, 06 and evaluate on 10 routes in the unseen Town05 (Prakash et al., 2021).
Revised: We collect 40K training data in Town01, 03, 04, 06 and evaluate on 10 routes in the unseen Town05 (Prakash et al., 2021).

Original: Due to the challenging scenarios in this task, we evaluate different pre-training approaches with the state-of-the-art image-based autonomous driving model TCP (Wu et al., 2022).
Revised: Due to the challenging scenarios in this task, we evaluate different pre-training approaches with the state-of-the-art image-based autonomous driving model TCP (Wu et al., 2022).

Original: The  metrics of this task are Driving Score, Route Completion, and Infraction Score (all the higher the better), where the main metric Driving Score is the product of Route Completion and Infraction Score.
Revised: The major metrics of this task are Driving Score, Route Completion, and Infraction Score (all the higher the better). Route Completion denotes the portion of the route completed by the agent.

Revised: Infraction Score is the number of infractions made along the route including pedstrain collisions, vehicle collisions, red light infractions, etc .

Revised: And the main metric Driving Score is the product of Route Completion and Infraction Score.

Original: Reinforcement Learning.
Revised: Reinforcement Learning.

Original: Proximal Policy Optimization (PPO) (Schulman et al., 2017) is used to train the CILRS (Codevilla et al., 2019) model initialized with different pre-trained weights in
Revised: Proximal Policy Optimization (PPO) (Schulman et al., 2017) is used to train the CILRS (Codevilla et al., 2019) model initialized with different pre-trained weights in

Original: CARLA Town01 environment.
Revised: CARLA Town01 environment.

Original: The reward shaping details follow Roach (Zhang et al., 2021).
Revised: The reward shaping details follow Roach (Zhang et al., 2021).

Original: We also conduct experiments to freeze the pre-trained visual encoder during training to further study the effectiveness of the pre-trained feature representations.[Table] [Table]
Revised: We also conduct experiments to freeze the pre-trained visual encoder during training to further study the effectiveness of the pre-trained feature representations.

Original: nuScenes Planning.
Revised: nuScenes Planning.

Original: This task involves trajectory planning in real-world dataset nuScenes (Caesar et al., 2020).
Revised: This task involves trajectory planning in real-world dataset nuScenes (Caesar et al., 2020).

Original: Given the current visual input, the model plans a 3-second trajectory (0.5 Hz), and the planned trajectory is compared with the ground truth log.
Revised: Given the current visual input, the model plans a 3-second trajectory (0.5 Hz), and the planned trajectory is compared with the ground truth log.

Original: We also calculate the collision rate, where a collision is defined as overlaps with future vehicles and pedestrians based on planned waypoints.
Revised: We also calculate the collision rate, where a collision is defined as overlaps with future vehicles and pedestrians based on planned waypoints.

Revised: The metric of this tasks includes (1) the L2 distance between predicted trajectory and ground truth trajectory, and (2) the collision rate.

Revised: Metrics are measured at different time lengths from 1s to 3s.

Original: The planning model used here is comprised of a visual encoder and a GRU-based planner to predict each waypoint auto-regressively.
Revised: The planning model used here is comprised of a visual encoder and a GRU-based planner to predict each waypoint auto-regressively.

Original: We use the official train-val split for training and evaluation.
Revised: We use the official train-val split for training and evaluation.

Original: 3. N UMERIC C OMPARISON ON D OWNSTREAM T ASKS
Revised: 3. N UMERIC C OMPARISON ON D OWNSTREAM T ASKS

Original: For imitation learning based closed-loop driving tasks, the evaluation results are shown in Table 13.
Revised: For imitation learning based closed-loop driving tasks, the evaluation results are shown in Fig.

Original:  We present the plot between episode return and environment steps of each method in Fig. 3 for the reinforcement learning experiments.
Revised: 3 and Table 1. We present the plot between episode return and environment steps of each method in Fig.for the reinforcement learning experiments.

Original: The open-loop nuScenes planning results are provided in Table 4.
Revised: The open-loop nuScenes planning results are provided in Table 2.

Original: We could observe that PPGeo outperforms other baselines by a large margin in all tasks.
Revised: We could observe that PPGeo outperforms other baselines by a large margin in all tasks.

Original: Note that the model is tested under a different number of fine-tuning samples from 10% (4K) to full 40K in the Navigation and Navigation Dynamic tasks.
Revised: Note that the model is tested under a different number of fine-tuning samples from 10% (4K) to full 40K in the Navigation and Navigation Dynamic tasks.

Original: In the case of the particularly small size of training samples, PPGeo still demonstrates competitive performance and has a larger improvement gap of over 100%.
Revised: In the case of the particularly small size of training samples, PPGeo still demonstrates competitive performance and has a larger improvement gap of over 100%.

Original: This validates the generalization ability of the pre-trained visual encoder, which is important when adapting to a new environment with very limited labeled data.
Revised: This validates the generalization ability of the pre-trained visual encoder, which is important when adapting to a new environment with very limited labeled data.

Original: In the more challenging and real-world style Leaderboard Town05-long task in Table 3, the model pre-trained with our method achieves the highest driving score and infraction score.
Revised: In the more challenging and real-world style Leaderboard Town05-long task in Table 1, the model pre-trained with our method achieves the highest driving score and infraction score.

Original: PPGeo well handles cases where the agent needs to stop, leading to much fewer vehicle collisions and red light infractions.
Revised: PPGeo well handles cases where the agent needs to stop, leading to much fewer vehicle collisions and red light infractions.

Original: Since ACO considers steering angles only during pre-training, its performance degrades on more challenging scenarios where brake and throttles are also important.
Revised: Since ACO considers steering angles only during pre-training, its performance degrades on more challenging scenarios where brake and throttles are also important.

Original: SelfD performs slightly better than ACO in complex cases while it significantly degenerates when the task data is limited, as affected by the unsatisfying pseudo labeling model.
Revised: SelfD performs slightly better than ACO in complex cases while it significantly degenerates when the task data is limited, as affected by the unsatisfying pseudo labeling model.

Original: ImageNet pre-training also shows competitive performance, which might credit to its ability of finding salient objects in the scene when the input contains little irrelevant information (see examples in Sec. 3.5).
Revised: ImageNet pre-training also shows competitive performance, which might credit to its ability of finding salient objects in the scene when the input contains little irrelevant information (see examples in Sec. 3.5).

Original: 3. D EPTH AND O DOMETRY E STIMATION
Revised: 3. D EPTH AND O DOMETRY E STIMATION

Original: In this part, we explore whether the large-scale training on uncalibrated data could benefit the depth and odometry estimation models as well and validate the effectiveness of first-stage training.
Revised: In this part, we explore whether the large-scale training on uncalibrated data could benefit the depth and odometry estimation models as well and validate the effectiveness of first-stage training.

Original: Specif-
Revised: Specifically, we employ the DepthNet and PoseNet trained after the first stage as initial weights for Mon-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:        
Revised:        

Original: [Figure]
Revised: [Figure]

Original:        ( S L V RG H  5 H W X U Q
Revised:        ( S L V RG H  5 H W X U Q

Original: [Figure]
Revised: [Figure]

Original:        
Revised:        

Original: [Figure]
Revised: [Figure]

Original:   

Original: [Figure]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: ically
Revised: ACO

Original: [Table]
Revised: [Table]

Original: we employ the DepthNet and PoseNet trained after the first stage as initial weights for Monodepthv2 (Godard et al., 2019)

Original: [Table]
Revised: [Table]

Original:  and conduct experiments on KITTI (Geiger et al., 2012).
Revised: odepthv2 (Godard et al., 2019), and conduct experiments on KITTI (Geiger et al., 2012).

Original: Results in Table 5 indicate that pre-training on large-scale driving videos could bring performance improvement to both depth and odometry estimation tasks, which is an additional harvest of our pre-training framework.
Revised: Results in Table 3 indicate that pre-training on large-scale driving videos could bring performance improvement to both depth and odometry estimation tasks, which is an additional harvest of our pre-training framework.

Original: We refer readers to Godard et al.
Revised: We refer readers to Godard et al.

Original: for details about the metrics of these tasks
Revised: for details about the metrics of these tasks

Original: [Table]
Revised: [Table]

Original: 3. V ISUALIZATION R ESULTS
Revised: 3. V ISUALIZATION R ESULTS

Original: Here we provide heatmaps of the feature representations learned by different pre-training methods using Eigen-Cam (Muhammad & Yeasin, 2020) to show the attended regions in Fig.4.
Revised: Here we provide heatmaps of the feature representations learned by different pre-training methods using Eigen-Cam (Muhammad & Yeasin, 2020) to show the attended regions in Fig.

Original: In many
Revised: In many

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: cases (Row 1&2), our model mainly concentrates on the lane in front of the ego vehicle
Revised: cases (Row 1&2), our model mainly concentrates on the lane in front of the ego vehicle

Original: [Table]
Revised: [Table]

Original: which is highly related to driving.
Revised: which is highly related to driving.

Original:  PPGeo well captures the specific cues causing the brake action including front vehicles (Row 3&4) and traffic lights (Row 5).
Revised: And our model PPGeo well captures the specific cues causing the brake action including front vehicles (Row 3&4) and traffic lights (Row 5).

Original: We also observe that the model pre-trained with ImageNet classification tends to capture salient objects in the image.
Revised: We also observe that the model pre-trained with ImageNet classification tends to capture salient objects in the image.

Original: This is helpful when the salient objects are straightforwardly related to the driving decision (Row 4); but it may focus on wrong objects when the input contains other irrelevant information (Row 2&3).
Revised: This is helpful when the salient objects are straightforwardly related to the driving decision (Row 4); but it may focus on wrong objects when the input contains other irrelevant information (Row 2&3) [Table]

Original: 3. A BLATIVE S TUDY
Revised: 3. A BLATIVE S TUDY

Original: We conduct ablative study as to different designs of PPGeo on the Navigation task in Table 6.
Revised: We conduct ablative study as to different designs of PPGeo on the Navigation task in Table 4.

Original: Training the visual encoder and DepthNet in a single stage simultaneously (Row 1) leads to an inferior performance, indicating that it is quite challenging for the visual encoder to learn the correct egomotion if depth estimation is also trained from scratch.
Revised: Training the visual encoder and DepthNet in a single stage simultaneously (Row 1) leads to an inferior performance, indicating that it is quite challenging for the visual encoder to learn the correct egomotion if depth estimation is also trained from scratch.

Original: Moreover, jointly optimizing the DepthNet in the second stage (Row 2, not frozen) degrades the depth estimation quality and harms the performance.
Revised: Moreover, jointly optimizing the DepthNet in the second stage (Row 2, not frozen) degrades the depth estimation quality and harms the performance.

Original: In Row 3, we observe that utilizing the PoseNet obtained in the first stage to provide pseudo label supervision directly leads to inferior results, since an inaccurate pseudo label impairs the learning process to great extent.
Revised: In Row 3, we observe that utilizing the PoseNet obtained in the first stage to provide pseudo label supervision directly leads to inferior results, since an inaccurate pseudo label impairs the learning process to great extent.

Original:  R ELATED W ORK4. P RE - TRAINING FOR NLP AND G ENERAL V ISION
Revised:  R ELATED W ORK

Revised: Pre-training for NLP and General Vision.

Original: Pre-training or representation learning has proved to be an essential key to the success of artificial intelligence.
Revised: Pre-training or representation learning has proved to be an essential key to the success of artificial intelligence.

Original: In the field of Natural Language Processing (NLP), with the powerful capability of Transformer (Vaswani et al., 2017), pre-training on large-scale datasets with large models then finetuning on downstream tasks has become the dominant paradigm (Kenton & Toutanova, 2019; Brown et al., 2020).
Revised: In the field of Natural Language Processing (NLP), with the powerful capability of Transformer (Vaswani et al., 2017), pre-training on largescale datasets with large models then fine-tuning on downstream tasks has become the dominant paradigm (Kenton & Toutanova, 2019; Brown et al., 2020).

Original: As for the field of Computer Vision, training specific downstream tasks with the supervised pre-trained weights of visual encoder via ImageNet classification task is widely adopted.
Revised: As for the field of Computer Vision, training specific downstream tasks with the supervised pre-trained weights of visual encoder via ImageNet classification task is widely adopted.

Original: Recently, unsupervised and self-supervised learning methods such as contrastive learning (He et al., 2020; Chen et al., 2020c;b) and masked image modeling (Bao et al., 2021; He et al., 2022; Xie et al., 2022; Peng et al., 2022) have gained impressive improvement over ImageNet pre-training on various vision benchmarks.
Revised: Recently, unsupervised and self-supervised learning methods such as contrastive learning (He et al., 2020; Chen et al., 2020c;b) and masked im- age modeling (Bao et al., 2021; He et al., 2022; Xie et al., 2022; Peng et al., 2022) have gained impressive improvement over ImageNet pre-training on various vision benchmarks.

Original: Very recent vision-language co-training approaches (Radford et al., 2021;
Revised: Very recent vision-language co-training approaches (Radford et al., 2021;

Original: Wang et al., 2022) demonstrate their extraordinary potential in the domain of multi-modal learning and applications.
Revised: Wang et al., 2022) demonstrate their extraordinary potential in the domain of multi-modal learning and applications.

Original: Yet, these generic representation learning methods adopt various data augmentation techniques to achieve translation and view invariance, while visuomotor driving sets in a highly dynamic environment.
Revised: Yet, these generic representation learning methods adopt various data augmentation techniques to achieve translation and view invariance, while visuomotor driving sets in a highly dynamic environment.

Original: In this work, we show that the ever-victorious pre-training methods may not be the optimal choice, and introduce a curated paradigm for visuomotor driving policy learning.
Revised: In this work, we show that the ever-victorious pre-training methods may not be the optimal choice, and introduce a curated paradigm for visuomotor driving policy learning.

Original: Pre-training for Visuomotor Applications.
Revised: Pre-training for Visuomotor Applications.

Original: Learning a control policy directly from raw visual input is challenging since the model needs to reason about visual pixels and dynamic behaviors simultaneously.
Revised: Learning a control policy directly from raw visual input is challenging since the model needs to reason about visual pixels and dynamic behaviors simultaneously.

Original: Moreover, training visuomotor models from scratch usually requires tons of labeled data or environment interactions.
Revised: Moreover, training visuomotor models from scratch usually requires tons of labeled data or environment interactions.

Original: To this end, recently, Shah & Kumar (2021) shows that feature representations from ResNet (He et al., 2016) pre-trained on ImageNet classification is helpful for
Revised: To this end, recently, Shah & Kumar (2021) shows that feature representations from ResNet (He et al., 2016) pre-trained on ImageNet classification is helpful for

Original: RL-based dexterous manipulation tasks.
Revised: RL-based dexterous manipulation tasks.

Original: Parisi et al.
Revised: Parisi et al.

Original: (2022) conducts extensive experiments on applying “off-the-shelf” pre-trained vision models in diverse control domains and validates their benefits to train control policies.
Revised: (2022) conducts extensive experiments on applying “off-the-shelf” pre-trained vision models in diverse control domains and validates their benefits to train control policies.

Original: CLIP (Radford et al., 2021) is also adopted in some embodied
Revised: CLIP (Radford et al., 2021) is also adopted in some embodied

Original: AI and robot navigation problems (Shah et al., 2022).
Revised: AI and robot navigation problems (Shah et al., 2022).

Original: Besides borrowing pre-trained weights for visuomotor tasks, researchers in robotics now desire a paradigm learning policy representations from raw data directly.
Revised: Besides borrowing pre-trained weights for visuomotor tasks, researchers in robotics now desire a paradigm learning policy representations from raw data directly.

Original: Xiao et al.
Revised: Xiao et al.

Original: ; Radosavovic et al.
Revised: ; Radosavovic et al.

Original: ; Seo et al.
Revised: ; Seo et al.

Original: ; Gupta et al.
Revised: ; Gupta et al.

Original: (2022) inherit the MIM spirit to realize visual pre-training for control tasks.
Revised: (2022) inherit the MIM spirit to realize visual pre-training for control tasks.

Original: Yang & Nachum (2021) investigates unsupervised representation learning objectives from D4RL environments (Fu et al., 2020), and Yamada et al.
Revised: Yang & Nachum (2021) investigates unsupervised representation learning objectives from D4RL environments (Fu et al., 2020), and Yamada et al.

Original: (2022) further adopts task-induced approaches to learn from prior tasks.
Revised: (2022) further adopts task-induced approaches to learn from prior tasks.

Original: However, compared with visuomotor driving, the visual inputs of such control tasks are less diverse which usually concentrate on objects and are much more compact.
Revised: However, compared with visuomotor driving, the visual inputs of such control tasks are less diverse which usually concentrate on objects and are much more compact.

Original: To our best knowledge, ACO (Zhang et al., 2022b) is the only pre-training method customized for autonomous driving.
Revised: To our best knowledge, ACO (Zhang et al., 2022b) is the only pre-training method customized for autonomous driving.

Original: By first training an inverse dynamic model on nuScenes (Caesar et al., 2020), they get pseudo steer labels of the driving videos and then construct the steer-conditioned discrimination for contrastive learning following MoCo.
Revised: By first training an inverse dynamic model on nuScenes (Caesar et al., 2020), they get pseudo steer labels of the driving videos and then construct the steer-conditioned discrimination for contrastive learning following MoCo.

Original: However, ACO ignores other crucial driving factors such as throttle and brakes, and its performance is largely limited by the inverse dynamic model.
Revised: However, ACO ignores other crucial driving factors such as throttle and brakes, and its performance is largely limited by the inverse dynamic model.

Original: SelfD (Zhang et al., 2022a) is not strictly designed for pre-training while it also makes use of vast amounts of videos to learn driving policies via semi-supervised learning.
Revised: SelfD (Zhang et al., 2022a) is not strictly designed for pre-training while it also makes use of vast amounts of videos to learn driving policies via semi-supervised learning.

Original: It acquires the pseudo labeling knowledge from the target domain.
Revised: It acquires the pseudo labeling knowledge from the target domain.

Original: These two methods both depend on the accuracy of pseudo labeling.
Revised: These two methods both depend on the accuracy of pseudo labeling.

Original: In contrast, we realize fully self-supervised learning through dense geometric reconstruction, evading the possible adverse effect.
Revised: In contrast, we realize fully self-supervised learning through dense geometric reconstruction, evading the possible adverse effect.

Original: Policy Learning for Autonomous Driving.
Revised: Policy Learning for Autonomous Driving.

Original: Visuomotor autonomous driving learns a driving policy directly from sensor inputs in an end-to-end manner (Codevilla et al., 2018; 2019;
Revised: Visuomotor autonomous driving learns a driving policy directly from sensor inputs in an end-to-end manner (Codevilla et al., 2018; 2019;

Original: Liang et al., 2018; Chen et al., 2020a; Prakash et al., 2021; Chen et al., 2021; Wu et al., 2022; Shao et al., 2022).
Revised: Liang et al., 2018; Chen et al., 2020a; Prakash et al., 2021; Chen et al., 2021; Wu et al., 2022; Shao et al., 2022).

Original: In essence, the inherent difficulty of the urban-style autonomous driving tasks makes such methods data-hungry.
Revised: In essence, the inherent difficulty of the urban-style autonomous driving tasks makes such methods data-hungry.

Original: Interfuser (Shao et al., 2022), the current top-1 method on the CARLA Leaderboard (CARLA, 2022), requires 3 million labeled data samples for imitation learning (behavior cloning specifically).
Revised: Interfuser (Shao et al., 2022), the current top-1 method on the CARLA Leaderboard (CARLA, 2022), requires 3 million labeled data samples for imitation learning (behavior cloning specifically).

Original: RL-based model MaRLn (Toromanoff et al., 2020) needs 20 million environment steps of interaction.
Revised: RL-based model MaRLn (Toromanoff et al., 2020) needs 20 million environment steps of interaction.

Original: The sample efficiency problem greatly impedes the real-world application of such approaches.
Revised: The sample efficiency problem greatly impedes the real-world application of such approaches.

Original: In this work, we propose a self-supervised pre-training pipeline to learn driving policy related representations on unlabeled driving videos, and pave the way for these visuomotor autonomous driving models to further achieve satisfying performance.
Revised: In this work, we propose a self-supervised pre-training pipeline to learn driving policy related representations on unlabeled driving videos, and pave the way for these visuomotor autonomous driving models to further achieve satisfying performance.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this work, we have proposed a fully self-supervised visuomotor driving policy pre-training paradigm PPGeo by modeling the 3D geometry of large-scale unlabeled driving videos.
Revised: In this work, we have proposed a fully self-supervised visuomotor driving policy pre-training paradigm PPGeo by modeling the 3D geometry of large-scale unlabeled driving videos.

Original: Taking a direct approach to infer the ego-motion and benefiting from the two-stage pre-training pipeline, we enable the visual encoder to learn driving policies based on single visual input.
Revised: Taking a direct approach to infer the ego-motion and benefiting from the two-stage pre-training pipeline, we enable the visual encoder to learn driving policies based on single visual input.

Original: Our method outperforms the peer pre-training approaches by a large margin on a series of visuomotor driving tasks.
Revised: Our method outperforms the peer pre-training approaches by a large margin on a series of visuomotor driving tasks.

Original: For its limitation, our method currently only considers the ego-motion for a single time step, and a future direction is to devise the framework to perform multi-step motion prediction which contains more information about driving decisions. 
Revised: For its limitation, our method currently only considers the ego-motion for a single time step, and a future direction is to devise the framework to perform multi-step motion prediction which contains more information about driving decisions. P OLICY P RE - TRAINING FOR A UTONOMOUS D RIVING VIA S ELF - SUPERVISED G EOMETRIC M ODELING Supplementary Materials In this Supplementary document, we first provide detailed network structures in Sec. A. More description and visual illustrations of the downstream tasks are discussed in Sec.

Revised: B. Last, we discuss limitations and common failure cases in Sec.

Revised: C.

Revised: A N ETWORK D ETAILS

Revised: For all experiments, the backbone of the visual encoder is ResNet-34 (He et al., 2016), and the detailed structure of it is provided in Table 5.

Revised: For DepthNet and PoseNet, we follow the same model structure as Godard et al.

Revised: with a two-layer MLP focal length head and a two-layer MLP optical center head added to the bottleneck of the PoseNet to predict the intrinsic matrix.

Revised: Please refer to Godard et al.

Revised: for model details.

Revised: For the Navigation, Navigation Dynamic, and Reinforcement Learning tasks, we use CILRS (Codevilla et al., 2019) and the model details are provided in Table 6.

Revised: For the Leaderboard Town05-long task, TCP (Wu et al., 2022) is chosen as our agent

Revised: [Table]

Revised: and we refer readers to Wu et al.

Revised: for model details.

Revised: For the nuScenes Planning, the trajectory planning model structure is shown in Table 7.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: B D OWNSTREAM T ASKS D ETAILS

Revised: For Navigation and Navigation Dynamic , training data is collected in Town

Revised: [Table]

Revised: and the closed-loop testing is conducted in Town02.

Revised: The maps of Town01 and Town02 are shown in Fig.

Revised: The agent needs to follow a series of sparse waypoints to navigate from the start point to the end point and avoid collisions.

Revised: The difference between Navigation and Navigation Dynamic is that there are other dynamic vehicles and pedestrians in the town.

Revised: Examples are provided in Fig.

Revised: The detailed results of the Successful Rate on these two tasks are provided in Table 8 and Table 9.

Revised: The Leaderboard-Town05-long task is more close to real-world urban driving, with different challenging scenarios added to the route.

Revised: The map of Town05 is shown in Fig.

Revised: 6 and some scenario examples are shown in Fig.

Revised: Besides comparing with pre-training methods on TCP (Wu et al., 2022), we also compare with other end-to-end autonomous driving methods on Leaderboard-town05-long task.

Revised: As shown in Table 10, using our pre-training method could bring improvements to end-to-end autonomous driving methods.

Revised: [Table]

Revised: Town

Revised: Town

Revised: Town

Revised: C L IMITATIONS

Revised: In this part, we analyze some failure cases and limitations of our method.

Revised: Since the visual encoder need to predict the future motion based on a single front-view image, there might be some factors that directly influence the driving decision not shown in the image ( e

Revised: [Table]

Revised: g. , vehicles behind the ego vehicle, factors related to the driver, navigation information).

Revised: Some of such cases are provided in Fig.

Revised: In these cases, the visual encoder does not get enough information to make the correct prediction.

Revised: These samples during training may hamper the learning process.

Revised: After training, one may use the difference between the prediction from PoseNet and that from visual encoder to filter out these samples, and re-train the visual encoder.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]