CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion
Abstract
Masked Image Modeling (MIM) has recently been established as a potent pretraining paradigm.
A pretext task is constructed by masking patches in an input image, and this masked content is then predicted by a neural network using visible patches as sole input.
This pre-training leads to state-of-the-art performance when finetuned for high-level semantic tasks, e.g . image classification and object detection.
In this paper we instead seek to learn representations that transfer well to a wide variety of 3D vision and lower-level geometric downstream tasks, such as depth prediction or optical flow estimation.
Inspired by MIM, we propose an unsupervised representation learning task trained from pairs of images showing the same scene from different viewpoints.
More precisely, we propose the pretext task of cross-view completion where the first input image is partially masked, and this masked content has to be reconstructed from the visible content and the second image.
In single-view MIM, the masked content often cannot be inferred precisely from the visible portion only, so the model learns to act as a prior influenced by high-level semantics.
In contrast, this ambiguity can be resolved with crossview completion from the second unmasked image, on the condition that the model is able to understand the spatial relationship between the two images.
Our experiments show that our pretext task leads to significantly improved performance for monocular 3D vision downstream tasks such as depth estimation.
In addition, our model can be directly applied to binocular downstream tasks like optical flow or relative camera pose estimation, for which we obtain competitive results without bells and whistles, i.e ., using a generic architecture without any task-specific design.
of the same scene, cross-view completion makes precise reconstruction possible, assuming that the model is able to understand both the scene geometry and the spatial relationship between the two images.
Introduction
Self-supervised learning for model pre-training has allowed to achieve state-of-the-art performance in a number of high-level computer vision tasks, such as image classification or object detection.
Contrary to traditional supervised learning, these models enable the use of unlabelled data via carefully designed pretext tasks.
A popular method for self-supervision is instance discrimination [12, 13, 17, 28, 37, 39, 83] which constructs a pretext task by learning representations that are invariant to various data augmentations.
More recently, Masked Image Modeling (MIM) [7, 16, 19, 32, 38, 60, 93] has emerged as a powerful alternative for self-supervision.
Inspired by BERT [24], these models are trained using an auto-completion pretext task.
An encoder, usually a Vision Transformer (ViT) [26], takes a partial view of an image input, obtained by splitting the image into patches and masking some of them, and encodes it into a latent representation.
The masked patches are then predicted by a decoder using the latent representation.
To solve the task, the model must leverage the context given by the visible portion and act as a prior for the ambiguous content that cannot be deduced from them.
In practice these models are typically trained on object-centric datasets such as ImageNet [67] and thus tend to learn high-level semantic information; that makes them well suited for tasks such as image classification or object detection [4, 38, 47].
In this paper, we propose a self-supervised training objective specially designed to learn 3D geometry from unlabeled data, named Cross-view Completion , or CroCo in short.
Given two images depicting the same scene, random parts of the first input image are masked and then predicted by the model using both (1) the visible parts of this first image, as well as (2) a second image called reference image as it depicts the same scene from a different point of view, see Figure 1.
While multi-view image completion has a long history in image editing [22, 92], we are the first to explore its potential as a self-supervised representation learning tool.
In contrast to single-view completion, the proposed pretext task of cross-view completion allows to perform masked image modeling conditionally on a second view.
In this case, most of the ambiguity can be resolved by reasoning about the scene geometry and spatial relationship between the two views.
This is illustrated in the reconstruction examples of our model in Figure 2.
Figure 3 provides an overview of our self-supervised model during pre-training.
We divide both images into sets of non-overlapping patches, denoted as tokens.
Most tokens from the first image are randomly discarded, and the remaining ones are fed to an image encoder, which we implement using a Vision Transformer (ViT) backbone [26].
All patches from the second (reference) image are encoded in a Siamese manner, i.e ., using the same encoder with shared weights.
The token latent representations output by the encoder from both images, including tokens to account for the masked patches of the first image, are then fed to a decoder whose goal is to predict the appearance of hidden patches.
To this aim, we use a series of transformer decoder blocks comprising cross-attention layers.
This allows non-masked tokens from the first image to attend tokens from the reference image, thus enabling cross-view comparison and reasoning.
Our model is trained using a simple pixel reconstruction loss over all masked patches, similar to MAE [38].
To finetune the model on downstream tasks that process only a single image, e.g . monocular depth estimation, the decoder can be discarded and we use the pre-trained encoder alone.
In the case of binocular tasks such as optical flow estimation, the original CroCo architecture is used as is.
For pre-training, CroCo relies on pairs of images depicting the same scene; in this work the pairs were obtained from synthetic renderings of indoor scenes produced with the Habitat [68] simulator.
We empirically show that high masking ratios, e.g . 90%, lead to the best pre-training performance.
Our model is evaluated on monocular downstream tasks, in particular depth estimation on the NYUv dataset [70] and a diverse set of dense 2D and 3D regression tasks taken from Taskonomy [91].
We show that CroCo leads to significantly better performance compared to existing MIM models, whether they were pre-trained on the same data or on ImageNet [67].
When evaluated on monocular high-level semantic tasks, such as ImageNet classification, CroCo obtains lower performance compared to established MIM models, however.
This essentially comes from our use of indoor scenes for pretraining, instead of highly semantic datasets like ImageNet, as empirically shown in our experiments.
Lastly, we demonstrate that CroCo can be applied to binocular downstream tasks in a straightforward manner without bells and whistles.
For instance, optical flow estimation can be performed by directly regressing 2 values per pixel using the decoder, and likewise, relative pose regression is achieved by simply appending a pose regression head to CroCo.
In both cases, we show that CroCo pre-training leads to competitive results in these 3D vision downstream tasks.
Related work
Self-supervised representation learning is a paradigm developed to learn visual features from largescale sets of unlabeled data [44], before finetuning the model on downstream tasks, e.g . classification or detection.
To achieve that, a pretext task is designed and exploits inherent data attributes to automatically generate surrogate labels.
The earliest principle behind most existing pretext tasks for computer vision revolves around purposefully removing some information from an image, e.g .
the color, the orientation or the ordering of a sequence of patches obtained from the image that the model then learns to recover [28, 34, 57, 58].
It has been shown that despite the lack of any semantic supervision, networks trained to recover this artificially removed information learn useful representations and facilitate the training of various supervised downstream tasks [1, 31].
Recently, the paradigm of instance discrimination has received a lot of attention, achieving highly competitive results in self-supervised visual representation learning [2, 13, 14, 17, 37, 39].
It seeks to learn outputs that are invariant to well designed classes of data augmentation.
Positive image pairs, obtained from the same instance by data augmentation, are pulled closer by the model, while samples obtained from different instances are pushed apart in the embedding space.
More recently, motivated by the success of BERT [24] in NLP and by the introduction of Vision
Transformers (ViT) [26], a variety of masked image prediction methods for self-supervised pretraining of vision models has been proposed.
Reminiscent of denoising autoencoders [78] or context encoders [60], and aiming to reconstruct masked pixels [3, 16, 26, 30, 38, 85], discrete tokens [7, 95] or deep features [5, 82], these methods have demonstrated the ability to scale to large datasets and models and achieve state-of-the-art results on various downstream tasks.
In particular, the masked autoencoder (MAE) [38] accelerates pre-training by using an asymmetric architecture that consists of a large encoder that operates only on unmasked patches followed by a lightweight decoder that reconstructs the masked patches from the latent representation and mask tokens.
MultiMAE [4] leverages the efficiency of the MAE approach and extends it to multi-modal and multitask settings.
Rather than masking input tokens randomly, the Masked Self-Supervised Transformer model (MST) [48] proposes to rely on the attention maps produced by a teacher network to dynamically mask low response regions of the input, and a student network is then trained to reconstruct it.
Self-supervised pre
-training for dense downstream tasks .
Seminal self-supervised methods [18, 37] were designed to output global image representations, thus encoding limited local information, and thereby hindering transferability of the learned models to downstream tasks involving dense per-pixel predictions, such as semantic segmentation.
To alleviate this issue, several self-supervised methods have been proposed to perform contrastive learning on dense local representations rather than global ones [51, 81, 84].
In contrast, InsLoc [87] proposes to paste image instances at various locations and scales onto background images, then predicts instance categories and foreground bounding boxes in the composed images.
In VADeR [61] and FlowE [86], dense image representations are learned for several dense semantic downstream tasks such as object detection and semantic segmentation, using multi-hierarchy features [80], while [90] combines multi-resolution parallel modules with local-window self-attention to improve the memory and computation efficiency of dense prediction.
These self-supervision methods lead to improved performance on dense semantic downstream tasks such as object detection or semantic/instance segmentation.
However, their design and evaluation protocols are not focused on 3D vision and lower-level geometric tasks such as depth, motion and flow estimation.
For such tasks, the self-supervision signal can be obtained via view synthesis, where the model is trained by enforcing photometric consistency [36, 56, 88, 97], or from RGB-D data, where the model is trained to match 2D visual features with 3D geometric representations [29, 41, 52].
Cross-view Completion Pre-training
In this section we present CroCo, our self-supervised pre-training method based on cross-view completion and tailored to 3D vision tasks.
Overview .
Figure 3 gives an overview of the CroCo architecture.
Let x 1 and x 2 be two images of the same scene taken from different viewpoints.
Both images are divided into N non-overlapping patches p 1 = { p 1 1 , . .
, p N 1 } , p 2 = { p 1 2 , . .
, p N 2 } , and a number n = ⌊ rN ⌋ of tokens from the first set p 1 is randomly masked, with r ∈ [0 , 1] being a masking ratio hyper-parameter.
We typically use r = 0 .
9 , i.e ., 90% of patches from p 1 are discarded.
We denote the remaining set of visible patches from the first image by ˜ p
= { p i 1 | m i = 0 } , with m i = 0 indicating that the patch p i 1 is not masked ( m i = 1 otherwise).
An encoder E θ processes ˜ p 1 and p 2 independently, and a decoder D ϕ takes the encoding E θ ( ˜ p 1 ) , conditioned on encoding E θ ( p 2 ) , in order to reconstruct p 1 :
[Equation]
Details on the encoder E θ .
A Siamese network denoted by E θ with shared weights θ , implemented as a ViT [26], serves to encode the two input images previously split into independent patch sets ˜ p 1 and p 2 .
In this study, we consider images of resolution 224 × 224 , with a patch size of 16 ×pixels.
Following ViT, the encoder consists in a linear projection of the flattened input RGB patches, to which sinusoidal positional embeddings are added, followed by a series of transformer blocks [77], each composed of a multi-head self-attention block and an MLP (Multi-Layer Perceptron).
Details on the decoder D ϕ .
The decoder D ϕ with weights ϕ takes as input the set of encoded tokens from the first image
E θ
( ˜ p 1 ) concatenated with a learned representation e mask that is repeated n times to account for the masked patches.
These tokens are iteratively processed by an attention-based block that also takes into account the encoded tokens E θ ( p 2 ) from the reference image x 2 .
A sinusoidal positional encoding is added to all tokens before being decoded by D ϕ .
We experiment with two different architectures for the decoder block that are illustrated in Figure 4, see Appendix B for detailed equations.
The first one, termed CrossBlock , consists of (a) multi-head self-attention on the tokens representing the first image; (b) multi-head cross-attention between these tokens and the tokens in E θ ( p 2 ) from the reference image, and (c) an MLP.
The second architecture, termed CatBlock , proceeds by concatenating the tokens from the two images, with an added learnable embedding for each of the two images, and then feeding these tokens to a series of standard transformer blocks, i.e ., each consisting of (a) multi-head self-attention and (b) an MLP layer.
Only the subset of tokens corresponding to the first image is taken into account for the final prediction.
These two attention blocks represent a trade-off between model size and computational cost.
Indeed, CrossBlock has more learnable parameters, due to the cross-attention module, but a lower computational cost as it avoids the quadratic complexity of computing self-attention over the joint token set of size 2 N .
Training the CroCo model .
The decoder produces one feature vector per patch token, which is processed by a fully-connected layer that outputs 3 values (R, G and B) per pixel for each patch, i.e ., 16 × 16 × 3 = 768 values for squared patches with side 16 .
These outputs serve as predicted reconstruction ˆ p 1 , evaluated using a reconstruction error such as the Mean Squared Error (MSE) loss between predictions and ground-truth values of the pixels, averaged over the set of all masked tokens, denoted p 1 \ ˜ p 1 .
Thus to perform self-supervised pre-training, the network is trained to minimize:
[Equation]
We additionally try a variant of the loss where each target patch is normalized using the mean and standard deviation of all pixel values within this patch.
Pre-training details .
We implement our CroCo model in PyTorch [59] and train the network forepochs using the AdamW optimizer [54].
We use a cosine learning rate schedule with a base learning rate of 1 . 5 × 10 − 4 for an effective batch size of 256; with a linear warmup in the first 40 epochs.
As encoder, we use a ViT-Base/16 backbone, i.e ., a series of 12 transformer blocks with 768 dimensions and 12 heads for self-attention with patches of size 16 × 16 .
For both CrossBlock and CatBlock decoders, we use a series of 8 blocks with 512 dimensions and 16 attention heads.
To account for the different dimensions of the encoder and the decoder, we apply a fully-connected layer between them.
Pre-training data.
We train our model on a set of synthetic image pairs of 3D indoor scenes derived from the HM3D [64], ScanNet [23], Replica [72] and ReplicaCAD [75] datasets as follows.
In each 3D scene, we randomly sample up to 1000 pairs of camera viewpoints with a co-visibility greater than 50%, and render these pairs of images using the Habitat simulator [68].
In total, we generated a dataset of 1,821,391 pairs, that we refer to as Habitat in this paper.
[Figure] [Figure]
ADE Semantic segm.
(mIoU) ↑
[Figure]
NYUv2 depth (Acc@
[Figure]
25) ↑
[Figure] [Figure]
Taskonomy (avg) ↓
Finetuning CroCo.
Our CroCo model can be used as pre-training to solve either monocular or binocular tasks, dense or not.
For monocular task, the ViT-Base encoder is used alone, while binocular tasks benefit from the pre-training of both the encoder and the decoder.
Experimental results
We evaluate CroCo on both monocular downstream tasks in Section 4.1, for which we provide ablations (Section 4.1.1) and a comparison to the state of the art (Section 4.1.2), and on binocular tasks in Section 4.2.
Training details for each task are available in the appendix.
4.1 Monocular transfer tasks
High-level semantic tasks .
While targeting 3D vision tasks, we still evaluate some high-level semantic tasks, namely image classification with linear probing on ImageNet-1K [67] and report top-1 accuracy.
We follow the exact same protocol as MAE [38] for that, with global average pooling for CroCo as we did not include a [CLS] token in our model.
Additionally, we report results on semantic segmentation on ADE20k [94] with 150 classes and 20,210 training images.
We follow the protocol of [4] and report mean Intersection-over-Union (mIoU) on the validation set, using the same
ConvNext [53] prediction head on top of the encoder.
3D vision tasks .
We evaluate monocular depth prediction on NYUv2 [70] (795 training and 655 test images) by reporting δ 1 , i.e ., the percentage of pixels that have an error ratio (max over prediction divided by ground-truth and its inverse) below 1 . 25 .
We additionally report results on the Taskonomy dataset [91] (800 training images) and report L1-loss on the ‘tiny’ test set (54,514 images) for the same 8 tasks as [4]: curvature, depth, edges, 2D keypoints, 3D keypoints, normal, occlusion and reshading .
For better clarity with decimal number, we report the L1-losses multiplied by 1000.
We also report the average ranking (rank.)
over the 8 tasks in tables.
For these tasks, we use a DPT head [65] on top of the encoder.
We additionally experiment on the monocular task of absolute camera pose regression in Appendix D.4 and the binocular task of stereo image matching in Appendix E.3.
4.1.1 Ablations
Masking ratio .
We first report in Figure 5 the performance on semantic segmentation (ADE), depth estimation (NYUv2) and Taskonomy when varying the masking ratio r using a CrossBlock decoder and a loss without normalization.
We observe that the best performance is obtained with a masking ratio of r = 90% .
This optimal ratio is higher than what was found for instance in MAE [38] for the auto-completion task.
We attribute this to the help provided by the reference image in cross-view completion.
We show some reconstruction examples in Figure 2.
Despite the small number of visible patches, this amount, combined with the reference view, seems to provide sufficient information to reconstruct the first image.
Note that the reconstructions tend to be blurry.
This is due to the MSE loss, but as noted in MAE [38], beyond a certain point, sharper reconstructions do not necessarily lead to better pre-training.
Normalized targets .
Regressing RGB values normalized by the mean and standard deviation within each patch has proven to be effective for MAE [38].
The first two rows of Table 1 compare the results with or without this normalization for CroCo.
We observe that it indeed consistently improves the performance on all tasks, and we therefore use it in the remainder of the experiments.
Normalized
Decoder
ADE
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
Pre
[Figure]
training epochs
NYUv2 depth (Acc@1.25)
[Figure] [Figure]
Pre-training epochs
Taskonomy (avg) ↓
Decoder architecture .
In Table 1, we compare decoders built with the two proposed attention blocks: CrossBlock which uses cross-attention to exchange information between the two images, and
CatBlock which concatenates the tokens from both images.
Note that CatBlock has a smaller number of learnable parameters but a higher number of FLOPs.
The performances obtained with the two proposed architectures are quite similar; CatBlock yielding slightly better performance for semantic segmentation or depth estimation while CrossBlock performs better on Taskonomy.
We favor the use of CrossBlock in what follows.
In Appendix C.3, we provide an ablation study on the number of blocks used in the decoder.
Leveraging the decoder .
Note that the pre-trained decoder can also be used when finetuning the model on monocular tasks: it suffices to feed the input image twice to the whole CroCo network, which in practice is done by duplicating the encoder output before passing it to the decoder.
While this significantly increases the computational cost of the model, it yields a consistent gain of performance from 40 .
6 to 41 .
on ADE, 86 .
1 to 88 .
1 on NYUv2, and an improvement on 6 out of 8 tasks on
Taskonomy, see further details in Appendix D.2.
Training profile .
In Figure 6 we show the performance on the downstream tasks as the pre-training progresses up to 400 epochs, when using CrossBlock or CatBlock in the decoder.
We observe that the performance on 3D vision tasks – depth estimation on NYUv2 and Taskonomy – reaches a plateau and thus we evaluate our model pre-trained for 400 epochs in what follows.
We additionally train a MAE model 1 on the Habitat dataset, using all available images ( i.e ., twice the number of pairs).
When pre-training on Habitat, both models achieve similar performance on the task of semantic segmentation, while CroCo significantly outperforms MAE on depth prediction and Taskonomy.
This shows the benefit of using cross-view completion pre-training, rather than pure auto-completion, for 3D vision downstream tasks.
Training pairs .
CroCo pre-training requires pairs of images, which we obtain in our experiments by sampling two different points of view using the Habitat simulator.
As an alternative, we evaluate using geometric transformations (homographies, rotations, scaling, crop, etc .) applied to an input image in order to generate the reference view; we report the results in Table 2 after 400 epochs of pre-training on Habitat.
We observe a clear drop in performance on all downstream tasks, i.e ., semantic segmentation, monocular depth estimation and Taskonomy.
One hypothesis is that in this case – when there exists a synthetic transform between the two images – the model can solve the
[Table] [Table]
pretext task of cross-view completion by directly fitting this transform, without reasoning about the geometry of the scene.
This in turn raises the question of how to sample optimal viewpoints when generating synthetic image pairs using the Habitat simulator.
To answer it, we study the relation between downstream performance and co-visibility in pre-training image pairs ( i.e ., how much visual content is shared between the two images) in Appendix C.2.
In short, it shows that using image pairs with a co-visibility ratio of approximately 0.5 leads to the best pre-trained models, and that on the one hand, very little co-visibility between the two images is sub-optimal as the task boils down to auto-completion, and on the other hand , if the two images composing a pair overlap too much, performance drops as the pre-training task becomes trivial.
4.1.2 Comparison to the state of the art
In Table 3, we compare the proposed CroCo pre-training strategy to the state of the art on monocular tasks.
In particular, we compare to DINO [14], a state-of-the-art self-supervised method based on instance discrimination, and to MIM methods with MAE [38] and MultiMAE [4].
The latter uses extra data modalities, namely depth and semantic segmentation generated by off-the-shelf supervised models for pre-training.
All three methods rely on the ImageNet-1K (IN1K) dataset [67] for pretraining.
This leads to high performance on high-level semantic tasks compared to our approach.
Note that DINO, which outputs a global representation, performs best on ImageNet-1K, while MIM-based approaches, which already output dense patch-level predictions during pre-training, obtain better results on dense tasks such as semantic segmentation or depth.
To measure the impact of the pre-training dataset, we report the performance of a MAE model trained on Habitat, i.e ., with exactly the same data and ViT-Base architecture as used for CroCo.
In this case, the performance on semantic tasks largely drops compared to pre-training on ImageNet ( e.g . 75.1% to 32.5% in classification on IN1K, see Table 3).
This confirms that the high performance on semantic tasks are mostly due to the use of IN1K for pre-training.
When evaluating 3D vision downstream tasks, such as depth estimation on NYUv2 or Taskonomy, CroCo significantly outperforms all other methods.
Interestingly, it even outperforms MultiMAE for depth estimation on NYUv2 with 85.6%
Acc@1.25 vs .
83.0%, while other approaches are below 80%.
On Taskonomy, CroCo performs best on 6 tasks out of 8 and is second best on the two other tasks, with an average rank of 1.25/5.
4.2 Applications to binocular tasks
We now empirically demonstrate that our CroCo model achieves competitive performance in two binocular tasks, namely optical flow and relative pose estimation, without any task-specific design.
[Table]
Optical Flow .
We treat optical flow as a straightforward regression task and do not change the pre-training architecture except for modifying the regression head to output two flow channels instead of 3 RGB color channels.
We finetune our network on the public 40,000 synthetic pairs from
AutoFlow [73], using a simple MSE loss on 224 × 224 crops, for 100 epochs, without any data augmentation besides color jittering.
We evaluate on the MPI-Sintel [10] dataset ( 1041 image pairs in the train split), on the clean and final renderings, using the average endpoint error (AEPE) metric.
To test on high resolution 1024 × 536 images, we regularly sample overlapping tiles of size 224 ×at the same position in both input images.
We regress the flow between each pair of corresponding tiles.
To recombine flow values, at each pixel location in the final prediction we use the flow value predicted by the nearest tile ( i.e ., based on the tile center).
In Table 4 we compare performance for various pre-training strategies and datasets on the MPI-Sintel training set.
Without pre-training, the network is unable to learn anything useful, as shown by the large AEPE values, despite thorough but unsuccessful hyper-parameter tuning.
This may be due to the high difficulty of the task coupled with the limited amount of training data.
The performance significantly improves when we initialize the encoder with an MAE pre-trained model.
With a CroCo initialization of both encoder and decoder weights, however, we again observe a large improvement with an average decrease of the error by almost 2 pixels w.r.t .
the best MAE model on both clean and final renderings.
Interestingly, we find that even with just 2 blocks in the decoder, a model pre-trained with CroCo still outperforms an MAE model having 8 decoder blocks (3.59/4.35 AEPE on clean/final, resp., compared to 4.63/5.24 AEPE for MAE; see Appendix E.1 for a complete ablation).
These results clearly demonstrate the benefits from pre-training with the cross-view completion task.
Our results (3.00/3.60 on clean/final, resp.) are slightly behind recent state-of-the-art approaches like
PWC-Net[74] (2.55/3.93), LiteFlowNet2 [42] (2.24/3.78) and RAFT [76] (1.43/2.71), all trained on
FlyingChairs [27] and FlyingThings [55] combined.
However, our CroCo model remains competitive for this task considering the simplicity of our approach.
In particular, note that (a) our model is trained from limited data without any sophisticated data augmentation, (b) our architecture is generic and not specially designed for optical flow prediction, unlike [74, 42, 76] which all rely on 4D cost volumes, (c) regression at test time is simple and straightforward, (d) the small 224 × 224 resolution used at test time hinders our performance on large displacements, a problem which we leave to future work.
We refer to Appendix E.1 for additional ablative experiments and comparisons.
Relative pose estimation .
We experiment on the relative pose regression (RPR) downstream task.
Given a pair of images, the goal is to predict the relative pose of the camera with respect to a reference view.
To do so, we feed to the CroCo model two images corresponding to normalized camera views, and we regress the rigid transformation (
R , t ) ∈
SO (3) × R 3 between these two views using a differentiable Procrustes layer [9] on top of the prediction head to ensure that R is a rotation matrix.
We finetune the RPR model with an MSE loss between the predicted relative pose ( R , t ) and a ground truth ( ˆ R , ˆ t ) :
∥ R − ˆ R ∥ 2 F + λ ∥ t − ˆ t ∥ 2 , with λ set to 100 in practice.
During training, we augment the training image pairs with random permutations, color jittering and virtual camera rotations.
We evaluate on the 7-scenes dataset [35], with a single model finetuned on the union of the official training sets of all scenes, and tested on the corresponding test splits.
Images are rescaled and cropped to a 224 × 224 resolution for simplicity.
At test time, for each image of the test split we retrieve the nearest image from the training set according to their AP-GeM-18 descriptors [66], and we predict the pose relative to this retrieved image.
For each scene, we report the standard median position and median orientation error in Table 5.
We compare results obtained with the model initialized with CroCo to that of a model with an encoder initialized with MAE (Habitat) and a decoder trained from scratch (last two rows).
Unsurprisingly,
[Table]
11.7cm, 4.53° 5.0cm, 3.46° *: fuse multiple pose predictions † : exploit temporal information and multi-step retrieval the former leads to significantly better results than the latter.
We furthermore compare our model pre-trained with CroCo to other state-of-the-art RPR methods which directly regress relative camera poses [6, 25] or essential matrices [96] as well as the the retrieval baseline consisting in predicting the identity as relative pose (see Table 5 upper part).
Our model outperforms all methods except
CamNet [25].
Note that the latter fuses multiple pose predictions, exploit temporal information and multi-step retrieval; in contrast our model directly predicts the relative pose from a (query, map) image pair, without any further processing, and is thus significantly simpler.
Discussion
We have introduced the novel task of cross-view completion for pre-training computer vision networks tailored to 3D vision downstream tasks.
Our experiments show that cross-view completion allows to learn representations better suited to 3D vision tasks than classical MIM with auto-completion, and straightforwardly transfer to monocular and binocular tasks.
A limitation of our model pretrained with cross-view completion is that it seems less tailored to high-level semantic tasks.
This is arguably more related to the choice of the dataset than to the pre-training task itself, and future work could explore the use of cross-view completion in conjunction with more object-centric datasets like ImageNet.
Our approach requires pairs of images depicting the same scene, and in this work, we leverage synthetic renderings only.
Future work could also extend it to real-world image pairs, which can be obtained without any supervision, for instance with Structure-from-Motion [49, 63] or geo-referencing.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Yes.
(b) Did you describe the limitations of your work?
Yes, in the Discussion section (Section 5).
(c) Did you discuss any potential negative societal impacts of your work?
No, we do not envision any potential negative societal impact beyond those that are generic to machine learning and data bias.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Yes. 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
N/A. (b) Did you include complete proofs of all theoretical results?
N/A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Not with the submission
but we are planning to release code if accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
All details are available in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
No, because pre-training is costly (about a week on 8 gpus), but our ablations with for instance the different decoder blocks concur towards the same conclusion that CroCo outperforms other pre-training strategies.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Yes, in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Yes.
(b) Did you mention the license of the assets?
Yes, in the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
No.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
N/A. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
N/A. 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
N/A. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
N/A. (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
N/A.
Appendix
This appendix is structured as follows.
In Section A, we first provide a list of acronyms used throughout the paper, followed by further details about our decoder architectures (Section B).
In
Section C we detail the parameter setting used in our pre-training and we provide an analysis about how the overlap between the training image pairs affects the pre-training and downstream performance (Section C.2) as well as an ablation on the impact of the decoder depth (Section C.3).
Section D then gives further details about monocular tasks, including a set of experiments on a complementary task, namely absolute pose regression (Section D.4).
In Section E, we give more details and visual examples concerning the binocular tasks as well as a complementary task, namely stereo matching (Section E.3).
Next, we list assets and computing resources in Section F. Finally, we provide additional reconstruction examples and introduce a video of sample scene reconstructions (Section G).
A List of acronyms
We provide the list of acronyms used in the paper, excluding acronyms related to papers or datasets with direct references.
acronym meaning
Acc@X
Accuracy at a certain error threshold X
AEPE
Average
EndPoint
Error
CroCo Cross-view Completion DINO DIstillation
with
NO labels
DPT Dense Prediction Transformer
FLOPs Floating-Point Operations
IN1K ImageNet-1K
mIoU mean Intersection-over-Union MAE Masked Auto-Encoder
MIM Masked Image Modeling MLP Multi-Layer Perceptron MSE Mean Squared Error NLP Natural Language Processing RPR Relative Pose Regression ViT
Vision Transformers [26]
B Details on the decoder architectures
In this section, we provide detailed equations about the blocks used in the CroCo model architecture.
For the encoder, we use a standard transformer block.
Let X ∈ R N × D be the N D -dimensional tokens as input to the transformer block.
The transformer block computes:
[Equation]
where W SQ , W SK , W SV are learnable parameters.
In practice, a bias is also learned and applied to these 3 projections but are omitted in the equations for the sake of clarity.
The attention itself is computed classically as:
[Equation]
where Proj denotes a projection layer, i.e ., a fully-connected layer.
We now provide equations for the two blocks we have tried in the decoder, namely the CrossBlock and the CatBlock .
In the CrossBlock decoder architecture, self-attention and cross-attention are used one after the other.
Let X, Y ∈ R N × D be the N input tokens to the block from the two views respectively.
The
CrossBlock output is computed by:
[Equation]
where W SQ , W SK , W SV denote learnable parameters for the self-attention and likewise W CQ , W CK , W CV for the cross-attention.
For the CatBlock decoder architecture, and following the same notations, we first concatenate X and
Y to form Z =
[ X + v 1 , Y + v 2 ] ∈ R 2 N × D before the first block, where v 1 , v 2 ∈ R D are learnable embeddings specifying the input view.
The CatBlock output is computed by:
[Equation]
C Further details on cross-view completion pre-training C.1 Detailed pre-training settings
We report below the detailed parameter setting we used in our pre-training.
Hyperparameters
Value
Optimizer
AdamW
[Equation]
[Figure]
1.5eWeight decay 0.
Adam β (0.9, 0.
[Equation]
[Figure]
Cosine
[Figure]
Warmup
[Figure]
rate 1eWarmup
epochs
[Table]
C.2 Ablation on overlaps between pre-training pairs
For pre-training, we use synthetic pairs generated using the Habitat simulator, keeping pairs with a co-visibility ratio over 0 . 5 .
Figure 7 presents some statistics about the distribution of viewpoints considered.
Intuitively, the choice of this co-visibility threshold was guided by two important observations: (a) if two images composing a pair overlap too little, the task boils down to autocompletion, therefore we set a threshold on the minimal co-visibility ratio of 0.5 in our main experiments, (b) if two images composing a pair overlap too much, the task becomes trivial, therefore we encourage large viewpoint changes between images in our training set.
In this section, we present an ablation to better understand what makes good pre-training pairs.
Formally, we define the visibility ratio v i,j of a view i with respect to an other view j as the ratio of pixels of image i that are visible in image j .
We ignore pixels where no geometry is rendered by the
Habitat simulator for this computation.
We similarly define the co-visibility ratio between two views i and j as min( v i,j , v j,i ) .
To study the influence of co-visibility on pre-training, we generate multiple training sets each composed of 700 , 000 image pairs with different co-visibility distributions, and pre-train CroCo on these sets for a number of steps equivalent to 200 epochs of the original dataset, while keeping all other parameters fixed.
Figure 8 provides examples of such training pairs.
We report in Figure 9 the performance achieved when pre-training with such pairs and then finetuning on
Taskonomy tasks.
Overall, we observe better performance for most 3D-related tasks (in particular for curvature , depth , keypoints3d , normal , reshading ) when pre-training with pairs with a co-visibility ratio close to 0.5, compared to pre-training with pairs of greater or lower co-visibility (blue curve in
Figure 9).
We also experimented with pairs chosen to have co-visibility ratios uniformly distributed over different value ranges and report results in Figure 9.
We find that pre-training with co-visibility ratios uniformly distributed over [0 , 1 . 0] leads to worse results than when exclusively sampling pairs with a co-visibility ratio within [0 . 4 , 0 . 5] .
This confirms that pairs having an intermediate co-visibility ratio are the most suitable for pre-training, although more extensive experiments would be required to strongly support these findings.
[Figure]
C.3 Ablation on decoder depth
In this section
[Table]
we report an ablation on the decoder depth,
i
[Table]
e ., varying the number of decoder blocks for a decoder using CrossBlock .
The results are shown in Table 6.
Note that in the main paper we used a decoder with 8 blocks.
We observe that the decoder depth has overall little impact on the performance for monocular tasks (semantic segmentation on ADE, depth prediction on NYUv2 or
Taskonomy dense tasks).
For the binocular task of optical flow estimation on MPI-Sintel, the error clearly decreases as the depth is increased, showing the importance of a deeper decoder when the decoder is also leveraged.
Note however that even with a decoder of depth 2, which means that the network comparing the two images is extremely shallow, a competitive performance can still be reached for optical flow.
In particular, the performance with 2 decoder blocks is still largely superior to that of a MAE-pre-trained network having a deep decoder (8 blocks) finetuned in the same conditions.
[Table]
D Further details and results for the monocular downstream tasks D.1 Detailed finetuning settings for monocular tasks
We provide in this section details of the finetuning settings used for our monocular tasks experiments.
Linear probing settings are summarized below, Note that they correspond exactly to the settings presented by the authors of MAE [38] with the LARS [89] optimizer and a large batch size.
[Table]
We further detail the training settings for semantic segmentation on ADE20k, monocular depth estimation on NYUv2 and Taskonomy regression tasks.
These settings closely match those presented by the authors of MultiMAE [4].
Hyperparameters
[Table]
NYUv
[Table]
Optimizer
AdamW [54]
AdamW [54] AdamW [54]
[Table] [Table]
Learning
[Table]
sched.
[Table]
Augmentation
Large
[Table]
D.2 Leveraging the decoder for monocular tasks
By default, for dense monocular tasks we append a DPT module [65] to the encoder of our CroCo model; this means that the decoder is discarded when finetuning.
We now discuss other possible ways of finetuning our CroCo model on downstream tasks.
• Using the encoder alone.
The simplest and most lightweight option is to use the encoder alone, without the decoder nor the DPT module, by appending an output linear prediction head, trained from scratch for the downstream task.
In Table 7 we see
by comparing rows 1 & 2 and rows 5 &
that removing the DPT module results in a significant degradation of performance across all tasks.
• Decoder instead of the DPT module.
The decoder of our CroCo model can be used instead of the DPT module.
While it does not fuse information from several layers at different depths, as the DPT module does, the decoder is trained to output dense predictions, and as such it should be easy to finetune for dense tasks other than RGB predictions.
To achieve this we initialize the full CroCo model with pre-trained weights, duplicate the encoded features from the input image to serve as input to the decoder, and simply replace the prediction head of the network by a new one trained from scratch.
In Table 7 we observe by comparing rows 2 & 3 as well as 6 & 7 that on most tasks, this leads to a very minor degradation of performance.
Furthermore, we found that finetuning the model this way was one order of magnitude faster than training a DPT module: convergence on NYUv2 requires approximately 1500 epochs for models without a decoder, against approximately 200 for models using the decoder.
• Decoder and DPT module.
The decoder can be used together with the DPT module, by extracting the features to be used as input to the DPT module from the CroCo decoder rather than from the encoder.
This increases the cost of running the model.
However, in Table 7 we see that this leads to consistent performance gains, by comparing rows 2 & 4, or rows 6 & 8.
• Frozen backbones.
In all cases, we can choose to freeze the backbone and only train the new prediction layers (DPT module or output prediction head when no DPT module is used).
Such an experiment is useful to probe what information the pre-trained model captures.
It can also help with over-fitting in cases where very little data is used to train for the downstream task.
Empirically, we observe by comparing the lower half of Table 7 to the upper part that freezing the backbone degrades the performance in a majority of cases.
On average, however, the performance remains surprisingly high, which demonstrates that frozen output features produced by the CroCo model are already meaningful representations for a wide diversity of 3D vision tasks.
[Table]
D.3 Visualization of monocular depth prediction and Taskonomy results
In Figure 10 we display input images from the NYUv2 dataset
Furthermore, in Figure 11 we present results on the Taskonomy dataset for the 8 dense regression tasks.
prediction ground truth prediction ground truth prediction ground truth c u r v a t u r e d e p t h e dg e s k e ypo i n t s 2d
k e ypo i n t s 3d no r m a l
o cc l u s i on r e s h a d
i ng
D.4 Application to absolute pose regression from a single image
Task and setting.
We apply our pre-trained model to the task of monocular absolute pose regression, where a model directly predicts the absolute camera pose of a given image in a fixed and given environment for which it was specifically trained.
In contrast to classical structure-based methods, camera localization methods based on deep neural networks directly regress the absolute pose of a query image [46, 45, 8, 79] and do not require database images or 3D maps at test time.
While these methods are in general significantly less accurate than structure-based methods [69, 71, 40, 43], they only require images and their corresponding camera poses as training data.
We use the evaluation code and protocol from AtLoc [79], a recent state-of-the-art improvement of the seminal PoseNet [46] work, which adds an attention module between the encoder output and the pose regressor heads to reweigh the encoded features.
The pose regressor is composed of two independent
MLP heads: one for predicting the camera pose and one for the camera rotation, represented as a quaternion.
In our experiments we append the same regression head (attention module and pose regressor heads) on top of the CroCo encoder, with a global average pooling layer inserted in-between to get a global image representation from the encoder tokens.
To finetune the model, we rely on the loss proposed in [8] and also used in [79]:
[Equation]
where β and γ are learned weights that balance the position loss and rotation loss and log q is the logarithmic form of a unit quaternion ( i.e ., the corresponding rotation vector).
To remove ambiguity between q and − q , all quaternions are restricted to the same hemisphere.
The finetuning parameters used for the pose regression are the followings:
Hyperparameters Value Optimizer Adam (Base learning rate = 5e-5; weight decay = 5e-4) Training epochs 500 (with batch size 64) dropout rate probability 0.5 weights β and γ
[Table]
We benchmark performance on the 7-Scenes dataset [35] on the test split of each scene considered independently.
We considered the RGB camera pose annotations from Kapture [43] and the images were rescaled and cropped to a 224 × 224 resolution with a constant 49° field of view (the largest value fitting in the original images).
Results .
We evaluate the impact of the choice of backbone and pre-training method in Table 8 as a function of the size of the training set.
The first row corresponds to a ResNet34 backbone, as used in
AtLoc, pre-trained with supervision on ImageNet.
Other rows correspond to a ViT-Base/16 backbone with different pre-training methods and datasets (MAE, MultiMAE, CroCo).
We observe that when using the full training set, all models achieve similar performance, except for the model initialized by
MultiMAE which consistently underperforms.
Note that CroCo achieves peak validation performance in approx.
50 − 100 epochs, while ResNet34 pre-trained models need approximately 500 epochs to converge.
To better assess the performance of the pre-training model, we significantly reduce the size of the training set and report results in Table 8 using only 5, 10 and 20% of the full training set.
We observe that the original AtLoc model, a pre-trained ResNet34 encoder, is unable to learn from such a small amount of data even with data augmentations (random color jittering and homographies simulating camera rotations).
Models pre-trained using MAE/MultiMAE perform better, but Croco pre-training leads to the best performance.
Finally, while a model trained from scratch (two last rows in Table 8) can achieve decent localization performances using the full training set compared to the other baselines, it generalizes poorly when trained using a more limited amount of data.
[Table]
E Further details and results on the binocular downstream tasks E.1 Optical Flow estimation
Experimental details .
We treat optical flow as a straightforward regression task and do not change the pre-training architecture except for modifying the regression head to output two flow channels instead of 3 RGB color channels: the two images are input as such in the Siamese encoders, and the decoder regresses two flow values ( u, v ) for each pixel using a simple linear head running independently for each output token.
We use a CrossBlock decoder with 8 layers (similar results are obtained with a CatBlock decoder).
Training details are provided in Table 9.
Qualitative Results .
We show in Figure 12 qualitative visualization of the flow predicted by our method.
Overall, we observe that the flow is correctly estimated, even in the case of varied and fast motion.
As a limitation, the estimation gets slightly inaccurate when occlusions happen, and blurry on extremely fine-grained motion such as hair tips, as for most methods.
Again, we wish to emphasize that the model was finetuned only on 40,000 images without any sort of elaborated data augmentation.
E.2 Relative pose regression
Model Architecture .
We replace the original head of the decoder pre-trained with CroCo by a simple head regressing a relative pose ( R , t ) ∈
SO (3) × R 3 .
This head consists of the following layers.
First, a linear projection reduces the dimension of tokens produced by the decoder to 64, in order to limit the computation costs.
Second, these tokens are flattened into a unique feature vector, which is
[Table]
processed by a MLP (with a hidden layer of size 1024 and ReLU activations) to regress a 12D output.
It is reshaped into an affine transformation ( M , t ) ∈ R 3 × 3 × R 3 .
Lastly we use a differentiable special Procrustes layer [9] to orthonormalize M into a rotation matrix R and to predict the relative pose ( R , t ) .
Experimental details .
The RGB camera pose annotations for the 7-scenes dataset [35] are obtained from Kapture [43].
To select training pairs, we extract global AP-GeM-18 descriptors [66] for every image, and match each training image with its 20 closest neighbors according to their descriptor similarity.
Our training set is composed of 26k images and 520k training pairs in total.
We rescale and crop the images to a square 224 × 224 resolution with a constant 49° field of view (the largest value fitting in the original images), and we augment the training image pairs with random permutations, color jittering and homographies that simulate camera rotation.
In our experiments, we trained models using the AdamW [54] optimizer using settings described in
Table 9.
We found the training to be quite sensitive to its initialization, and we had to try multiple random seeds to achieve decent performances on the training set when training the decoder from scratch with an encoder pre-trained with MAE on Habitat.
Using an encoder pre-trained with MAE on ImageNet furthermore always led to poor performance on both the training and test sets in our experiments.
We had none of these issues however when using an encoder and decoder pre-trained using CroCo.
E.3 Stereo image matching
Task and settings .
In this section, we experiment on the binocular downstream task of stereo matching [62].
We finetune a pre-trained CroCo model to predict pixel-wise disparity values, by simply replacing the linear prediction head.
The model takes two rectified images as input and predicts the disparity of every pixel by matching corresponding pixels in the images.
We use a
MSE-log loss for finetuning and report the error using the 3-pixel 5% discrepancy [15, 20].
We evaluate the stereo matching downstream task on the Virtual KITTI dataset [11] which consists of 5 synthetic sequences cloned from the KITTI tracking benchmark [33].
The dataset additionally provides 9 variants of these 5 sequences under different weather conditions ( e.g . ‘fog’, ‘rain’, etc .) and modified camera configurations ( e.g . rotation to left or right by 15 or 30 degrees).
We downscale
VKITTI images to a resolution of 224 × 742, preserving the aspect ratio and crop to 224 × 736.
Additional details on the finetuning hyper-parameters and settings are described in Table 9.
Results .
In Table 10 we report results obtained on all 10 weather condition and camera orientation variants in the VKITTI dataset.
We compare results obtained when finetuning the CroCo model and the MAE model with a randomly initialized decoder, pre-trained on Habitat.
We observe that
CroCo pre-training leads to significantly better results for all scenarios.
Next, we compare to the state-of-the-art methods for stereo matching, in particular, PSMNet [15], LaC-GweNet [50] and
LEAStereo [20].
Finetuning a model pre-trained with CroCo achieves results competitive with the state of the art, without any task-specific model design, such as spatial pyramid pooling [15], hierarchical neural search in 4D feature volume [20] or local similarity patterns for explicit neighbor relationships [50].
We point out that, in contrast to the other methods, our model is directly finetuned on VKITTI without pre-training on the large SceneFlow dataset [55].
In Figure 13 we visualize the disparity prediction by our method, for three different conditions in the VKITTI dataset
[Table] [Table]
F Compute resources, code and dataset assets F.1 Floating point operations (FLOPs)
We calculate the number of floating point operations (or FLOPs) in the most commonly accepted manner, i.e ., counting additions and multiplications as separate floating-point operations.
To that aim, we borrow and slightly adapt the flops_computation.py code from [21].
Note that most of the FLOPs comes from matrix-matrix multiplication operations in the attention and feed-forward layers.
For a standard ViT (encoder only), the number of FLOPs can thus be approximated as
[Equation]
where
[Equation]
For instance, for ViT-Base/16, setting the number of blocks L = 12 , the number of tokens N = 14and the embedding dimension D
= 768 , we find 34.7 GFLOPs with the formula above, which is close to the 35.3 GFLOPs calculated including all other operations (layer norms, patch embeddings, etc .).
Note that we report exact theoretical FLOPs (counting all operations) in the paper.
F.2 Compute resources
Pre-training a CroCo model for 400 epochs takes about 64 GPU-days on NVIDIA V100.
For instance on a 4-GPU server, this is about two weeks.
F.3 Assets used in this submission
We provide below an overview of assets used in our experiments and their licenses.
Asset
License
Habitat pre-training HM3D [64] academic, non-commercial research [hyperlink] ScanNet [23] non-commercial research and education [hyperlink] Replica [72] non-commercial research and education
[hyperlink] ReplicaCAD [75]
Creative Commons Attribution 4.0 International (CC
[Table]
G Further visual examples G.1 Video of sample CroCo reconstructions
We attach a video of reconstruction results obtained using the proposed CroCo model on some validation scenes from the HM3D dataset [64], unseen at training time (see Figure 14).
For each scene, we consider a constant reference input image and try to reconstruct a sequence of target images from masked inputs, each frame being processed independently using CroCo.
The capability of CroCo to solve this challenging task suggests that it is somehow able to capture the 3D layout of the scene, as well as the relative pose of the masked view with respect to the reference view.
G.2 Additional reconstruction examples
We provide additional reconstruction examples in Figure 15, where we compare reconstructions obtained using our CroCo model pre-trained with an RGB loss and a reference input image ( CroCo
RGB ) with reconstructions obtained after replacing this reference image by an image of random uniform noise, independent at each pixel ( CroCo RGB random noise ref. ).
Cross-view completion enables a decent reconstruction of the masked image in general, except for areas non-visible in the reference image ( e.g . CroCo RGB , left part of the painting in the third column).
When replacing the reference image by some random noise, the reconstruction problem becomes similar to inpainting and some parts of the scene may be wrongly reconstructed due to the lack of available information ( e.g . wrong size for the window in the first column, missing ice maker in the fridge in the second column).
We also compare reconstructions obtained with CroCo and MAE models pre-trained on Habitat to reconstruct normalized patches ( CroCo norm and MAE norm , last two rows).
We use patch statistics from the target image to un-normalize these reconstructions for visualization purposes, which explains that the mean color of each reconstructed patch is relatively well reconstructed in all cases.
Yet, we observe that CroCo produces more detailed reconstructions than MAE, the latter often appearing quite tessellated due to an inconsistent reconstruction of normalized patches.
R e f e r e n ce i npu t
M a s k e d i npu t
T a r g e t
C r o C
o R G
B
C r o C
o R G
B r a ndo m no
i s e r e f .
M A E no r m
C r o C o no r m