Self-Supervised Pretraining for 3D Vision Tasks by Cross-View Completion
Anonymous Author(s)
AffiliationAddress email
Abstract
Masked Image Modeling (MIM) has recently been established as a potent pretraining paradigm.
A pretext task is constructed by masking patches in an inputimage, and this masked content is then predicted by a neural network using visible patches as sole input.
This pretraining leads to state-of-the-art performancewhen finetuned for high-level semantic tasks, e.g . image classification and objectdetection.
In this paper we instead seek to learn representations that transfer wellto a wide variety of 3D vision and lower-level geometric downstream tasks, suchas depth prediction or optical flow estimation.
Inspired by MIM, we propose anunsupervised representation learning task trained from pairs of images showingthe same scene from different viewpoints.
More precisely, we propose the pretexttask of cross-view completion where the first input image is partially masked, andthis masked content has to be reconstructed from the visible content and the secondimage.
In single-view MIM the masked content often cannot be inferred preciselyfrom the visible portion only, so the model learns to act as a prior influenced byhigh-level semantics.
In contrast, this ambiguity can be resolved with cross-viewcompletion from the second unmasked image, on the condition that the model isable to understand the spatial relationship between the two images.
Our experiments show that our pretext task leads to significantly improved performance formonocular 3D vision downstream tasks such as depth estimation.
In addition, ourmodel can be, by its design, directly applied to binocular downstream tasks such asoptical flow or relative camera pose estimation, for which we obtain competitiveresults without bells and whistles, i.e ., using a generic architecture without anytask-specific design.
precise reconstruction possible, assuming that the model is able to understand both the scene geometry and the spatial relationship between the two images.
Introduction
Self-supervised learning for model pretraining has allowed to achieve state-of-the-art performancein a number of high-level computer vision tasks, such as image classification or object detection.
Contrary to traditional supervised learning, these models enable the use of unlabelled data viacarefully designed pretext tasks.
A popular method for self-supervision is instance discrimination [9,10, 13, 22, 30, 32, 67] which constructs a pretext task by learning representations that are invariant tovarious data augmentations.
More recently, Masked Image Modeling (MIM) [6, 12, 15, 26, 31, 48, 76]has emerged as a powerful alternative for self-supervision.
Inspired by BERT models [18], thesemodels are trained using an auto-completion pretext task.
An encoder, usually a Vision Transformer(ViT) [20], takes a partial view of an image input, obtained by splitting the image into patches andmasking some of them, and encodes it into a latent representation.
The masked patches are thenpredicted by the decoder using the latent representation.
To solve the task, the model must leveragethe context given by the visible portion, and act as a prior for the ambiguous content that cannot bededuced from them.
In practice these models are typically trained on object-centric datasets such as
ImageNet and thus tend to learn high-level semantic information; that makes them well suited fortasks such as image classification or object detection [3, 31, 36].
In this paper, we propose a self-supervised training objective specially designed to learn 3D geometryfrom unlabeled data, named Cross-view Completion .
Given two images depicting the same scene,random parts of the first input image are masked and then predicted by the model using both (1) thevisible parts of this first image, as well as (2) a second image called reference image as it depicts thesame scene from a different point of view (see Figure 1).
While multi-view image completion has along history in image editing [16, 75], we are the first to explore its potential as a self-supervisedrepresentation learning tool.
In contrast to single-view completion, the proposed pretext task ofcross-view completion allows to perform masked image modeling conditionally on a second view.
Inthis case, most of the ambiguity can be resolved by reasoning about the scene geometry and spatialrelationship between the two views.
This is illustrated in the reconstruction examples of our modelthat we termed CroCo which standards for Cross-view Completion, in Figure 2. Figure 3 provides an overview of our self-supervised training model.
We divide both images intosets of non-overlapping patches, denoted as tokens.
Most tokens from the first image are randomlydiscarded, and the remaining ones are fed to an image encoder, which we implement using a Vision Transformer (ViT) backbone [20].
All patches from the second (reference) image are encoded in a Siamese manner, i.e ., using the same encoder with shared weights.
The token latent representationsoutput by the encoder from both images, including tokens to account for the masked patches of thefirst image, are then fed to a decoder whose goal is to predict the appearance of hidden patches.
Tothis aim, we use a series of transformer decoder blocks comprising cross-attention layers.
This allowsnon-masked tokens from the first image to attend tokens from the reference image, thus enablingcross-view comparison and reasoning.
Our model is trained using a simple pixel reconstructionloss over all masked patches, similar to MAE [31].
To finetune the model on downstream tasks thatprocess only a single image, e.g . monocular depth estimation, the decoder can be discarded and weuse the pretrained encoder alone.
In the case of binocular tasks such as optical flow estimation, theoriginal CroCo architecture is used as is.
For pretraining, CroCo relies on pairs of images depicting the same scene; in this work the pairs wereobtained from synthetic renderings of indoor scenes produced with the Habitat [55] simulator.
Weempirically show that high masking ratios, e.g . 90%, lead to the best pretraining performance.
Ourmodel is evaluated on monocular downstream tasks, in particular depth estimation on the NYUv2dataset [56] and a diverse set of dense 2D and 3D regression tasks taken from Taskonomy [74].
We show that CroCo leads to significantly better performance compared to existing MIM models,whether they were pretrained on the same data or on ImageNet.
When evaluated on monocular highlevel semantic tasks, such as ImageNet classification, CroCo obtains lower performance comparedto established MIM models, however.
This essentially comes from our use of indoor scenes forpretraining, instead of highly semantic datasets like ImageNet [54], as empirically shown in ourexperiments.
Lastly, we demonstrate that CroCo can be applied to binocular downstream tasks ina straightforward manner without bells and whistles.
For instance, optical flow estimation can beperformed by directly regressing 2 values per pixel using the decoder, and likewise, relative poseregression is achieved by simply appending a pose regression head to CroCo.
In both cases, we showthat CroCo pretraining leads to competitive results in these 3D vision downstream tasks.
Related work
Self-supervised representation learning is a paradigm developed to learn visual features from largescale sets of unlabeled data [35], before finetuning the model on downstream tasks, e.g . classificationor detection.
To achieve that, a pretext task is designed and exploits inherent data attributes toautomatically generate surrogate labels.
The earliest principle behind most existing pretext tasksfor computer vision revolves around purposefully removing some information from an image, e.g .the color, the orientation or the ordering of a sequence of patches obtained from the image thatthe model then learns to recover [22, 27, 45, 46].
It has been shown that despite the lack of anysemantic supervision, networks trained to recover this artificially removed information learn usefulrepresentations and facilitate the training of various supervised downstream tasks [1, 25].
Recently, the paradigm of instance discrimination has received a lot of attention, achieving highlycompetitive results in self-supervised visual representation learning [11, 10, 13, 30, 32].
It seeks tolearn outputs that are invariant to well designed classes of data augmentation.
Positive image pairs,obtained from the same instance by data augmentation, are pulled closer by the model, while samplesobtained from different instances are pushed apart in the embedding space.
More recently, motivated by the success of BERT [18] in NLP and by the introduction of Vision
Transformers (ViT) [20], a variety of masked image prediction methods for self-supervised pretraining of vision models has been proposed.
Reminiscent of denoising autoencoders [63] or contextencoders [48], and aiming to reconstruct masked pixels [2, 12, 20, 24, 31, 69], discrete tokens [6, 78]or deep features [4, 66], these methods have demonstrated the ability to scale to large datasets andmodels and achieve state-of-the-art results on various downstream tasks.
In particular, the maskedautoencoder (MAE) [31] accelerates pretraining by using an asymmetric architecture that consistsof a large encoder that operates only on unmasked patches followed by a lightweight decoder thatreconstructs the masked patches from the latent representation and mask tokens.
MultiMAE [3] leverages the efficiency of the MAE approach and extends it to multi-modal and multitask settings.
Ratherthan masking input tokens randomly, the Masked Self-Supervised Transformer model (MST) [37]proposes to rely on the attention maps produced by a teacher network to dynamically mask lowresponse regions of the input, and a student network is then trained to reconstruct it.
Self-supervised pretraining for dense downstream tasks .
Seminal self-supervised methods [14, 30]were designed to output global image representations, thus encoding limited local information, andthereby hindering transferability of the learned models to downstream tasks involving dense per-pixelpredictions, such as semantic segmentation.
To alleviate this issue, several self-supervised methodshave been proposed to perform contrastive learning on dense local representations rather than globalones [68, 65, 39].
In contrast, InsLoc [71] proposes to paste image instances at various locations andscales onto background images, then predicts instance categories and foreground bounding boxesin the composed images.
In VADeR [49] and FlowE [70], dense image representations are learnedfor several dense semantic downstream tasks such as object detection and semantic segmentation,using multi-hierarchy features [64], while [73] combines multi-resolution parallel modules with localwindow self-attention to improve the memory and computation efficiency of dense prediction.
Theseself-supervision methods lead to improved performance on dense semantic downstream tasks such asobject detection or semantic/instance segmentation.
However, their design and evaluation protocolsare not focused on 3D vision and lower-level geometric tasks.
To tackle such 3D downstream tasks(depth, motion, flow), self-supervision signal can be obtained via view synthesis, [80, 29, 44, 72]where the model is trained by enforcing photometric consistency or from RGB-D data [23, 40, 33],where the model learns to match visual 2D features with 3D geometric representations.
Cross-view Completion pretraining
In this section we present CroCo, our self-supervised pretraining method based on cross-viewcompletion and tailored to 3D vision tasks.
Overview .
Figure 3 gives an overview of the CroCo architecture.
Let x 1 and x 2 be two images ofthe same scene taken from different viewpoints.
Both images are divided into N non-overlappingpatches p 1 = { p 1 1 , . .
, p N 1 } , p 2 = { p 1 2 , . .
, p N 2 } , and a number n = ⌊ rN ⌋ of tokens from the firstset p 1 is randomly masked, with r ∈ [0 , 1] being a masking ratio hyper-parameter.
We typically user = 0 .
9 , i.e ., 90% of patches from p 1 are discarded.
We denote the remaining set of visible patchesfrom the first image by ˜ p
= { p i 1 | m i = 0 } , with m i = [-
-]0 indicating that the patch p i 1 is not masked( m i = 1 otherwise).
An encoder E θ processes ˜ p 1 and p 2 independently, and a decoder D ϕ takes the encoding E θ ( ˜ p 1 ) ,conditioned on encoding E θ ( p 2 ) , in order to reconstruct p 1 :
[Equation]
Details on the encoder E θ .
A Siamese network denoted by E θ with shared weights θ , implementedas a ViT [20], serves to encode the two input images previously split into independent patch sets˜ p 1 and p 2 .
In this study, we consider images of resolution 224 × 224 , with a patch size of 16 × 16pixels.
Following ViT, the encoder consists in a linear projection of the flattened input RGB patches,to which sinusoidal positional embeddings are added, followed by a series of transformer blocks [62],each composed of a multi-head self-attention block and an MLP (Multi-Layer Perceptron).
Details on the decoder D ϕ .
The decoder D ϕ with weights ϕ takes as input the set of encoded tokensfrom the first image
E θ
( ˜ p 1 ) concatenated with a learned representation e mask that is repeated n timesto account for the masked patches.
These tokens are iteratively processed by an attention-basedblock that also takes into account the encoded tokens E θ ( p 2 ) from the reference image x 2 .
Asinusoidal positional encoding is added to all tokens before being decoded by D ϕ .
We experimentwith two different architectures for the decoder block that are illustrated in Figure 4.
The first one,termed CrossBlock , consists of (a) multi-head self-attention on the tokens representing the first image;(b) multi-head cross-attention between these tokens and the tokens in E θ ( p 2 ) from the referenceimage, and (c) an MLP.
The second architecture, termed CatBlock , proceeds by concatenating thetokens from the two images, with an added learnable embedding for each of the two images, and thenfeeding these tokens to a series of standard transformer blocks, i.e ., each consisting of (a) multi-headself-attention and (b) an MLP layer.
Only the subset of tokens corresponding to the first image istaken into account for the final prediction.
These two attention blocks represent a trade-off betweenmodel size and computational cost.
Indeed, CrossBlock has more learnable parameters, due to thecross-attention module, but a lower computational cost as it avoids the quadratic complexity ofcomputing self-attention over the joint token set of size 2 N .
Training the CroCo model .
The decoder produces one feature vector per patch token, which isprocessed by a fully-connected layer that outputs 3 values (R, G and B) per pixel for each patch,e.g ., 16 × 16 × 3 = 768 values for squared patches with side 16 .
These outputs serve as predictedreconstruction ˆ p 1 , evaluated using a reconstruction error such as the Mean Squared Error (MSE) lossbetween predictions and ground-truth values of the pixels, averaged over the set of all masked tokens,denoted p 1 \ ˜ p 1 .
Thus to perform self-supervised pretraining, the network is trained to minimize:
[Equation]
( x 1 , x 2 )
[Equation]
We additionally try a variant of the loss where each target patch is normalized using the mean andstandard deviation of all pixel values within this patch.
Pretraining details .
We implement our CroCo model in PyTorch [47] and train the network for 400epochs using the AdamW optimizer [42].
We use a cosine learning rate schedule with a base learningrate of 1 . 5 × 10 − 4 for an effective batch size of 256; with a linear warmup in the first 40 epochs.
Asencoder, we use a ViT-Base/16 backbone, i.e ., a series of 12 transformer blocks with 768 dimensionsand 12 heads for self-attention with patches of size 16 × 16 .
For both CrossBlock and CatBlockdecoders, we use a series of 8 blocks with 512 dimensions and 16 attention heads.
To account for thedifferent dimensions of the encoder and the decoder, we apply a fully-connected layer between them.
Pretraining data.
We train our model on a set of synthetic image pairs of 3D indoor scenes derivedfrom the HM3D [51], ScanNet [17], Replica [57] and ReplicaCAD [60] datasets as follows.
In each3D scene, we randomly sample up to 1000 pairs of camera viewpoints with a co-visibility greaterthan 50%, and render these pairs of images using the Habitat simulator [55].
In total, we generated adataset of 1,821,391 pairs, that we refer to as Habitat in this paper.
Finetuning CroCo.
Our CroCo model can be used as pretraining to solve either
Masking ratio
ADE Semantic segm.
(mIoU) ↑ 0 .0 .0 .0 .0 .
Masking ratio
NYUv2 depth (Acc@1.25) ↑ 0 .0 .0 .0 .0 .
Masking ratio
Taskonomy
Experimental results
We evaluate CroCo on both downstream monocular tasks in Section 4.1, for which we provideablations (Section 4.1.1) and a comparison to the state of the art (Section 4.1.2), and on binoculartasks in Section 4.2.
Training details for each task are available in the supplementary material.
4.1 Monocular transfer tasks
High-level semantic tasks .
While targeting 3D vision tasks, we still evaluate some high-levelsemantic tasks, namely image classification with linear probing on ImageNet-1K [54] and reporttop-1 accuracy.
We follow the exact same protocol as MAE [31] for that, with global average poolingfor CroCo as we did not include a [CLS] token in our model.
Additionally, we report results onsemantic segmentation on ADE20k [77] with 150 classes and 20,210 training images.
We follow theprotocol of [3] and report mean Intersection-over-Union (mIoU) on the validation set, using the same
ConvNext [41] prediction head on top of the encoder.3D vision tasks .
We evaluate monocular depth prediction on NYUv2 [56] (795 training and 655 testimages) by reporting δ 1 , i.e ., the percentage of pixels that have an error ratio (max over predictiondivided by ground-truth and its inverse) below 1 . 25 .
We additionally report results on the Taskonomydataset [74] (800 training images) and report L1-loss on the ‘tiny’ test set (54,514 images) for thesame 8 tasks as [3]: curvature, depth, edges, 2D keypoints, 3D keypoints, normal, occlusion andreshading .
For better clarity with decimal number, we report the L1-losses multiplied by 1000.
Wealso report the average ranking (rank.)
over the 8 tasks in tables.
For these tasks, we use a DPThead [52] on top of the encoder.
We additionally experiment on the task of absolute camera poseregression in the supplementary material.
4.1.1 Ablations
Masking ratio .
We first report in Figure 5 the performance on semantic segmentation (ADE),depth estimation (NYUv2) and Taskonomy when varying the masking ratio r using a CrossBlockdecoder and a loss without normalization.
We can observe that the best performance is obtainedwith a masking ratio of r = 90% .
This optimal ratio is higher than what was found for instance in
MAE [31] for the auto-completion task.
We attribute this to the help provided by the reference imagein cross-view completion.
We show some reconstruction examples in Figure 2.
Despite the smallnumber of visible patches, this amount, combined with the reference view, seems to provide sufficientinformation to reconstruct the first image.
Note that the reconstructions tend to be blurry.
This is dueto the MSE loss, but as noted in MAE [31], beyond a certain point, sharper reconstructions do notnecessarily lead to better pretraining.
Normalized targets .
Regressing RGB values normalized by the mean and standard deviation withineach patch has proven to be effective for MAE [31]. The first two rows of Table 1 compare the resultswithout or with this normalization for CroCo.
We observe that it indeed consistently improves theperformance on all tasks, and we therefore use it in the remainder of the experiments.
Decoder architecture .
In Table 1, we compare decoders built with the two proposed attentionblocks: CrossBlock which uses cross-attention to exchange information between the two images, and
CatBlock which concatenates the tokens from both images.
Note that CatBlock has a smaller numberof learnable parameters but a higher number of FLOPs.
The performances obtained with the two
[Table]
Pretraining epochs
ADE Semantic
[Figure]
[Table]
[Figure]
Pretraining epochs
[Table]
[Table] [Table]
Pretraining epochs
Taskonomy
[Table]
avg) [Table] proposed architectures are quite similar; CatBlock yielding slightly better performance for semanticsegmentation or depth estimation while CrossBlock performs better on Taskonomy.
We favor the useof CrossBlock in what follows.
In the supplementary material, we provide an ablation study on thenumber of blocks used in the decoder.
Leveraging the decoder .
Note that the pretrained decoder can also be used when finetuning themodel on monocular tasks: it suffices to feed the input image twice to the whole CroCo network,which in practice is done by duplicating the encoder output before passing it to the decoder.
While thissignificantly increases the computational cost of the model, it yields a consistent gain of performancefrom 40 .
6 to 41 .
on ADE, 86 .
1 to 88 .
1 on NYUv2, and an improvement on 6 out of 8 tasks on
Taskonomy, see further details in the supplementary material.
Training profile .
In Figure 6 we show the performance on the downstream tasks as the pretrainingprogresses up to 400 epochs.
We observe that the performance on 3D vision tasks – depth estimationon NYUv2 and Taskonomy – reaches a plateau and thus we evaluate our model pretrained for 400epochs in what follows.
We additionally train a MAE model 1 on the Habitat dataset, using allavailable images ( i.e ., twice the number of pairs).
When pretraining on Habitat, both models achievesimilar performance on the task of semantic segmentation, while CroCo significantly outperforms
MAE on depth prediction and Taskonomy.
This shows the benefit of using cross-view completionpretraining, rather than pure auto-completion, for 3D vision downstream tasks.
4.1.2 Comparison to the state of the art
In Table 2, we compare the proposed CroCo pretraining strategy to the state of the art on monoculartasks.
In particular, we compare to DINO [11], a state-of-the-art self-supervised method based oninstance discrimination, and to MIM methods with MAE [31] and MultiMAE [3].
The latter usesextra data modalities, namely depth and semantic segmentation generated by off-the-shelf supervisedmodels for pretraining.
All three methods rely on the ImageNet-1K (IN1K) dataset [54] for pretraining.
This leads to high performance on high-level semantic tasks compared to our approach.
Note that
DINO, which outputs a global representation, performs best on ImageNet-1K, while MIM-basedapproaches, which already output dense patch-level predictions during pretraining, obtain betterresults on dense tasks such as semantic segmentation or depth.
[Table] [Table]
To measure the impact of the pretraining dataset, we report the performance of a MAE model trainedon Habitat, i.e ., with exactly the same data and ViT-Base architecture as used for CroCo.
In this case,the performance on semantic tasks largely drops compared to pretraining on ImageNet ( e.g . 75.1% to32.5% in classification on IN1K, see Table 2).
This confirms that the high performance on semantictasks are mostly due to the use of IN1K for pretraining.
When evaluating 3D vision downstreamtasks, such as depth estimation on NYUv2 or Taskonomy, CroCo significantly outperforms all othermethods.
Interestingly, it even outperforms MultiMAE for depth estimation on NYUv2 with 85.6%
Acc@1.25 vs .
83.0%, while other approaches are below 80%.
On Taskonomy, CroCo performs beston 6 tasks out of 8 and is second best on the two other tasks, with an average rank of 1.25/5.
4.2 Applications to binocular tasks
We now empirically demonstrate that our CroCo model achieves competitive performance in twobinocular tasks, namely optical flow and relative pose estimation, without any task-specific design.
Optical Flow .
We treat optical flow as a straightforward regression task and do not change thepretraining architecture except for modifying the regression head to output two flow channels insteadof 3 RGB color channels.
We finetune our network on the public 40,000 synthetic pairs from
AutoFlow [58], using a simple MSE loss on 224 × 224 crops, for 100 epochs, without any dataaugmentation besides color jittering.
We evaluate on the MPI-Sintel [8] dataset ( 1041 image pairs inthe train split), on the clean and final renderings, using the average endpoint error (AEPE) metric.
Totest on high resolution 1024 × 536 images, we regularly sample overlapping tiles of size 224 × 224at the same position in both input images.
We regress the flow between each pair of correspondingtiles.
To recombine flow values, at each pixel location in the final prediction we use the flow valuepredicted by the nearest tile ( i.e ., based on the tile center).
In Table 3 we compare performance for various pretraining strategies and datasets on the MPI-Sinteltraining set.
Without pretraining, the network is unable to learn anything useful, as shown by thelarge AEPE values, despite thorough but unsuccessful hyper-parameter tuning.
This may be due tothe high difficulty of the task coupled with the limited amount of training data.
The performancesignificantly improves when we initialize the encoder with an MAE-pretrained model.
With a CroCoinitialization of both encoder and decoder weights, however, we again observe a large improvementwith an average decrease of the error by almost 2 pixels w.r.t .
the best MAE model on both clean andfinal renderings.
Interestingly, we find that even with just 2 blocks in the decoder, a model pretrainedwith CroCo still outperforms an MAE model having 8 decoder blocks (3.59/4.35 AEPE on clean/final,resp., compared to 4.63/5.24 AEPE for MAE; see the supplementary material for a full ablation).
These results clearly demonstrate the benefits from pretraining with the cross-view completion task.
[Table]
11.7cm, 4.53° 5.0cm, 3.46° *: [Table] predictions † : exploit
[Table] information and multi [Table] retrieval
Our results (2.80/3.54 on clean/final, resp.) are slightly behind recent state-of-the-art approaches like
PWC-Net[59] (2.55/3.93), LiteFlowNet2 [34] (2.24/3.78) and RAFT [61] (1.43/2.71), all trained on
FlyingChairs [21] and FlyingThings [43] combined.
However, our CroCo model remains competitivefor this task considering the simplicity of our approach.
In particular, note that i) our model is trainedfrom limited data without any sophisticated data augmentation, ii) our architecture is generic and notspecially designed for optical flow prediction, unlike [59, 34, 61] which all rely on 4D cost volumes,iii) regression at test time is simple and straightforward, iv) the small 224 × 224 resolution used attest time hinders our performance on large displacements, a problem which we leave to future work.
We refer to the supplementary material for additional ablative experiments and comparisons.
Relative pose estimation .
We experiment on the relative pose regression (RPR) downstream task.
Given a pair of images, the goal is to predict the relative pose of the camera with respect to a referenceview.
To do so, we feed to the CroCo model two images corresponding to normalized cameraviews, and we regress the rigid transformation (
R , t ) ∈
SO (3) × R 3 between these two views usinga differentiable Procrustes layer [7] on top of the prediction head to ensure that R is a rotationmatrix.
We finetune the RPR model with an MSE loss between the predicted relative pose ( R , t ) ∈ SO (3) × R 3 we can drop ∈
SO (3) × R 3 here and a ground truth ( ˆ R , ˆ t ) :
∥ R − ˆ R ∥ 2 F + λ ∥ t − ˆ t ∥ 2 ,with λ set to 100 in practice.
During training, we augment the training image pairs with randompermutations, color jittering and virtual camera rotations.
We evaluate on the 7-scenes dataset [28],with a single model finetuned on the union of the official training sets of all scenes, and tested on thecorresponding test splits.
Images are rescaled and cropped to a 224 × 224 resolution for simplicity.
Attest time, for each image of the test split we retrieve the nearest image from the training set accordingto their AP-GeM-18 descriptors [53], and we predict the pose relative to this retrieved image.
For each scene, we report the standard median position and median orientation error in Table 4.
Wecompare results obtained with the model initialized with CroCo to that of a model with an encoderinitialized with MAE (Habitat) and a decoder trained from scratch (last two rows).
Unsurprisingly,the former leads to significantly better results than the latter.
We furthermore compare our modelpretrained with CroCo to other state-of-the-art RPR methods which directly regress relative cameraposes [5, 19] or essential matrices [79] as well as the the retrieval baseline consisting in predictingthe identity as relative pose (see Table 4 upper part).
Our model outperforms all methods except
CamNet [19].
Note that the latter fuses multiple pose predictions, exploit temporal information andmulti-step retrieval; in contrast our model directly predicts the relative pose from a (query, map)image pair, without any further processing, and is thus significantly simpler.
Discussion
We have introduced the novel task of cross-view completion for pretraining computer vision networkstailored to 3D vision downstream tasks.
Our experiments show that cross-view completion allows tolearn representations better suited to 3D vision tasks than classical MIM with auto-completion, andstraightforwardly transfer to monocular and binocular tasks.
A limitation of our model pretrained withcross-view completion is that it seems less tailored to high-level semantic tasks.
This is arguably morerelated to the choice of the dataset than to the pretraining task itself, and future work could explorethe use of cross-view completion in conjunction with more object-centric datasets like ImageNet.
Ourapproach requires pairs of images depicting the same scene, and in this work, we leverage syntheticrenderings only.
Future work could also extend it to real-world image pairs, which can be obtainedwithout any supervision, for instance with Structure-from-Motion [38, 50] or geo-referencing.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
Yes.
(b) Did you describe the limitations of your work?
Yes, in the Discussion section (Section 5)
(c) Did you discuss any potential negative societal impacts of your work?
No, we do not envision any potential negative societal impact beyond those that are
generic to machine learning and data bias.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
Yes.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
N/A.
(b) Did you include complete proofs of all theoretical results?
N/A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
Not with the submission
but we are planning to release code if accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
All details are available in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
No, because pretraining is costly (about a week on 8 gpus), but our ablations with for
instance the different decoder blocks concur towards the same conclusion that CroCo
outperforms other pretraining strategies.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Yes, in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
Yes.
(b) Did you mention the license of the assets?
Yes, in the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
No.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
N/A.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
N/A.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
N/A.
(b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
N/A.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
N/A.