Paper:
Predictive coding theories suggest that the brain learns by predicting observations at various levels of abstraction. One of the most basic prediction tasks is view prediction: how would a given scene look from an alternative viewpoint? Humans excel at this task. Our ability to imagine and fill in missing visual information is tightly coupled with perception: we feel as if we see the world in 3 dimensions, while in fact, information from only the front surface of the world hits our (2D) retinas. This paper explores the connection between view-predictive representation learning and its role in the development of 3D visual recognition. We propose inverse graphics networks, which take as input 2.5D video streams captured by a moving camera, and map to stable 3D feature maps of the scene, by disentangling the scene content from the motion of the camera. The model can also project its 3D feature maps to novel viewpoints, to predict and match against target views. We propose contrastive prediction losses that can handle stochasticity of the visual input and can scale view-predictive learning to more photorealistic scenes than those considered in previous works. We show that the proposed model learns 3D visual representations useful for (1) semi-supervised learning of 3D object detectors, and (2) unsupervised learning of 3D moving object detectors, by estimating motion of the inferred 3D feature maps in videos of dynamic scenes. To the best of our knowledge, this is the first work that empirically shows view prediction to be a useful and scalable self-supervised task beneficial to 3D object detection.

Review:
This paper deals with turning a 2.5D video representation into a 3D representation of an environment or a scene. The authors introduce self-supervised methods to pretrain the 2d-3d projection with a contrastive loss, which is the neural backbone for multiple other tasks. They then assess their approach on numerous tasks such as 3D-object detection, 3D-moving object detection, and 3D motion estimation. The authors also evaluate the transferability of the features in a challenging sim2real setting.
It is dense paper with multiple modules (2d-3d, ergomotion, memory, etc.) and concept. Still, the authors make it accessible by concise paragraphs, highlighting key equations (The enum + eq 1 and 2 are quite useful), and well-designed sketch (Figure1). I had some difficulties digging into the visual head component for each task as I was not familiar with this topic. However, the authors always explain their choices in a few lines and refer to the related papers for technical details in a meaningful way.
I am pretty convinced with the experiments, especially Sim2Real, in Tab1, where the baselines are clears and make sense.
I appreciated the limitation section, which is transparent and honest, and clearly states the strength and weaknesses (such as image downscaling) of the approach. Besides, the code and the data should be released, which is always a positive point.
Remarks:
 - Latent map update: running average is a simple and efficient mechanism, it also makes sense as you are dealing with big 3D tensors. Yet, have you tried other update mechanisms?
 - A natural follow-up to this paper is Contrastive Predictive Losses (which had several successes in pure vision setting[1]). Did you already try this approach?
 - In visual CPC papers [1] (or since the early days of visual representation learning!), data transformation has been applied to improve model performance. Would it make sense to apply it to I_{n+1}, D_{n+1} ?
 - Although the authors assess their approach with RGB-D, the models were still trained on 2.5D video. It would have been useful also to assess a pretraining on pure RGB-D data
However, I have two (somehow related) concerns. First of all, the machine learning novelties are rather small, contrastive losses are now widespread, and the models are closed to Tung et al. as mentioned by the authors.  However, I believe the paper to be a substantial contribution in vision, as they show the feasibility of their approach on large scale scenarios and over a highly diverse set of tasks. Again, the authors also release the code, making the paper a valuable baseline for the following work. On my side, I am impressed by the sim2real env. 
My second concern is the following, it is a high quality vision paper, and I am curious why the authors chose ICLR over CVPR. Besides, the tasks are vision-oriented, and 2.5D vision is not common in the ML community. Having said that, the proposed approach is pretty generic, can be applied to RGB-D (more common in ML), and require few expert knowledge in vision (only the Egomotion module).
As a result, I would advocate for clear accept if we assess vision-based contribution for ICLR; otherwise, I would only recommend weak accept the paper is solely based on ML contributions (the paper is still sound, well-written, with numerous experiments and with a semi-generic architecture)