Paper:
While supervised object detection and segmentation methods achieve impressive accuracy, they generalize poorly to images whose appearance significantly differs from the data they have been trained on. To address this in scenarios where annotating data is prohibitively expensive, we introduce a self-supervised approach to detection and segmentation, able to work with monocular images captured with a moving camera. At the heart of our approach lies the observations that object segmentation and background reconstruction are linked tasks, and that, for structured scenes, background regions can be re-synthesized from their surroundings, whereas regions depicting the object cannot. We encode this intuition as a self-supervised loss function that we exploit to train a proposal-based segmentation network. To account for the discrete nature of the proposals, we develop a Monte Carlo-based training strategy that allows the algorithm to explore the large space of object proposals. We apply our method to human detection and segmentation in images that visually depart from those of standard benchmarks, achieving competitive results compared to the few existing self-supervised methods and approaching the accuracy of supervised ones that exploit large annotated datasets.

Review:
This submission proposes a self-supervised segmentation method, that learns from single-object videos by finding the region where it can segment an object, remove the entire bounding box around it, inpaint it, then finally put the object back. The loss is a balance between a reconstruction error and a negative inpainting error (high error means object is probably present, due to the weak correlation with the background).
My decision is Weak Accept. I like the method very much and think it’s a clever and well-executed algorithm. The reason for being Weak is because the experimental evidence could be stronger, especially comparing with Croitoru et al. and Rhodin et al. The paper leaves some open problems, inviting future work to be built on top of it (i.e. leveraging time more and handling multiple objects better). I think it should be accepted to promote such future work.
Method:
The method is interesting and clever. Similar efforts have been made, such as Bielski & Favaro and Crawford & Pineau. However, key contributions relax some of the contrived requirements of these past methods (e.g. simple foreground translation over background; requirement of plain background). Thanks to the inpainter, the importance sampler, and avoidance of collapsing into trivial solutions, this paper is able to put together a method that works on moving cameras and fairly complex scene semantics (still limited to one object though). Actually getting this to converge with only two loss terms balanced against each other is impressive. Having certain heads of the network trained only on one of the two terms seems to be a key contribution. 
Experimental results:
The comparison with Rhodin et al. on H36M is characterized as “slightly” lower, but I would call 71% vs. 58% a significant difference. Of course, I understand that Rhodin et al. relies on the static background, so this is not a fair comparison.
Ski-PTZ-Dataset should then offer a better comparison, but here the method struggles to compete with Croitoru et al. 2019, which has a 11-point higher F-measure. On Handheld190k, the dataset proposed in this paper, is where the method finally shines, but still only offers a 1-point F-measure improvement over Croitoru et al. That being said, considering how different the methods are, and how Croitoru et al. requires two-stage training, there are many benefits to this method. Croitoru et al. also relies on video to extract the object features, and this requirement is not as explicit in this work. Actually, that brings me to one question I had. The paper states that “as long as videos or picture collections of a single object in front of the same scene are available.” I didn’t quite understand why this must be trained on multiple images of the same scene. If the inpainter is general to any background scenery, couldn’t it work on single-images as well? The conclusion even says you do not use temporal cues.
Other:
I don’t think it’s that meaningful to include precision/recall in the tables. It is also not that meaningful to point out that your method’s precision is higher than that of Croitoru et al., when the F-measure is shy 11 points. The reason is because many points on the precision/recall could be constructed simply by applying for instance a gamma curve on the segmentation predictions. The high precision is clearly at the cost of a low recall, and another point on this tradeoff curve could be presented. This is why F-measure and average precision are much better.