Paper:
Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint-level annotations. A popular approach is to factorize an image into a pose and appearance data stream, then to reconstruct the image from the factorized components. The pose representation should capture a set of consistent and tightly localized landmarks in order to facilitate reconstruction of the input image. Ultimately, we wish for our learned landmarks to focus on the foreground object of interest. However, the reconstruction task of the entire image forces the model to allocate landmarks to model the background. This work explores the effects of factorizing the reconstruction task into separate foreground and background reconstructions, conditioning only the foreground reconstruction on the unsupervised landmarks. Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the foreground object of interest. Furthermore, the rendered background quality is also improved, as the background rendering pipeline no longer requires the ill-suited landmarks to model its pose and appearance. We demonstrate this improvement in the context of the video-prediction task.

Review:
The paper presents an unsupervised approach for learning landmarks in images or videos with single objects by separating the representation of the image into foreground and background and factorizing the representation of the foreground into pose and appearance. It builds upon previous work [Jakab 2018, Lorenz 2019] who proposed to train by reconstructing the original image from one version of the image with perturbed appearance and another with perturbed pose. It extends this approach by introducing an additional separation of foreground and background in the image. 
Strengths:
+ Nicely motivates the approach of separating foreground and background
+ Fewer landmarks are needed than in previous work
+ Approach seems beneficial for video prediction
+ Clear and well written
+ Detailed description of architecture and training
 
Weaknesses:
- The changes and improvements feel somewhat incremental
- Some uncertainty about the solidity of the evaluation/comparability with baselines
Results on CelebA somewhat weak
Overall the paper is well written, easy to follow, presents a straightforward extension of previous work and appears to show an improvement. I’m thus generally supportive of the paper.
One question I’d like to see addressed in the response is about the evaluation: as you state, the details about cropping are not known for previous work, introducing some uncertainty into the external comparisons. It would be great if you could provide some more details about the steps you took to verify that your internal baseline is indeed comparable to previous work (e.g. Lorenz 2019). For instance, on CelebA your baseline seems to fall short of Lorenz 2019, which may suggest that your substantial looking improvement on BBC Pose is indeed due to more favorable cropping.
The second concern is about the results on CelebA presented in Fig. 6 in the Appendix: It looks like the background net reconstructs almost the entire image. I would have expected that hair style, shape of ears or existence of a beard would equally warrant landmarks and appearance. It seems a bit odd that the foreground is so focused on the central part of the face. Do you have an explanation for that?
Minor comments:
- Test in Fig. 1 too small and not readable when printed
- Fig. 1 seems to be missing an arrow pointing from image to appearance encoder
- Why are there more than 8 points in Fig. 2 for “Ours8” (and more than 12 for “Ours12”)?