Paper:
Unsupervised learning is an important tool that has received a significant amount of attention for decades. Its goal is 'unsupervised recovery,' i.e., extracting salient factors/properties from unlabeled data. Because of the challenges in defining salient properties, recently, 'contrastive disentanglement' has gained popularity to discover the additional variations that are enhanced in one dataset relative to another. Existing formulations have devised a variety of losses for this task. However, all present day methods exhibit two major shortcomings: (1) encodings for data that does not exhibit salient factors are not pushed to carry no signal; and (2) introduced losses are often hard to estimate and require additional trainable parameters. We present a new formulation for contrastive disentanglement which avoids both shortcomings by carefully formulating a probabilistic model and by using non-parametric yet easily computable metrics. We show on four challenging datasets that the proposed approach is able to better disentangle salient factors.

Review:
Contributions:
1. This paper fixes two problems that appeared in common contrastive disentanglement methods.
2. The paper evaluates its method in multiple experiments.
Overall, this paper does a nice contribution to improving existing methods of contrastive disentanglement. My major concern is the novelty in this paper since all the contributions can be summarized into adding two additional loss terms, which I would like to discuss below.
1. [The missing term KL[q(s|y)||p_y(s)]]. One major claim of this paper is to add back the additional term KL[q(s|y)||p_y(s)] in order to push the encoder q(s|y) to converge to a point mass \delta\{s=0\} for background images y. According to the mathematical formula, this term is natural, but the major concern is the non-overlapping support between q(s|y) and p_y(s), since in the paper they assume p_y(s)=\delta\{s=0\}. This paper claims they fix this problem. But I do not think the solution is satisfactory, because they use another term E_q[\mu_{\phi,s}(y)^2+\sigma2_{\phi,s}(y)] to replace KL[q(s|y)||\delta\{s=0\}]. If I get it correct, the replacing term E_q[\mu(y)^2+\sigma2(y)]=H(q(s|y)) is the entropy of q(s|y). But if we check the true divergence term, KL[q(s|y)||\delta\{s=0\}]=-E_q[\log\delta\{s=0\}]-H(q(s|y)) that consists of a negative entropy term. This means the behavior of the proposed loss tries to minimize the entropy while the original term tries to maximize the entropy, regardless of it is ill-defined. Thus, the replacement of the loss term seems reasonable to me at first glance but does not quite directly fix the problem of an ill-defined KL-divergence. A more reasonable solution is to replace the KL divergence with the Wasserstein distance. Is there any difficulty if we do that?
2. [The additional distributional matching term W_2^2(q_t(z),q_b(z))]. This term makes perfect sense to me, even though it is just added artificially. I'm more curious about the way to evaluate the gradient. Normally, computing Wasserstein distance is hard due to a hard linear programming matching algorithm. In this paper, the authors seem to use an existing library to compute the optimal transportation matrix. Is that a heavy computational burden? Or one could resort to some entropy regularization methods with sinkhorn iterative solvers. I'm curious about how these two methods compare with each other.
3. Another concern is the numerical result seems not as good as the qualitative results.
Overall, I think this paper does make some changes regarding previous works on contrastive disentanglement. But their main contributions need some more explanation and justification.