Paper:
Learning with Relaxed Supervision
For weakly-supervised problems with deterministic constraints between the latent variables and observed output, learning necessitates performing inference over latent variables conditioned on the output, which can be intractable no matter how simple the model family is. Even finding a single latent variable setting that satisfies the constraints could be difficult; for instance, the observed output may be the result of a latent database query or graphics program which must be inferred. Here, the difficulty lies in not the model but the supervision, and poor approximations at this stage could lead to following the wrong learning signal entirely. In this paper, we develop a rigorous approach to relaxing the supervision, which yields asymptotically consistent parameter estimates despite altering the supervision. Our approach parameterizes a family of increasingly accurate relaxations, and jointly optimizes both the model and relaxation parameters, while formulating constraints between these parameters to ensure efficient inference. These efficiency constraints allow us to learn in otherwise intractable settings, while asymptotic consistency ensures that we always follow a valid learning signal.

Review:
# Summary of paperThe paper proposes a method for handling models with latent variables where the latent variables have a complex, deterministic relationship with the output variables. Computing marginal probability in such models is intractable because it requires summing over the latent variables. The proposed method forms an alternate probability distribution that decomposes a functional representation of the determinism, which essentially penalizes disagreement with the constraints. The relaxation is theoretically analyzed, showing that the effect of the relaxation on loss is bounded by the inverse of the minimum value in the relaxation vector beta (which ranges from 0 to infinity), that the amount of data needed to learn increases by at most a similar ratio. The authors then provide analysis for the tradeoff between efficiency and these negative effects in setting the relaxation vector, which uses sampling to learn and adjust the relaxation parameter in a manner that guarantees tractable inference.The method is tested on synthetic data for translation and parsing. The evaluation essentially only compares against different variants of the proposed approach, so it doesn't compare against any existing methods for similar problems. I don't see any discussion of this either. It seems the approaches described in the 2nd paragraph of the paper would be useful comparisons. # QualityThis is a very good paper, with an innovative technical contribution, well evaluated and analyzed results, and good presentation. I'm actually left with a feeling the virtues of the proposed approach are too good. What are the weaknesses of this approach? What is the catch? This problem is related to the lack of comparison, in discussion or experiments, to other perhaps-less-principled approaches to problems with intractable hard constraints.# ClarityIt is obvious to me that the authors took a lot of care in presenting what could otherwise be difficult, dense material. The writing is some of the best I have seen, with useful examples, readable, but technically precise exposition, and well thought out structure. # OriginalityThe idea of relaxing hard constraints is not particularly new, but I think using the idea in this way is. # SignificanceThe approach should apply to a variety of applications, and the technical ideas may extend to other problems.