Paper:
Many notions of fairness may be expressed as linear constraints, and the resulting constrained objective is often optimized by transforming the problem into its Lagrangian dual with additive linear penalties. In non-convex settings, the resulting problem may be difficult to solve as the Lagrangian is not guaranteed to have a deterministic saddle-point equilibrium. In this paper, we propose to modify the linear penalties to second-order ones, and we argue that this results in a more practical training procedure in non-convex, large-data settings. For one, the use of secondorder penalties allows training the penalized objective with a fixed value of the penalty coefficient, thus avoiding the instability and potential lack of convergence associated with two-player min-max games. Secondly, we derive a method for efficiently computing the gradients associated with the second-order penalties in stochastic mini-batch settings. Our resulting algorithm performs well empirically, learning an appropriately fair classifier on a number of standard benchmarks.

Review:
The authors propose a method to generate predictions under fairness constraints. The main idea is to take linear fairness constraints, and replace them with weak squared penalties plugged into the objectives, which supposed to help in cases where the loss function is not convex. The penalty coefficients are chosen by cross-validation, and the effectiveness of this approach is demonstrated empirically.
In Sec. 3.1, the authors point out several shortcomings of using linear penalties (using Lagrange multipliers) for non-convex losses. These seem valid. Sec. 3.2, however, is not clear on why exactly replacing the linear penalties with quadratic penalties solves these issues. I'm hoping the authors can clarify the following points:
1) The authors note that, for quadratic penalties, \lambda->0 means no constraints, and \lambda->\infty means hard constraints. Isn't this also true for linear constraints?
2a) Why do linear penalties have unique \lambda_k for each constraint k, but the quadratic objective has only a single \lambda for all constraints?
2b) Why can CV over \lambda be used for quadratic constraints - what is the justification? And, more importantly, why *can't* it be used with linear constraints? If it can, then this should be one of the baselines compared to in the experiments.
3) What is the criterion optimized for by CV - accuracy or the constraints? Different parts of the paper give different answers to this question. For example, "... may be easily determined via standard hyperparameter optimization methods" vs. "tuning \lambda to directly satisfy the desired fairness metrics". Or even more unclear - "choose \lambda ... so that the final solution gives the desired fairness-accuracy trade-off". How is the desired trade-off defined?
4) If there is a trade-off between fairness and accuracy, and no clear-cut criterion for evaluation is pre-defined, then the evaluation procedure should compare methods across this trade-off (similarly to precision-recall analysis).
5) The authors differentiate between cases where the loss is either convex or non-convex. This is confusing - most losses are convex, and non-linearity appears when they are composed with non-linear predictors. Is this the case here? If so, the fairness constraints are no longer linear, and they're quadratic counterpart is no longer quadratic. It would be helpful if the authors specify where the non-linearity comes from, and what they assume about the loss and predictors.
6) Why is it important to show that the quadratic constraints can be written as an expectation? Isn't the square of an expectation always an expectation of pairs? How does the double summation/integral effect runtime?
7) It would be helpful if the authors differentiate between loss/constraints over the entire distribution vs. over a given sample set.