Paper:
FEATURE-WISE BIAS AMPLIFICATION
We study the phenomenon of bias amplification in classifiers, wherein a machine learning model learns to predict classes with a greater disparity than the underlying ground truth. We demonstrate that bias amplification can arise via an inductive bias in gradient descent methods that results in the overestimation of the importance of moderately-predictive "weak" features if insufficient training data is available. This overestimation gives rise to feature-wise bias amplificationa previously unreported form of bias that can be traced back to the features of a trained model. Through analysis and experiments, we show that while some bias cannot be mitigated without sacrificing accuracy, feature-wise bias amplification can be mitigated through targeted feature selection. We present two new feature selection algorithms for mitigating bias amplification in linear models, and show how they can be adapted to convolutional neural networks efficiently. Our experiments on synthetic and real data demonstrate that these algorithms consistently lead to reduced bias without harming accuracy, in some cases eliminating predictive bias altogether while providing modest gains in accuracy.

Review:
Summary:
In this paper the authors identify a specific source of marginal class probability bias that occurs when using logistic regression models. Using synthetic and real datasets they demonstrate this bias and explore characteristics of the data that exacerbate the issue. Finally, they propose two methods for correcting this bias in logistic regression models and neural network models with logistic output layers and evaluate these methods on several benchmark datasets.
Review:
Overall, I found the paper well-written, the problem well-motivated, and the proposed methods clear and reasonable. While I have a few concerns about presentation and experimentation, these are issues that can easily be remedied and I recommend acceptance.
Major comments:
- The authors repeatedly say that gradient descent is the cause of the bias amplification (e.g. Section 2.2 title, "...features that are systematically overestimated by gradient descent.", "... i.e., a consequence of gradient descent's inductive bias.", "... gradient descent may lead to systematic bias..."). The inductive bias they describe is coming from the use of logistic regression, not the use of gradient descent. Specifically, a logistic regression model has a convex likelihood, which means that regardless of what algorithm is used to maximize the likelihood, it should converge to the same point. In fact, most off-the-shelf implementations of logistic regression do not use vanilla gradient descent. Further, gradient descent may be used to estimate the parameters of any number of models which may or may not have the same inductive bias the authors describe.
- I thought the related work section was well-written and would strongly recommend moving it to the beginning of the paper as it motivates the entire problem. I also think it could be helpful to ground the technical definitions of bias amplification in a meaningful example.
- I think that the experimental setup for comparing \ell_1 regularization to the proposed feature selection methods is not quite fair. In particular, the hyperparameters of the "expert" method are selected to minimize bias subject to the constraint that loss not increase. In contrast, the \ell_1 regularization hyperparameter is selected purely to minimize bias. Instead, I would select the \ell_1 regularization hyperparameter in the same way as the expert method, that is, to minimize bias subject to a constraint on loss. In general, I think hyperparameters should be selected using the same criterion for all methods.
- The authors make a point of highlighting results on the "prostate" which showed a large increase in accuracy along with a large decrease in bias. I think the paper would benefit from some exploration of why this happened. Specifically, it would be valuable to answer the question: what are the properties of the "prostate" dataset that make this method so effective and are these properties general and identifiable a priori?
- Section 2, paragraph 2, line 5: The stated goal in this paragraph is "minimizing 0-1 loss on unknown future i.i.d. samples". As stated in the introduction, this is, in fact, not the goal. The goal is to minimize loss while also minimizing bias. A larger criticism that I would have of this work is: if minimizing bias is a first order goal, then why are we using empirical risk minimization in the first place? Put another way, why use post-hoc correction for an objective function that does not match our actual stated goals rather than using an objective function that does?
Minor comments:
- Section 1, paragraph 4, line 2: "Weak" is not clearly defined here. Is it different than "moderately-predictive"?
- Section 2.1, last paragraph, line 1: I understand what the authors are saying when they say "Bias amplification is unavoidable", but it is avoidable by changing our objective function. I would consider rewording this statement to something like "Using an ERM objective will lead to bias amplification when the learning rule..."
- Equation 4: I believe h should be changed to f in this equation.
- Equation 6: L is not defined anywhere.
- Table 1: As defined in equation 1, B_D(h_s) should be between 0 and 1. Also, the accuracy results for the glioma dataset have the wrong result in bold.
- Section 4, methodology paragraph, line 5: forthe --> for the
- Section 5, paragraph 5, lines 5-6: Feature selection is not used "only to improve accuracy". For example, Kim, Shah, and Doshi-Valez (2015) use feature selection to improve interpretability (https://beenkim.github.io/papers/BKim2015NIPS.pdf).