  Classification with abstention has gained a lot of attention in recent years
as it allows to incorporate human decision-makers in the process. Yet,
abstention can potentially amplify disparities and lead to discriminatory
predictions. The goal of this work is to build a general purpose classification
algorithm, which is able to abstain from prediction, while avoiding disparate
impact. We formalize this problem as risk minimization under fairness and
abstention constraints for which we derive the form of the optimal classifier.
Building on this result, we propose a post-processing classification algorithm,
which is able to modify any off-the-shelf score-based classifier using only
unlabeled sample. We establish finite sample risk, fairness, and abstention
guarantees for the proposed algorithm. In particular, it is shown that fairness
and abstention constraints can be achieved independently from the initial
classifier as long as sufficiently many unlabeled data is available. The risk
guarantee is established in terms of the quality of the initial classifier. Our
post-processing scheme reduces to a sparse linear program allowing for an
efficient implementation, which we provide. Finally, we validate our method
empirically showing that moderate abstention rates allow to bypass the
risk-fairness trade-off.
