  A critical problem in deep learning is that systems learn inappropriate
biases, resulting in their inability to perform well on minority groups. This
has led to the creation of multiple algorithms that endeavor to mitigate bias.
However, it is not clear how effective these methods are. This is because study
protocols differ among papers, systems are tested on datasets that fail to test
many forms of bias, and systems have access to hidden knowledge or are tuned
specifically to the test set. To address this, we introduce an improved
evaluation protocol, sensible metrics, and a new dataset, which enables us to
ask and answer critical questions about bias mitigation algorithms. We evaluate
seven state-of-the-art algorithms using the same network architecture and
hyperparameter selection policy across three benchmark datasets. We introduce a
new dataset called Biased MNIST that enables assessment of robustness to
multiple bias sources. We use Biased MNIST and a visual question answering
(VQA) benchmark to assess robustness to hidden biases. Rather than only tuning
to the test set distribution, we study robustness across different tuning
distributions, which is critical because for many applications the test
distribution may not be known during development. We find that algorithms
exploit hidden biases, are unable to scale to multiple forms of bias, and are
highly sensitive to the choice of tuning set. Based on our findings, we implore
the community to adopt more rigorous assessment of future bias mitigation
methods. All data, code, and results are publicly available at:
https://github.com/erobic/bias-mitigators.
