  Adversarial learning can learn fairer and less biased models of language than
standard methods. However, current adversarial techniques only partially
mitigate model bias, added to which their training procedures are often
unstable. In this paper, we propose a novel approach to adversarial learning
based on the use of multiple diverse discriminators, whereby discriminators are
encouraged to learn orthogonal hidden representations from one another.
Experimental results show that our method substantially improves over standard
adversarial removal methods, in terms of reducing bias and the stability of
training.
