  Societal biases resonate in the retrieved contents of information retrieval
(IR) systems, resulting in reinforcing existing stereotypes. Approaching this
issue requires established measures of fairness regarding the representation of
various social groups in retrieved contents, as well as methods to mitigate
such biases, particularly in the light of the advances in deep ranking models.
In this work, we first provide a novel framework to measure the fairness in the
retrieved text contents of ranking models. Introducing a ranker-agnostic
measurement, the framework also enables the disentanglement of the effect on
fairness of collection from that of rankers. Second, we propose an adversarial
bias mitigation approach applied to the state-of-the-art Bert rankers, which
jointly learns to predict relevance and remove protected attributes. We conduct
experiments on two passage retrieval collections (MS MARCO Passage Re-ranking
and TREC Deep Learning 2019 Passage Re-ranking), which we extend by fairness
annotations of a selected subset of queries regarding gender attributes. Our
results on the MS MARCO benchmark show that, while the fairness of all ranking
models is lower than the ones of ranker-agnostic baselines, the fairness in
retrieved contents significantly improves when applying the proposed
adversarial training. Lastly, we investigate the trade-off between fairness and
utility, showing that through applying a combinatorial model selection method,
we can maintain the significant improvements in fairness without any
significant loss in utility.
