Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search (2021)

Abstract: Internet search affects people’s cognition of the world, so mitigating biases in search results and learning fair models is imperative for social good. We study a unique gender bias in image search in this work: the search images are often gender-imbalanced for gender-neutral natural language queries. We diagnose two typical image search models, the specialized model trained on in-domain datasets and the generalized representation model pre-trained on massive image and text data across the internet. Both models suffer from severe gender bias. Therefore, we introduce two novel debiasing approaches: an in-processing fair sampling method to address the gender imbalance issue for training models, and a post-processing feature clipping method base on mutual information to debias multimodal representations of pre-trained models. Extensive experiments on MS-COCO and Flickr30K benchmarks show that our methods significantly reduce the gender bias in image search models.

## introduction internet information is shaping people's minds. the algorithmic processes behind modern search engines, with extensive use of machine learning, have great power to determine users' access to information @xcite . these information systems are biased when results are systematically slanted in unfair discrimination against protected groups @xcite . gender bias is a severe fairness issue in image search. figure 1 shows an example: given a genderneutral natural language query "a person is cooking", only 2 out of 10 images retrieved by an image search model @xcite depict females, while equalized exposure for male and female is expected. such gender-biased search results are harmful to society as they change people's cognition and worsen gender stereotypes @xcite . mitigating gender bias in image search is imperative for social good. in this paper, we formally develop a framework for quantifying gender bias in image search results, where text queries in english foot_0 are made genderneutral, and gender-balanced search images are expected for models to retrieve. to evaluate model fairness, we use the normalized difference between masculine and feminine images in the retrieved results to represent gender bias. we diagnose the gender bias of two primary families of multimodal models for image search: (1) the specialized models that are often trained on in-domain datasets to perform text-image retrieval, and (2) the generalpurpose representation models that are pre-trained on massive image and text data available online and can be applied to image search. our analysis on ms-coco @xcite and flickr30k @xcite datasets reveals that both types of models lead to serious gender bias issues (e.g., nearly 70% of the retrieved images are masculine images). to mitigate gender bias in image search, we propose two novel debiasing solutions for both model families. the specialized in-domain training methods such as scan @xcite often adopt contrastive learning to enforce image-text matching by maximizing the margin between positive and negative image-text pairs. however, the gender distribution in the training data is typically imbalanced, which results in unfair model training. thus we introduce a fair sampling (fairsample) method to alleviate the gender imbalance during training without modifying the training data. our second solution aims at debiasing the large, pre-trained multimodal representation models, which effectively learn pre-trained image and text representations to accomplish down-stream applications @xcite @xcite ; rad-figure 1 : gender bias in image search. we show the top-10 retrieved images for searching "a person is cooking" on the flickr30k @xcite test set using a state-of-the-art model @xcite . despite the gender-neutral query, only 2 out of 10 images are depicting female cooking. @xcite . we examine whether the representative clip model @xcite embeds human biases into multimodal representations when they are applied to the task of image search. furthermore, we propose a novel post-processing feature clipping approach, clip, that effectively prunes out features highly correlated with gender based on their mutual information to reduce the gender bias induced by multimodal representations. the clip method does not require any training and is compatible with various pre-trained models. we evaluate both debiasing approaches on ms-coco and flickr30k and find that, on both benchmarks, the proposed approaches significantly reduce the gender bias exhibited by scan and clip models when evaluated on the gender-neutral corpora, yielding fairer and more gender-balanced search results. in addition, we evaluate the similarity bias of the clip model in realistic image search results for occupations on the internet, and observe that the post-processing methods mitigate the discrepancy between gender groups by a large margin. our contributions are four-fold: (1) we diagnose a unique gender bias in image search, especially for gender-neutral text queries; (2) we introduce a fair sampling method to mitigate gender bias during model training; (3) we also propose a novel post-processing clip method to debias pre-trained multimodal representation models; (4) we conduct extensive experiments to analyze the prevalent bias in existing models and demonstrate the effectiveness of our debiasing methods. ## gender bias in image search in an image search system, text queries may be either gender-neutral or gender-specific. intuitively, when we search for a gender-neutral query like "a person is cooking", we expect a fair model returning approximately equal proportions of images depicting men and women. for gender-specific queries, an unbiased image search system is supposed to exclude images with misspecified gender information. this intention aligns with seeking more accurate search results and would be much different from the scope of measuring gender bias in gender-neutral cases. therefore, we focus on identifying and quantifying gender bias when only searching for gender-neutral text queries. ## mitigating gender bias in image search there are two fashions of multimodal models for the image search task. one is to build a specialized model that could embed image and text into representation vectors with measurable similarity scores. the other is to use general-purpose imagetext representations pre-trained on sufficiently big data and compute a particular distance metric. we focus on two representative models, scan @xcite and clip @xcite , for both fashions. for the first fashion, we propose an in-processing learning approach to ameliorate the unfairness caused by imbalanced gender distribution in training examples. this approach builds on contrastive learning but extends with a fair sampling step. the in-processing solution requires full training on in-domain data examples. for the second fashion, we propose a post-processing feature clipping technique to mitigate bias from an information-theoretical perspective. this approach is compatible with pre-trained models and is light to implement without repeating training steps. ## related work fairness in machine learning a number of unfair treatments by machine learning models were reported recently @xcite @xcite , and the literature has observed a growing demand and interests in proposing defenses, including regularizing disparate impact @xcite and disparate treatment @xcite , promoting fairness through causal inference @xcite , and adding fairness guarantees in recommendations and information retrieval @xcite @xcite . the existing fair machine learning solutions can be broadly categorized as pre-processing @xcite @xcite , inprocessing, and post-processing approaches. preprocessing algorithms typically re-weight and repair the training data which captures label bias or historical discrimination @xcite @xcite . in-processing algorithms focus on modifying the training objective with additional fairness constraints or regularization terms @xcite @xcite . post-processing algorithms enforce fairness constraints by applying a post hoc correction of a (pre-)trained classifier @xcite . in this work, the fair sampling strategy designed for the contrastive learning framework could be considered as an in-processing treatment, while the clip algorithm is in the post-processing regime that features an information-theoretical clipping procedure. our contribution highlights new challenges of reducing gender bias in a multimodal task and specializes new in-processing and postprocessing ideas in the domain of image search. social bias in multi-modality implicit social bias related to gender and race has been discussed in multimodal tasks including image captioning @xcite , visual question answering @xcite , face recognition @xcite , and unsupervised image representation learning @xcite . for example, @xcite shows that models trained on unbalanced data can amplify bias, and injecting corpus-level lagrangian constraints can calibrate the bias amplification. @xcite demonstrates the association between the word embeddings of occupation and gendered concepts correlates with the imbalanced distribution of gender in text corpora. there are also a series of debiasing techniques in this area. @xcite propose to surgically alter the embedding space by identifying the gender subspace from gendered word pairs. @xcite extend the bias component removal approach to the setting where the sensitive attribute is non-binary. data augmentation approaches remove the implicit bias in the training corpora and train the models on the balanced datasets @xcite . our work complements this line of research by examining gender bias induced by multimodal models in image search results. our focus on gender bias in the gender-neutral language would offer new insights for a less explored topic to the community. gender bias in online search systems our work is also closely connected to studies in the hci community showing the gender inequality in online image search results. kay et al. (2015) articulate the gender bias in occupational image search results affect people's perceptions of the prevalence of men and women in each occupation. kay et al. (2015) compare gender proportions in occupational image search results and discuss how the bias affects people's perceptions of the prevalence of men and women in each occupation. singh et al. (2020) examine the prevalence of gender stereotypes on various digital media platforms. otterbacher et al. (2017) identify gender bias with character traits. nonetheless, these works do not attempt to mitigate gender bias in search algorithms. our work extends these studies into understanding how gender biases enter search algorithms and provides novel solutions to mitigating gender bias in two typical model families for image search. ## conclusion in this paper, we examine gender bias in image search models when search queries are genderneutral. as an initial attempt to study this critical problem, we formally identify and quantify gender bias in image search. to mitigate the gender bias perpetuating two representative fashions of image search models, we propose two novel debiasing algorithms in in-processing and post-processing manners. when training a new image search model, the in-processing fairsample method can be used to learn a fairer model from scratch. meanwhile, the clip algorithm can be used for lightweight deployment of pre-trained representation models with accessible gender information.