Women Wearing Lipstick: Measuring the Bias Between an Object and Its Related Gender (2023)

Abstract: In this paper, we investigate the impact of objects on gender bias in image captioning systems. Our results show that only gender-specific objects have a strong gender bias (e.g., women-lipstick). In addition, we propose a visual semantic-based gender score that measures the degree of bias and can be used as a plug-in for any image captioning system. Our experiments demonstrate the utility of the gender score, since we observe that our score can measure the bias relation between a caption and its related gender; therefore, our score can be used as an additional metric to the existing Object Gender Co-Occ approach.

## introduction visual understanding of image captioning is an important and rapidly evolving research topic @xcite . recent approaches predominantly rely on transformer @xcite and the bert based pre-trained paradigm @xcite to learn cross-modal representation @xcite @xcite . while image captioning models achieved notable benchmark performance in utilizing the correlation between visual and co-occurring labels to generate an accurate image description, this often results in a gender bias that relates to a specific gender, such as confidently identifying a woman when there is a kitchen in the image. the work of @xcite tackles the problem of gender bias in visual semantic role labeling by balancing the distribution. for image captioning, @xcite consider ameliorating gender bias via a balance classifier. more recently, @xcite measured racial and gender bias amplification in image captioning via a trainable classifier on an additional human-annotated existing caption dataset for the gender bias task @xcite . to close the full picture of gender bias in image captioning, in this work, unlike other works, we examine the problem from a visual semantic relation perspective. we therefore propose a gender score via human-inspired judgment named belief revision @xcite which can be used to ( 1 ) discover bias and (2) predict gender bias without training or unbalancing the dataset. we conclude our contributions are as follows: (1) we investigate the gender object relation for image captioning at the word level (i.e. object-gender) and the sentence level with captions (i.e. gender-caption); ( 2 ) we propose a gender score that uses gender in relation to the visual information in the image to predict the gender bias. our gender score can be used as a plug-in for any out-of-the-box image captioning system. figure 1 shows an overview of the proposed gender bias measure for image captioning. ## visual context information in this work, we investigate the relation between gender bias and the objects that are mainly used in image captioning systems, and more precisely, the widely used manually annotated image caption datasets: flickr30k @xcite and coco dataset @xcite . however, we focus mainly on the coco dataset as the most used dataset in gender bias evaluation. coco captions is an unbalanced gender bias dataset with a 1:3 ratio bias towards men @xcite @xcite . the dataset contains around 120k images, and each image is annotated with five different human-written captions. to obtain the visual context o from each image i, we use out-of-the-box classifiers to extract the image context information o(i). specifically, following @xcite , the objects extracted from all pre-trained models are obtained by extracting the top-3 object class/category (excluding person category) from each classifier after filtering out instances with (1) the cosine distance between @xmath0 ## gender object distance the main component of our gender score (next section) is the semantic relation between the object and the gender. therefore, in this section, we investigate the semantic correlation between the object and the gender in the general dataset (e.g. wiki). more specifically, we assume that all images contain a gender man, woman or gender neutral person, and we aim to measure the cosine distance between the object o, objects with context i.e. caption y from the image i and its related gender a ′ ∈ {man, woman, person}. in this context, we refer to the cosine distance as the closest distance (i.e. semantic relatedness score) between the gender-to-object via similarity measure e.g. cosine similarity. we apply our analysis to the most widely used human-annotated datasets in image captioning tasks: flikr30k and coco captions datasets. we employ several commonly used pre-trained models: (1) word-level: glove @xcite , fasttext @xcite and word2vec @xcite out-of-the-box as baselines, then we utilize gender neutral glove @xcite , which balances the gender bias; informativeness of bias information: as the visual context probability p(c o ) approaches 1 and in consequence is less informative (very frequent objects have no discriminative power since they may co-occur with any gender) 1 -p(c o ) approaches zero, causing α to get closer to 1, and thus, a smaller revision of p(g ′ y ). therefore, as we described in the visual context information section 2, we leverage a threshold and semantic filter visual context dataset from resnet, inception-resnet v2 based faster r-cnn object detector, and clip to extract top-k textual visual context information from the image. the extracted object is used to measure the gender-to-object bias direct relation. relatedness between hypothesis and bias information: likelihood revision occurs if there is a close correlation between the hypothesis and the new information. as the sim(y, o) (gender, object), gets closer to 1 (higher relatedness) α gets closer to 0, and thus hypothesis probability is revised (i.e. gender bias) and raised closer to 1. therefore, the initial hypothesis will be revised or backed off to 1 (no bias) based on the relatedness score. in our case, we employ sentence-bert to compute the cosine distance (section 3) by using object o as context for the caption y with associated gender a. our experiments apply visual semantics between the object or object with context i.e. caption and its related gender information to predict object-togender related bias. the proposed visual-gender scores are (1): cosine distance, which uses the similarity score to try to estimate the proper gender as shown in in this work, we hypothesize that every image has a @xmath1, woman} or gender neutral (i.e. person), and our rationale is that each model suffers from right for the wrong reasons (hen- ## conclusion in this work, we investigate the relation between objects and gender bias in image captioning systems. our results show that not all objects exhibit a strong gender bias and only in special cases does an object have a strong gender bias. we also propose a gender score that can be used as an additional metric to the existing object-gender co-occ method.