Gender Biases in Automatic Evaluation Metrics for Image Captioning (2023)

Abstract: Model-based evaluation metrics (e.g., CLIPScore and GPTScore) have demonstrated decent correlations with human judgments in various language generation tasks. However, their impact on fairness remains largely unexplored. It is widely recognized that pretrained models can inadvertently encode societal biases, thus employing these models for evaluation purposes may inadvertently perpetuate and amplify biases. For example, an evaluation metric may favor the caption “a woman is calculating an account book” over “a man is calculating an account book,” even if the image only shows male accountants. In this paper, we conduct a systematic study of gender biases in model-based automatic evaluation metrics for image captioning tasks. We start by curating a dataset comprising profession, activity, and object concepts associated with stereotypical gender associations. Then, we demonstrate the negative consequences of using these biased metrics, including the inability to differentiate between biased and unbiased generations, as well as the propagation of biases to generation models through reinforcement learning. Finally, we present a simple and effective way to mitigate the metric bias without hurting the correlations with human judgments. Our dataset and framework lay the foundation for understanding the potential harm of model-based evaluation metrics, and facilitate future works to develop more inclusive evaluation metrics.

## introduction pretrained model-based evaluation metrics such as bertscore @xcite , clipscore @xcite , and gptscore @xcite have shown promising performance, achieving stronger correlations with human judgments over n-gram matching-based evaluation metrics such as bleu @xcite , rouge @xcite , figure 1 : an image-caption pair example from the pao-evalbias dataset. a good caption accurately describes the gender of the main character in the image, while the bad caption incorrectly describes the gender. clipscore can assign a higher score to the caption that is incorrect (0.75 vs. 0.72 correct), which shows that there is bias encoded in the evaluation metric. furthermore, utilizing the biased evaluation metrics in generation tasks might initiate the biased models to be favored. and cider @xcite across various generation tasks. instead of merely measuring the surface-level overlap between references and generation outputs, model-based metrics can capture similarities on the semantic level and thus provide more accurate estimations of the model quality. despite the promising results, it is widely recognized that pretrained models encode societal biases, including but not limited to gender, racial, and religious biases @xcite @xcite @xcite @xcite . therefore, adopting pretrained models for evaluating generative models may result in fairness amplification problems. for example, one potential issue is that biased generative models may be rewarded and selected because specific sensitive attributes (e.g., gender) are favored by biased model-based evaluation metrics. moreover, when using such evaluation metrics in reinforcement learning from ai feedback (rlaif), there is a potential risk of further amplifying these biases in the models. there are a few prior works that have pointed out issues regarding languageonly evaluation metrics @xcite . regarding fairness, @xcite constructed a dataset based on winobias @xcite and systematically investigated different metrics. however, they focus on synthetic model generations and failed to analyze the implications and harm of biased metrics in real-world scenarios. as a results, it is hard to draw insights from their works in terms of practical applications. moreover, they leave out studies of biases encoded in cross-modal evaluation metrics such as clip-score. as we see an increase in the variety of multimodal generation tasks such as image captioning and multimodal summarization @xcite , it is crucial to evaluate the cross-modal metrics specifically designed for these tasks. in this paper, we perform a systematic study of gender biases in cross-modal generation evaluation metrics using image captioning tasks. following previous research @xcite , we classify gender expression instead of biological sex or gender identity. we limit our analysis to two genders (man and woman) in this study, but it is important to note that gender is non-binary. we acknowledge this limitation and refer readers to the ethics statement section for a more in-depth discussion on this topic. for the study, we first collect a large-scale dataset, pao-evalbias, consisting of 92,049 images of people of 88 professions, in 52 activities, and with 39 objects. figure 1 provides an imagecaption pair example from the dataset. then, we use the proposed dataset to analyze potential gender biases in automatic evaluation metrics, and how biased evaluation metrics can affect generation models through reinforcement learning. we also propose a simple method that combines model-based and n-gram matching-based evaluation metrics to reduce gender biases, while maintaining high correlations with human judgments for generation quality. the highlights of our findings include: ## bias evaluation for evaluation metrics we aim to identify and quantify potential gender biases in evaluation metrics for language generation models. to do this, we first gather a dataset in section 2.1. then, we formally define gender biases and conduct a comprehensive analysis of image captioning evaluation metrics on our dataset in section 2.2. ## impact on generation models because the model-based evaluation metric contains biases, we posit that these biases may lead to severe consequences in real-world applications. to test this, we experiment with fiber @xcite , a strong image captioning model pretrained on 10m image-caption pairs and then finetuned on the coco captioning karpathy-split data @xcite . our goal is to examine the impact of gender biases pre-encoded in evaluation metrics on generation models. specifically, we reveal that the existing image-captioning models contain gender biases, and using biased model-based metrics will make this kind of biased model more favorable over lessbias ones (more details in section 3.1). based on these findings, we further investigate whether using a biased metric as a reward may amplify biases in both the generation model and evaluation metric under the reinforcement learning setting (more details in section 3.2). ## a hybrid similarity metric while model-based metric contains biases, n-gram matching-based metrics can hardly encode gender biases. therefore, it is natural to combine n-gram matching-based with model-based metrics to alle-viate gender biases. motivated by this, we investigate if adding clipscore and cider together without normalization for model evaluation (denoted as clipscore+cider) can harness the benefits of both model-based and n-gram matching-based evaluation metrics, which has demonstrated effective in other tasks @xcite . formally, we obtain the new evaluation score with ## conclusion we analyze the gender biases issue of model-based evaluation metrics on image captioning tasks and investigate its potential impact on image captioning generation models. to do this, we create our own dataset and conduct a thorough analysis of the gender bias present in various evaluation metrics across multiple concepts. we also discuss the consequences of these biases in real-world applications and propose a hybrid metric as a solution to mitigate the issue. experiments show that using biased model-based evaluation metrics cannot distinguish between biased and debiased model generations and amplifies the model-encoded gender biases through reinforcement learning. the proposed hybrid similarity evaluation metric can significantly reduce gender biases, while maintaining a stronger correlation with human judgments than existing metrics. in the future, we plan to expand our analysis to include other protected attributes such as race and ethnicity, as well as other language generation tasks. additionally, we aim to continue developing more effective methods for removing bias from generation evaluation metrics.