Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts (2025)

Abstract: Large vision-language models (LVLMs) have recently achieved significant progress, demonstrating strong capabilities in open-world visual understanding. However, it is not yet clear how LVLMs address demographic biases in real life, especially the disparities across attributes such as gender, skin tone, age and race. In this paper, We empirically investigate visual fairness in several mainstream LVLMs by auditing their performance disparities across demographic attributes using public fairness benchmark datasets (e.g., FACET, UTKFace). Our fairness evaluation framework employs direct and single-choice question prompt on visual question-answering/classification tasks. Despite advancements in visual understanding, our zero-shot prompting results show that both open-source and closed-source LVLMs continue to exhibit fairness issues across different prompts and demographic groups. Furthermore, we propose a potential multi-modal Chain-of-thought (CoT) based strategy for unfairness mitigation, applicable to both open-source and closed-source LVLMs. This approach enhances transparency and offers a scalable solution for addressing fairness, providing a solid foundation for future research and practical efforts in unfairness mitigation. The dataset and code used in this study are publicly available at this GitHub Repository.

## introduction large vision-language models (lvlms) have successfully encoded images and text into a shared latent space, enabling a better visual reasoning @xcite . pre-trained @xcite across lvlms in our experiments. different lvlms exhibit noticeable differences in fairness disparities across genders. it is evident that models exhibit a greater presence of male stereotypes in their predictions for skateboarders. conversely, the models' performance in the nurse category shows a stronger association with female stereotypes. lvlms can accurately interpret images and extract semantics by meticulously designing natural language instructions (also known as "prompts"), providing additional information for traditional vision tasks such as classification @xcite , segmentation @xcite , and visual question answering @xcite . although many lvlms have achieved remarkable results in improving accuracy @xcite @xcite @xcite , their performance across different demographic groups, such as race and gender, remains understudied, leading to the perpetuation of unfairness @xcite . for example, even if the model's prediction attributes are unrelated to race, gender, and age, these factors can still interfere with the training process due to typically socially biased samples or demographically imbalanced pre-training data, which risks propagating unfairness into model inference and leading to disparate impacts across groups. this oversight is critical as it can lead to unfair outcomes, potentially reinforcing harmful stereotypes @xcite , as illustrated in figure 1 from our experiments. moreover, existing studies @xcite @xcite have not adequately addressed the need for fairness evaluation specifically designed for the contemporary large model settings. it is essential to systematically study the impact of various demographic attributes on lvlms performance. models such as clip @xcite and @xcite have been assessed using datasets like fairface @xcite , utkface @xcite , and celeba @xcite , but the images in these datasets primarily focus on facial features, providing limited information. furthermore, the architectures of clip and vit differ significantly from modern lvlms, which makes them less suitable for evaluating the full capabilities of lvlms in fairness contexts. recently, some researchers have taken advantage of diffusion models' ability to generate large-scale synthetic images to investigate bias in popular lvlms @xcite . while synthetic images allow for large datasets, they may introduce biases that distort fairness evaluations. in this study, we empirically provide a detailed evaluation of lvlms from a fairness perspective by proposing a novel evaluation framework. this framework uses real, annotated images and incorporates both direct questions and single-choice question-instructed prompts on visual question answering/classification tasks, based on the facet @xcite and utkface @xcite benchmark. our approach analyzes the models' ability to accurately interpret images while assessing fairness related to visual attributes such as gender, skin tone, and age. by building on previous methods, our framework offers a more comprehensive and accurate evaluation of lvlms fairness, providing insights into how these models handle real-world visual biases and establishing a solid foundation for future unfairness mitigation strategies. in addition, we introduce a multi-modal chainof-thought (cot)-based strategy to mitigate unfairness, which can be applied to both open-source and closed-source models. this strategy not only improves lvlms' performance in addressing fair-ness concerns but also offers a straightforward and scalable solution for real-world applications. we summarize the contribution of this work as follows: ## conclusion and future work in this paper, we presented a novel visual fairness evaluation framework for investigating demographic fairness in lvlms. the experimental results demonstrated significant fairness gap across gender, skin tone, and age in both open-source and closed-source lvlms. we propose a multi-modal cot mitigation strategy that enhances fairness by using rationale-based sub-questions to guide more accurate predictions. in future work, we will explore more diverse datasets, including socioeconomic, cultural, and other fine-grained attributes, to better understand when and why fairness issues arise. building on these insights, we aim to design improved mitigation strategies that combine tuningand prompt-based methods for more effective fairness intervention.