Brown Like Chocolate: How Vision-Language Models Associate Skin Tone with Food Colors (2025)

Abstract: We investigate how Vision-Language Models (VLMs) leverage visual features when making analogical comparisons about people. Using synthetic images of individuals varying in skin tone and nationality, we prompt GPT and Gemini models to make analogical associations with desserts and drinks. Results reveal that VLMs systematically associate darker-skinned individuals with brown-colored food items, with GPT showing stronger associations than Gemini. These patterns are amplified in Thai versus English prompts, suggesting language-dependent encoding of visual stereotypes. The associations persist across manipulation checks including position swapping and clothing changes, though presenting individuals alone yields divergent language-specific patterns. This work reveals concerning associations in VLMs’ visual reasoning that vary by language, with important implications for multilingual deployment.

## introduction vision-language models (vlms) are increasingly used in creative and decision-making applications, yet their processing of human visual features remains inadequately understood. while these models demonstrate impressive capabilities in visuallinguistic tasks @xcite , they may encode problematic associations between physical appearance and abstract concepts. this paper examines how vlms create analogical associations between individuals' skin tones and food items across languages. extensive research has documented biases in language models and their multimodal counterparts. foundational work demonstrated that word embeddings encode gender stereotypes through analogical reasoning tasks @xcite while facial analysis algorithms exhibit significant accuracy disparities across different skin tones @xcite . text-to-image systems similarly underrepresent darker skin tones and amplify societal biases @xcite . recent work examining vlms reveals complex patterns of multimodal biases. vlms often select stereotypical captions even when presented with anti-stereotypical images @xcite . smaller models perform substantially worse than larger variants on bias benchmarks @xcite . studies using controlled image sets demonstrate that vlms produce significantly different responses based on perceived gender or race of depicted individuals (fraser and kiritchenko, 2024), while systematic probing reveals biased associations across multiple dimensions @xcite . these findings suggest that biases permeate both language and visual modalities in ai systems. despite this growing body of work, there remains a research gap in understanding how vlms process visual features when making creative analogical associations across different languages, particularly for low-resource languages. to address this gap, we study how vlms form analogical associations about people when prompted in thai and english. our research questions are: (r1) do vlms exhibit languagedependent associations in mapping people to color-coded food/drink analogies? (r2) to what extent do non-facial factors (e.g., clothing color, spatial position, isolated framing) account for these associations? we focus on thai for two reasons. first, thai is a low-resource language, underrepresented in pretraining, instruction tuning, and safety evaluation. second, thailand presents substantial within-country variation in skin phototypes across populations, ranging from very light to tan and to darker tones @xcite . thai, therefore, offers a practical testbed for language-conditioned analogical associations while avoiding a simplistic, single-race framing. we probe two model families, gpt and gemini, of varying sizes using controlled synthetic portraits that vary in skin tone and nationality, and we report sensitivity analyses that manipulate clothing color, spatial position, and isolation. our analysis reveals that vlms consistently associate individuals with darker skin tones with brown-colored items, with these biases being significantly amplified in thai language prompts and smaller models. through systematic manipulation checks, we demonstrate that these associations persist across changes in position and clothing, suggesting biases in how vlms process human appearance for creative tasks. this study contributes to the growing body of work for ai safety by establishing an evaluation framework for investigating visual biases in vlms through analogical reasoning. image generation. we generated synthetic images using google's imagen 4, each image (see figure 1 ) depicts two individuals with controlled variations in skin tone and nationality. our use of synthetic images generated by imagen 4 provides essential experimental control by isolating skin tone and nationality as the targeted varying factors while holding constant environmental lighting, pose, expression, and background-eliminating confounds inherent in real photographs such as different photographers, equipment, styling choices, and socioeconomic signals. this approach also addresses critical ethical concerns by avoiding privacy violations and potential harm to identifiable individuals when investigating how vlms make analogical comparisons. the generation prompt follows this template: "a realistic image of two 25-year-old women side by side with some space between them in a bedroom with white walls. the woman on the left is [nationality] with [skin tone] skin wearing light pink casual clothes. the woman on the right is [nationality] with [skin tone] skin wearing light blue casual clothes. both are smiling and looking directly. the photo is taken from a phone's front camera showing only the upper half of their bodies." using this template, we systematically varied skin tone and nationality to create five conditions: thai with very light skin (tl), thai with tan skin (tt), european with very light skin (eu), and african american with dark skin (aa). due to resource constraints, we explored five pairs: tt-tl (main pair), tt-eu, tt-aa, tl-aa, and eu-aa. we controlled for gender (female) and clothing colors (pink and blue) across all conditions. five unique images were generated for each pairing. questions. we designed questions across four topics: cake types, macaron flavors, drink types, and dessert types. these categories were selected because their answers naturally span the color spectrum, including both dark/brown tones (e.g., chocolate, coffee) and light/bright tones (e.g., vanilla, strawberry). each question prompts models to assign one food item to each person in the image. for example: "if we were both cakes, what type of cake would we each be? answer only the type in order, left person first and then right person. separate the answers with commas." we tested questions in both thai (th) and english (eng), created and verified by bilingual proofreaders. complete question sets are provided in appendix a.1. models. given computational constraints, we evaluated four models from two leading providers: gpt-4.1-mini and gpt-4.1-nano from openai, and gemini-2.5-flash and gemini-2.5-flash-lite from google. all models were configured with @xmath0.0. each image-question pair was processed four times. in total, there are (5 skintone/nationality conditions + 3 sensitivity conditions (see 3.2)) x 5 images x 4 questions x 2 languages x 4 models x 4 @xmath1. ## discussion this study reveals that vlms exhibit systematic biases in analogical reasoning tasks, associating individuals with darker skin tones with brown-colored food and beverage items across multiple question types. model family differences further underscore the heterogeneity of bias manifestation, with gpt models consistently showing stronger associations than gemini models. interestingly, the results reveal languagedependent effects, where thai prompts elicited substantially stronger skin tone-color associations than english prompts. the differences between languages in the "alone" condition between languages are notable: thai prompts maintained strong associations while english prompts showed minimal effects. this disparity could stem from limitations in training data representation across languages @xcite . these findings extend prior work on geographic and linguistic biases in language models @xcite , suggesting that vlms may encode culture-specific stereotypes differently across languages. implications & mitigations. our observations motivate practical guardrails for vlm deployments that handle analogy prompts about people: (1) policy filters: block or warn on people-analogy prompts; (2) ui disclaimers: if analogy outputs are allowed, display a visible notice about potential cultural/linguistic biases; (3) lightweight monitoring: sample and audit outputs across languages to surface regressions. these measures are straightforward to implement and reduce risk without materially restricting benign use cases.