Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models (2025)

Abstract: Recent work has demonstrated that the unequal representation of cultures and socioeconomic groups in training data leads to biased Large Multi-modal (LMM) models. To improve LMM model performance on underrepresented data, we propose and evaluate several prompting strategies using non-English, geographic, and socioeconomic attributes. We show that these geographic and socioeconomic integrated prompts favor retrieving topic appearances commonly found in data from low-income households across different countries leading to improved LMM model performance on lower-income data. Our analyses identify and highlight contexts where these strategies yield the most improvements.

## introduction a lack of diversity in popular ai datasets @xcite leads to unequal model performance, further widening the technological gap between well-represented and underrepresented communities. while data from higher-income western communities are readily available online, lower-income and non-western data are often missing @xcite . as a result, cost-effective methods like web scraping fail to produce diverse datasets. one approach to building large datasets leverages lmm models to filter uncurated data based on image-text association strength scores @xcite . for instance, openai's clip vit-b/32 @xcite was used to filter web-scraped images to create the laion-5b dataset @xcite . however, foundation lmm models like clip perform unequally across cultures and socioeconomic groups, favoring higher-income and western images @xcite . datasets filtered by lmm models reflect the model's biases @xcite , often excluding underrepresented data and worsening the lack of diversity in ai models. @xcite demonstrates this by showing that the laion-5b dataset closely resembles data from western countries, such as the united states and canada while differing from non-western countries' data. this leads to lmm models with uneven performance on data drawn from different locations and income groups. therefore, our paper seeks to answer the following question: how do we improve the performance of lmm models on lower-income and non-western data? we tackle performance inequality in lmm models @xcite through prompting that transfers the cultural knowledge embedded in language @xcite @xcite . our goal is to improve the performance of lmm models on data from households with non-western and lower socioeconomic status. specifically, as shown in fig. 1 , we pose several research questions to evaluate the role of non-english languages, as well as prompts with geographic and socioeconomic attributes, to retrieve more diverse images. our contributions are summarized as follows. first, we show that a naive prompt translationbased approach fails to adequately address the performance gap of lmm models on lower-income data. second, we establish that geographic and socioeconomic attribute integrated prompts improve lmm performance on lower-income data. we identify contexts where these prompts work best by conducting an in-depth analysis of lmm models' understanding of these attributes and their effects on recall across data from different countries. lastly, we share insights from our analysis demonstrating how these attributes drive a perspective shift that benefits the retrieval of lowerincome data. ## methodology we propose prompting strategies that account for language, location, and socio-economic attributes and analyze how these prompts affect the performance of a multilingual lmm model on data across different socio-economic groups, primarily focusing on lower-income data. ## research questions we perform several analyses to answer three research questions that uncover and mitigate limitations in the performance of lmm models across different countries and socioeconomic groups. we calculate the cosine similarities between image and translated prompt text embeddings for each image-topic pair across english and 40 non-english languages, generating 41 alignment scores per image. the alignment scores with default english prompts serve as our baseline. we compute recall scores by selecting the top n images with the highest alignment scores for each topic, where n represents the number of ground truth images. we then group and analyze the recall scores across different countries and image income classes and present our findings below. native translated prompts perform consistently worse than english prompts on lower-income images from their respective countries. we focus our analysis on images from the two lowest image income groups, i.e., poor and low-middle as grouped in 3.1. after excluding 20 countries without data for these income groups (e.g., russia, turkey), we retain 39/59 countries and 28/40 non-english languages for the study. each country is paired with its native non-english language, and we compare recall scores for the native translated prompts to those for the default english prompts. the average recall across all countries and scores from four countries are displayed in figure 2 . for 35 out of 39 countries, the native translated prompts underperform compared to the default english prompts. the exceptions include burkina faso, nigeria, pakistan and tanzania, where native translated prompts in french (diff. of 1.0), hausa (diff. of 0.2), urdu (diff. of 0.7) and swahili (diff. of 1.5), respectively, outperform english prompts. overall, native translated prompts generally fail to retrieve diverse images, as depicted by the example using french prompt in fig. 1 . the best-performing non-english language often differs from the country's native language. we analyze the recall scores for lower-income images across 28 language prompts used in different countries and find that the bestperforming language prompts often differ from the countries' non-english major languages. specifically, in 24 out of 39 countries, non-english language prompts outperform the default english prompts, yet these top-performing languages are not typically spoken in the respective countries. as illustrated in figure 3 , for 37/39 countries, the language with the highest recall score (highlighted in yellow) differs from the country's primary non-english language (highlighted in cyan), with exceptions in indonesia and pakistan, where they coin- arabic bengali burmese chinese creole ewe french hausa hindi indonesian khmer kinyarwanda korean mongolian nepali oromo portuguese serbian shona somali spanish swahili tagalog thai ukrainian urdu vietnamese zulu languages bangladesh bolivia brazil burkina faso burundi cambodia cameroon china colombia cote d'ivoire ethiopia ghana guatemala haiti kenya india indonesia jordan mongolia myanmar nepal nigeria pakistan palestine peru philippines rwanda serbia somalia south africa south korea tanzania thailand togo tunisia ukraine united states vietnam zimbabwe countries 0.0 0.1 0.2 0.3 0.4 0.5 0.6 figure 3 : recall scores for lower income images from 39 countries and 28 languages. the cyan highlight shows the recall for a country's native translated language, the yellow highlight shows the best-performing language recall, and the red shows the recall for the language that is both the native and highest performing for that country. best viewed in color. ## lessons learned we highlight key insights learned from our findings and present them below. ## conclusion in this paper, we addressed the uneven performance of lmm models across different countries and income levels. we explored three attributeintegrated prompting strategies: (1) translation of text prompts to native non-english languages, (2) addition of geographic information, and (3) addition of socioeconomic attributes. we found that integrating geographical and socioeconomic information into prompts enhances lmm model performance on images from lower-income households and retrieves more diverse label representations. furthermore, we identified and highlighted the contexts where the proposed prompting techniques work best and shared our insights to improve representation in lmm models and datasets. our code can be used to evaluate the performance of other lmm models and datasets and is publicly available at analysis for uplifting lower-income data.