The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning (2025)

Abstract: Large Multimodal Models (LMMs) exhibit impressive performance across various multimodal tasks. However, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of most data and models. Conversely, multi-agent models have shown significant capability in solving complex tasks. Our study evaluates the collective performance of LMMs in a multi-agent interaction setting for the novel task of cultural image captioning. Our contributions are as follows: (1) We introduce MosAIC, a Multi-Agent framework to enhance cross-cultural Image Captioning using LMMs with distinct cultural personas; (2) We provide a dataset of culturally enriched image captions in English for images from China, India, and Romania across three datasets: GeoDE, GD-VCR, CVQA; (3) We propose a culture-adaptable metric for evaluating cultural information within image captions; and (4) We show that the multi-agent interaction outperforms single-agent models across different metrics, and offer valuable insights for future research.

## introduction large multimodal models (lmms) demonstrate remarkable performance across various multimodal tasks. despite these achievements, their effectiveness in cross-cultural contexts remains limited due to the predominantly western-centric nature of most data and models @xcite . conversely, multi-agent models have proven to be highly capable, often excelling in solving complex tasks @xcite . in this paper, we propose to evaluate and analyze the collective performance of lmms as multi-agent the picture depicts pasca and cozonac, foods that are traditionally made in romania at easter. the food is placed on a kitchen table decorated with traditional red and white towels. a: the food is pasca and cozonac, made in romania at easter. q: what room is in the picture? a: this looks like a kitchen … q: what is on the table ? a: on the table are red and white towels and food. q: what kind of food is that? figure 1 : in a multi-agent setting, three lmm agents, each embodying a curious persona and drawing upon knowledge from distinct countries (india, china, and romania), participate in a question-and-answer dialogue centered around an image. a fourth agent then summarizes their discussion, creating a culturally enriched image caption. models in the novel multimodal task of culturally enriched image captioning. culture is a complex and elusive concept. as @xcite show, culture is a multifaceted concept meaning different things to different people at different times. this complexity is apparent in various cultural expressions, such as proverbs, social norms, and other contextdependent elements. in our work, we adopt the definition provided by @xcite and focus on visual cultural elements such as food, drinks, clothing, traditions, rituals, and behaviors. culture is strongly tied to our group-oriented human nature, which allows us to learn from one another over generations. furthermore, as sociologists and anthropologists have demonstrated, our progress as a species is primarily due to our cooperative nature, rather than individual knowledge @xcite . inspired by the success of human collective intelligence, we conceptualize the culturally enriched image captioning task as a "social task". specifically, we frame it as a dialogue between three agents from different cultures who seek to learn about each other's cultures through an image. they engage in asking questions and sharing insights, akin to human collaborative problem-solving. a moderator provides examples of initial questions and highlights key visual cultural aspects to focus on. the conversation is then summarized into a comprehensive cultural image description (see figure 1 ). our findings indicate that this multi-agent approach yields better results than single-agent methods. we summarize our contributions as follows. first, inspired by collective intelligence, we propose mosaic, a novel multi-agent framework to improve cross-cultural image captioning performance. second, we share a dataset of 2,832 culturally enriched image captions in english, for images from three different countries: china, india, and romania, across three datasets: geode, gd-vcr, and cvqa. third, we introduce a cultureadaptable metric for evaluating cultural information within image captions. finally, we show that multi-agent interaction surpasses singleagent (and culturally fine-tuned) models across different metrics, and provide actionable steps for future work. at the same time, large multimodal models (lmms) have extended the capabilities of traditional language models by integrating several data modalities such as text, videos, and images. lmms such as llava @xcite , @xcite or lens @xcite have shown promising results in complex vision-language tasks due to their pretraining on terabytes of image and language data with billion-parameters @xcite to the best of our knowledge, our study is the first to employ lmms in a multi-agent setting for a cross-cultural multimodal understanding task. ## mosaic: a framework for cultural image captioning we introduce mosaic, a framework for multi-agent interactions, as shown in figure 2 , to tackle cultural image captioning, a complex task that involves not only describing the visual content of the image but also capturing the cultural elements it represents. the framework consists of a multiagent model, a cultural benchmark, and evaluation metrics, as described below. ## evaluation and results we assess the influence of multi-agent interaction on image captioning by comparing our multi-agent interaction model, mosaic, with single-agent models (blip-2, llava-13b) and a human baseline. llava-13b. llava-1.5 13b 1 is an end-toend trained large multimodal model that connects pre-trained clip vit-l/14 visual encoder and the vicuna llm @xcite , using a projection matrix for general multimodal understanding. ## lessons learned and actionable steps our findings reveal the performance of multi-agent lmms in cultural image captioning, highlighting lessons learned and suggesting steps to enhance cultural richness in future models. develop efficient cross-cultural techniques. current approaches to improving cross-cultural understanding in lmms often rely on fine-tuning. however, we show that interaction-based models outperform fine-tuned, non-interaction models (figure 6 c), highlighting both the effectiveness and efficiency of multi-agent lmms. for instance, llava-13-ft-all requires 54 hours on a single nvidia a100 gpu to generate captions (12h for fine-tuning on 9000 wit images and 42h for inference). in contrast, mosaic completes the task in 47 hours with only inference. given these findings, future work should focus on multi-agent models to improve sustainability and accessibility. lmms focus on culture; humans focus on correctness. lmms tend to be more culturally specific in their responses, while humans provide more accurate answers (section 4.4). the main sources of errors in lmms stem from poor object recognition and hallucinations-instances where the model generates incorrect or fabricated information. future work can integrate a state-of-the-art object recognition system to enhance caption accuracy. based on our findings, we share ideas for future work. our dataset and models can be accessed at https://github.com/michigannlp/mosaic .