Pearl: A Multimodal Culturally-Aware A rabic Instruction Dataset (2025)

Abstract: Mainstream large vision-language models (LVLMs) inherently encode cultural biases, highlighting the need for diverse multimodal datasets. To address this gap, we introduce PEARL, a large-scale Arabic multimodal dataset and benchmark explicitly designed for cultural understanding. Constructed through advanced agentic workflows and extensive human-in-the-loop annotations by 37 annotators from across the Arab world, PEARL comprises over 309K multimodal examples spanning ten culturally significant domains covering all Arab countries. We further provide two robust evaluation benchmarks (PEARL and PEARL-LITE) along with a specialized subset (PEARL-X) explicitly developed to assess nuanced cultural variations. Comprehensive evaluations on state-of-the-art open and proprietary LVLMs demonstrate that reasoning-centric instruction alignment substantially improves models’ cultural grounding compared to conventional scaling methods. PEARL establishes a foundational resource for advancing culturally-informed multimodal modeling research. All datasets and benchmarks are publicly available.

## introduction mainstream large vision-language models (lvlms) predominantly encode western perspectives, often neglecting diverse cultural traditions. recent efforts have partially addressed this issue through multilingual datasets @xcite and benchmarks like culturevlm @xcite , gimmick @xcite , cvqa @xcite , and related initiatives @xcite . however, some of these resources lack sufficient cultural depth, often relying on translated content, limited image sets, narrow topical coverage, and simplistic factual questions unsuitable for evaluating advanced lvlm capabilities. furthermore, they frequently assume local cultural homogeneity, overlooking nuanced regional variations even within shared cultural artifacts. thus, developing high-quality, culturally nuanced multimodal resources that adequately challenge contemporary lvlms remains significantly underexplored, especially for many global regions, including the arab world. addressing this gap, we introduce pearl, a largescale arabic multimodal instruction dataset and benchmark explicitly designed for cultural understanding. to ground our work, we operationalize 1 https://github.com/ubc-nlp/pearl culture through the lens of cultural heritage. that is, the focus of the work is on the enduring traditions, values, and practices validated by native speakers @xcite . this framing provides a clear scope that distinguishes deeply rooted customs from contemporary, globalized practices. our approach aligns with recent efforts to systematically measure and model cultural dimensions in language models, which highlight the importance of heritage and regional specificity @xcite . pearl is the outcome of an extensive collaborative effort involving a diverse community of 37 contributors, all of whom are authors of this work, spanning the arab region. by leveraging advanced agentic workflows alongside iterative human-in-theloop refinement, our dataset integrates sophisticated llm and lvlm outputs with the nuanced cultural expertise of native annotators. covering ten culturally significant domains, pearl authentically captures arab cultural heritage. moreover, we structure it around ten distinct question categories designed to test sophisticated lvlm capabilities, including hypothesis formation, problem-solving, comparative analysis, and chronological sequencing. to specifically evaluate subtle cultural variations overlooked by existing benchmarks, we also introduce pearl-x (figure 3 , a novel benchmark highlighting culturally shared yet visually distinct concepts (e.g., coffee) across different arab contexts. unique among benchmarks, pearl-x incorporates both text-to-single-image and text-tomultiple-image pairings, enabling richer assessments of lvlm performance on complex multimodal tasks. we leverage our benchmarks to systematically evaluate a range of open and proprietary lvlms across diverse sizes and capabilities. our results demonstrate that models incorporating reasoning-based cultural alignment substantially outperform those relying solely on conventional scaling approaches. the rest of this paper is organized as follows: section 2 reviews related work on multilingual and cultural benchmarks. section 3 details our data collection, annotation pipeline, and benchmark design. section 5 describes the evaluation protocol and experimental setup. section 6 presents our findings and analysis. section 7 introduces the novel pearl-x benchmark .finally, section 8 concludes the paper and outlines future directions. multi-cultural vqa datasets. relatively few vqa datasets prioritize cultural diversity in their image curation and qa generation processes. for instance, culturalvqa @xcite incorporates images representing cultural concepts (2, 378 image-qa pairs, 11 countries) but is limited to english. the alm-bench @xcite benchmark covers cultural aspects from 73 countries and 100 languages (2, 929 images, 22, 763 qa pairs, 13 domains). its qa pairs were automatically generated and translated using gpt-4o, followed by human verification. maxm @xcite is a test-only vqa dataset in seven languages (five scripts), featuring 2, 142 auto-generated and human-verified qa pairs. cvqa @xcite provides qa pairs in the local languages of 30 countries, alongside english translations for 5, 239 images (10, 374 questions). more extensively, cultureverse @xcite ) contains 74, 959 images covering 19, 682 cultural concepts from 188 countries, with its gpt-4o-generated qa pairs validated through automated and manual checks. arabic cultural vqa. despite the growing interest in vqa, arabic cultural representation in existing datasets remains sparse. the henna dataset @xcite is specifically dedicated to arabic culture, comprising 1, 132 images from 11 arabic countries. other datasets offer minimal arabic content: alm-bench @xcite includes 168 images from saudi arabia, uae, and egypt; cvqa @xcite ) contains approximately 200 egyptian images; and culture-verse @xcite features only seven libyan and 272 egyptian images. more recently, jeem @xcite introduced a benchmark for five arabic dialects, consisting of 10.89k qa pairs and 2, 178 images across 13 cultural domains. while jeem improves arabic vqa coverage, it is limited to four question categories and highlights the ongoing need for broader and deeper cultural representation. to the best of our knowledge, pearl is the first large-scale, culturally diverse arabic multimodal benchmark carefully constructed through extensive human supervision, covering a wide range of cultural domains and challenging question types requiring reasoning and deep cultural knowledge. a comparative analysis with existing datasets is presented in ## our datasets apart from our main benchmarks (see sections §4.1 and §7), our pipeline produces several highquality datasets suitable for various purposes: (1) culturally-relevant images, comprising 12, 637 manually selected images reflecting distinct countrylevel cultural nuances across ten domains; (2) augmented captions, a set of 135k carefully crafted captions via agents derived from arabic wikipedia articles. these enriched captions serve as a visual knowledge base, providing contextually rich and interconnected information beyond standard image descriptions, and are publicly released for community use; (3) automated q&a pairs, including ## pearl-x benchmark while working on pearl, we observed existence of cultural element or practices prevalent across multiple regions or countries that, despite a fundamental similarity, exhibit subtle local variations in appearance, preparation, usage, or style. for instance, the traditional headband known as the agal " ", widely worn in countries such as the uae, saudi arabia, and other gulf nations, displays regional differences in shape, color, and style. motivated by this cultural insight and supported iteratively by the chatgpt-o3 model, foot_4 we manually identified 61 such culturally shared concepts, each observed across at least two foot_5 arab countries. 8 a complete list of the identified shared concepts is provided in appendix d.2. we then manually located images from publicly accessible web resources. on average, we gathered approximately three representative images per concept, for a total of 347 images, ensuring a rich visual depiction of cultural diversity. we then developed mcq and true/false questions exploiting the collected images. we include two categories of questions, differing based on whether we feed the model a single image or multiple images. single-image questions focus on aspects specific to one country through an individual image, whereas multiple-image questions target variations among countries regarding the particular shared concept depicted across several images. the next step was to generate questions based on manually developed templates that we provide to chatgpt-o3 along with the images and the name of each shared concept, enabling it to generate relevant questions. the prompt we developed to generate the questions is shown in appendix d.9. in total, we produce 367 questions, split into 177 single-image and 190 multiple-image questions. finally, we conduct a thorough human review of all generated questions, ensuring that the questions and answers are accurate, meaningful, error-free, and diverse. figure 3 demonstrates the workflow used to develop pearl-x benchmark. figure 4 shows examples of the shared concepts for both single and multiple image questions. additionally, appendix d.3 presents sample questions based on one-image and multiimage templates used for generating shared concept questions. ## conclusion in this paper, we presented pearl, an arabic multimodal instruction dataset and benchmarking suite tailored to enhance the cultural understanding capabilities of large vision-language models. pearl addresses critical gaps in existing resources by encompassing diverse culturally-authentic materials across ten domains from the arab world. our rigorous annotation and agentic workflows, combined with the expertise of 37 local annotators, ensure high-quality, culturally-relevant content. comprehensive evaluations confirm the superior performance of models explicitly optimized for reasoning tasks over parameter scaling, underscoring the importance of culturally-aware alignment methods. the specialized pearl-x benchmark further allows nuanced assessment of cross-country cultural variations, setting the stage for more culturally sensitive model development.