SANSKRITI : A Comprehensive Benchmark for Evaluating Language Models’ Knowledge of I ndian Culture (2025)

Abstract: Language models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce , a benchmark designed to evaluate language models’ comprehension of India’s rich cultural diversity. Comprising of 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture namely rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife and personalities, providing a comprehensive representation of India’s cultural tapestry. We evaluate on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models(SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, sets a new standard for assessing and improving the cultural understanding of LMs. We will share the dataset and findings publicly to support research on inclusive and culturally aware AI systems.

## introduction language models (lms) have revolutionized the way we interact with technology, becoming indispensable tools in applications such as natural language understanding, content generation, knowledge discovery, and decision-making @xcite @xcite . models like large language models (llms), indic language models (ilms), and small language models (slms) are increasingly integrated into workflows across industries, enabling seamless communication and efficient problem-solving @xcite foot_0 . these advancements, however, come with the critical challenge of ensuring that lms cater to diverse populations by recognizing and reasoning about their linguistic and cultural contexts @xcite . while lms excel in syntactic and semantic language understanding, their effectiveness depends significantly on their ability to comprehend cultural diversity @xcite . a model's capacity to recognize cultural contexts is crucial for its application in education, governance, healthcare, and entertainment in culturally rich societies @xcite . the inability to incorporate cultural nuances risks perpetuating biases, stereotypes, or inaccuracies, thereby alienating underrepresented communities. models that can capture such diversity not only perform better but also foster inclusivity and equity in ai-driven solutions @xcite . india's unparalleled cultural diversity makes it an ideal testbed for evaluating the cultural competence of lms. the country is home to 28 states and 8 union territories, each characterized by distinct traditions, festivals, cuisines, music, dance forms, architectural styles, and historical narratives @xcite . for example, states like rajasthan and tamil nadu exhibit unique artistic and culinary traditions, while festivals such as diwali, eid, and onam highlight diverse religious and regional practices @xcite . this cultural richness presents significant challenges for lms. existing benchmarks like tydi qa @xcite and xquad @xcite primarily focus on multilingualism, overlooking cultural depth. models trained on global datasets often fail to capture region-specific nuances, such as the significance of kathakali performances in kerala or madhubani art in bihar @xcite . this gap in understanding not only limits the utility of these models in culturally complex regions but also hinders their acceptance by local populations @xcite . motivation for this research : the absence of culturally refined benchmarks, especially based on different nuanced attributes, inspired the creation of sanskriti, a dataset designed to evaluate the cultural understanding of lms in the indian context. unlike existing datasets like @xcite , which are smaller and do not include all indian states and union territories, sanskriti focuses on cultural aspects across all 28 states and 8 union territories of india, providing a complete picture of the country's rich cultural diversity. the motivation behind this research lies in bridging the gap between technological advancements in lms and their real-world applicability in culturally diverse societies. we developed the sanskriti dataset to benchmark cultural understanding in india, utilizing diverse data sources to cover key attributes like rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities. a team of 40 annotators curated 21,853 questions across four categories: association, country prediction, general awareness, and state prediction. research questions: this research aims to address the following questions: contributions: our key contributions are summarized as follows: ## conclusion in this work, we introduced sanskriti, a benchmark to evaluate language models' understanding of india's cultural diversity through over 21,000 curated question-answer pairs spanning 28 states and 8 union territories. sanskriti meets the need for culturally nuanced datasets by covering attributes like arts, festivals, and cuisines. model evaluations revealed significant gaps, especially for certain states, highlighting biases likely rooted in training data limitations. sanskriti dataset ensured quality, precision and cultural sensitivity, setting a new standard for inclusive ai research. by releasing sanskriti publicly, we aim to advance the development of ai systems that are culturally aware and capable of serving diverse populations effectively. looking forward, we plan to expand sanskriti by adding more attributes to capture nuanced aspects of indian culture, such as regional folklore and ecological heritage. additionally, we aim to build a visual question-answering dataset and a multilingual version of sanskriti, enabling broader accessibility and applications across diverse linguistic and cultural contexts.