Does Mapo Tofu Contain Coffee? Probing LLM s for Food-related Cultural Knowledge (2025)

Abstract: Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain—a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs’ ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.

## fmlama construction to assess whether llms encode and access cultural information, we develop fmlama, a mul-ticultural, multilingual dataset focusing on culinary knowledge. the designed framework can be adapted to other cultural domains. step #1: obtain countries set. following zhou et al. (2023), we use countries of food origin to delineate cultural groups. this method leverages countries as proxies for cultural identity, encapsulating diverse traditions, values, and norms that reflect the breadth of human civilizations across geographical boundaries @xcite . step #2: acquire food instances. we utilize sparql to query wikidata, extracting a vast array of food-related data. this approach exploits wikidata's rdf triple structure to gather detailed information on food instances, offering a rich source of comprehensive food knowledge. i. class. for our food-focused dataset, we concentrate on the dish class and employ two approaches to find food instances: we exclude dishes with multiple origin countries to maintain cultural specificity. ultimately, our constructed dataset, fmlama, comprises 33,600 dish instances, detailed by name, origin, ingredients, and optionally, materials and images, encompassing 128 cultural country groups and 250 languages. the average number of ingredients in fmlama dataset is 2.04. examples and statistics are provided in appendix a. ## analysis: designing better prompts we examine the impact of different prompt settings on cross-cultural knowledge exploration, focusing on cultural background references, multilingual prompts, and code-switching scenarios. ## error analysis a manual inspection of a data sample 11 followed by an automated evaluation on the complete data reveals recurring mistake patterns: (1) coarsegrained confusions (foreign ingredients, e.g., coffee in a chinese dish), where the main issue lies in the model's lack of cultural attribute awareness for 9 the probing results comparison on other multilingual llms are shown in appendix d.2. ## conclusion this study presents an automated method for generating extensive cultural knowledge datasets, exemplified by the creation of fmlama, a diverse, food-centric dataset that spans multiple cultures and languages. we introduce novel metrics for cultural knowledge evaluation in llms, emphasizing the influence of cultural context and language in the probing process. our findings reveal a predominant bias towards american culture in llms when using english prompts, a bias that diminishes with prompts in other languages. interestingly, incorporating explicit cultural cues in prompts enhances llms' cultural knowledge access. the study also highlights the scarcity of culturally diverse knowledge across languages, pointing to a potential root of observed biases in llms.