F oodie QA : A Multimodal Dataset for Fine-Grained Understanding of C hinese Food Culture (2024)

Abstract: Food is a rich and varied dimension of cultural heritage, crucial to both individuals and social groups. To bridge the gap in the literature on the often-overlooked regional diversity in this domain, we introduce FoodieQA, a manually curated, fine-grained image-text dataset capturing the intricate features of food cultures across various regions in China. We evaluate vision–language Models (VLMs) and large language models (LLMs) on newly collected, unseen food images and corresponding questions. FoodieQA comprises three multiple-choice question-answering tasks where models need to answer questions based on multiple images, a single image, and text-only descriptions, respectively. While LLMs excel at text-based question answering, surpassing human accuracy, the open-sourced VLMs still fall short by 41% on multi-image and 21% on single-image VQA tasks, although closed-weights models perform closer to human levels (within 10%). Our findings highlight that understanding food and its cultural implications remains a challenging and under-explored direction.

## introduction one of the most popular dishes in china is hotpot, which comes in many varieties, as shown in figure 1 the hotpot varieties from sichuan and chongqing are celebrated for their flavorful broths, with chili peppers and sichuan peppercorns that create a unique numbing-spicy sensation. the variation among regional cultures within a country highlights the challenges that language models face in understanding cultural knowledge and contextspecific information in the food domain. ## related work multilingual multimodal datasets multimodal systems are typically evaluated on english due to the widespread availability of english-language datasets. however, there are some examples of research on training and evaluating models beyond english for image captioning @xcite , image-sentence retrieval @xcite , visual reasoning @xcite , and questionanswering @xcite . this paper focuses on chinese visual question answering, with fine-grained attributes in the food domain. food datasets in recent years, most food datasets have been designed for food image classification @xcite , food captioning @xcite , and recipe-focused generation and question answering @xcite @xcite . for culture knowledge probing in the food domain, some of the recent datasets span multiple countries and include broad cultural or regional metadata @xcite @xcite . however, they often use country as a proxy for culture, such as the country of origin for the food. for example, @xcite english. investigating cultural differences within a country remains an under-explored area @xcite . fine-grained vision-language understanding @xcite quantified the fine-grained vision-language understanding capabilities in existing models, focusing on aspects within the general domain. later works focus on the culture understanding in vlms @xcite . however, current fine-grained vl datasets @xcite @xcite ; hendricks and nematzadeh, 2021) are often framed as binary classification tasks, which limits their difficulty. concurrently with our work, @xcite and @xcite have created culturally-diverse question-answering datasets across multiple countries. our multi-choice vision question answering dataset that focuses on chinese regional differences aims to advance the boundaries of fine-grained understanding in the context of food and culture. ## foodieqa: dataset annotation china, with its expansive territory and long history, has cultivated rich and diverse food culture and traditions. focusing on regional food culture differences, our dataset collection contains five distinct phases. 1) selection of cuisine types inside china; 2) collection of private images; 3) individual dish annotation; 4) visual question formulation; 5) text question formulation. ## analysis in this section, we explore which factors are important for fine-grained understanding of chinese food culture. non-public images are crucial for fair evaluation. we incorporate user-uploaded non-public images into our dataset to prevent data contamination during evaluation. to verify the importance of preserving these non-public images for fair evaluation, we compare model performance using websourced images instead. specifically, we manually searched with dish names to obtain web images for 171 out of 256 questions in the single-image vqa task. as shown in please see comparison examples in appendix f.2. dish names could be helpful clues for some of the models. as discussed in section 4.2, over 73.4% of single-image questions require multi-hop reasoning, which typically involves identifying the dish and then leveraging related knowledge to answer the questions. to determine whether the identification of the food image and the utilization of visual information are bottlenecks for the models, we compare their performance on single-image vqa when provided with the dish name in the question. the results in favorite food of the models. in figure 9 , we compare model performance on multi-image vqa tasks for questions grouped by food categories and cuisine types. this analysis provides insight into n o r t h e a s t s h a n g h a i g u iz h o u n o r t h w e s t x in ji a n g h u n a n f u ji a n o t h e r s s ic h u a n i n n e r m o n g o li a / j ia n g x i/ a n h u i z h e ji a n g j ia n g s u g u a n g d o n g m a n t is ## conclusion we introduce foodieqa, a multimodal dataset designed to evaluate fine-grained understanding of chinese food culture through multi-image, singleimage, and text-only multiple-choice questions. our experiments, which focus on regional cultural differences and detailed visual features, reveal that understanding food and its cultural context remains a complex and under-explored task. we find that comparing food across multiple images-similar to the common scenario of people browsing menus-is particularly challenging. all open-source models underperform human accuracy by more than 40% in this task. this suggests that our dataset offers a more accurate assessment of the suitability of state-of-the-art models for real-world applications in the food domain. our analysis of language and prompt templates indicates that models can be sensitive to the language in which questions are asked-bilingually trained chinese-english models perform better in chinese, while other multilingual models are stronger in english. we also demonstrate the effectiveness of incorporating visual features compared to text-only settings in this context. improved models or methods for understanding food culture may be essential for future progress in the foodieqa challenge. looking ahead, we aim to expand the dataset to include dishes from other countries and regions. following jacovi et al. (2023), we make our dataset a public benchmark on huggingface at lyan62/foodieqa with the cc by-nc-nd 4.0 license. all of our data annotation and verification tools are freely available for re-use at github.com/lyan62/foodieqa. we encourage the community to create foodie datasets for their own language and culture groups.