I ndi F ood VQA : Advancing Visual Question Answering and Reasoning with a Knowledge-Infused Synthetic Data Generation Pipeline (2024)

Abstract: Large Vision Language Models (VLMs) like GPT-4, LLaVA, and InstructBLIP exhibit extraordinary capabilities for both knowledge understanding and reasoning. However, the reasoning capabilities of such models on sophisticated problems that require external knowledge of a specific domain have not been assessed well, due to the unavailability of necessary datasets. In this work, we release a first-of-its-kind dataset called IndiFoodVQA with around 16.7k data samples, consisting of explicit knowledge-infused questions, answers, and reasons. We also release IndiFoodKG, a related Knowledge Graph (KG) with 79k triples. The data has been created with minimal human intervention via an automated pipeline based on InstructBlip and GPT-3.5. We also present a methodology to extract knowledge from the KG and use it to both answer and reason upon the questions. We employ different models to report baseline zero-shot and fine-tuned results. Fine-tuned VLMs on our data showed an improvement of ~25% over the corresponding base model, highlighting the fact that current VLMs need domain-specific fine-tuning to excel in specialized settings. Our findings reveal that (1) explicit knowledge infusion during question generation helps in making questions that have more grounded knowledge, and (2) proper knowledge retrieval can often lead to better-answering potential in such cases. The data and code is available at https://github.com/SLSravanthi/IndifoodVQA.

## introduction visual question answering (vqa) was initially introduced as a mechanism to compare the ability of machines to behave like a human @xcite . since the advent of chatbots like chatgpt that show a high degree of understanding, they have become a common interface for human-machine interaction, where humans frequently ask questions based on specific domains to solve various problems. for instance, a restaurant foot_0 data and code are available at indifoodvqa. chatbot should excel in food-related queries and images, while fashion chatbots should specialize in recognizing delivered clothing items within images. while humans are extremely efficient at answering questions involving a single domain both before and after undergoing proper training, the same cannot always be said about language models. to develop such models, substantial domain-specific data is essential. the primary necessity here is to get datasets that enable vlms to show capabilities to understand and reason based on both prevalent and external knowledge. there have been numerous works pertaining to the requirement of commonsense knowledge @xcite @xcite in vqa, most using day-to-day images from datasets such as ms-coco @xcite and knowledge entities from generic kgs like conceptnet @xcite . only recently has attention grown towards a higher degree of reasoning according to knowledge in a particular area of interest @xcite . however, a big subset of curated datasets have been made by crowdsourcing efforts, which albeit being of high quality, are not easy to scale. with most state-of-the-art (sota) llms trained on huge chunks of data, this can be a big bottleneck. in this work, we present a framework that leverages domain-specific knowledge and the superior capabilities of llms in text generation to create a reasoning benchmark with minimal human effort. our contributions are: figure 1 : examples from our dataset indifoodvqa, that require multiple reasoning steps. the second example shows a situation where the externally infused triples were used to reason on the generated question and answer. the selection of the food domain, specifically indian cuisine, is driven by its extraordinary diversity and daily significance. present object detectors and vision encoders encounter challenges when tasked with identifying these food items, often confusing them with western food dishes. this inherent bias also gets incorporated into the sota vlms, which serves as additional motivation for choosing the niche domain of indian food. ## experimental setup in this section, we describe the experimental setup used to establish the baselines. the dataset has been split into the train, validation, and test sets in a ratio of 70 : 10 : 20, thus consisting of 11, 709, 1661, and 3346 questions. the split into the test set has been done maintaining a roughly equal number the random model gives a random answer and a random reason from questions belonging to the same type in the train set, and i under the model name stands for vlms with an instruction-tuned base llm. knowledge refers to the type of triples presented to the models during inference, as explained in section 5.1. no kg means inference without any external knowledge, 1-hop and 2-hop are for inference with the triples extracted by the corresponding method, and original refers to inference with the triples given to gpt-3.5 during question generation. the bold values are the best accuracy scores by the 4 models and the best metric on reason generation across different models. ## conclusion we developed a novel domain-specific vqa generation pipeline using the existing large models and domain-specific knowledge from our curated kg indifoodkg. to the best of our knowledge, this is the first synthetic data generation pipeline that uses both external knowledge and the model's internalized knowledge for creating vqa data. we have evaluated the performance of various baselines to establish the quality of the proposed dataset and showed how existing llms generally do not demonstrate good zero-shot performance when constrained to a domain. our results showed a 15% improvement in accuracy with a fine-tuned llava model over the best-performing zero-shot vlm. through this endeavor, our aim is to expedite multimodal research in fields where generating data at scale is a costly and labor-intensive task. given the extensive training datasets used by contemporary llms, evaluating their effectiveness when incorporating external knowledge not present during training becomes increasingly critical. assessing these models with knowledge pertaining to lessexplored fields offers an optimal approach for such evaluation. additionally, these datasets can serve as crucial benchmarks for detecting biases in sota vlms. the architecture of our pipeline allows for seamless replacement of its components with elements from other domains, facilitating the creation of benchmarks and conducting studies in lowresource domains. detailed insights into the generalizability of our model to diverse domains are dis-cussed in appendix f. our research also prompts potential modifications in both retrieval and modeling techniques to enhance the off-the-shelf domainrelevant performance of versatile llms.