NLKI : A Lightweight Natural Language Knowledge Integration Framework for Improving Small VLM s in Commonsense VQA Tasks (2025)

Abstract: Commonsense visual–question answering often hinges on knowledge that is missing from the image or the question. Small vision-language models (sVLMs) such as ViLT, VisualBERT, and FLAVA therefore lag behind their larger generative counterparts. To study the effect of careful commonsense knowledge integration on sVLMs, we present an end-to-end framework (NLKI) that (i) retrieves natural language facts, (ii) prompts an LLM to craft natural language explanations, and (iii) feeds both signals to sVLMs across two commonsense VQA datasets (CRIC, AOKVQA) and a visual-entailment dataset (e-SNLI-VE). Facts retrieved using a fine-tuned ColBERTv2 and an object information-enriched prompt yield explanations that largely cut down hallucinations while lifting the end-to-end answer accuracy by up to 7% (across three datasets), making FLAVA and other models in NLKI match or exceed medium-sized VLMs such as Qwen-2 VL-2B and SmolVLM-2.5B. As these benchmarks contain 10–25% label noise, additional finetuning using noise-robust losses (such as symmetric cross-entropy and generalised cross-entropy) adds another 2.5% in CRIC and 5.5% in AOKVQA. Our findings expose when LLM-based commonsense knowledge beats retrieval from commonsense knowledge bases, how noise-aware training stabilises small models in the context of external knowledge augmentation, and why parameter-efficient commonsense reasoning is now within reach for 250M models.

## introduction early visual question answering (vqa) work already recognised that images and questions alone are often insufficient and called for the use of external knowledge sources @xcite @xcite . factoid vqa now routinely benefits from graph-or web-scale retrieval @xcite @xcite , yet the commonsense variant remains under-explored, especially for small vision-language models (svlms) such as vilt, vi-sualbert and flava. recent attempts either bypass pre-trained small vlms @xcite or limit themselves to factual qa @xcite . a big challenge in commonsense knowledge retrieval is a lack of a unified source -no single knowledge graph, corpus, or model covers everyday physics, social conventions, and object affordances. large language models (llms) have emerged as promising but noisy reservoirs of such knowledge @xcite . these challenges motivate our four research questions: rq1) to what extent can commonsense facts retrieved from knowledge bases and llmgenerated explanations help small vlms?; and rq2) is one better suited than the other for commonsense reasoning in vqa systems?; rq3) can noise-robust losses mitigate the pervasive label noise in the context of external knowledge augmentation?; and to draw relevance to recent progress in lightweight generative ai, we analyse rq4) how do medium-size generative vlms (≤ 4b) fare in commonsense tasks, in comparison to svlms (with or without knowledge integration)? to probe rq1-2, we test the three svlms above (≤ 240m parameters) on cric and aokvqa @xcite along with a visual entailment dataset, e-snli-ve @xcite . using ground-truth explanations (along with the image and question) lifts accuracy by up to 10 to 12%, signalling the necessity and scope for utilising text-based knowledge. we therefore contrast (i) colbertv2-retrieved facts from commonsense knowledge bases (kbs) and (ii) free-form textual explanation produced by llama-3-8b from dense/region captions, list of objects, and retrieved facts. a richer visual context helps curb hallucination, while simple architectural tweaks temper the effect of occasional noisy snippets. because label noise dominates these datasets, rq3 tests whether generalised cross-entropy (gce) @xcite and symmetric cross-entropy (sce) @xcite preserve the gains and further boost the performance of knowledge integration, while mitigating the effects of label noise. finally, addressing rq4, we evaluate qwen-2, phi-3-vision, minicpm and smolvlm (≤ 4b) and show that they too lack commonsense without explicit knowledge integration, underscoring the urgency of lightweight, knowledge-aware methods. overall, we make the following contributions: ## related work knowledge-based vqa systems pose a challenging task, as they need the models to comprehend visual and textual data while utilising external knowledge to derive correct answers. for instance, as depicted in fig. 2 , "is there a place that is blue and liquid?". the correct result "ocean" has to be deduced using the visual information, common knowledge that it is "liquid" and "blue". symbolic structured knowledge is helpful in this context, for instance, existing literature has used knowledge graphs to retrieve facts, using knowledge graph embeddings and formal language queries @xcite @xcite . beyond kgs, logic-based formalisms have also been used to combine knowledge with reasoning, e.g., probabilistic soft logic for visual puzzle solving in @xcite . for unstructured knowledge, @xcite showed that the dense-passage-retrieval (dpr) technique performs the best by encoding both the question and the knowledge sentences to text embedding using bert. this method uses cosine similarity (or the dot product) between vectors to retrieve relevant knowledge text. ## problem formulation in visual question answering (vqa), some questions require knowledge beyond what is directly visible in an image. for example, identifying a giraffe as an "even-toed ungulate" or recognising a sofa as "furniture used for sitting" requires commonsense reasoning. ambiguities in commonsense datasets further challenge models, as multiple correct answers may exist, making implicit knowledge crucial for accurate predictions. addressing these gaps is essential for improving the vqa system's contextual understanding. here, we lay down the task more formally. the input for our task comprises an image (i) and a natural language question (q), and the objective is to answer the question while utilising a set of implicitly required commonsense knowledge facts (s) (k ′ ). we utilise the ground-truth answer (a) and the ground-truth explanation/knowledge (f * ), when available, as supervision. ## knowledge-augmented visual qa framework here, we explore two types of knowledge sources: traditional commonsense knowledge corpora (textual versions) and pre-trained large language models. following @xcite , we can represent a retrieval or knowledge-augmented vqa system probabilistically as follows: to generalise to llm-generated explanations, we can think of the reader as a model that outputs a paragraph relevant to the question. we will utilise the following notations wherever required: ## discussion learnings from gold-label baseline setup and effect of additional context on accuracy. when does knowledge integration help? impact in the context of generative vlms. ## conclusion our study shows that pairing external commonsense knowledge with noise-aware training @xmath0. a lightweight nlki pipeline-colbert-v2 retrieval plus llama-3.1-8b generated type-5 explanations-boosts the performance of vilt from 24 → 38 and flava from 46 → 54 % on aokvqa, outmatching 1-4b parameter generative baselines at a fraction of the compute. noise audits reveal 17-23% faulty labels in cric and aokvqa (virtually none in e-snli-ve); swapping vanilla ce loss for symmetric ce adds up to +3%, while a ce + gce mix is best at moderate noise, and vanilla ce suffices for clean data. finetuned colbert raises retrieval precision, and enriching prompts with dense/region captions curbs hallucination, though one fact per question is optimal given vlm context limits. in sum, nlki plus a dataset-aware robust loss turns 250m-parameter svlms into noise-resilient commonsense vqa engines that rival or beat multi-billion-parameter models without extra inference cost.