One More Modality: Does A bstract M eaning R epresentation Benefit Visual Question Answering? (2025)

Abstract: Visual Question Answering (VQA) requires a vision-language model to reason over both visual and textual inputs to answer questions about images. In this work, we investigate whether incorporating explicit semantic information, in the form of Abstract Meaning Representation (AMR) graphs, can enhance model performance—particularly in low-resource settings where training data is limited. We augment two vision-language models, LXMERT and BLIP-2, with sentence- and document-level AMRs and evaluate their performance under both full and reduced training data conditions. Our findings show that in well-resourced settings, models (in particular the smaller LXMERT) are negatively impacted by incorporating AMR without specialized training. However, in low-resource settings, AMR proves beneficial: LXMERT achieves up to a 13.1% relative gain using sentence-level AMRs. These results suggest that while addition of AMR can lower the performance in some settings, in a low-resource setting AMR can serve as a useful semantic prior, especially for lower-capacity models trained on limited data.

## introduction the task of visual question answering (vqa) @xcite challenges models to answer natural language questions about the content of an image. for example, given an image of a scene with sheep and the question "how many sheep are there?" the model must identify relevant visual elements and interpret the question to produce an accurate answer (see figure 1 ). recent advances in multimodal learning have led to substantial gains in vqa performance @xcite @xcite @xcite , powered by transformer-based vision-language models (vlms) pretrained on large-scale image-text datasets. these models typically operate on raw textual and visual embeddings and lack integration of deeper, structured semantic representations. however, prior work in visual semantic parsing and caption generation @xcite @xcite @xcite has demonstrated that incorporating formal semantic structures can enhance both the controllability and expressiveness of model outputs. given that vqa sits at the intersection of computer vision and language understanding, and promising prior work incorporating semantic representations into other vision-related tasks @xcite , we explore the integration of a semantic representation as an additional modality for this task. abstract meaning representation (amr; @xcite is a rooted, directed graphbased representation of meaning that broadly captures "who does what to whom," where the nodes in the graph correspond to concepts in the sentence and edges denote the relationship between those concepts. we hypothesize that the amr of an image description often encodes sufficient informa- question: how many sheep are there? (s / stand-01 :arg1 (g / sheep :quant 3 :mod (m / mountain)) :arg2 (a / amongst :op1 (r / rock :location (m2 / mountain)))) vision encoder language encoder amr bart encoder crossmodality encoder vqa answer classifier roi features word embedding word embedding (s / stand-01 :arg1 (g / sheep :quant 3 :mod (m / mountain)) :arg2 (a / amongst :op1 (r / rock :location (m2 / mountain)))) vision encoder amr bart encoder qformer qformer llm projection question: how many sheep are there? a b tion to answer associated questions. this motivates our central inquiry: can integrating amr into the vqa pipeline improve model performance, particularly under data or model capacity constraints? to investigate this, we frame the following research questions: rq1 how does the presence of amr in the training process impact vqa quality? rq2 how does model size impact the effectiveness of amr in vqa? rq3 can amr compensate for limited image data in low-resource settings? to investigate these questions, we augment vlms with amr graphs parsed from image descriptions in the coco dataset @xcite . we compare two architectures: lxmert @xcite , a first-generation cross-modal transformer, and blip-2 @xcite , a more advanced model that leverages frozen language models. to assess amr's impact under varied resource conditions, we experiment with fine-tuning both sentence-level and document-level (docamr) representations @xcite , and analyze different configurations including frozen model components and varying contributions from visual versus semantic inputs. our contributions include: ## methods & experiments in this section, we detail the data preparation procedures (section 2.1), model configurations (section 2.2), and evaluation metrics (section 2.3) used in our experiments. ## results we now present our experimental results addressing each of our three research questions. ## qualitative analysis to better understand model behavior in lowresource settings (as reported in building on this, we set out to examine whether including amr is more helpful in scenarios where the type of question aligns directly to a semantic role included in amr, thus indicating that there is a specific piece of information included in the amr that would correspond with the answer for that question type. we identify cases in which amr's structured semantics align with specific vqa question types; to do so, we identify a subset of questions that exhibit explicit correspondences with amr edge roles, as outlined in in the amr graph contains the correct answer to the associated question. similarly, questions like "what is in the..." may correspond to the :part-of relation, reflecting compositional structure. as shown in figure 4 , error rates are similar across all question types for the three models. we observe that amr-augmented models exhibit slightly lower error rates than their image-only counterparts for the question types "what is in the..." and "what is on the..." across both blip-2 and lxmert. this suggests that amr may help retain part-whole relationship information, likely due to the explicit use of the :part-of role in amr graphs. we also note modest performance gains for the question "what time..." which corresponds to the :time role, particularly in the do-camr variant. overall, the error rate reductions are more pronounced in lxmert than in blip-2. these observations support our earlier findings: lxmert, with its smaller model size and shallower cross-modal attention, is more sensitive to the structure of the amr input and performs better with sentence-level amr. meanwhile, blip-2's deeper architecture and advanced fusion mechanisms make it relatively insensitive to the specific form of amr used, maintaining stable behavior across both variants. however, it is not always the case that we observe a lower error rate for amr and docamr on these questions identified as having explicit amr roles. for example, while the amr model achieves the lowest error rate on "how many" questions in lxmert, it performs worse than the image-only baseline for the same question type in blip-2. in addition to raw error rate, we assess the quality these answers (and thus the models with and without amr) via bertscore @xcite and bleu @xcite , comparing predicted answers against ground truth responses. a prediction is considered correct if its similarity score exceeds a predefined threshold. we then calculate the percentage of correct answers for each model variant under the low-resource setting. for blip-2, consistent with earlier findings, the benefits of amr augmentation are minimal and largely unnoticeable under both bertscore and bleu-based thresholding. in contrast, lxmert shows noticeable improvements (see while performance gains are not uniform across all question types, our analysis indicates that incorporating amr can lead to targeted improvements, particularly when the semantics of a question aligns closely with explicit amr edge roles. these findings underscore the potential of amr to preserve and leverage specific semantic information, especially in low-resource settings where visual context alone may be insufficient. 5 related work vqa @xcite has gained significant research momentum over the past decade. early neural approaches primarily adopted encoder-decoder architectures @xcite @xcite , combining cnn-based image encoders with rnn-based question decoders. inspired by advances in attention-based machine translation @xcite , prior work has begun integrating visual attention mechanisms to improve question grounding @xcite @xcite @xcite @xcite . more recently, the rise of large-scale image-text datasets and the success of transformer-based vlm pretraining approaches have significantly boosted performance in vision-language tasks @xcite @xcite @xcite @xcite . these developments, along with breakthroughs in large language models @xcite @xcite @xcite and vlms @xcite , have led to the emergence of multi-stage, modular architectures-often referred to as 'socratic' models @xcite . in such architectures, a vision model and a language model are connected via intermediate projection or reasoning modules that enable interaction while keeping both components largely frozen. recent works that adopt this paradigm @xcite @xcite @xcite @xcite typically use frozen vision encoders and llms, bridging them with lightweight interface modules designed to align their representational spaces. however, current vlms largely ignore explicit high-level semantic representations. prior work has explored scene graphs @xcite @xcite and their utility in downstream visionlanguage tasks @xcite @xcite @xcite . more recent research highlights the value of structured semantic representations-such as semantic role labeling (srl; @xcite and amr @xcite -in improving performance on downstream multimodal tasks @xcite @xcite . additionally, as is shown in this work, amr has been shown to be particularly useful for engineering tasks in lowresource settings @xcite . the integration of semantic representations into modern pretrained vlms remains underexplored; in this work, we explore the impact of incorporating high-level semantic representations, specifically amr, into vlms for the targeted task of vqa-with a focus on leveraging amr in lowresource vqa settings. ## conclusion in this work, we explore the utility of incorporating semantic information, specifically abstract meaning representation, as an additional modality in the task of vqa. our findings indicate that current vlms do not consistently benefit from amr augmentation in standard training settings. experimenting on both a first-generation model (lxmert; @xcite and a more advanced model (blip-2; @xcite , we observe that the absence of amr-aware pretraining makes it challenging for these models to effectively integrate semantic structures as an additional modality. however, in our low-resource experiments using only 2.25% of the original training data, amr integration consistently improves performance across both model architectures. this suggests that amr can serve as an effective inductive bias, particularly when training data is limited, to compensate for a lack of visual data. one bottleneck in incorporating amr into the vqa pipeline is the lack of readily available amrs for images. developing an effective system for parsing images directly into amr-like semantic representations @xcite is a promising direction for future work. we generate silver-standard sentence-and document-level amrs for the vqa dataset to support reproducibility and future research on amr for vqa.