From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation (2023)

Abstract: Addressing the challenge of adapting pre-trained vision-language models for generating insightful explanations for visual reasoning tasks with limited annotations, we present ReVisE: a Recursive Visual Explanation algorithm. Our method iteratively computes visual features (conditioned on the text input), an answer, and an explanation, to improve the explanation quality step by step until the answer converges. We find that this multi-step approach guides the model to correct its own answers and outperforms single-step explanation generation. Furthermore, explanations generated by ReVisE also serve as valuable annotations for few-shot self-training. Our approach outperforms previous methods while utilizing merely 5% of the human-annotated explanations across 10 metrics, demonstrating up to a 4.2 and 1.3 increase in BLEU-1 score on the VCR and VQA-X datasets, underscoring the efficacy and data-efficiency of our method.

## introduction explanations for visual reasoning are important in real-world applications @xcite like assistive technologies @xcite and interactive learning @xcite , but collecting human annotations for these explanations is expensive. the use of language models (lms) and pre-trained visionlanguage models (vlms) have shown promise in explanation generation @xcite . however, generating high quality explanations remains a considerable challenge when annotations are scarce @xcite . previous work has aimed to ameliorate this issue by focusing on enhancing model architecture and subsequent finetuning using large amounts of human-annotated explanations @xcite . nonetheless, such techniques, reliant on extensive fine-tuning, fall short in the face of limited annotations. thus, we propose an approach to amplify the model's own reasoning capabilities during inference to generate highquality explanations. recent research has demonstrated the efficacy of step-by-step reasoning in language and multimodal reasoning, particularly in contexts where samples are limited @xcite @xcite . as such, we adopt a phased approach, integrating visual and linguistic components for stepby-step vision-language explanation. in this work, we introduce the recursive visual explanation (revise) -a method for generating visual reasoning explanations that surpasses previous methods while using merely 5% of the human-annotated explanations. initially, we finetune blip-v2 @xcite to generate explanations on 5% of the dataset. during inference, we generate an initial explanation, then iteratively generate new explanations based on the preceding one. each step involves computing new visual features, guided by the preceding sentence. this sentence and the new visual features then serve as inputs to generate a new sentence in the next step. crucially, revise serves as a dynamic, selfcorrecting mechanism by progressively redirecting visual attention on the image and regenerating the explanation over steps. additionally, re-vise generates pseudo-ground truth explanations for few-shot self-training, producing pseudo-labels that considerably aid self-improvement compared to traditional pseudo-labels. we evaluate revise on four vision-language natural language explanation (vl-nle) taskse-snli-ve @xcite , vqa-x @xcite , aok-vqa @xcite , and vcr @xcite . our results show improvements across ten evaluation metrics, with enhancements of up to 4.2 and 1.3 in the bleu-1 score on vcr and vqa-x respectively. furthermore, self-training using our method's pseudo-ground truth explanations shows consistent progress compared with traditional generationbased self-training. further in-depth ablation studies elucidate the impact of revise and the insights behind it, indicating that sentences can effectively guide visual attention, and that sentence structure and phrasing style are pivotal for few-shot self-training. our contribution are summarized as follows: vision-language natural language explanation (vl-nle) vl-nle tasks demand a comprehensive understanding and reasoning across both vision and language modalities @xcite . there are two prevailing strategies: the first is a modular approach integrating two separate modules-one for predicting an answer, and another for generating an explanation-represented by works such as e-ug @xcite , pj-x (park et al., 2018), fme @xcite , rtv @xcite , and qa-only @xcite . the second approach is a unified one that uses a single model to generate an answer and explanation simultaneously; relevant works include nlx-gpt @xcite and ofa-x mt @xcite . our work utilizes the more efficient and training-effective unified approach. however, these methods fall short in effectively integrating the reasoning alignment between vision and language, a gap we address with revise. vision-language reasoning vision-language reasoning is a cornerstone of vision-language explanation generation. ## method in this section, we first provide an overview of the architecture of blipv2 @xcite and how we trained blipv2 for vl-nle tasks. then, we provide a detailed introduction and pseudo code for revise. finally, we discuss how revise is employed for self training. ## experiments in this section, we first introduce the basic settings including the task formulation, training details, baselines, and metrics. then, we provide detailed experiment results and in-depth analysis for the results. ## conclusion in this paper, we introduce revise, a method for generating natural language explanations for visual reasoning tasks with a small number of annotations. we demonstrate its high performance in generating language explanations using significantly less data compared to existing models, enabling the model's self-correction mechanism, and effectively facilitating few-shot self-training by generating robust pseudo-explanations. our work raises the question of whether recursive procedures like revise can be used to improve performance in other multimodal domains as well.