Paper: RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering (ICLR-2024)

Reviewer comment (Experiments):
The paper focuses on VQA tasks, and it's not clear how's performance of the proposed method when it was adapted to other vision-language tasks.

Author response:
**A2:** We thank the reviewer for the suggestion. We are happy to point out that we did apply our Rapper on another VL-NLE (natural lange explanation) task of Visual Entailment (VE), as listed in the lower part of Table 1 of our main paper. 

We note that, VE requires one to assign a label determining the relationship between a premise image and a textual hypothesis (i.e., “entailment”, “neutral”, and “contradiction”), while giving an explanation for supporting the aforementioned relationship [6]. And, VE is tyically applied in cross-modal retrieval, etc. applications. In Table 1 of our main paper, we report the performance on the **e-SNLI-VE** benchmark, in which we observed that our Rapper performed favorably against NLXGPT [7] and S3C [8] in terms of different NLG metrics (i.e., CIDEr, SPICE, and ROUGE-L). We will add the definition of VE with the above additional clarification to Sect. 4.1 of our main paper.

[6] Do et al. e-snli-ve: Corrected visual-textual entailment with natural language explanations. CVPRW 2020.

[7] Sammani et al. Nlx-gpt: A model for natural language explanations in vision and vision-language tasks. CVPR 2022.

[8] Suo et al. S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning. CVPR 2023.

Reviewer comment (Evaluation):
No human evaluation is conducted regarding the generation quality.

Author response:
**A3:** We thank the reviewer for suggesting additional experiments for subjective evaluation. Following the evaluation setting/process applied in NLXGPT [7] and S3C [8] (i.e., two SOTAs on VQA-NLE), we randomly select 200 test samples from the VQA-X [5] dataset with correctly predicted answers. Then, subjective evaluation is performed by 3 different annotators. Note that each annotator has to select one out of 4 choices: *yes, weak yes, weak no,* and *no*, as a response for whether the explanation justifies the answer. And, these 4 decisions are numerically mapped to 1, 2/3 , 1/3 , and 0, respectively. With averaged results obtained for each method, we present the performance comparisons in the following table. From this table, we see our proposed Rapper is preferable by the users in terms of subjective plausibility and faithfulness assessment. This conclusion also aligns with the objective quantification evaluation of Table 1 presented in the main paper.


| Method | Human score regarding plausibility (&uarr;) | Human score regarding faithfulness (&uarr;) |
|-|-|-|
| NLXGPT [7] | 0.771 | 0.795 |
| S3C [8] | 0.797 | 0.811 |
| *Rapper* (Ours) | **0.845** | **0.859**  |

Reviewer comment (Evaluation):
How does the complexity of the two-stage approach impact the computational efficiency of the model?

Author response:
**A1:** We thank the reviewer for suggesting further analysis for the computational cost. To discuss the trade-off between computational cost and our model performance, we now present the total and trainable model parameters of Rapper and its ablated variations (as listed in the upper part of Table 3). 

From the table below, we see that Rapper significantly improves the metrics of natural language generation (e.g., CIDEr, SPICE, and Rough-L) over both the Baseline (Rapper w/o KD and w/o RLNF, a one-stage approach) and Rapper w/o RLNF (a two-stage approach). It is worth noting that, the Baseline only deploys the proposed reasoning module $M$ that is based on BLIP2 [1] and has a total of 3.82B parameters, including 187M trainable parameters (i.e., Qformer, initialized from BERT [2]) and 3.63B freezed parameters (i.e., ViT [3] and OPT [4]). On the other hand, Rapper w/o RLNF and the full version of Rapper are composed of the proposed rationale generator $G$ and reasoning module $M$. For such 2-stage frameworks, both modules are based on BLIP2 and share 3.63B freezed parameters, with 187M trainable parameters for each (and thus a total of 374M trainable parameters). It can be seen that, the two-stage methods do require a larger number (2X) of trainable parameters than the one-stage method does. However, the increased amount is in the same order (187M), and it is remarkably less than the total amount of parameters (~4B).

As for the training/inference stages, the runtime estimates are also listed in the table below, which are performed on 8/1 Nvidia V100 GPU(s) on the VQA-X [5] dataset. From the table below, we see that the two-stage method of Rapper w/o RLNF requires 2X computation time for both training and testing, and the full version of Rapper needs additional 3X computation time due to the RL optimization scheme. Nevertheless, even with RLNF, training can be done in 18hrs without utilizing an advanced GPU environment. And, acceleration of LLM training/inference is still an active research topic in the community. We sincerely thank the reviewer for suggesting the above analysis, which will be added to the revised version for the completeness of our discussions.


| Method | Total / Trainable param. | Training time (hr) | Inference time (ms/sample) | B@4  | Rough_L | CIDEr | SPICE | Acc. |
|-|-|-|-|-|-|-|-|-|
| Baseline (Rapper w/o KD w/o RLNF) | 3.82B / 187M | 3 | 167.7 | 29.3 | 51.6 | 112.1 | 22.3 | 85.0 |
| Rapper w/o RLNF | 4.01B / 374M | 6 | 291.3 | 31.2 | 52.5 | 120.2 | 24.2  | 86.6 |
| *Rapper* | 4.01B / 374M | 18 | 291.3 | **31.8** | **52.9** | **124.0** | **24.5**  | **87.3** |

[1] Li, Junnan, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML 2023.

[2] Devlin et al. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019.

[3] Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.

[4] Zhang et al. Opt: Open pre-trained transformer language models. arXiv 2022.

[5] Park et al. Multimodal explanations: Justifying decisions and pointing to the evidence. CVPR 2018.

Reviewer comment (Theory):
Could you elaborate on how the RLNF stage specifically tackles the hallucination problem in NLE?

Author response:
**A4:**  We thank the reviewer for giving us the opportunity to clarify our proposed method. 

Hallucination in VQA refers to the problem when the output is not related to the visual image [9]. With a rationale generator $G$ and a reasoning module $M$ introduced in our Rapper, the proposed RLNF scheme tackles the hallucination problem by enforcing deployed rationale generator $G$ to inject the visual content (of the input image) into the produced rationales. This is achieved by performing the proposed RLNF to optimize the rationale generator $G$ with the reward calculated from the outputs (i.e., answers and explanations) of the pre-trained/freezed reasoning module $M$. The reason why we use the reasoning module output to guide the learing of $G$ is that, generating faithful explanations and accurate answers requires $G$ to input $M$ with rationales associated with proper *visual* clues. Note that, optimization of $G$ is a non-differentiable operation, since the guidance from $M$ cannot be back-propagated to $G$ due to the fact that the rationales from $G$ are text prompts instead of soft features. This is the reason why we propose an RL strategy to perform the above end-to-end training scheme.

We also note that, the core idea and learning scheme of our RLNF aligns with the hypothesis from Kadavath, et al. [10], which trains the model towards a faithful one when it exhibits greater confidence in their output answers. As confirmed in Table 2, our Rapper achieved the best performance on RefCLIPScore (i.e, metric for evaluating faithfulness). We thank the reviewer again for giving us the opportunity to clarify how our RLNF tackles hallucination in NLE.

[9] Ji, et al., Survey of hallucination in natural language generation. ACM 2023.

[10] Kadavath, et al. Language models (mostly) know what they know. arXiv 2022.

Reviewer comment (Evaluation):
The role of using rationale to improve implausibility and hallucination is unclear. It is well-known that chain-of-thought can improve reasoning. However, it is unclear to me if adding one more step, i.e., rationale could really mitigate hallucination.

Author response:
**A1:** We thank the reviewer for giving us the opportunity to clarify the raised issues. 

We first explain **how our work is fundamentally different from CoT promoting [1].** In NLP, the use of CoT has been a common technique, which utilizes a pre-trained LLM with manually-designed prompts with in-context learning (ICL) techniques, so that additional reasoning descriptions can be produced by that LLM. However, since the above technique is designed for language data, extending CoT to vision and language is not a trivial task. More specifically, it is not clear how to determine prompting conseutive examples which allow pre-trained vision-language models like LLAVA [2] to fully extract the information from both visual and text modality data for vision-language explanation tasks [3]. Another concern is that, since CoT (for NLP) requires manually selected prompts and does not provide additional facts related to the output, the output would suffer from hallucination [4]. For our Rapper, the deployed rationale generator is designed to exploit language-based facts and visual content from cross-modality data inputs (i.e., with our learning schemes via knowledge distillation (KD) and Reinforcement Learning from NLE Feedback (RLNF), as explained in the following paragraph). With the learned rationales as the text prompts, plausible and faithful natural language explanations (NLEs) for VQA can be performed. Based on the above explanations, we hope the reviewer could see **why CoT cannot be directly applied for reasoning in vision-language tasks.**

We now explain **how rationale learning helps mitigate implausibility and hallucination problems**. As noted in our paper, *implausibility* refers to the problem when NLEs are irrelevant to the questions or contradictory to the established supporting facts [5]. As discussed in Sect. 3.2 (Plausible NLE Generation) and depicted in Fig. 2(A), we tackle this problem by learning rationales enriched with language-based facts from a pre-trained LLM. That is, we deploy a rationale generator $G$ in Rapper, which is trained via distillating the knowledge from LLM. With such rationales with language-based facts serving as additional text prompts, together with the image-question pair as the inputs, the subsequent reasoning module $M$ can be trained to provide answers with plausible explanations. 

On the other hand, *hallucination* refers to the problem when the output explanation is not related to the visual image [6]. As noted in Sect. 3.3 (Faithful NLE Generation) and illustrated in Fig. 2(B), this is tackled by the proposed Reinforcement Learning from NLE Feedback (RLNF) technique, which jointly trains $G$ and $M$ for enforcing the learned rationale to fully exploit the visual content. The motivation of our RLNF aligns with the hypothesis of Kadavath, et al. [7], which views a learning model towards a faithful one when it exhibits greater confidence in their output answers. In other words, with the proposed RL scheme and objective rewards, the outputs of our reasoning module $M$ need to be supported by the rationale (i.e., predticed by the generator $G$) containing the visual content from the input image. As confirmed in Table 2, our Rapper achieved the best performance on faithful NLE for VQA in terms of RefCLIPScore. We thank the reviewer again for giving us the opportunity to clarify how our Rapper mitigates the problems of implausibility and hallucination.


[1] Wei et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 2022.

[2] Liu et al. Visual instruction tuning. NeurIPS 2023.

[3] Li et al. Mimic-it: Multi-modal in-context instruction tuning. arXiv 2023.

[4] Dhuliawala et al. Chain-of-verification reduces hallucination in large language models., arXiv 2023.

[5] Majumder et al., Knowledge-grounded self-rationalization via extractive and natural language explanations. ICML 2022.

[6] Ji et al. Survey of hallucination in natural language generation. ACM 2023.

[7] Kadavath et al. Language models (mostly) know what they know., arXiv 2022.

Reviewer comment (Experiments):
While the method is sound, I’m not very convinced that we cannot just use large vision language models and perform a chain-of-thought style prompting (which was actually the inspiration of this method)? How does large vision language models (e.g. BLIP-family or LLAVA models) perform?

Author response:
**A2:** We thank the reviewer for the positive feedback. We are glad to perform additional experiments as suggested. As noted in our response to Q1, we would like to point out that existing VL models like LLAVA [2] or BLIP-family [8, 9] do *not* support in-context learning (ICL). Nevertheless, we are happy to take the suggestion from the reviewer to assess the use of VL models with properly designed ICL. We now consider the VQA-X dataset and compare Rapper with Otter [10] (i.e., a VL model built on LLaMA7B). For Otter, we follow the ICL setting in PICa [11] to random sample 16 samples as the demonstration prompts for ICL. 

* Filtered results on the VQA-X dataset

|Method|Bleu4|Meteor|Rough-L|CIDEr|SPICE|RefCLIPScore|Acc.|
|-|-|-|-|-|-|-|-|
|Otter w/ ICL|4.9|14.9|26.7|29.0|13.0|63.68|39.6|
|*Rapper* (ours)|**31.8**|**24.3**|**52.9**|**124.0**|**24.5**|**67.05**|**87.3**|

* Unfiltered results on the VQA-X dataset

| Method|Bleu4|Meteor|Rough-L|CIDEr|SPICE|
|-|-|-|-|-|-|
| Otter w/ ICL|4.5|12.5|24.9|22.0|9.7|
| *Rapper* (ours)|**30.0**|**23.3**|**51.3**|**116.0**|**23.2**|

From the above tables, we see that Otter with ICL did not achieve comparable explanation performances as ours for both filtered and unfiltered cases (i.e., explanations associated with correct answers or not). This suggests that simple combination of VL models with ICL would not sufficiently address the VQA with natural language explanation, and a properly design VL model like ours would be desirable. 

[8] Li et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML 2023.

[9] Dai et al. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. NeurIPS 2023.

[10] Li et al. Otter: A multi-modal model with in-context instruction tuning. arXiv 2023.

[11] Yang et al. An empirical study of gpt-3 for few-shot knowledge-based vqa. AAAI 2022

Reviewer comment (Evaluation):
In ablation study table, the impact of proposed method is small. Especially, RLNF effect is small.

Author response:
**A3:** We thank the reviewer for pointing this out, and we are glad to clarify the performance improvements. In the upper part of Table 3, we perform ablation studies to verify the effectiveness of KD and RLNF. From this table, we see that the introduction of KD produced improved performance over the baseline model (e.g.,  CIDEr: 112.1 -> 120.2 and SPICE: 22.3 -> 24.2). This confirms the use of KD for predicting plausible outputs. Since our RLNF is deployed to mitigate the hallucination problem **based on plausible prediction**, adding RLNF to KD only increased the performance with smaller margins (e.g., CIDER: 120.2 -> 124.0 and SPICE: 24.2 -> 24.5). However, such improvements are expected. Recall that, as we explained in Q1, our Rapper is based on the introduction of KD and RLNF for providing plausible and faithful language explanation, and we cannot simply utilize RLNF without enforcing the NLE to be plausible. 

Nevertheless, even with simply KD introduced and without RLNF, our model already performed favorably against SOTAs like S3C [12] and NLXGPT [13] (e.g., list CIDER/SPICE numbers, as shown in Table 1). And the full version of Rapper (i.e., with KD and RLNF) achieved the best results, which confirms our proposed learning schemes.

[12] Suo et al. S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning. CVPR 2023.

[13] Sammani et al. Nlx-gpt: A model for natural language explanations in vision and vision-language tasks. CVPR 2022.

Reviewer comment (Writing):
A clear definition of “rationale” is not presented in the paper. Only mentioned that it is like an “intermediate” just like in chain-of-thought prompting.

Author response:
**A4:** We thank the reviewer for the suggestion. Although rationale-based prompting techniques have been presented in [14, 15], a proper definition should be given in our paper for clarity purposes. More precisely, in our work, rationales are the text prompts that generated by the rationale generator $G$, which are injected with language-based facts and visual contents. As illustrated in Fig. 1, the reasoning module $M$ takes the learned rationales as the text prompts, together with the image and question inputs, realizing plausible and faithful NLE. We will add the above definition to the fourth paragraph of the introduction.

[14] Zhang et al. Multimodal chain-of-thought reasoning in language models. arXiv 2023.

[15] Krishna et al. Post hoc explanations of language models can improve language models. arXiv 2023.