Paper: FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback (ICLR-2024)

Reviewer comment (Experiments):
The experiments conducted in this study are limited in scale. The choice of a relatively weak baseline and potential data distribution shifts raises concerns about the fairness of the comparisons.

Author response:
* **Baseline selection:** Our main goal with this work is to provide a benchmark for developing and evaluating new methods that addresses the problem of model alignment with human preferences for vision-language tasks. To this end, we selected 10 baseline models from 3 different classes of models (OCR-only, Figure-only, and Figure-Caption), that have been shown to perform well for the general image-to-text tasks. We believe that our current baselines and RLHF method would provide the researchers with the knowledge needed to further develop better performing models and evaluate how different  models  perform when utilizing human-preference data. To this end, we believe our baseline models are justified and we hope that  our proposed benchmark and RLHF method leads to further innovation in this area.

Reviewer comment (Experiments):
The main technical novelty seems to be `HUMAN FEEDBACK PREDICTION MODEL`. However, there is limited discussion on why such an explicit prediction model can help RLHF. A fair comparison would be to show how the proposed method performs better than an end-to-end RLHF system.

Author response:
* **Utility of Human-Feedback prediction model:**  When compared with the online RL formulation of RLHF, both our offline-RLHF formulation and the online-RLHF methods require a reward prediction model pre-trained on human-preference data. However, the key difference between the two approaches is that once we infer the reward signal for each figure-caption pair, in our proposed offline RLHF method, we don't require the reward model for policy optimization. We proposed the offline upside-down RL method for the RLHF formulation as it would allow other researchers to compare their image-captioning models easily without the need to setup and train separate reward models.

    We don't claim that our proposed RLHF formulation is better or worse than the online RLHF method. However, as shown in previous works [[1]](https://arxiv.org/abs/2206.11871) offline RL methods are more stable, scalable and easier to use than online RL methods. We believe that for researchers focusing on developing new models for image-to-text task, our RLHF framework allows a simpler framework for evaluating their models for the human-preference realignment tasks.

Reviewer comment (Evaluation):
The authors mentioned `As can be seen from the results, our model is able to achieve good results on the validation set. This highlights that our human-feedback prediction model demonstrates out-of-sample generalization and proves the statistical significance of our model.` How different is the data distribution? How would you measure the generalization ability?

Author response:
* **Clarification on out-of-sample generalization:**  For the human-feedback prediction model, we do not claim that it can generalize to out-of-distribution data. Instead, we claim that it shows out-of-sample generalization. We evaluated the out-of-sample generalizability of our human-feedback prediction model in the following experiments:

    1) We computed the statistics of the predictions of trained feedback prediction model on the train set with that of human-annotated data. We show the results below. From the table, we can see that the model predictions are similar to the human-annotated data.

    | **# Fig-Caption Pairs** | **Human Feedback** | **Median** | **Mean** | **Std** | **Min** | **Q1** | **Q3** |
    |-------------------------|-------------------|------------|----------|---------|---------|--------|--------|
    | **Actual**              | 438               |                 |              |             |              |             |              |
    |                         |                   | **Helpfulness** | 3    | 3.01 | 1.19 | 1   | 2   | 3   |
    |                         |                   | **Takeaway**    | 2    | 2.16 | 1.22 | 1   | 1   | 2   |
    |                         |                   | **Visual**      | 2    | 2.11 | 1.08 | 1   | 1   | 2   |
    |                         |                   | **OCR**         | 4    | 3.83 | 0.80 | 1   | 4   | 4   |
    | **Predicted**           | 106,834           |                 |              |             |              |             |              |
    |                         |                   | **Helpfulness** | 2.89 | 2.89 | 1.07 | -1.27 | 2.17 | 3.61 |
    |                         |                   | **Takeaway**    | 1.95 | 2.06 | 1.03 | -1.06 | 1.33 | 2.66 |
    |                         |                   | **Visual**      | 1.91 | 2.02 | 1.01 | -1.23 | 1.31 | 2.63 |
    |                         |                   | **OCR**         | 3.88 | 3.84 | 0.83 | 0.018  | 3.32 | 4.41 |

    2) We  also conducted a 5-fold cross validation experiment using our annotated samples. We constructed a train/val set of the original 438 annotated samples by randomly sampling 219 samples to be selected as train set and remainder as validation set. We report our results (mean squared error and standard deviation over 5 runs) below:

    Helpfulness: $0.082 \pm 0.12$ | Visual: $0.076 \pm 0.2$ | Takeaway: $0.087 \pm 0.17$ | OCR: $0.095 \pm 0.13$

Reviewer comment (Theory):
**Time Complexity Analysis.** This paper claims that using offline reward-conditioned behavioral cloning for model optimization is computationally efficient. It is not convinced, that you should compare the time complexity analysis between the offline RLHF and online RLHF methods.

Author response:
* **Computational Efficiency:** We stated that our offline reward-conditioned behavioral cloning method is computationally efficient to online RL methods for human-preference alignment of model predictions in the context that, for once, we have a trained reward model (Human-feedback prediction model), and we have inferred the reward signal for individual  (figure, caption) tuple to define our trajectory of (reward, state, action), we do not require the reward prediction model for optimizing the policy. Thus the model optimization step is only dependent on the image-to-text model selected as policy. On the other hand, for online-RL methods, we require the initial supervised fine-tuned model, the policy model, and the reward model together to optimize the policy. Thus, from the perspective of runtime and compute memory requirements, our proposed offline-RL method for learning from human feedback is computationally efficient. 

    Furthermore, previous work [[1]](https://arxiv.org/abs/2206.11871), [[2]](https://arxiv.org/abs/2305.18290) in the context of natural language generation have highlighted that online-RLHF method is often unstable and substantially increase the complexity of the training setup because online methods often require more computational resources per step due to real-time interactions with environment and may require multiple gradient updates per step. Offline RLHF approach can be computationally more efficient per sample because they process a fixed dataset without the need for real-time interactions with the environment. We will clarify this better in the text.

Reviewer comment (Novelty):
**The proposed method is not novel enough.** The framework of the RLHF for figure-to-text can not provide more insight for the understanding of the read-aligned figure-caption pairs. It may not be enough for the technical contribution.

Author response:
* **Novelty Concerns and Evaluation of reader-alignment after RLHF training:**  Based on our qualitative and quantitative, we do find that our proposed RLHF method does improve  the quality of the captions. To further evaluate the generated captions, we conducted a **small scale human evaluation experiment**. Specifically, we randomly select 100 figures from the Test set of our proposed benchmark and generate captions using the BLIP and BLIP-RLHF models. We present the triplet of Figure, corresponding BLIP and BLIP-RLHF generated captions (after randomizing the order of the two captions) to 10 human subjects. Each human subject is asked to rank the two captions based on which caption they think is better. We ask the subjects to specifically consider helpfulness, visual-descriptiveness, OCR alignment and takeaway while ranking individual pair of captions. To guide the subjects, we first explain each metric [helpfulness, visual-descriptiveness, OCR alignment and takeaway] and present each human subject with 100 samples from our human annotated dataset with individual figures, groundtruth caption and the corresponding metric scores (recorded in 5-point Linkert scale). From our study, we find that on average 85% of the time, BLIP-RLHF generated caption was selected as the better caption relative to BLIP generated caption. From our small-scale study, we conclude that RLHF does improve the quality of the captions when compared to fine-tunning existing vision language models for the task of figure-caption generation.

    We want to highlight that our work provides a new benchmark for developing and evaluating new methods that addresses the problem of model alignment with human preferences for vision-language tasks. We believe that the availability of our benchmark would allow more in-depth evaluation of new RLHF methods as currently, the majority of benchmarks addressing human-preference alignment problems are specific to NLP literature.  To this end, our contribution of a new human-preference alignment benchmark, evaluation evaluating models and proposing a simple but effective RLHF method should be positively considered.

Reviewer comment (Novelty):
The applicability of the method is narrow since it is targeting for figure captioning task only. It will be more inspiring if it can be extended to general image/video captioning.

Author response:
* **Narrow applicability:** We agree with the reviewer that having a benchmark based on general image/video-to-text tasks is important. We specifically focused on figure-captioning task as one of our goals with the proposed benchmark is to allow researchers to evaluate different RLHF frameworks. To this end, we would like to obtain high-quality human-feedback data that is not ambiguous. For image captioning task with natural images, it is difficult to avoid ambiguity when collecting human preference data. This ambiguity emerges because it is difficult to evaluate different captions for a given natural image objectively. There could be multiple valid captions that correctly explains an image. Hence the human preference data can have high variance, making it difficult for downstream applications. On the other hand, scientific figures are more structured, and we can utilize domain experts to collect high-quality human preference data. This allows us to create a benchmark that can be used to accurately compare different models and RLHF frameworks.

    Nonetheless, our current framework can be easily extended to general image-captioning tasks, as we use general-purpose image-to-text models, and our proposed RLHF method can be applied to the new domain too. However, we consider this extension as out of scope because of the above-mentioned reason.

**General Comment:** We again thank the reviewer for acknowledging our work. We truly appreciate the concerns about the applicability of our method as we also share the same thoughts that  a similar framework targeting general image/video captioning is important. We believe that our proposed benchmark is a strong step in this direction and would allow the researchers to efficiently benchmark their methods.

Reviewer comment (Novelty):
I found it a bit confusing to the two contributions in this paper together as a "framework", as I don't feel they share synergy.

Author response:
* **Confusion regarding framework description:**  We referred to the two components (Human Feedback Prediction model and Upside-Down RLHF method) as a framework because they are jointly used to align an image-captioning model’s output with human preference. The human feedback prediction model is used to generate human feedback scores which is then used as reward signal along with the state ( input figure image) and action (output figure caption) to define the single-step trajectory of experience in a contextual bandit environment [[1]](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf). The triplet of ( reward, state, action) is then used to update policy (image-to-text model) using the Upside-Down RL formulation [[2]](https://arxiv.org/abs/1912.02875). However, to avoid confusion, we will modify the text to make it more clear how they share synergy.