Paper: Close the Gap: Lightweight Image Captioning via Retrieval Augmentation (ICLR-2024)

Reviewer comment (Novelty):
The main point of this paper is a $\textbf{lightweight}$ image captioning system. However, the proposed method is only lightweight during training in terms of training time. For inference, the model needs to encode the image, retrieve several captions, and then feed the retrieved captions into an LLM. The overall procedure may even be slower than a vanilla VL model for image captioning. For training, the proposed method did $\textbf{NOT}$ save trainable parameters compared to recent VL models like LLaVA, which only has a single fc layer to bridge the vision and language modalities. As the reviewer can tell from the paper, the "lightweight" part is only on the training speed and resources. This greatly reduced the contribution of this paper in the reviewer's opinion.

Author response:
We **explicitly define lightweight** in the second paragraph of the introduction as the **number of trainable parameters.** The overall method is **NOT slower than other vanilla VL approaches.** We mention in Section 3.2 that inference is on-par with SmallCap (the most lightweight approach thus far). Furthermore, **LLaVa consists of a pre-training and a fine-tuning stage**, where the entire LM is trained during the latter. Even when considering only the pre-training stage of LLaVa, which only trains the linear mapping, **our method is more lightweight**, since the **embedding space of CLIP is 4 times smaller than the embedding space of LLama, which results in 4 times fewer trainable parameters. **

Reviewer comment (Evaluation):
The performance of the proposed model is far below current SoTA models. Even though the proposed method saves training time and resources, the performance decrease is disproportional.

Author response:
We admit that we did not reach SOTA on any benchmark. However, our method is comparable with other lightweight captioning approaches, while being more efficient. Further, **we could slightly improve performance** by using a different prompting strategy, as highlighted in our revised version.

Reviewer comment (Experiments):
More details and analyses about the retrieval set should be discussed. For example, what retrieval set does the author use during training and testing/validation? What retrieval set does the author use for different datasets such as COCO and Flickr? Is the performance sensitive to the quality or distribution of the retrieval set?

Author response:
Generally, the retrieval set (denoted datastore in the paper) contains the captions from the training set. This is the same during training and evaluation. Only in case when the self-improvement loop is applied, we add synthetic (i.e., self-generated) captions for the training images. We do not use additional data sources. To evaluate out-of-distribution performance we **will add results on NoCaps** in the revised version.

Reviewer comment (Novelty):
There exists other retrieval augmented image captioning works that achieve reasonable performance such as REVEAL [1], RA-CM3 [2], HAAV [3], etc. Current approaches typically train the model jointly with the retrieval.

[1] https://arxiv.org/abs/2212.05221

[2] https://arxiv.org/abs/2211.12561

[3] https://arxiv.org/abs/2305.16295

Author response:
Thank you for making us aware of additional relevant literature, we will add a discussion on these works in our revised manuscript.

Reviewer comment (Evaluation):
Generalizability Issue: The method is heavily relied on the referenced dataset which poses a critical limitation of the method. While the method is efficient, it may not generalize well to all types of images or domains, especially those significantly different from the reference dataset. If the testing dataset is very different from the reference set, it may cause catastrophic outcomes by error propagation and require much longer rounds of iteration. This would diminish the efficiency advantage. Further, if the reference dataset has limited topic/semantic coverage, then the potential hallucination issue would also happen as the LLM was only asked to summarize the referenced captions. No matter how many iterations would be conducted, the hallucination may not be removed.

Author response:
We agree with the reviewer that our method is limited by the contents of the reference set (denoted datastore in the paper), that is the training set. However, it is **a general property of machine learning algorithms that their generalization abilities are restricted to the training distribution.** We do not claim that our method generalizes beyond the training distribution. 

A new image **does not need to** be exactly covered by captions contained in the reference set. Several captions can contain objects contained in a new image and the **FLAN-T5 can interpolate between them** and find a suitable caption for a new image. We will add some qualitative examples on that in our revised version.

**Questions:**
- If a concept of a new image does not appear in the training captions then it will also not appear in the generated caption. However, **this problem is not specific to our method, but applies to any other image captioning pipeline.**
- We believe there is a misunderstanding. Our method does **not rely on any additional data source.** It only retrieves from the training set, whereas other image captioning pipelines are trained end-to-end on the training data. As any other method **we rely on the assumption that the test distribution follows the training distribution.**
- We will **add results on NoCaps** for a linear mapping trained on the MS-COCO dataset to evaluate the out-of-distribution performance of our method. Since our method uses the training captions as reference set, our method would not produce appropriate captions for test images that substantially differ from training images.

Reviewer comment (Experiments):
Limited Experiment Issue: The performance is benchmarked on specific datasets. Since both Flickr and COCO are two very common datasets and share large similarity in terms of image distribution, there might be a lack of evidence on how the model performs on out-of-distribution data or in more diverse real-world scenarios.

Author response:
We **will add results on NoCaps** for a linear mapping trained on the MS-COCO dataset to evaluate the out-of-distribution performance of our method.

**Questions:**
- We will **add results on NoCaps** for a linear mapping trained on the MS-COCO dataset to evaluate the out-of-distribution performance of our method. Since our method uses the training captions as reference set, our method would not produce appropriate captions for test images that substantially differ from training images.

Reviewer comment (Evaluation):
LLM Issue: The method also heavily relies on LLM, however the LLM never saw the image but only can summarize. The iterative refinement with synthetic captions could potentially introduce or amplify errors if the synthetic captions are not of high quality.

Author response:
We agree that adding low-quality synthetic captions can amplify errors. This is the reason why **we threshold synthetic captions according to a certain metric**, which can be any of CIDEr-D, SPICE, B@4, ROUGE-L, etc. After additional experiments we found that **thresholding according to the CIDEr-D metric works the best** and usually leads to improvements across metrics.

Reviewer comment (Evaluation):
Computational Resources: While the model has fewer parameters, the computational cost and efficiency are not only about the number of parameters but also about the complexity of operations and the size of the input data. For every image, the method requires pre-computation of all the reference image and text data, not to mention the following newly updated synthetic captions for every iteration.

Author response:
We believe that there is a misunderstanding regarding the self-improvement loop. The self-improvement loop is only part of the training algorithm and does not run during evaluation/inference. Further, our **reference set only contains texts which are pre-computed once.** During self-improvement, **synthetic captions are embedded once** and added to the reference set.  We will make this more clear in the paper.

Reviewer comment (Evaluation):
Metric Sensitivity: The paper proposes a new metric for evaluation, but it’s not clear how sensitive the results are to the choice of metric and whether the proposed metric has been validated extensively.

Author response:
There must be a misunderstanding, since **we did not propose a new metric.**

Reviewer comment (Novelty):
Severely Limited Novelty/Details of the Caption: A good caption is not just a high-level summarization of the image but should also cover essential fine-grained details. The characteristics of this method determine it cannot include fine-grained details in the caption.

Author response:
A new image **does not need to** be exactly covered by captions contained in the reference set. Several captions can contain objects contained in a new image and the **FLAN-T5 can interpolate between them** and find a suitable caption for a new image. We will add some qualitative examples on that in our revised version.

**Questions:**
- If a concept of a new image does not appear in the training captions then it will also not appear in the generated caption. However, **this problem is not specific to our method, but applies to any other image captioning pipeline.**

Reviewer comment (Presentation):
The method presented is not one of image captioning; rather, it involves the enhancement of captions or the retrieval of text related to images. This is predicated on the assumption that there exists a preliminary set of fairly accurate captions for each image from which to draw. To better reflect this and to prevent any potential overstatement or confusion, a slight modification of the title is recommended.

Author response:
We believe there is a misunderstanding. After the **retrieval of related captions from the training set,** we use a LM (FLAN-T5) to generate a new caption for the input image. As long as the retrieved captions cover different aspects of the input image, the **LM can combine them to a new caption that fits the image.** We make this more clear in the revised version.

Reviewer comment (Theory):
The utility of the proposed method is constrained by a significant precondition: a repository of captions must be supplied to enable the retrieval process via mapping weights. Typically, we have only the image at our disposal without a pre-existing collection of initial or possible captions to draw from.

Author response:
Again, we believe that there is a misunderstanding. **Our method does not rely on any additional data source.** It only retrieves from the training set, whereas other image captioning pipelines are trained end-to-end on the training data. Thus, **training captions are generally available,** which is what we retrieve from during testing. Therefore, we **only require an input image** and no additional data. We try to clarify this in the revised version.

Reviewer comment (Experiments):
The comparison in Table 1 appears somewhat skewed. The proposed method relies on the availability of a predetermined pool of captions for retrieval, in contrast to the baseline models which do not require such a resource.

Author response:
We believe the **comparison in Table 1 is adequate** since it compares our method to others that aim at **parameter efficiency, i.e. training as few parameters as possible.** All compared methods (including ours) only use the training set for training and evaluate on the test set. Further, **one of our baselines (SmallCap) uses retrieval** and stores the training data in its datastore (same as ReCap).

Reviewer comment (Experiments):
How to compare your method to the large multi-modal models, such as llava, instructBLIP, BLIP2, and MiniGpt-4? Any comparison on the image captions generated by those large multi-modal models?

Author response:
**Caption quality compared to large-scale models:** We believe this is an unfair comparison, since **all of the mentioned methods train much more parameters on datasets that are orders of magnitude larger** than the ones we use. Further, they usually consist of **two-stage training** and use **language models of much larger size.**

Reviewer comment (Novelty):
Overall the idea lacks novelty and depth. The key idea of doing retrieval augmented captioning is taken from prior work. Although the paper differs in how the training is being done (just doing least squares instead of training some layer), the performance is mostly below the previous works on most metrics (e.g. on Bleu small cap vs recap is 36 vs 29, numbers are similar on spice metric)

Author response:
We agree that retrieval augmentation has been done in prior works. However, these works require end-to-end training which we alleviate due to the linear mapping to bridge the modality gap. Further, **our self-improvement loop is a novel approach to augment the retrieval datastore with synthetic captions.**

Reviewer comment (Experiments):
Authors have shown experiments on cross-dataset transfer, why didn't they consider other datasets such wizwiz, MSR-VTT.

Author response:
We will add experiments on additional datasets, i.e. VizWiz and MSR-VTT in the future version.

Reviewer comment (Novelty):
I agree that the finding about clip-score is interesting but it was not adding value to the key idea in the paper

Author response:
We believe that our analysis on CLIP-score does add value. The main reason we considered it is that we searched for an appropriate metric to **threshold our synthetic captions in the self-improvement loop.** We observed that **when optimizing for CLIP-score, our method started to hallucinate** which resulted in a decrease in all metrics except for CLIP-score itself.  We make this more clear in our revised version.

Reviewer comment (Novelty):
The author proposes bridging the modality gap through a cost-effective approach using a linear mapping optimized through a least-squares solution. However, there is a need for a more in-depth discussion on how this method differs from existing solutions in the realm of joint models.

Author response:
We **extend the related work section** to highlight the **differences to existing approaches to mitigate or avoid the modality gap.**

To the best of our knowledge we are the **first to efficiently bridge the modality gap for image captioning** which, as opposed to existing approaches, alleviates end-to-end training of the captioning pipeline. We elaborate the **difference to other vision-language approaches** in the related work section. Further, **our self-improvement loop is novel** and provides a way to leverage synthetic captions for retrieval augmentation. Finally, our analysis on metrics provides a new insight that CLIP-score is not reliable for evaluation of image captioning.

Reviewer comment (Novelty):
The utilization of a "linear mapping optimized via a least-squares solution" is a fundamental constraint optimization technique, which may raise questions about the novelty of the method. To enhance the novelty, the author should provide a comparative analysis with existing joint model solutions found in the Visual Question Answering (VQA) literature and other multimodal joint representation studies to highlight the distinctive aspects and advancements introduced by the proposed approach.

Author response:
We **extend the related work section** to highlight the **differences to existing approaches to mitigate or avoid the modality gap.**

To the best of our knowledge we are the **first to efficiently bridge the modality gap for image captioning** which, as opposed to existing approaches, alleviates end-to-end training of the captioning pipeline. We elaborate the **difference to other vision-language approaches** in the related work section. Further, **our self-improvement loop is novel** and provides a way to leverage synthetic captions for retrieval augmentation. Finally, our analysis on metrics provides a new insight that CLIP-score is not reliable for evaluation of image captioning.

Reviewer comment (Evaluation):
Computational Complexity: The text mentions that certain computations, such as least-squares linear model fitting, can be done on a CPU within minutes. It would be valuable to provide a brief discussion or estimate of the computational complexity involved in each step, giving readers an idea of the method's efficiency.

Author response:
We added asymptotic complexity of the retrieval (O(n)) and the computation of the least squares solution via pseudoinverse (O(d³)), where d denotes the dimensionality of the CLIP space and n the number of embeddings stored in the datastore. We want to highlight that **neither of these components are a bottleneck of our method.** The **embeddings for the datastore are pre-computed once before training.** As mentioned in Section 3.2, inference time is approximately equal to the one of SmallCap, which, to the best of our knowledge, is the most lightweight approach that currently exists.

Reviewer comment (Theory):
Iterative Self-Improvement: The iterative self-improvement process is described, but the rationale behind choosing high-scoring captions for augmentation could be clarified. Explaining why high-scoring captions are selected and how this contributes to the refinement process would add depth to the method's justification.

Author response:
We **threshold synthetic captions** in order to achieve a certain **trade-off between quality and diversity.** We ran additional experiments and found that thresholding according to the CIDEr-D metric works the best. Since CIDEr-D is based on n-grams, a high threshold would enforce captions as close as possible to reference captions, hence no additional information. Lowering the threshold results in more diverse synthetic captions that can be combinations of reference captions for different images.

Reviewer comment (Experiments):
The hyperparameters "k" and "l" are introduced but not thoroughly explained. Providing more insight into the rationale behind choosing specific values for these hyperparameters or discussing their impact on the results would strengthen the method's transparency.

Author response:
**W** is a **linear mapping from CLIP image space to the CLIP language space.** The hyperparameter **k** is the number of retrieved captions per image. We show in the appendix (Table 9 and Table 10) that it **slightly affects the downstream captioning performance.** The hyperparameter l is only used during the self-improvement cycle, to generate several **diverse captions.** In our revised version we avoid the need for this hyperparameter by using nucleus sampling.

Reviewer comment (Reproducibility):
The paper misses one of the core aspects of machine learning practice: readability and reproducibility of results. The core mechanism of the proposed explanation method is not clear. The author should provide an algorithm or pseudocode to reproduce the results, which is missing from the paper.

Author response:
We agree that reproducibility is very important, which is why **we added our code in the supplementary material.** Further, to enhance understanding of our method, we **added algorithms for ReCap as well as the self-improvement loop.**