Paper: Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR-2024)

Reviewer comment (Novelty):
The method itself is very intuitive and not quite novel, which is more like an engineering extension of existing models like VLM and LLM. Additionally, the proposed method does not achieve better results on all evaluation datasets. In Table 1, CIReVL underperforms SEARLE on CIRR dataset in terms of CIRR, which shows the unstability.

Author response:
__1. Novelty and results on CIRR:__    
Indeed, our method is an application of off-the-shelf VLMs and LLMs. However, their particular recombination and reuse to transfer it to the difficult task of compositional image retrieval is novel. This is particularly interesting, as unlike previous approaches that leverage off-the-shelf VLMs as well and have to train on large amounts of data, our approach requires no training at all, while the modular structure and operation in the language domain makes our approach interpretable and intervenable as well. 

This strongly contrasts our approach from existing ones! In addition, as our scaling laws study shows, our setup can be easily scaled up to account for modular improvements over time, without having to retrain. As such, we strongly believe that our work is an important contribution to the field of compositional image retrieval.
In addition, we outperform existing, trained methods on CIRCO and Fashion-IQ significantly, while on CIRR (with noisy supervision, lack of multiple positive annotation, etc. as shown in previous works and noted in the paper), we are able to match previously obtained results

> (...) the novelty of this paper is not significant enough"

It would be great if the reviewer could provide additional context for this statement, as in our reply, we do highlight that our method and framework is novel. Even if the single components themselves are not novel on their own, their particular recombination and reuse is. Other approaches also leverage large-scale, off-the-shelf pretrained models and existing network architectures.
Unlike these approaches however, our setup does not require large amounts of additional training data on top.

In addition to that, our method is first and foremost motivated by important practical constraint listed above, which significantly differentiates it from existing approaches, namely it being
* interpretable,
* intervenable,
* easily extendable,
* and providing better performance.

In addition to that, it also provides an important sanity check for future method that do incorporate additional training data.

> (...) the performance and inference time of the proposed method still limits its real application

Large-scale models find ubiquitous usage in real world applications, ranging from LLM services to recent large-scale multimodal models. In addition, our performance significantly outperforms existing methods on all benchmarks but one (and comparable scores in that case), while not requiring any training. Furthermore, the ability to freely interchange models is a property that especially allows for more realistic deployment based on cost-performance trade-offs.

Reviewer comment (Experiments):
This work only adopts datasets of everyday life and natural scenes (CIRR, CIRCO, GeneCIS) as evaluation. It is necessary to include datasets of various domains, such as fashion domain datasets like fashioniq and fashion200k to evaluate the generalization ability. Furthermore, for GeneCIS dataset, the comparison against other ZSCIR methods such as Pic2word and SEARLE should be included for complete comparison.

Author response:
__2. Results on Fashion benchmarks:__    

We thank the reviewer for the suggestion. We have now updated our paper with the results on the Fashion-IQ benchmark. For the detailed results, we refer to our shared comment. In summary, we find that CIReVL significantly outperforms existing methods on all three subtasks, e.g. improving average Recall@10 by nearly 6pp (28.29% versus 22.89% for SEARLE) with a ViT B/32 backbone. In addition, the simple scalability allows us to hot-swap the retrieval model to achieve additional 4pp (increasing from 28.29% to 32.19%).

Reviewer comment (Evaluation):
The proposed framework is not very time-efficient since for each query, there will be two auto-regressive processes: query image captioning and target caption generation, which are very time-consuming. Both processes involve large models like VLM and LLM, which may further lead to low efficiency for image search. Therefore, it is necessary to compare the retrieval efficiency (time) with the previous methods.

Author response:
__3. Inference time:__    
Indeed, while existing methods also utilize large VLMs, the use of distinct captioning and reasoning modules elicits additional runtime. On our hardware setup (V100 / A100 GPUs) when compared to trained methods, we find inference times to increase by ~1s. However, the modularity also allows our setup to directly benefit from efficiency changes within each module (as e.g. seen in the cost & inference time reductions for LLMs over the past year), allowing one system to remain operational for a long period of time. This stands in particular contrasts to other ZS-CIR methods that require costly training to incorporate any pipeline changes or shifts in expected datasets (e.g. Pic2Word trains the mapping network on 8 GPUs on the complete CC3M dataset).

Reviewer comment (Evaluation):
Incorrect experimental results: The Recall@10 of CIReVL (ViT-G/14∗) on CIRR dataset in Table 1 is obviously incorrect.

Author response:
__2. & 3. Typo in the result and errors in table formatting:__   
We thank the reviewer for pointing out these errors, and have rectified these in the updated draft.

Reviewer comment (Presentation):
Inconsistent formatting of table data: In some cases, the data is presented with two decimal places, while in others, only one decimal place is used. Furthermore, there is a case with a mix of formatting in Table 3. The authors may revise the paper more carefully.

Author response:
__2. & 3. Typo in the result and errors in table formatting:__   
We thank the reviewer for pointing out these errors, and have rectified these in the updated draft.

Reviewer comment (Evaluation):
The paper effectively combines the existing powerful VLM and LLM models. However, it would be better to provide a more insightful analysis of the caption and reason processes. For example, from Table 3, compared to utilizing different LLM models, the use of various state-of-the-art (SOTA) captioning models has a relatively minor impact on retrieval. Are there any potential explanations for this phenomenon? Does this mean the choice of caption model is not strict, as long as the model can catch the main object or attribute of images? I think it would be better to provide more analysis.

Author response:
__4. On the differing impact of captioning and LLM modules:__    
The reviewer rightly points out that particular choices in the image captioning module have less impact on the overall compositional image retrieval process as opposed to changes in the reasoning module (the choice of LLM). We do believe this to be primarily due to two reasons. First, the performance gaps between the tested captioning models is much less evident compared to the gap between available foundational LLMs (Llama versus GPT-4 have vastly different reasoning capabilities). This transitions to the second aspect, in that - as correctly noted - for all benchmarks, only a subset of crucial image properties have to be captured. This can be sufficiently done by all captioning models. Indeed, when looking at failure cases such as in Fig. 3, we find that a lack of captioning quality to be one of the main error sources - something which a notably better captioning model could fix in the future.

Given the modularity of our setting, something that can be simple hot-swapped, as opposed to having to retrain a full pipeline from scratch!
Finally, we believe that a more detailed investigation should involve the development of benchmarks which cannot be easily described through single, in parts simplistic captions. This is something we leave for future work to tackle.

Reviewer comment (Writing):
The description about other works in Section 3.1 is a bit obscure for me who is not so familiar with the task but proficient other related tasks.

Author response:
__5 & 6 & 7. Writing Suggestions, missing references in 3.2 and missing baseline references:__   
We thank the reviewer for their valuable feedback. We have updated our paper accordingly, by adding a more detailed breakdown of the compositional image retrieval task, as well as referencing related ZS-CIR methods in the beginning of Section 3.2 (particularly Pic2Word & SEARLE). In addition, we have included the “text-only”, “image-only” and “image+text” baseline across our experimental results. We do note that the interpretation of our results is not impacted by this additional inclusion, though agree that it belongs there for completeness. Finally, for our GeneCIS experiments, we have included information about our utilized backbone network, which follows the one utilized in the original GeneCIS paper.

Reviewer comment (Experiments):
Experiments: (a) In Table 1, the author only presents the experimental results of ‘image only’ and ‘image+text’ methods for reference while missing that of the ‘text only’ method under the ViT-B/32 setting. However, the authors mention that the results on CIRR benchmark are primarily dependent on the modifying instruction while the actual reference image has much less relation to the target image, which indicates that the ‘text-only’ method can be an important reference for measuring the performance of the proposed method. Better to present the result of ‘text-only’ method and make a fair comparison and discussion. (b) In Table 1, the authors miss the results of ‘image only’, ‘text only’ and ‘image+text’ method for ViT-L/14 and ViT-G/14 settings. Better to include them for reference. (c) When evaluating on the GeneCIS benchmark, the authors do not specify the architecture of the vision backbone adopted in the experiment. Better to specify it clearly for reference.

Author response:
__5 & 6 & 7. Writing Suggestions, missing references in 3.2 and missing baseline references:__   
We thank the reviewer for their valuable feedback. We have updated our paper accordingly, by adding a more detailed breakdown of the compositional image retrieval task, as well as referencing related ZS-CIR methods in the beginning of Section 3.2 (particularly Pic2Word & SEARLE). In addition, we have included the “text-only”, “image-only” and “image+text” baseline across our experimental results. We do note that the interpretation of our results is not impacted by this additional inclusion, though agree that it belongs there for completeness. Finally, for our GeneCIS experiments, we have included information about our utilized backbone network, which follows the one utilized in the original GeneCIS paper.

Reviewer comment (Writing):
Writing: Some parts of the paper don’t flow well. The overall writing requires further improvement.

Author response:
__9. Writing__   
We have included all suggestions made by the reviewer across previously mentioned points. If there is anything else the reviewer would like to see addressed on the writing end, we will gladly incorporate that as well.

Reviewer comment (Evaluation):
This paper seems to be a technical report with the application of current state-of-the-art foundation models. Although directly using existing models is convenient, it inevitably introduces errors in each processing step. How to address the cumulative errors resulting from these multiple models?

Author response:
* _Error propagation through module-specific errors (also Q1):_ By combining different modules, error propagation can indeed happen - an incorrect caption results in a faulty modified retrieval caption, and incorrect reasoning can turn a sufficient caption into a wrong modified version. We do not claim our approach to be void of errors, however unlike existing works, our modular pipeline makes these actually much more transparent, and allows us to separate and reason by various error types (compared to black-box errors in previous approaches). We provide several studies for this in our paper, where we first separate failure cases in Fig. 3 and 4.3. And indeed, an incorrect caption affects the final reasoning, and incorrect reasoning affects an otherwise sufficient caption. However, the interpretability and modularity clearly separates these error sources. Indeed, our  intervention examples in Fig. 4 show how errors within each domain can be easily corrected individually to correct the complete retrieval process. Finally, beyond actual interpretability of errors and the possibility to intervene on them, our overall performance on most benchmarks is much stronger than previous trained methods, resulting in an overall smaller number of total errors.

Reviewer comment (Experiments):
Could the authors provide experimental results of CIR works with a supervised training pattern? Based on the current results in this paper, the overall zero-shot retrieval performance appears to be poor. Do those supervised methods also yield unsatisfactory performance? If so, it seems that more significant efforts are needed to address the fundamental issues in CIR task; If not, does the zero-shot approach still have its value?

Author response:
__Q2. Supervised CIR__   
The main goal of our approach (as well as other ZS-CIR methods) is to have a general purpose model that performs well across multiple benchmarks (i.e without specializing to a single setting) in the absence of expensive curated data. Most supervised CIR methods instead focus on training separate models specific for each benchmark, thereby losing generality. We illustrate this in the Appendix with the Combiner architecture, where we see that our approach nearly matches the performance of specific Combiner models on all of the benchmarks. Still, even the fully supervised CIR approaches have lackluster performance, nearly matched by our general purpose approach (even without scaling respective modules). 

This is partly due to the in parts ambiguous nature of the task (see e.g. Fig. 3 for an example), and consequently the lack of benchmarks that can account and evaluate for this ambiguity (for an image and a corresponding instruction, there are multiple suitable candidates - but these are generally not fully covered in existing benchmarks). However, as the strong performance of our training-free zero-shot approach showcases, it is also because of currently insufficient understanding on how to build more effective (ZS-)CIR systems.

Reviewer comment (Experiments):
How you get the 6 curated samples for the Rs @K metric in Table 1?

Author response:
__Q3. CIRR evaluation details:__    
The CIRR evaluation protocol has 2 settings. The first is where the target image has to be retrieved from the entire gallery. The second setting is the subset evaluation, where there are 6 handcrafted hard-negatives from which the correct target image must be chosen. We will highlight this information in the final draft.

Reviewer comment (Experiments):
For reasoning, the experiments show LLM models obviously affect the performance. I wonder whether the same is true for the prompt template. Except for the prompt mentioned in Appendix A, were there any alternative prompts explored during the experimentation?

Author response:
__Q4. Alternate Prompts:__   
We test various alternative prompts, but find performance to be robust as long as the particular task remains sufficiently well described within it - this means that changes in general formulation have limited impact. Further gains can be achieved by extending the task description with general task examples, as has been shown in LLM prompting.

Reviewer comment (Theory):
The entire process relies on the underlying assumption that the image captioning module (e.g., BLIP-2 or CoCa) can provide a "detailed caption" that captures all information, which is a strong assumption. A lot depends on the accuracy and completeness of the captioning module.

Author response:
__1. Is a caption sufficiently descriptive?__   
We agree that a simple textual caption is limited in its ability to fully reflect the content of an image. This is something we also specifically highlight throughout the paper, particularly when addressing failure cases (Fig. 3) and their addressal through interventions in Fig. 4.

Generally however, when compared to current methods for Zero-shot Compositional Image Retrieval (ZS-CIR) which focus on mapping an input image to a single token in the textual embedding space, CIReVL is far less restrictive - generating a 10-20 word caption which can leverage e.g. 50+ tokens to represent the information in the image.

In addition, operating in the language space allows our approach to be inherently interpretable, and actually intervenable.
Given these benefits, our work particularly tries to emphasize how far one may push a language-centric compositional image retrieval approach, and successfully shows that compared to existing methods, current benchmarks can be effectively tackled by purely representing each image with a caption/description. 

Future work should focus on developing benchmarks that have information that cannot be easily described with a single description. In the current benchmarks, these cases are not the majority, while we do illustrate a potential example in Fig. 3.

Reviewer comment (Evaluation):
Even if the captioning module is highly accurate, it may omit some "less important" details that are needed for subsequent changes. For example, if the caption omits the color of the sky and focuses on the foreground, a query like "make the colour of the sky darker" cannot be fulfilled by the rest of the pipeline.

Author response:
> While the captioning module may be super accurate, it can miss some "less important" details that I want to change. Fig-4 seems like a patchwork to me -- if a human has to modify the generated image caption, he/she might as well write it in the first place.

This is of course true - the captioning system is not perfect, but given the modular nature, can simply we replaced in a plug-and-play fashion if needed. However, we show that even with very general open-source captioning models, that very strong CIR performance can be achieved.

Regarding the human just writing the caption themselves, this is exactly the point of the intervenability, and a byproduct of our inherent interpretability. 
If the human believes that insufficient context is extracted from an image, and wants to guarantee a more faithful retrieval, they can also write the caption themselves.
Such an intervention into the retrieval process is not possible for other trained systems, and is simply a bonus point on top of the modular and interpretable CIR framework.

Reviewer comment (Theory):
This limitation can cause the entire pipeline to fail, even without any fault of the captioning module, since it is not possible to describe every minor or seemingly unimportant detail in the image.

Author response:
__1. Is a caption sufficiently descriptive?__   
We agree that a simple textual caption is limited in its ability to fully reflect the content of an image. This is something we also specifically highlight throughout the paper, particularly when addressing failure cases (Fig. 3) and their addressal through interventions in Fig. 4.

Generally however, when compared to current methods for Zero-shot Compositional Image Retrieval (ZS-CIR) which focus on mapping an input image to a single token in the textual embedding space, CIReVL is far less restrictive - generating a 10-20 word caption which can leverage e.g. 50+ tokens to represent the information in the image.

In addition, operating in the language space allows our approach to be inherently interpretable, and actually intervenable.
Given these benefits, our work particularly tries to emphasize how far one may push a language-centric compositional image retrieval approach, and successfully shows that compared to existing methods, current benchmarks can be effectively tackled by purely representing each image with a caption/description. 

Future work should focus on developing benchmarks that have information that cannot be easily described with a single description. In the current benchmarks, these cases are not the majority, while we do illustrate a potential example in Fig. 3.

Reviewer comment (Novelty):
It is unclear how the proposed method is different from VisProg (https://arxiv.org/pdf/2211.11559.pdf), as it feels that the same benefits could be achieved by using VisProg.

Author response:
__2. Differences to Visual Programming:__  
While related, there are several significant differences between our proposed approach and Visual Programming. 
Visual Programming aims to solve a variety of vision and language tasks at the same time using an LLM to control the APIs to various modules. 

However, all the tasks tackled by Visual Programming typically relate to working with a single image at a time, and don’t account for large-scale applications in which e.g. retrieval over an expansive gallery set has to be performed. To possibly adapt the Visual Programming method to this task, one would need to precompute the image features for all the images in the gallery set, provide sufficient in-context examples and a task description to enable the LLM to reason over the description, and provide particular instructions on how to find the correct final image. All of these modifications involve significant specialization to the CIR task and move the end result far away from the Visual Programming generalist. 

In general, VisProg is a general framework which leverages different specialist modules and figures out ways to recombine them. Consequently, CIReVL should be regarded more as an expert, interpretable compositional image retrieval module that can then be leveraged by generalist agents via e.g. VisProg.

Reviewer comment (Novelty):
[Novelty & Literature review] There are a wider number of papers in the field since last year and some of those papers requires mention (at least the ones before the ICLR submission deadline). The most similar paper ((https://arxiv.org/pdf/2310.05473.pdf) proposes the same approach but with an additional training method for merging the automatically generated image caption and the desired modification's text description. This paper contains additional papers/baselines to cite/include. It would be great if there is a way to understand whether the proposed training-free method would perform better or worse compared to the paper mentioned above.

Author response:
__1. Novelty of the proposed approach:__   
We thank the reviewer for pointing out the additional reference. However, the paper is contemporary to our submission (i.e. it was published on arxiv after the ICLR deadline), thus we could not have provided a comparison with it/known about its existence. We acknowledge several related works in Section 2 of our main paper, and highlight crucial differences between these works and our approach. 

Most works on compositional image retrieval aim to train/fine-tune models specific for each dataset (i.e., training separate models for CIRR, Fashion-IQ etc.). Zero-shot Compositional Image Retrieval (ZS-CIR) instead aims to train a single model that generalizes across multiple benchmarks without the use of paired (source image, text, target image) data. The only works that explicitly focus on this task (and that we compare with) are Pic2Word [Saito et al 2023] and SEARLE [Baldrati et al. 2023], which are based on a different principle, i.e. trained textual inversion. Both methods already leverage off-the-shelf, large-scale pre-trained models as part of their pipeline.

In our work, we go one step further and propose the first training-free approach for this task, that only relies on off-the-shelf pre-trained models, while allowing for easier scalability, interpretability and the possibility of interventions.

Reviewer comment (Experiments):
[Experiments] Baseline PALAVRA attack different set of problems, uses different datasets and metrics. It does not mention CIR task in the text. Not clear if PALAVRA serves as a baseline.

Author response:
__2. PALAVRA as a baseline:__   
We agree that PALAVRA has not been proposed for the CIR task. However, since it is conceptually very similar to Pic2Word and SEARLE, it was originally introduced as a comparison in these works. To retain direct, comprehensive comparability, we also utilize PALAVRA as a corresponding baseline.

Reviewer comment (Experiments):
FashionIQ is yet another dataset which other papers use for CIR evaluations, it might be worth considering it in the future.

Author response:
__3. Results on Fashion-IQ:__   
We thank the reviewer for the suggestion. We have now updated our paper with the results on the Fashion-IQ benchmark. For the detailed results, we refer to our shared comment. In summary, we find that CIReVL significantly outperforms existing methods on all three subtasks, e.g. improving average Recall@10 by nearly 6 percentage points (pp) (28.29% versus 22.89% for SEARLE) with a ViT B/32 backbone. In addition, the simple scalability allows us to hot-swap the retrieval model to achieve additional 4pp (increasing from 28.29% to 32.19%).