Paper: Simplifying Referred Visual Search with Conditional Contrastive Learning (ICLR-2024)

Reviewer comment (Novelty):
This task is similar to composed image retrieval. Composed image retrieval aims to find the target image based on the reference image and text description. I have some doubts about the contribution and meaning of the task.

Author response:
Our RVS task is different from Composed Image Retrieval in that in RVS the object in the query image and in the target image are exactly the same. The use case is retrieving that object from a catalog, given a picture that contains many other different objects. The goal of the model is thus to be able to extract the correct information from the query image. In contrast, in CIR, the target object is not the same as any of the objects present in the query image. The goal of a CIR model is to modify the query image such that it hallucinates an object that is similar to what is in the catalog. Both tasks have relevant applications, but they are sufficiently different to be addressed with different methods. Indeed, we show that models made for CIR perform poorly in RVS, which is intuitive: they are not designed to extract the relevant information from the image, but to modify it to hallucinate an object that matches the textual description.

Reviewer comment (Novelty):
The structure of the model is simple. It lacks innovation. The description of the model is not specific enough. Contrastive learning is often used in the task of composed image retrieval, so it is not an innovative method.

Author response:
It works, and the goal of the paper was explicitly to show that complex pipelines are not needed to reach good performances. Simplification of complex tasks so they become accessible to more people and robust has often more impact than a complex system prone to brittleness

Reviewer comment (Experiments):
Experiments are basically a comparison with other models, but the ablation experiment of your own model and visualization is lacking.

Author response:
Due to space limitations, we rejected 2 ablations in the Appendix related to our method: how to condition (symmetric/asymmetric) and where to condition (insertion depth), with plenty of retrieval results, attention visualization and PCA, as well as failure cases. Squeezing them into the main paper would force us to cut related work or experimental descriptions.

Reviewer comment (Writing):
You say it extracts referred embedding using weakly-supervised training. Why it is a weakly-supervised training?

Author response:
We call it weakly-supervised because we don’t have access to localized information (such as bounding boxes) about the object of interest. In the computer vision literature, this is referred to as weak supervision, in contrast to having access to bounding box or segmentation maps which are usually referred to as supervised methods.

Reviewer comment (Reproducibility):
In fig2, it shares weight between the two vision transformers. In this model, it is whether all parameters are shared or only parts of the parameters are shared. And I want to know why to share the weight.

Author response:
Our method uses a single model to process both unconditional and conditional images, so the weights are the same. It is less memory intensive to have the same model and we did not find any negative impact.

Reviewer comment (Experiments):
In table1 and table2, you say you report bootstrapped means and standards deviations for 0K distractors, but I don’t see the result of the 0K distractors. In addition, your model is similar to some models that used in the task of composed image retrieval. FashionIQ is also a dataset about clothes. Do you try this dataset using your method?

Author response:
Thanks for pointing out the mistake in the table caption, we removed the results with 0 distractors to make the table more legible. The results were not really interesting as performances in this very simple setting tend to saturate. 

We are currently running experiments on FashionIQ and will report the results here. However please note that this is a different task and we do not expect state-of-the-art results.  

As announced, we ran additional experiments on FashionIQ and posted the results as an [Official Comment](https://openreview.net/forum?id=Z8uFGTNXIF&noteId=47CMQgCUha).

Reviewer comment (Experiments):
It may be not appropriate to use the entire image (even given the conditions) to search for an part area, such as pants.

Author response:
While surprising, the point of this paper is to demonstrate that it is currently the best performing approach (see results in Table 2).

Reviewer comment (Experiments):
Why not crop the part area according to the given condition and then use it to search?

Author response:
Very good point. Maybe it wasn’t stressed enough in the paper but it is exactly our baseline with GroundingDINO + ViT (Section 5.2, §Detection-based Baseline, and Table 1).

Reviewer comment (Experiments):
How to collect the LAION-RVS-Fashion dataset? How to ensure the accuracy of the labels? The original labels are not clear.

Author response:
We will add a script for direct downloading to a public repository (using img2dataset on the provided parquet files).

We are very confident on the labels since the tuples are made from a limited number of shop websites using rule-based parsing.

Reviewer comment (Novelty):
Although this paper claims its target task is new, I still consider it to belong to multi-modal fashion retrieval. That is, one can include classes, attributes, captions, or even negative prompts in the textual conditions, and then leverage LLMs to process them uniformly.

Author response:
We agree that our RVS task belongs to multi-modal fashion retrieval. However, it has not been tackled in the literature, contrary to text-to-image or image-to-text retrieval and composed image retrieval for which known datasets and benchmarks exist.

For text, we do use a T5-XL to produce our embeddings and enable free-form conditioning. We find experimentally that using a single word (~category) as input works well (Fig. 9 in the appendix) even though BLIP2 captions are longer.

Reviewer comment (Experiments):
The failure cases suggest image features are dominant, and hence the proposed method or the task might not be as convenient as it claims. For example, what if the user wants to find clothes with similar styles but different colors, or of the same brand? Moreover, the text conditions seem rather simple, so whether the proposed method can handle fine-grained queries is unclear.

Author response:
Composed Image Retrieval is a different task: in our task we need to retrieve the exact garment depicted on the image. We think that paying more attention to the image is a good property, text is only here to designate an object in the scene. Unifying RVS & CIR is indeed an interesting direction, as stated in our conclusion.

Regarding fine-grained text conditioning, we are currently running experiments on a Fashion CIR dataset (FashionIQ) and will report the results here. However please note that this is a different task and we do not expect state-of-the-art results.

Reviewer comment (Experiments):
The comparison between the proposed method and SOTAs might be unfair, e.g., ASEN is implemented partially and it only uses attributes. Other baselines with similar architectures, like FashionBert should be considered as well. Besides, how is the performance of the proposed method on other fashion retrieval benchmarks?

Author response:
We fully trained ASEN using the author’s implementation and reported the best obtained results (attained during stage 1). The results obtained with stage 2 are 14.6 R@1 against2M. We believe this is due to the asymmetry of this task in which the gallery is composed of simple images containing only a single object that break key assumptions of ASEN. This result highlights the difference of our RVS task compared to other retrieval tasks for which ASEN was designed.

Since FashionBERT/UNITER are designed for cross-modal retrieval, it is unclear how they could be used for conditional retrieval. A naive approach would result in a quadratic cost with respect to the number of products which is prohibitive here (in our case, 2k queries times 2M distractors results in 4B forward passes which would take more than a year of compute at 100 forward passes per second). 

To our knowledge, there exists no other public RVS benchmark, where the task is to conditionally extract an embedding of a subpart of the image. Our method has not been designed for unconditional Image to Text retrieval or Text to Image retrieval tasks.

We are currently running experiments on a Fashion CIR dataset (FashionIQ) and will report the results here. However please note that this is a different task and we do not expect state-of-the-art results.