Paper: Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization (ICLR-2024)

Reviewer comment (Writing):
Figure 1.a and section 3.1 did not clearly explain why there is uncertainty in triplet matching, and it is recommended to give examples of situations where the description is unclear or the source image is inaccurate.

Author response:
A: Thank you. We will modify the figures and provide more samples. 
Usually, we do not know whether the triplet annotation is exactly one-to-one, and there may exist many false-positive candidates.
For instance, for coarse-grained matching, the short caption "change red and short sleeves" can match many different T-shirt candidates in the gallery; 
In contrast, we also have many fine-grained annotations in the dataset, which could exactly match one target object, such as, "change no sleeves with a black-white Micky mouse on the clothes". 
Therefore, facing such triplet annotation uncertainty, we intend to propose a method that could automatically learn multi-grained matching with flexibility.

Reviewer comment (Reproducibility):
Since this paper mainly built the network using existing works [1][2], but did not describe their implementation, it is recommended to provide a more detailed explanation of the details of the network.

Author response:
A: Thank you. We will provide a more comprehensive description of the network architecture details in Figure 2, and provide more details in implementation details. In particular, we adopt the same compositor structure in the existing work CosMo for a fair comparison, which contains two modules: the Content Modulator (CM) and the Style Modulator (SM). The CM uses a Disentangled Multi-modal Non-local block (DMNL) to perform local updates to the reference image feature, while the SM employs a multimodal attention block to capture the style information conveyed by the modifier text. The outputs of the CM and SM are then fused using a simple concatenation operation, and the resulting feature is fed into a fully connected layer to produce the final image-text representation.

We will add the above-mentioned details to the final version.

Reviewer comment (Writing):
There are non-standard punctuation marks, such as P8, the third line of "Parameter sensitivity of the balance weight", etc. It is recommended to check the whole paper.

Author response:
A: Thank you. We will refine expressions for clarity and formality, and conduct a comprehensive review to ensure proper punctuation marks throughout the entire paper.

Reviewer comment (Novelty):
The technical novelty and contribution are limited. It is trivial that adding noise to features can enhance the robustness. The other components in this method are trivial.

Author response:
**Q1. Technical novelty and adding noise is too simple.**

A: Thank you.
 
(1) Simple is our advantage. It is worth noting that the performance improvement we achieved, especially in Recall@50, is nontrivial. On the three public datasets, i.e., FashionIQ, Fashion200k, and Shoes, the proposed method has achieved +4.03\%, + 3.38\%, and + 2.40\% Recall@50 accuracy over a strong baseline, respectively. In this paper, we propose a new unified method to learn both fine- and coarse-grained matching during training. Additionally, our method is complementary, and when combined with SOTA, it can further enhance overall performance.

(2) Adding noise is not an arbitrary idea. Actually, the noise assumption is designed in Eq.1 according to uncertainty. We derive the final uniform loss in Eq.5, and show the conventional fine-grained loss is a special case of Eq.5.

Reviewer comment (Evaluation):
The qualitative results are insufficient. The authors should provide more qualitative results, and provide in-depth analyses on the cases where the proposed method outperforms or underperforms the baselines.

Author response:
**Q2. Insufficient qualitative result?**

A: We appreciate you feedback regarding the qualitative results and the need for a more comprehensive analysis of the cases where our proposed method outperforms or underperforms the baselines. We have provided qualitative results in Figure 3. Can the reviewer specify what additional results of our experiment are needed? We are willing to provide them in the final version.

Reviewer comment (Evaluation):
Provide more qualitative results and analyses.

Author response:
**Q2. Insufficient qualitative result?**

A: We appreciate you feedback regarding the qualitative results and the need for a more comprehensive analysis of the cases where our proposed method outperforms or underperforms the baselines. We have provided qualitative results in Figure 3. Can the reviewer specify what additional results of our experiment are needed? We are willing to provide them in the final version.

Reviewer comment (Novelty):
The novelty is not well explained, please explain the difference from existing methods. The authors have to provide convincing proof to show why their proposed learning strategy is significantly different from the former methods. In Table 4(b), the author has provided a very simple discussion and experiment. However, it would be beneficial to include additional comparison methods and provide a more detailed discussion.

Author response:
**Q1-1. Explain novelty. The difference from existing methods with convincing proof**

A: Thank you. Existing methods merely focus on the fine-grained search, by harnessing positive and negative pairs during training. This pair-based paradigm only considers the one-to-one distance between a pair of specific points, which is not aligned with the one-to-many coarse-grained retrieval process and compromises the recall rate. 
In an attempt to fill this gap, we introduce an uncertainty learning approach to simultaneously modeling the coarse- and fine-grained retrieval by considering the multi-grained uncertainty.  
It is worth noting that the proposed method is focused on better learning objectives, which is complementary to existing works on compositor design. 

**Q1-2 / Q3. The baseline selection? More additional comparison methods and a more detailed discussion in Table 4(b). More SOTA methods.**

A: Thank you. Considering the availability of open-source code, we select a widely-used method, CosMo, as baseline. We also show the scalability of our method to a recent baseline, Clip4Cir in Table 4(b). 
In Table 1, we have compared ARTEMIS and DCNet, both within the last three years.
We will further add FashionViL and Pic2Word in comparison as follows.  
 
| Average on FashionIQ | R@10      | R@50      |
| -------------- | ----- | ----- |
| ARTEMIS [1] | 26.05  | 50.29 |
| DCNet [2] | 27.78 | 53.89  |
| Pic2Word [4] | 24.70   | 43.70   |
| FashionViL [3]    | 26.60 | 52.74 |
| FashionViL* [3]  | 31.21 | 57.04  |
| Ours   | **33.17**  | **61.39** |
(* Note that FashionViL introduces extra datasets for training. )

[1] Delmas et al. Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. In ICLR, 2022.

[2] Kim et al. Dual Compositional Learning in Interactive Image Retrieval. In AAAI, 2021.

[3] Han et al. FashionViL: Fashion-Focused Vision-and-Language Representation Learning. In ECCV, 2022

[4] Saito et al. Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval. In CVPR, 2023

Besides, as suggested, we adopt a strong cross-modality baseline BLIP [5], and add our method. We observe that the proposed method could improve the performance as follows. 

| Shoes | R@10      | R@50      |
| -------------------- | --------- | --------- |
| BLIP  | 43.93 | 70.21 |
| BLIP + Ours   | 48.40 | 75.29 |

[5] Li et al. BLIP: Bootstrapping Language-Image Pre-training. In ICML 2022

Reviewer comment (Experiments):
The comparison baseline of this paper is not sufficiently novel, consider exploring alternative methods that may be more unique. Recent three years of state-of-the-art methods should be included in the comparison experiment section to validate the effectiveness of the proposed method.

Author response:
**Q1-2 / Q3. The baseline selection? More additional comparison methods and a more detailed discussion in Table 4(b). More SOTA methods.**

A: Thank you. Considering the availability of open-source code, we select a widely-used method, CosMo, as baseline. We also show the scalability of our method to a recent baseline, Clip4Cir in Table 4(b). 
In Table 1, we have compared ARTEMIS and DCNet, both within the last three years.
We will further add FashionViL and Pic2Word in comparison as follows.  
 
| Average on FashionIQ | R@10      | R@50      |
| -------------- | ----- | ----- |
| ARTEMIS [1] | 26.05  | 50.29 |
| DCNet [2] | 27.78 | 53.89  |
| Pic2Word [4] | 24.70   | 43.70   |
| FashionViL [3]    | 26.60 | 52.74 |
| FashionViL* [3]  | 31.21 | 57.04  |
| Ours   | **33.17**  | **61.39** |
(* Note that FashionViL introduces extra datasets for training. )

[1] Delmas et al. Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. In ICLR, 2022.

[2] Kim et al. Dual Compositional Learning in Interactive Image Retrieval. In AAAI, 2021.

[3] Han et al. FashionViL: Fashion-Focused Vision-and-Language Representation Learning. In ECCV, 2022

[4] Saito et al. Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval. In CVPR, 2023

Besides, as suggested, we adopt a strong cross-modality baseline BLIP [5], and add our method. We observe that the proposed method could improve the performance as follows. 

| Shoes | R@10      | R@50      |
| -------------------- | --------- | --------- |
| BLIP  | 43.93 | 70.21 |
| BLIP + Ours   | 48.40 | 75.29 |

[5] Li et al. BLIP: Bootstrapping Language-Image Pre-training. In ICML 2022

Reviewer comment (Writing):
As mentioned, the main contribution of this paper includes three points. The description of the first contribution is not clear enough.

Author response:
**Q4. The description of the first contribution.**

A: Thank you. In the first contribution, we hope to point out a long-exist misalignment problem, which compromises the training. The problem is due to the multi-grained annotation of the datasets. For instance, "change red" is a coarse-grained feedback, while "change red and short sleeves with a blue white Micky mouse on the clothes" is a fine-grained feedback. It is challenging to learn the multi-grained by the conventional one-to-one metric learning losses, like infoNCE loss, sphere loss and etc. We will modify the first point of motivation as follows: 

*We pinpoint a training/test misalignment in real-world image retrieval with text feedback, specifically between fine-grained metric learning and the practical need for coarse-grained inference. Traditional metric learning mostly focuses on one-to-one alignment, adversely impacting one-to-many coarse-grained learning. This problem identification underscores the gap we address in our approach.*