Paper: Retrieval-based Disentangled Representation Learning with Natural Language Supervision (ICLR-2024)

Reviewer comment (Experiments):
The retrieval baselines are split into primary baselines and advanced baselines. The authors position themselves in the primary group, arguing that their method does not need advanced techniques as in the advanced baseline group. I am not sure how 'advanced' is defined here, and thus whether this is a fair distinction. This is important, as VDR outperforms the primary baselines, but does not necessarily outperform the advanced baselines.

Author response:
**W1(1): How 'advanced' baseline is defined and whether this is a a fair distinction?**

The "advanced baselines" refer to baselines that employ sophisticated techniques known for significantly improving the performance number but orthogonal to the retrieval design. These techniques often introduce substantial computational cost and tuning effort. 

While we have detailed these techniques in cross-modal scenario in Section 4.4, below we offer additional details about these techniques in text-to-text scenario:
- Retrieval-oriented pre-training utilizes large-scale pre-training tasks`[1]` specifically designed to boost retriever. 
- Specialized negative sampling is well known as crucial in contrastive learning and training retriever.`[2]` 
- Knowledge distillation`[3]` is a process where knowledge from a larger model (usually cross-encoder) is transferred to a simpler one (bi-encoder). While this improves performance, it also increases complexity and computational demands.
- Access to Wikipedia data during training potentially benefit the performance on datasets constructed from Wikipedia, such as the NQ dataset. `[4]`

[1] Chang, Wei-Cheng, et al. "Pre-training tasks for embedding-based large-scale retrieval." 

[2] Lin, Sheng-Chieh, Jheng-Hong Yang, and Jimmy Lin. "In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval." 

[3] Gou, Jianping, et al. "Knowledge distillation: A survey." 

[4] Ren, Ruiyang, et al. "A thorough examination on zero-shot dense retrieval."

Reviewer comment (Experiments):
One of the most important aspects of VDR is the disentangled representation. This is evaluated with a small case study of only 6 images (and some more in the appendix) and a small scale human evaluation. Except for the small size of the data samples, it is also not clear how the images were selected. Are these cherry picked, or random? There is also little information about the setup of the human evaluation, making it difficult to judge these results. Given that this is such an important part of the paper, I would have expected a larger scale evaluation.

Author response:
**W1(2): Issues related to evaluations.**

- The examples for case study and human evaluation were randomly selected.
- We provide more details about the setup and analysis of human evaluation in Appendix H.
- In addition to case studies and human evaluations, our framework's explainability can be also demonstrated through a retrieval benchmark. This is significant for the following reasons. Our framework utilizes disentangled representations, which are notably sparse (< 2% activation). This sparse nature ensures that the similarity between query-target pairs ($q_i$ and $p_i$) is partially dependent on the number of dimensions they both activate. Achieving high performance in retrieval tasks require the representations of each pair ($q_i$, $p_i$) assgin large values to the dimensions that reflect their data characteristics in common,  while assigning lower values to dimensions representing other potential targets $p_j$ in the large corpus. Such a mechanism indirectly validates the effectiveness and reliability of our disentangled representations.

Reviewer comment (Writing):
Page 2: "In practice, individuals employ natural language to interpret and distinguish objects." There is no source to back up this claim, and it is also not entirely clear what the authors mean by this, for example, babies can distinguish objects, before they can speak?

Author response:
**W2: A couple of unsubstantiated claims and citation error for AdamW.**

Thank you for your valuable feedback. We apologize for any confusion caused by our previous statement. We have revised the statement to better reflect the intended meaning:
1. "In practice, individuals commonly use natural language to convey the  distinctive features of objects, thereby aiding in the differentiation of the discussed object from others."
2. We have revised the definition of IR to more closely align with the broader perspective in Manning, 2009.
3. We have corrected the citation of AdamW in the revised version.

Reviewer comment (Writing):
The definition of Information Retrieval on page 3 is very narrow. It does not follow the broader definition given in Manning, 2009, which the authors cite. The other source, Zhao et al., 2022, is a survey paper on dense text retrieval methods, which is also only a tiny fraction of the entire field of Information Retrieval.

Author response:
**W2: A couple of unsubstantiated claims and citation error for AdamW.**

Thank you for your valuable feedback. We apologize for any confusion caused by our previous statement. We have revised the statement to better reflect the intended meaning:
1. "In practice, individuals commonly use natural language to convey the  distinctive features of objects, thereby aiding in the differentiation of the discussed object from others."
2. We have revised the definition of IR to more closely align with the broader perspective in Manning, 2009.
3. We have corrected the citation of AdamW in the revised version.

Reviewer comment (Writing):
Minor detail: the authors cite Kingma & Ba (Adam) for AdamW, instead of Loshchilov & Hutter.

Author response:
**W2: A couple of unsubstantiated claims and citation error for AdamW.**

Thank you for your valuable feedback. We apologize for any confusion caused by our previous statement. We have revised the statement to better reflect the intended meaning:
1. "In practice, individuals commonly use natural language to convey the  distinctive features of objects, thereby aiding in the differentiation of the discussed object from others."
2. We have revised the definition of IR to more closely align with the broader perspective in Manning, 2009.
3. We have corrected the citation of AdamW in the revised version.

Reviewer comment (Theory):
The current work focusses on disentanglement, and I am wondering what the authors consider to be the difference between disentanglement and grounding in this case?

Author response:
**Q1: What the authors consider to be the difference between disentanglement and grounding in this case?**

Thank you for the insightful question. 
- Disentangled representations, originating from Bengio's early work`[5,6]`, is defined as a property of "good" representations for machine learning, which focuses on the representations and the representation learning approach.
- Grouding`[7]` generally refer to linking abstract knowledge (often in textual form) to tangible real-world examples, enabling AI systems to comprehend and interact with their environments. Grounding emphasizes practical applications and task-oriented solutions.
- Disentangled representations benefit grounding tasks and a variety of other applications by improving interpretability, controllability, debuggability. For instance, both CLIP and VDR systems can identify relevance between images and text. While VDR employs disentangled representations, it offers a more nuanced perspective by pinpointing specific elements (tokens) that facilitate the matching process. 

[5] Bengio, Yoshua. "Learning deep architectures for AI." 

[6] Bengio, Yoshua, Aaron Courville, and Pascal Vincent. "Representation learning: A review and new perspectives." 

[7] Chandu, Khyathi Raghavi, Yonatan Bisk, and Alan W. Black. "Grounding'Grounding'in NLP."

Reviewer comment (Evaluation):
The authors claim that VDR outperforms the baselines with a significant margin, can the authors give the results of a significance test that backs up this claim?

Author response:
**Q2: Can the authors give the results of a significance test that backs up this claim?**

Thank you for your valuable feedback and for kindly sharing the examples to us. In response to your concerns, we have conducted comprehensive significance testing to reinforce the robustness of our work. We have included detailed analyses in Appendix K of our paper. 

Briefly, we conducted two-sided t-tests comparing VDR against baselines with identical parameter counts and training pipelines. Below tables are the results of significance tests where $\triangle$ means the improvement is significant with p<0.01. Our results demonstrated significant improvements (p<0.01) in the majority of text-to-text retrieval datasets and across all recall measures for cross-modal retrieval. 

|               | DPR   | VDR$_{t2t}$      |
|---------------|-------|------------------|
| ArguAna       | 40.8  | 48.8$\triangle$  |
| Climate-FEVER | 16.2  | 18.1$\triangle$  |
| DBPedia       | 30.4  | 37.6$\triangle$  |
| FEVER         | 63.8  | 74.8$\triangle$  |
| FiQA          | 23.7  | 29.3$\triangle$  |
| HotpotQA      | 45.2  | 68.4             |
| NFCorpus      | 26.1  | 32.7$\triangle$  |
| NQ            | 43.2  | 45.8$\triangle$  |
| SCIDOCs       | 10.9  | 15.4$\triangle$  |
| SciFact       | 47.4  | 67.6$\triangle$  |
| TREC-COVID    | 60.1  | 69.0             |
| Touché-2020   | 22.1  | 27.7             |


| Model       | MSCOCO i2t R@1 | R@5 | R@10 | t2i R@1 | R@5 | R@10 | F30k i2t R@1 | R@5 | R@10 | t2i R@1 | R@5 | R@10 |
|-------------|-|-|-|-|-|-|-|-|-|-|-|-|
| CLIP-BERT   | 23.9 | 47.8 | 60.3 | 13.6 | 33.8 | 45.1 | 44.1 | 71.2 | 80.7 | 27.8 | 54.7 | 65.9 |
| VDR$_\mathrm{cm}$ | 30.9$\triangle$ | 54.5$\triangle$ | 65.4$\triangle$ | 17.4$\triangle$ | 38.1$\triangle$ | 49.7$\triangle$ | 51.0$\triangle$ | 79.3$\triangle$ | 86.7$\triangle$ | 32.4$\triangle$ | 60.1$\triangle$ | 70.7$\triangle$ |

Reviewer comment (Theory):
Since the latent representations presented are tied to an underlying vocabulary, that inherently limits the kind of representations that the model can potentially learn. Because most of the modern day tokenizers are learned based on statistical properties, especially for models with larger vocabulary sizes (where each unit itself may not represent a meaningful entity), the representations learned may not be interpretable. The authors do showcase this limitation in one of the failure cases in the Appendix (w.r.t the tokenization of the word Giraffe).

Author response:
**W1: Unit itself may not represent a meaningful entity.**

While our analysis show  most tokens in BERT vocabulary are interpretable, we acknowledge that this may not be universally applicable to all tokenizers. In cases where most tokens are not individually meaningful, we suggest pre-defining a set of textual concepts and measuring the data-concept relevance at a coarse-level, as conventional retrieval does.

Reviewer comment (Evaluation):
The human evaluation setup for BLIP captioning is artificially detrimental to the image-captioning model. Specifically, the evaluators extracted 5 tokens from the caption to best reflect the image. However, the criterion of using top-k discrete tokens is a limitation of the proposed method and thus should not be imposed on the image captioning model. At the very least, using the complete caption should also be used (potentially as an upper bound for the task of which set of tokens better describe the image).

Author response:
**W2: Human evaluation with complete BLIP caption.**

Thank you for your valuable suggestion. We conducted an assessment on complete BLIP captions on the same sample set, and have detailed it in Appendix H. The evaluation revealed a 76% satisfaction rate, compared to 85% for the top-5 tokens manually selected from these captions. This discrepancy likely arises because BLIP captions, typically 2-10 words long, are succinct, making the top-5 tokens effective for summarizing core ideas. Furthermore, complete captions, while thorough, can risk in introducing specific but inaccurate details (like location and quantity), leading to a reduction in overall satisfaction.

Reviewer comment (Experiments):
How stable is the proposed approach to increasing the vocabulary size? Concretely, would the performance be impacted if the underlying text encoder was XLM-Roberta (with a vocabulary of 250k)?

Author response:
**Q1: How stable is the proposed approach to increasing the vocabulary size?**

We experiments on mBERT (bert-base-multilingual-uncased) encoder with vocabulary size of 110k, as detailed in Appendix E. We observed a minor performance drop compared to BERT encoder (42.6 vs. 44.5). This slight decline is likely due to the gap between multilingual encoder and English-focused tasks but not the increased vocabulary size itself, as VDR consistently outperforms DPR by about 23% in both type of encoders. This finding align with our analysis in Section 3.3. The "silver bullet" here is that pretrained masked language models can provide a well-established gating distribution for subsequent dimension-wise supervision, which remains effective irrespective of vocabulary size changes.

| |DPR(mBERT)|VDR(mBERT)|DPR(BERT)|VDR(BERT)|
|-|--|---|---|---|
|ArguAna|37.2|46.7|40.8|48.6|
|Climate-FEVER|14.6|14.7|16.2|17.6|
|DBPedia|31.3|36.4|30.4|39.0|
|FEVER|62.7|69.7|63.8|74.0|
|FiQA|21.6|27.5|23.7|28.8|
|HotpotQA|45.6|63.1|45.2|65.5|
|NFCorpus|25.2|33.3|26.1|33.0|
|NQ|43.2|43.8|43.2|47.2|
|SCIDOCs|10.7|14.3|10.9|15.3|
|SciFact|48.2|66.7|47.4|67.3|
|TREC-COVID|57.3|66.7|60.1|67.8|
|Touché-2020|21.9|27.7|22.1|29.8|
|Avg|34.9|42.6|35.8|44.5|

Reviewer comment (Experiments):
On a similar line of thought, what do the authors think about the feasibility of being able to decouple the vocabulary of the latent vector and the vocabulary used by the query encoder? Concretely, would it be possible to have an XLM-Roberta encoder with the latent vectors being represented by a BERT vocabulary?

Author response:
**Q2: Would it be possible to have an XLM-Roberta encoder with the latent vectors being represented by a BERT vocabulary?**

The reviewer raises an interesting point! Integrating an XLM-Roberta encoder with BERT vocabulary involves training a new linear projection in the DST head. This setup parallels the challenges encountered in cross-modal scenarios (Section 3.3). The feasibility varies, depending on whether we have a viable gating to start with:  
1. Both Q and P side have well-established gating distribution (e.g., text-to-text senario). Effective supervision for downstream task optimization are achievable. 
2. Only text-side has well-established gating distribution (e.g., cross-modal senario). Substantial training (more data, larger batches) and enhancement (contrastive mask) is necessary for valid supervision.
3. Neither side has well-established gating distribution (e.g., Appendix C). The bi-encoder struggle to conduct valid supervision and likely fail to learn useful representations.

Therefore, substituting only one BERT-based encoder with XLM-Roberta is feasible. However, this may forfeit the advantages of a pre-trained MLM model, and a substantial training cost would be required to train this projection from scratch.

Reviewer comment (Experiments):
For the text-to-text retrieval scenario, does the non-parametric inference still work without using the non-parametric entry loss?

Author response:
**Q3: Does the non-parametric inference still work without using the non-parametric entry loss?**

Our ablation study (Appendix I) reveals that while the non-parametric inference remains functional without this loss (denoted as "w/o bow" below), there is a noticeable decline in performance in NDCG@10 from 44.6 to 40.3.

||VDR|VDR(w/o bow)|VDR_a|VDR_a(w/o bow)|
|---|:-------:|:------:|:------:|:-------:|
|ArguAna|48.6|43.5|48.8|41.4|
|Climate-FEVER|17.6|18.4|18.1|15.5|
|DBPedia|39.0|38.3|37.6|34.4|
|FEVER|74.0|73.7|74.8|70.7|
|FiQA|28.8|28.1|29.3|25.3|
|HotpotQA|65.5|64.5|68.4|63.2|
|NFCorpus|33.0|33.2|32.7|32.3|
|NQ|47.2|44.9|45.8|38.8|
|SCIDOCs|15.3|14.7|15.4|14.0|
|SciFact|67.3|65.7|67.6|66.0|
|TREC-COVID|67.8|68.0|69.0|60.7|
|Touché-2020|29.8|29.4|27.7|21.4|
|Avg|44.5|43.5|44.6|40.3|

Reviewer comment (Reproducibility):
Would it be possible to elaborate more on how the analysis for Figure 4 (b) was done? Is it considering only the patch embeddings for the max pooling operation in the DST head?

Author response:
**Q4: Would it be possible to elaborate more on how the analysis for Figure 4 (b) was done?**

Your understanding is correct! We only aggregate the selected patch representations for the max pooling rather than pooling across all patches in DST head.