Paper: Training-free Deep Concept Injection Enables Language Models for Crossmodal Tasks (ICLR-2024)

Reviewer comment (Novelty):
The paper claims to be the first to demonstrate the ability of Pre-trained Language Models (PLMs) to perform zero-shot cross-modal tasks without any training. However, there exist similar works, such as Tip-Adapter [1], which should be discussed and compared to provide context and clarify the novelty of this approach.

Author response:
- Thanks for the reference. But the provided reference doesn’t change the position of our paper. **Our work is focused on crossmodal tasks that require modality fusion, but the Tip-adapter is designed only for unimodal classification tasks, which is fundamentally different from ours**. We have cited it in the revised main paper.

Reviewer comment (Reproducibility):
The concept library seems to be extremely important in the whole DCI pipeline; however, how to construct this concept library given different input datasets or domains seems to be too simple for evaluating if this pipeline could be adapted to more general settings. The authors should consider adding more details to this concept library construction process for the reviewers to evaluate the contribution of this paper.

Author response:
- From our results in Tables 1 and 2 and our analysis in Sec 4.2 and 4.3, the dataset-specific variants DCI-A and DCI-LM achieve better results, but we would also like to highlight that: **even with the generic vision vocabulary, our training-free DCI can already achieve comparable results with SOTA that requires training, which has already validated the main message that current paradigm of training projection and adaptation layers are not necessary.**
- We indeed included the details of the vocabulary construction in Sec. 3.3 and G.1. The generic vocabulary is obtained from existing computer vision benchmarks – VisualGenome and OpenImages. The DCI-A dataset-specific vocabulary is obtained from the answer vocabulary used in the open-ended setting for each dataset.
- We indeed included detailed quantitative results and analysis in Sec. F.2 for ablation study on DCI-LM. We generally observe that DCI-LM helps to remove from the concept recognition results some too generic visual concepts that are not relevant to the question.

Reviewer comment (Theory):
The overall architecture seems extremely simple and efficient. Given the good performance, the authors should have considered providing more insights on why this simple augmentation strategy, or put it another way, weighting input and features from the concept library, could help improve vision-language tasks. Is this augmentation enough?

Author response:
- We indeed provided insightful analysis in Sec. 4.4 and 4.5. In Sec. 4.4, we specifically study how the projection layers can be replaced with different training-free variants, and we find that most of the ability of the learned projection layers and the adaptation layers can be instantly
obtained with the simple constructed projection layer, as long as we use concepts as the bridge. This observation is aligned with our first insight that concepts can be directly understood by the language model, as summarized in the Introduction.
- In Sec 4.5, we find that the augmentation in the intermediate feature space helps the model attend more to extracted concepts that are relevant to the correct answer, which is aligned with our second insight that the feed-forward process is the other ignored complementary pathway for modality fusion.
- In Sec. 3.2.2, we also show the insights on how the injection process is equivalent to a feed-forward network with constructed weights, which again shows that as long as we speak the “language” that PLMs understand, the training process can be removed.
- To further understand the potential of the proposed method, we also provide Plug-and-play results, results using ViT pretrained on ImageNet instead of CLIP, and Fine-tuning results in Appendix B, D, and F.1. We find that for models that have already been trained, our model can still provide help. When there is finetuning involved, DCI is also a better strategy to fuse visual information compared to the existing approach. More importantly, when there is no high-quality vision-language contrastive model like CLIP available, we show that with a generic classifier, our DCI method can still achieve better results compared to the existing approach.
- Our method certainly has its limitations, which we discuss in detail in Appendix E. Overall, the main aim of this paper is not to completely replace existing approaches but to open up new possibilities for future research on better designs of modality fusion and new paths towards more efficient and versatile utilization of PLMs.

Reviewer comment (Theory):
As shown in Eqn. (8), the input features of feed-forward networks guide the aggregation of the output features of feed-forward network, which is hard to comprehend. More explanations are required.

Author response:
- Thanks for the suggestion. Eq. 8 adds the representation of concepts weighted based on the conditional distribution from Eq. 5 to the representation of the current word being processed, in the output space of each feed-forward network. $\lambda$ controls the strength of the augmentation. We have added this explanation in the revised paper.
- We indeed included the intuitive explanation of this process in the last paragraph of Sec 3.2.2: 
> Intuitively, as verified in Figure 4, the intermediate representation of “[mask]” could not be close to the answer “hat” but after adding the representation of observed concepts, the model can make the correct prediction.

Reviewer comment (Reproducibility):
The authors adopt PLM to extract the features of textual visual concepts. However, when the length of the textual visual concepts exceeds one, it would be better to elucidate the specific feature extraction process.

Author response:
- We include this detail in Appendix G.1 due to space limit. We simply apply averaging over the tokens when the concept consists of multiple tokens.

Reviewer comment (Presentation):
There exists ‘?’ and ‘-’ in Table 1. It would be better to explain the meaning of these quotes.

Author response:
- Thanks for the suggestion. “-” means not applicable and “?” means unclear from the original paper.
- We have included it in the revised version.

Reviewer comment (Reproducibility):
While the ablation studies regarding hyper-parameters are included in the appendix, it would be beneficial to mention the best hyper-parameter settings in the implementation details to make the paper more concrete.

Author response:
Thanks for the suggestion. We have added it to the revised main paper.

Reviewer comment (Reproducibility):
This method demonstrates remarkable performance. Could the authors consider releasing the source code upon acceptance?

Author response:
- Thanks for the question. We indeed plan to, as we mentioned in Sec. G.1.

Reviewer comment (Experiments):
The paper describes two variants of the DCI method in Section 3.2.1 and Section 3.2.2, however the variations presented in the experiments are based on the vocabulary selection method (DCI, DCI-A, DCI-LM). The authors have not assessed the performances of the previously described variants.

Author response:
### Two Injection Mechanisms
- As discussed in the introduction, Figures 1 and 3, Sec. 3.2, Sec. 3.2.1 and Sec. 3.2.2 are complimentary pathways that combine to form Deep Concept Injection.
- We indeed included qualitative analysis in Figure 4 and Sec. 4.5 to analyze the effect of the constructed adaptation layers. We find that, with the help of the constructed adaptation layers, we successfully augment the intermediate feature to help the model attend more to extracted concepts relevant to the correct answer.
- We also included quantitative analysis in the appendix due to space limit. In Sec. F.3, we quantitatively analyze the contribution of the two mechanisms and observe that combining the two injection mechanisms yields the best result.

Reviewer comment (Reproducibility):
In experiment section, the visual encoder has not been mentioned explicitly. Authors need to add that information in the tables as well as description.

Author response:
### Visual encoders used
- We indeed included the visual encoders used in the experiments in Sec. 4.1. As a summary, to facilitate fair comparisons, we use the same CLIP ViT-L/14 as FrozenBiLM when comparing with it, and we use the same Q-Former based on ViT-g as BLIP-2 when comparing with it.
- We also included the information in the caption of Tables 1 and 2.
- We are happy to indicate this further if the reviewer kindly provides more detailed suggestions.

Reviewer comment (Reproducibility):
The vocabulary addition is a major step of the algorithm. Certain details and variations in vocabulary are missing: 1. What is the total number of visual concepts taken in the experiments, 2. What is the difference in output vocabulary of DCI-LM vs other variants. In equation 10, how are authors going from output of LLM to visual concepts. Question by itself can generate very open ended responses from LLM. Similarly for DCI-A, what is the number of top frequent words taken as dictionary.

Author response:
### Vocabulary construction and ethic question
- We indeed include the details of the vocabulary construction in Sec. 3.3 and G.1. We don’t expect additional bias to be introduced beyond that of the VisualGenome ontology.
- We indeed included detailed quantitative results and analysis in Sec. F.2 due to the space limit. We generally observe that this process helps remove from the concept recognition results some too generic visual concepts that are irrelevant to the question. We will further add some qualitative examples of the difference between the retrieved concepts when the generic and narrowed vocabulary are used.
- For DCI-LM, as shown in Eq. (10), we didn’t let the PLM decode in an open-ended manner. We use it to measure the relevance of each word in the generic vocabulary to the question.
- Once we obtain the conditional probability given Eq. (10), as explained in the last sentence of page 6, we just select the most probable $n_C$ words as the concept vocabulary for retrieval and injection usage.
- We didn’t change the size of the vocabulary for DCI-A. We just use it as it is, based on the setting of open-ended video/visual question answering. Therefore, it varies from dataset to dataset.

Reviewer comment (Evaluation):
In Table 2, there is not much difference between the results of BLIP-2 and proposed DCI variants. Authors have not explained the reason behind no significant change in output accuracy wrt original BLIP-2.

Author response:
### Comparison with BLIP2
- Thanks for the good question. As we mentioned in Sec G.2, all the hyper-parameters are tuned based on iVQA validation set to avoid fitting other datasets for zero-shot evaluation, which means that current hyper-parameters are not optimal for comparisons with BLIP-2.
- Our results do not rely on any training, which is not expected to achieve significantly higher accuracy because of its great efficiency. The comparable results we have obtained are already quite surprising and validate the research hypothesis that the projection and adaptation layers are not necessary to be learned.