Paper: Attribute Recognition with Image-Conditioned Prefix Language Modeling (ICLR-2024)

Reviewer comment (Novelty):
Though the solution is sound according to the new development tendency, I do not find too many new insights and "surprising" parts. There are many works using captioning via LLM to solve nearly all other directions in visual understanding, e.g., action/object recognition, visual relationship understanding, VQA, etc. Please give a discussion covering more domains to analyze the contribution.

Author response:
**Q1:** Thank you for the comment. We want to clarify that the main insight of our work is the formulation of attribute learning as a knowledge retrieval problem to extract relevant knowledge embedded in large vision-language models (VLM) through the means of generative prompting. We also establish the **fundamental connection between conditional probability graphs and prompt template construction** in the context of prefix language models, which we illustrate in figure 2, and recognize its importance and applications for visual attribute learning. To further elaborate, this formulation allows us to flexibly extract relevant knowledge of object-attribute-image relationships from the pretrained foundational models for downstream tasks outside of the foundation model settings, as we mentioned in paragraph 4 of the introduction. To our knowledge, we are the first to propose this **novel view of treating attribute learning as a language modeling and knowledge retrieval problem** and in applying VLMs towards visual attribute recognition, as we discussed in paragraph 4 of Related Work.

To further highlight the difference between our work and other captioning work in visual understanding, we would like to reiterate parts of the Approach section in the paper. One of our central ideas is to formally express attribute learning as a task of learning image-object-attribute conditional probabilities, which naturally leads to generative prompting in a learning distillation paradigm  - constructing custom probability dependencies between objects and attributes at inference time (with 4 examples given in section 3.3 and in figure 2) and use it to extract relevant knowledge on image-object-attribute relationship acquired during standard large-scale pretraining in the foundation VLM model, as we discussed in paragraph 6 in section 3.3 and illustrated in figure 1.

Reviewer comment (Novelty):
Compositional learning and its zero-shot setting (CZSL) is challenging with previous paradigms given the fixed train and test sets. Now, we have huge datasets like LAION and many other datasets beyond images like text. More discussion or insights about the new weapons and the CZSL would be a better contribution.

Author response:
**Q2:** Thank you for the comment. We will include additional discussions and citations on compositional zero-shot learning in the final submission and discuss how it relates to attribute learning and our work.

Reviewer comment (Experiments):
Though previous non-LLM works are "old", they can still be used as the baselines.

Author response:
**Q3:** We have included numerous non-LLM baselines in our experiment table for comparison. We would love to add additional important ones we have not yet included if you could kindly bring them to our attention.

Reviewer comment (Writing):
The method part is kinda of too brief.

Author response:
**Q4:** We will expand and elaborate the method section in our final submission.

Reviewer comment (Experiments):
More experiments on more datasets would be more solid to support the claims.

Author response:
**Q5:** We examined a list of potential dataset and found HICO and V-COCO to be more relevant, but not necessary for expanding on for our work’s central idea due to overlapping domain (COCO) and the focus on relational attributes.

Reviewer comment (Writing):
Typo: many wrong quotation marks ”xx”.

Author response:
**Q6 Q7:** We will fix these issues in the final submission.

Reviewer comment (Presentation):
Some weird green boxes appeared.

Author response:
**Q6 Q7:** We will fix these issues in the final submission.

Reviewer comment (Evaluation):
Detailed analysis of the long-tailed distribution and results?

Author response:
**Q9:** Thank you for the comment. Our model performs significantly better on the less frequent categories in the distribution long tail, as shown in table 3. SCoNE and TAP’s slightly higher overall mAP and head mAP does not necessarily mean they are superior models, since they represent performance on frequently observed categories where it is easy for models to **fit to the dataset**. Our model’s performance on the long tail shows that it fits more to the prior knowledge carried by the foundation models.

Please see below for qualitative results on the least frequent attributes in VAW:
| Attribute   | SCoNE mAP | Our mAP |
|-------------|-----------|---------|
| nylon       |    0.6984 |  0.5333 |
| bell shaped |    0.6955 |  0.9167 |
| braided     |    0.3893 |  0.7046 |
| styrofoam   |    0.3591 |  0.3354 |
| spiral      |    0.2294 |  0.8605 |
| kissing     |    0.0409 |  0.4085 |
| wallpapered |    0.5293 |  0.8956 |
| smoking     |    0.1966 |  0.3671 |
| stucco      |    0.3774 |  0.5914 |
| cubed       |    0.1102 |  0.4258 |
| **TAIL MEAN**   |     **0.480** |   **0.594** |

Reviewer comment (Novelty):
As a machine learning research, the technical significance and novelty seem weak. This work looks like a simple prompt engineering paper to me. To claim the technical significance, the authors should try with more various prompts and compare the results.

Author response:
**Q1:** Thank you for your comment. We want to clarify that the main contribution of our work is not the prompt engineering, but the formulation of attribute learning as a knowledge retrieval problem to extract relevant knowledge embedded in large vision-language models (VLM) through the means of generative prompting. Our work advances the state of the art by reframing the attribute learning problem to **solve it through leveraging standard pretrained VLMs**. We also establish the **fundamental connection between conditional probability graphs and prompt template construction** in the context of prefix language models, which we illustrate in figure 2, and recognize its importance and applications for visual attribute learning. To further elaborate, this formulation allows us to flexibly extract relevant knowledge of object-attribute-image relationships from the pretrained foundational models for downstream tasks outside of the foundation model settings, as we mentioned in paragraph 4 of the introduction. To our knowledge, we are the first to propose this **novel view of treating attribute learning as a language modeling and knowledge retrieval problem** and in applying VLMs towards visual attribute recognition, as we discussed in paragraph 4 of Related Work.

Reviewer comment (Theory):
As the main contribution of this paper is to replace contrastive prompting with generative prompting for attribute recognition, the authors should provide a theoretical explanation of why generative prompting is better than contrastive prompting. Otherwise, the technical contribution might be weak as a machine learning paper.

Author response:
**Q2:** Thank you for your comment. To reiterate section 3.1 on page 4, generative prompting is better than contrastive prompting because it **explicitly decomposes and represents the dependency in text input** by taking word order and relation into consideration, which also **align closer with a prefixLM’s pretraining-acquired knowledge** of diverse compositions of object-attribute dependencies. Given a nonsensical pair like “sky is parking”, the conditional probability term for “parking”, P(“parking” | v, s0, “sky”, “is”), would make the joint probability very small. CLIP-based contrastive prompting does not model dependencies in the text and has the problem of discrepancy between pretraining objective and downstream tasks, as it learned to align text to image without structures, resulting in a focus on objects, but is later tasked with understanding finer attributes that were not salient in pretraining objectives.

Reviewer comment (Experiments):
The experiment is also weak. In Table 3, the performance of the proposed method is not noticeably better than the baseline models (e.g., TAP).

Author response:
**Q3:** Thank you for the comment. Our model performs significantly better on the less frequent categories in the distribution long tail, as shown in table 3. SCoNE and TAP’s slightly higher overall mAP and head mAP does not necessarily mean they are superior models, since they represent performance on frequently observed categories where it is easy for models to **fit to the dataset**. Our model’s performance on the long tail shows that it fits more to the prior knowledge carried by the foundation models.

Additionally, the methods should not be directly compared because our focuses on **cross-domain knowledge extraction and learning distillation**, instead of constructing a task-specific model with specialized modules and training procedures. SCoNE and TAP both rely on object mask supervision and custom datasets during training, require task-specific modules, and are trained and tested in the same domain.

Reviewer comment (Experiments):
There should be more comprehensive experimental results. For example, how about the result with other LLM models than CoCa? Also, in Tables 4 and 5, why didn’t the authors compare the proposed model with the state-of-the-art methods? Finally, it would be better to evaluate the proposed method on other attribute recognition datasets, such as UTZappos or MIT datasets, or HOI detection datasets, such as HICO or V-COCO datasets.

Author response:
**Q4:** Thank you for the comment. CoCa was selected because it integrates both CLIP based training as well as captioning based training in the same model, allowing for a fair comparison, and because we can finetune it for comparisons with baselines.

Table 4 and 5 highlights the difference between probability metamodels in a zero-shot setting. Most of the baselines do not have zero-shot capability. TAP does (embedding space projection, similar to contrastive prompting), but its implementation is not available to us. 

Thank you for the list of potential datasets we can explore. We carefully examined each dataset and found HICO and V-COCO to be more relevant, but not necessary for expanding on for our work’s central idea due to overlapping domain (COCO) and the focus on relational attributes.