Paper: Language-Informed Visual Concept Learning (ICLR-2024)

Reviewer comment (Evaluation):
The performance of the method is mostly shown in qualitative evaluations. The quantitative evaluation only shows the performance of image generation by modifying some concept axes (and human evaluation). I think the paper would be better if it came with an objective quantitative evaluation of the obtained concepts themselves in some ways (though I didn’t come up with any good approaches for this).

Author response:
**Q1 - Additional quantitative evaluation**

To better quantify the performance on capturing nuanced colors, we designed a new metric based on color statistics for the task of concept editing as an additional quantitative evaluation. 

Specifically, we compare the color of a 16 × 16 patch cropped around the center (focusing on the object) from both the input image and the image generated by the method being evaluated. The original size of the image is  64 × 64. We then compute the MSE of the mean color of the two patches, and report the average of the metric across all examples in Figure 11 in Table 1. Results suggest that our method captures colors more accurately compared to the baselines.

Reviewer comment (Evaluation):
Related to Point 1, I’m not sure if CLIPScore is really sensitive to arbitrary combinations of concepts. Some references or experimental results may help understand the experiment.

Author response:
**Q2 - Additional Qualitative results accompanying the CLIP-score evaluation**

In addition to the results in the main paper, in Figure 11 of the `Rebuttal PDF`, we provide more qualitative examples accompanying the experiments in Table 1. From these visual examples, we observed that the color nuances captured by ours are more accurate compared to the BLIP-2 baseline. However, since the CLIP-based metric specified in Section 4.3 of the paper cannot fully capture the minute differences, the BLIP-2 baseline still  achieves comparable scores to our method despite this evident gap in visual results. To better quantify such visual differences in colors, we further designed a new metric based on color statistics, as explained in Q3 above.

Reviewer comment (Writing):
The paper’s purpose is not sufficiently clear. Is it to learn concepts for image generation? Or is it for some other downstream tasks?

Author response:
**Q3 - Purpose of the framework**

The goal of visual concept learning is to extract visual representations in a structured manner akin to human perception. Humans interpret and organize the visual world into a hierarchy of abstract concepts, such as `object categories`, `colors`, `styles` etc. With this abstracted visual concept representation, we can easily picture a *new* visual instance with various concept compositions, such as `a red banana`.

In this paper, we aim at developing a method that can automatically learn to extract disentangled image representations along a number of language-specified concept axes (`category`, `color`, etc).
With these disentangled representations, we can then recompose them to *generate new compositions of visual concepts*, capturing visual nuances which would otherwise often exceed the limitations of language.

Reviewer comment (Writing):
What is concept representation learning? Is concept learning just the mutual translation of text and images?

Author response:
**Q1 - What is visual concept representation learning?**

The goal of visual concept learning is to extract visual representations in a structured manner akin to human perception. Humans interpret and organize the visual world into a hierarchy of abstract concepts, such as object `categories`, `colors`, `styles` etc. With this abstracted visual concept representation, we can easily picture a *new* visual instance with various concept compositions, such as `a red banana`.

In this paper, we aim at developing a method that can automatically learn to extract disentangled image representations along a number of language-specified concept axes (`category`, `color`, etc).
With these disentangled representations, we can then recompose them to *generate new compositions of visual concepts*, capturing visual nuances which would otherwise often exceed the limitations of language.

Reviewer comment (Experiments):
In the experiments, the authors primarily focus on conducting investigations using synthetic datasets. However, it raises concerns about the generalizability of the conclusions/findings obtained from synthetic datasets to real-world datasets.

Author response:
**Q2 - Real-world datasets**

We provide new inference results on real-world images in Figures 1 to 5 from the Rebuttal PDF, spanning various types of objects, including furniture, kitchen utensils, fruits, and artwork. Despite *only* being trained on images generated by diffusion-based models, the concept encoders generalize well to diverse, complex real-world images, including *unseen* types of objects, materials, colors, and styles. Note that some of the examples originally presented in the main paper also came from real photos, such as Figure 5 from the main paper.

Reviewer comment (Experiments):
The datasets used for the experiment were small and simple. It is not guaranteed that the claimed conclusions could be maintained when this framework is applied on more complex datasets (with much more concepts). The idea of using anchors to ensure that the embeddings are disentangle is great, however more experiments on larger datasets should be done to prove it. Given that there are only 2 to 3 concepts in each domain, the sparsity of concepts might be one of the reasons why the embeddings are disentangle.

Author response:
**Q1 - More complex datasets**

It is worth highlighting that despite training on simple, synthetic datasets, the model generalizes well to diverse, complex *real-world images* as discussed in the general response above. Figures 1-5 show results on real-world photos of various types of objects, including furniture, kitchen utensils, fruits, and artwork. Despite the *unseen* types of objects, materials, and colors, the model still extracts disentangled and compositional concept embeddings, showing the strong generalization capability of the proposed framework to complex real-world images.

Reviewer comment (Evaluation):
The effectiveness of L^{anchor} is not fully explained. The L^{anchor} is omitted during the test-time optimization procedure to avoid “over-committing to the coarse text anchors”. However, in the ablation experiment, the paper claims “disabling L^{anchor} deteriorates the performance”. It seems kind of contradictory; the paper should explain more about why “disabling L^{anchor}” is desired during one phase but it leads to unsatisfactory results in general evaluation.

Author response:
**Q2 - Omitting the anchor loss during test-time optimization**

During *training time*, $L^{anchor}$ encourages disentanglement of the concept embeddings across different concept axes while remaining in the subspace of the text embedding along their respective axes, as explained in Section 3.2. During *test time*, the concept encoders have already learned to encode only axis-specific concepts. The main objective is then to find a concept embedding within a specified axis that is consistent with the input image, which can be enforced by $L^\text{recon}$ (Equation (1)) alone. Omitting $L^{anchor}$ is beneficial as it helps the model to better preserve visual details of the input instance which are hard to capture with texts, *e.g.*, the style of paintings in Figure 24.

Reviewer comment (Evaluation):
The ablation test is not fully explained. In the “Editing Category” column, the results of “w/o Encoder & L^{anchor}_k” are actually higher than the results of “w/o L^{anchor}_k” in two metrics. This does not fully conform to the conclusion, quote, “and further removing the encoder decreases the overall decomposition performance”.

Author response:
**Q4 - Explanations on ablation study results**

Thanks for pointing this out. In fact, both ablation baselines ‘w/o Encoder & $L^{anchor}_k$’ and ‘w/o $L^{anchor}_k$’ fail to learn disentangled concept embeddings, as illustrated in Figure 8 in the main paper as well as the additional results in Figure 11 of the `Rebuttal PDF`.
The CLIP-based score is no longer indicative of the performance difference of the two baselines.
We have carefully updated the conclusion in Section 4.4 in the paper.

Reviewer comment (Writing):
On page 5, in sentence “text encoder c_{theta} of the T2I diffusion model…”, it should be “part of the text encoder c_{theta}”. Because a text encoder should take “text” as input rather than “text embeddings”. The original sentence might be confusing.

Author response:
**Q6 - Other clarifications**

Thanks for pointing them out. We revised the paper accordingly, marked in blue.

Reviewer comment (Writing):
On page 5, in formula (1), there is no explanation about the N and U notation. I assume they represent “multivariate normal distribution” and “uniform distribution” respectively. It would be more clear to annotate.

Author response:
**Q6 - Other clarifications**

Thanks for pointing them out. We revised the paper accordingly, marked in blue.

Reviewer comment (Writing):
On page 5, in sentence “so that they can directly inserted into the text embeddings”, it seems that a “be” is missed.

Author response:
**Q6 - Other clarifications**

Thanks for pointing them out. We revised the paper accordingly, marked in blue.

Reviewer comment (Presentation):
The paper should mention the backbone T2I model earlier. It is first mentioned on page 5; it would be better to do it earlier.

Author response:
**Q6 - Other clarifications**

Thanks for pointing them out. We revised the paper accordingly, marked in blue.

Reviewer comment (Evaluation):
Limited evaluation (only one set of quantitative numbers).

Author response:
**Q2 - Additional quantitative evaluation**

To better quantify the performance on capturing nuanced colors, we designed a new metric based on color statistics for the task of concept editing as an additional quantitative evaluation. 

Specifically, we compare the color of a 16 × 16 patch cropped around the center (focusing on the object) from both the input image and the image generated by the method being evaluated. The original size of the image is  64 × 64. We then compute the MSE of the mean color of the two patches, and report the average of the metric across all examples in Figure 11 in Table 1. Results suggest that our method captures colors more accurately compared to the baselines.

Reviewer comment (Presentation):
Table 1: Are you reporting CLIP score and human evaluation on same table?? Please point out CLEARLY that these are two different metrics in the Table caption. Or please separate into two Tables. This is highly confusing.

Author response:
**Q5 - Other clarifications.**

- We have updated Table 1 in the main paper to make sure the CLIP-based scores and human evaluation scores are clearly separated.
- We’ve included the proper citation for DeepFloyd. We apologize for missing it.
- Thanks for the reference! We’ve included a discussion in the related work section. 
- The quantitative evaluation for the BLIP-based baseline is added to Table 1.

Reviewer comment (Novelty):
DeepFloyd - please cite appropriately.

Author response:
**Q5 - Other clarifications.**

- We have updated Table 1 in the main paper to make sure the CLIP-based scores and human evaluation scores are clearly separated.
- We’ve included the proper citation for DeepFloyd. We apologize for missing it.
- Thanks for the reference! We’ve included a discussion in the related work section. 
- The quantitative evaluation for the BLIP-based baseline is added to Table 1.

Reviewer comment (Novelty):
The work in [1] explores language defined concept axes in video domain - maybe an interesting comparison to discuss in related work.

Author response:
**Q5 - Other clarifications.**

- We have updated Table 1 in the main paper to make sure the CLIP-based scores and human evaluation scores are clearly separated.
- We’ve included the proper citation for DeepFloyd. We apologize for missing it.
- Thanks for the reference! We’ve included a discussion in the related work section. 
- The quantitative evaluation for the BLIP-based baseline is added to Table 1.