Paper: Neural Language of Thought Models (ICLR-2024)

Reviewer comment (Novelty):
Novelty: The major limitation of this paper is the lack of theoretical novelty, since it seems the idea of disentangling object-level slots into blocked-level factors as well as the autoregressive transformer decoder for image reconstruction have been introduced in prior work SysBinder. It’s not elaborated on how the proposed approach is developed further in representing discrete semantics, especially when there’s not a large margin between the performance of SVG and other baselines.

Author response:
While our work builds on the foundation of SysBinder, it introduces significant and crucial extensions. Unlike SysBinder, which lacks the abilities to model prior distributions and perform sampling, we observe that by discretizing the block representations, we can train a novel semantic prior transformer. This enables us to achieve the **first semantic and composable modeling of representations and distributions from raw images**, which can be seen as an advancement towards learning a semantic language of visual scenes, similar to words in text.

We want to clarify that while SysBinder and SVQ both use a transformer decoder for reconstruction, SVQ additionally is able to train a separate autoregressive transformer decoder as the Semantic Prior on top of the discrete latents (Section 3.1 and Figure 2b), allowing for sampling of new scenes. **This ability is not supported by SysBinder and is a major contribution of our work and a key motivation for obtaining discrete latents.** Our generated samples are shown to be better than several baselines in terms of quantitative metrics (FID and generation accuracy) as well as qualitatively as shown in Figures 3 and 4 and discussed in Section 5.1.

Reviewer comment (Reproducibility):
Experiment – Missing implementation details: The specification of multiple hyper-parameters (e.g. number of blocks, dimension of vectors) of SVG and other implementation details is never found.

Author response:
We also already provide the hyperparameters and additional training and implementation details of our experiments in section B in the Appendix in the supplementary material.

Reviewer comment (Experiments):
Experiment – Unexplained task and constraints: The artificial designs of the constraints of the scene (metric Generation Accuracy) as well as the odd-one-out downstream task are not explained.

Author response:
In the beginning of Section 5 in the Dataset section, we describe the 2D Sprites dataset:

”In the 2D Sprites datasets, objects of varying shapes and colors are placed in a scene. In total, there are 7 possible colors and 12 possible shapes. In each image, one object has a single property that is unique from the other objects. All other properties are shared by at least two objects. This structure allows us to evaluate if the prior correctly models the dependencies between the properties of the scene.”

We then describe at the beginning of section 5.1.1 the Generation Accuracy metric:

”We additionally calculate generation accuracy by manually inspecting 128 images per model to check if the generated images follow the constraints of the dataset. That is, each image must have exactly one object that has a unique property. All other properties in the scene will have at least one duplicate among the other objects.”

For the downstream task, we describe in section 5.2.1:

”We modify the dataset by first dividing each image into four quadrants and ensuring exactly one object will be in each quadrant. As in our previous experiments, one object has a single property that is unique from the other objects. The goal of the task is to identify the quadrant of the odd-one-out object.”

Please let us know if there is still any confusion about this metric and task based on our description in the submitted manuscript. We would be glad to revise the text to make this more clear.

Reviewer comment (Experiments):
I would be happy to improve my score if my concerns are addressed, particularly if the method works on real, more complex datasets than CLEVR.

Author response:
Thank you for your support! We are glad to hear that we have addressed most of your concerns.

We also ran an additional experiment on a more realistic dataset to address your concern about applying SVQ to more realistic datasets. We use a dataset where the objects are taken from the Google Scanned Objects [5]. We present the results in the updated supplementary material in Appendix A.7. We use two versions of SVQ. SVQ-small uses the same hyperparameters as we used for the CLEVR-Hard dataset (see Table 11 in Appendix) and SVQ-large increases the codebook size to 256 and increases the size of the transformer decoder from 8 layers, 4 heads, model size 192 to 16 layers, 8 heads, model size 512. We see that while SVQ-small is able to generate objects from the dataset, the objects are smoothed out, resulting in a high FID score. Increasing the model size to SVQ-large significantly improves the quality of the generated scenes and the FID score, providing some evidence that SVQ can be scaled to work on more realistic datasets.

Reviewer comment (Experiments):
How sensitive are the results to the codebook size and number of blocks? It seems that for CLEVR we can a priori know these hyper-parameters, but for real-world or more complex datasets how would we set these?

Author response:
This is an important question. The codebook size and number of blocks are currently hyperparameters like the number of heads or convolution filters in other models. Our method does not require the true codebook size or number of blocks as it learns in an unsupervised way, but there exists a range of hyperparameters that make it work better. Making the model work on more complex real-world scenes is definitely an important future direction. 

To answer your question, we ran an ablation on the number of blocks for the 2D Sprites (3 obj) dataset. We are additionally running an ablation on the codebook size and will respond with the results once those experiments are finished running.

The results for the ablation on number of blocks are shown in the table below. We see that when the number of blocks is too small, the model performs poorly and fails to generate scenes corresponding to the data distribution. For a sufficiently large number of blocks, the model is able to segment the scene, but Generation Accuracy decreases when the number of blocks is too large. We suspect this to be because with more blocks, the model may require a higher capacity prior, which we kept fixed in this ablation.

| Number of Blocks | FID | Generation Accuracy (in %) |
| --- | --- | --- |
| 1 | 465.60 | 0.00 |
| 2 | 80.71 | 1.56 |
| 4 | 7.76 | 75.78 |
| 8 | 6.61 | 75.00 |
| 16 | 7.17 | 55.47 |
| 32 | 8.74 | 54.69 |  

We’ve additionally ran ablations on the codebook size for the 2D Sprites (3 obj) dataset. We present the results below. We also tried smaller codebook sizes of 4 and 16, but found that for codebook sizes smaller than 32, the SVQ model did not converge well, resulting in black reconstructions. Similar to the number of blocks ablations, we see that for larger codebook sizes, the FID scores are similar, but generation accuracy decreases for codebook sizes larger than 64. This again may be because the larger codebook sizes require a higher capacity prior, which was fixed in these ablations.

| Codebook Size | FID | Generation Accuracy (in %) |
| --- | --- | --- |
| 32 | 7.31 | 76.56 |
| 64 | 6.61 | 75.00 |
| 96 | 6.64 | 51.56 |
| 128 | 7.58 | 48.44 |
| 256 | 8.73 | 32.81 |

Reviewer comment (Experiments):
It seems natural to compare the ARI with slot-attention or a variant of slot-attention. Is there a reason why the method shouldn't be compared in this manner?

Author response:
Thank you for this suggestion. It is true that as an object-centric learning method, it seems natural to include FG-ARI as an evaluation metric as is typically done in previous works. We originally did not include this because object segmentation was not a key focus of our work. Instead, we emphasized the ability to **generate new samples through a composition of high-level semantic concepts** as a key contribution and instead focused our main evaluations on FID, generation accuracy, and qualitative samples. Since our method adds an additional discrete bottleneck, we also do not expect SVQ to improve segmentation performance over previous methods. Nonetheless, we agree with the reviewer that it is important to include this metric to see the effect of the discrete bottleneck on segmentation performance. We will include these numbers in an updated version of the manuscript and summarize the results below. We see that when compared to SysBinder, SVQ performs similarly in terms of FG-ARI on CLEVR-Easy and CLEVR-Hard and slightly underperforms on CLEVR-Tex. Compared to vanilla Slot Attention, SVQ achieves higher FG-ARi on all 3 datasets. **Importantly, note that SVQ achieves this by incorporating the crucial abilities of prior distribution modeling and sampling, which are not present in Slot Attention, SLATE, or SysBinder.**

|  | Slot Attention | SLATE | SysBinder | SVQ |
| --- | --- | --- | --- | --- |
| CLEVR-Easy | 85.85 | 91.65 | 92.58 | 91.37 |
| CLEVR-Hard | 81.29 | 76.79 | 90.43 | 90.48 |
| CLEVR-Tex | 24.67 | 73.85 | 78.12 | 70.93 |

Reviewer comment (Experiments):
The VQ-VAE/dVAE model may require larger model capacity to handle these tasks as they work on the local patch level.

Author response:
> The VQ-VAE/dVAE model may requires larger model capacity to handle this tasks as they works on the local patch level.
> 

For VQ-VAE, we had used the same 20-layer PixelCNN as proposed in the original paper. For dVAE, which is a more direct ablation of the semantic prior because it shares the same image decoder as SVQ, we had the same suspicion that perhaps a larger capacity decoder could improve the results. We therefore ran several experiments with larger capacity priors (12-layer and 16-layer transformers compared to 8-layer from our paper) on CLEVR-Hard to see the effect. We present the results below:

| Model | FID |
| --- | --- |
| dVAE (8-layer) | 65.89 |
| dVAE (12-layer) | 61.74 |
| dVAE (16-layer) | 60.75 |
| SVQ (8-layer) | 43.12 |

We see that while increasing the size of the transformer for the prior does slightly improve the FID for dVAE, it still underperforms when compared to SVQ, indicating that the performance issue may not be solely due to model capacity.

Nevertheless, we would like to emphasize that, although we use generation quality as one of the evaluation metrics, the **main contribution of the paper is not to propose a better image synthesizer but to answer the question “how can we learn semantic and composable representations (like language of visual thoughts) from raw images similar to the role of words in text.** Although patch-level quantized representation is known to be good for image generation, it does not provide semantic (or conceptual) and composable representation as we do.

Reviewer comment (Experiments):
The current experiments only show results on synthetic data, which is somewhat weak.

Author response:
> Overall I think this paper shows some interesting results and could inspire people. But the current experiments only show results on synthetic data which is somewhat weak.
> 

In order to evaluate SVQ on a more realistic dataset, we use a dataset where the objects are taken from the Google Scanned Objects [5]. We present the results in the updated supplementary material in Appendix A.7. We use two versions of SVQ. SVQ-small uses the same hyperparameters as we used for the CLEVR-Hard dataset (see Table 11 in Appendix) and SVQ-large increases the codebook size to 256 and increases the size of the transformer decoder from 8 layers, 4 heads, model size 192 to 16 layers, 8 heads, model size 512. We see that while SVQ-small is able to generate objects from the dataset, the objects are smoothed out, resulting in a high FID score. Increasing the model size to SVQ-large significantly improves the quality of the generated scenes and the FID score, providing some evidence that SVQ can be scaled to work on more realistic datasets.

Additionally, since the rebuttal period is ending soon, we kindly ask the reviewer to check if our rebuttal addresses all of your concerns.

Reviewer comment (Experiments):
In Tab.4, the authors only compare with dVAE and VQ-VAE Indices, which makes the comparison unfair as SVQ Indices also perform badly. Why not also showcase the VQ-VAE code (like Tab.5) as well as dVAE feature (using the code id to index the weight of the first layer in the decoder)?

Author response:
> In Tab.4, the authors only compare with dVAE and VQ-VAE Indices, which makes the comparison unfair as SVQ Indices also perform bad. Why not the authors also showcase the VQ-VAE code (like Tab.5) as well as dVAE feature (using the code id to index the weight of the first layer in the decoder)?
> 

Thank you for this suggestion. As suggested, we’ve ran these additional experiments and will update the manuscript. We’ve reproduced the results below (new results highlighted in **********bold**********). We see that while the VQ-VAE Codebook performs better in the 2D Odd-One-Out experiments when compared with the other patch-based methods, **it still take more steps to reach 98% ID accuracy and has lower OOD Accuracy than SVQ.** The continuous version of dVAE does not seem to improve performance on the CLEVR-Hard downstream task. We will include these results in an updated version of the paper.

2D Odd-One-Out Downstream Experiment:

| Model | Steps to 98% (ID) | OOD Accuracy % |
| --- | --- | --- |
| dVAE Discrete | 37,000 | 26.7 |
| **dVAE Continuous** | 32,000 | 29.5 |
| VQ-VAE Indices | 77,000 | 24.0 |
| **VQ-VAE Codebook** | 54,500 | 55.6 |
| SysBinder | 27,000 | 67.6 |
| SVQ Indices | 77,000 | 46.8 |
| SVQ Codebook | 27,000 | 99.1 |

CLEVR-Hard Property Comparison Downstream Experiment:

| Model | ID Accuracy % | OOD Accuracy % |
| --- | --- | --- |
| dVAE Discrete | 27.52 | 19.87 |
| **dVAE Continuous** | 24.51 | 20.51 |
| VQ-VAE Indices | 24.53 | 17.74 |
| VQ-VAE Codebook | 23.73 | 18.80 |
| SysBinder | 79.60 | 70.09 |
| SVQ Indices | 68.21 | 64.53 |
| SVQ Codebook | 75.86 | 71.15 |

Reviewer comment (Experiments):
The Tab.5 comparison seems not fair as well. VQ-VAE / dVAE are not designed for object-centric tasks so it is expected they will not perform well on this task; on the other hand, SysBinder shows comparable and even better performance on this task.

Author response:
> The Tab.5 seems not a fair comparison as well. VQ-VAE / dVAE are not designed for the object-centric tasks so it is expected they will not perform well on this task, on the other hand, SysBinder shows comparable and even better performance on this task.
> 

Here, we respectfully disagree. We believe that the experiment comparing SVQ with VQ-VAE and dVAE is fair and serves an important purpose. The VQ-VAE and dVAE baselines for the downstream task utilize an architecture similar to the Vision Transformer (ViT) architecture, where patch-codes are used as inputs and transformer layers are employed for predictions. The results highlight that the general ViT architecture struggles to handle reasoning tasks effectively, despite the presence of transformer layers. This outcome may not be immediately expected, as ViTs and their variants are widely recognized as powerful representation learners. We believe that this experiment is crucial in supporting the motivation behind our semantic representations that go beyond patch representations.

It is worth noting that while SysBinder performs slightly better in the ID setting and slightly worse in the OOD setting compared to SVQ, it lacks the ability to provide a prior distribution and sampling. The comparison to SysBinder is, therefore, still crucial because it suggests that SVQ's representations remain effective even after incorporating prior modeling and sampling through the addition of the discrete bottleneck.

Reviewer comment (Experiments):
The paper lacks ablation studies, which makes it hard for the reader to understand which part plays the critical role and is useful to support the authors' claim. For example, what would happen when M varies and becomes smaller than the actual number of attributes?

Author response:
> Following 3, the paper lacks of ablation studies, which makes the reader hard to understand which part plays the critical role and is useful to support the author's claim, for example, what would happen when M varies and becomes smaller than the actual number of attributes?
> 

Thank you for this suggestion. We ran an ablation on the number of blocks for the 2D Sprites (3 obj) dataset. We are additionally running an ablation on the codebook size and will respond with the results once those experiments are finished running.

The results for the ablation on number of blocks are shown in the table below. **We see that when the number of blocks is too small, the model performs poorly and fails to generate scenes corresponding to the data distribution.** For a sufficiently large number of blocks, the model is able to segment the scene, but Generation Accuracy decreases when the number of blocks is too large. We suspect this to be because with more blocks, the model may require a higher capacity prior, which we kept fixed in this ablation.

| Number of Blocks | FID | Generation Accuracy (in %) |
| --- | --- | --- |
| 1 | 465.60 | 0.00 |
| 2 | 80.71 | 1.56 |
| 4 | 7.76 | 75.78 |
| 8 | 6.61 | 75.00 |
| 16 | 7.17 | 55.47 |
| 32 | 8.74 | 54.69 |

Reviewer comment (Experiments):
The authors have shown some interesting codebook analysis results. It would be great if they could also showcase whether the learned codebook is composable and can perform controllable generation (other than based on reconstruction) by solely manipulating the code.

Author response:
> The authors have shown some interesting codebook analysis results, would be great if the authors could also showcase whether the learned codebook is composable and can perform controllable generation (other than based on reconstruction) and by solely manipulate the code.
> 

Thank you for this suggestion. One thing we’d like to point out is that **the Latent Traversal experiments (Figure 6) already provides controllable generation results**. Note that the results are created by modifying the code for one block while keeping the other blocks fixed. Thus, they are not based on reconstruction and can be seen as a form of controllable generation. For example, in Figure 6, if we look at the bottom left image of the middle column (Block 3), we see the effect of changing this block on Slot 1 (originally the teal ball) and Slot 4 (originally the gray cylinder). The effect is similar in both cases — the object is enlarged and moved closer to the camera.

Reviewer comment (Novelty):
Recently there is a paper presenting a similar idea (group level quantization https://arxiv.org/abs/2309.15505), which may give the authors some inspiration as well. Note that this is a concurrent work so there is no request to compare with it.

Author response:
> Recently there is a paper present similar idea (group level quantization https://arxiv.org/abs/2309.15505), which may give the authors some inspiration as well, note that this is a concurrent work so I'm not asking the authors to compare with them.
> 

Thank you for pointing this out. We will update our related works to include this work and consider it for future iterations of the model.