Paper: Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection (ICLR-2024)

Reviewer comment (Novelty):
The authors fail to convince me, why we need to generate answers in arbitrary positions ? I am eager to see what is the advantage of generating an arbitrary position over generating the right-bottom answer, not only the final performance, but also the rationale or the motivation behind this design.

Author response:
We hope that RAISE can **understand the overall rule of RPM panels**, rather than learning to map given contexts to bottom-right images. The model's understanding of overall rules is illustrated through the ability to generate target images at arbitrary positions. Moreover, the arbitrary-position generation ability **can help RAISE solve abstract reasoning tasks besides RPM**. The experiment in Section 4.5 shows that RAISE can solve the odd-one-out problems by predicting the image at each position and comparing the prediction results with the realistic images to determine the rule-breaking image in the panel. If the model only predicts the bottom-right images, solving odd-one-out problems can be difficult as the rule-breaking images are not always located at the bottom-right.

Reviewer comment (Experiments):
Too many hyper-parameters should be tuned.

Author response:
In general, neural networks require inductive biases to learn attributes from images. Some generative solvers introduce artificially designed structures in representation learning, e.g., PrAE and ALANS design specific representations to encode attributes like object size, color, etc. While RAISE introduces inductive biases via hyperparameters on network structures and weights in the loss. By adjusting the hyperparameters, RAISE can adapt to different types of data and automatically encode attributes into latent concepts.

Reviewer comment (Experiments):
Experiments are not enough. I suggest using PGM-Neutral, PGM-Interpolation and PGM-Extrapolaton to confirm the effectiveness of RAISE.

Author response:
We choose to evaluate models on RAVEN/I-RAVEN since their samples contain fewer noise attributes, and we can control the noise in RPMs to study the influence of noise in generative solvers by changing the settings of RAVEN/I-RAVEN, while PGM does not provide a similar way to control noise. We also refer to the configurations of PGM to generate two held-out configurations based on Center and O-IC and conduct experiments in Section 4.5 to evaluate the performance of models under OOD samples.

Reviewer comment (Experiments):
The authors also should clearly state why to use rule annotations, but not the answer images. Which indeed violates the RPM question.

Author response:
RAISE uses images of correct answers to supervise the prediction results in training but does not require distractors in candidate sets. We think that there is no conflict between using candidate sets and rule annotations. The generative RPM solvers like PrAE and ALANS use both candidate sets and rule annotations in training. **Rule annotations can guide the generative solvers to generate target images with specific rules.** We also attempt to **reduce the use of rule annotations** by supporting semi-supervised training settings. RAISE does not use candidate sets for supervision, mainly because **it is trained via arbitrary-position generation, but the datasets do not provide candidate sets for non-bottom-right images**. We find that without using candidate sets, RAISE can generate better results than the compared solvers. During the testing phase, we evaluate the generative solvers in the conventional setting of RPM problems, which is to select answers from candidate sets based on the generated target images.

Reviewer comment (Novelty):
This is a generative model designed specifically for RPM, and it is unclear how one would leverage this work or its findings in any other context.

Author response:
**Q1. How to generalize RAISE away from the RPM setting?**

We provide three examples to illustrate the way to generalize RAISE away from the RPM setting.

1. RPM can be regarded as an **analogical reasoning task**. The first two rows are reference examples following specific rules, and models need to apply the rules to the third row to produce predictions (the third row is the problem panel where the first two images are queries, and the third image is the solution). RAISE can solve other analogical reasoning tasks by concatenating reference examples and problem panels into RPM-like matrices. The reasoning process of RAISE supports learning latent concepts and atomic rules on matrices of different shapes and generating solutions for given queries.

2. The **odd-one-out** experiment is another example. In this experiment, we leverage the arbitrary-position generation ability of RAISE to find the image that breaks the panel rules. We can set each image on the panel as the target image, generate the prediction of targets using RAISE, and find the one having the largest prediction error as the rule-breaking image.

3. Regarding image sequences as one-row matrices, we can use RAISE to **generate the missing frames in sequences**. For example, we can represent object appearances and positions in latent concepts, and different patterns of movement in atomic rules. RAISE can parse the movement pattern based on the given frames, and then apply it to the latent concept of position to generate the missing frames.

Reviewer comment (Novelty):
It is equally a “straightforward” application of an ELBO-based generative process. It is well executed, but not surprising and I did not find particularly interesting pieces of technical/insightful choices throughout the paper.

Author response:
**Q2.  Any specific insights / technical novelty compared to previous works?**

The novelty of this paper lies in the definition of rules and how we realize atomic rules in the latent space via learnable networks. RAISE defines the **overall rules on the panel** instead of mapping the context images at eight positions to the bottom-right one or predicting the third image from the previous two images in a row. Therefore, RAISE **can predict answers at arbitrary and even multiple positions** conditioned on the given context and solve non-RPM reasoning tasks like odd-one-out.

Reviewer comment (Experiments):
It is unfortunate that some amount of rule annotation is still used. It oscillates between amounts (5% for non-grids, 20% for O_IG and 100% for 2x2 and 3x3 grids), but I was wondering what the performance would have been with 0% supervision, as this seems like the “optimal” solution target for RPMs.

Author response:
**Q6. How needed was the rule supervision? Do you have numbers when you drop this to 0?**

Without the supervision of rule annotations, the accuracy of RAISE is

|          Models           |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----------------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|        Transformer        | 40.1/64.0 | 98.4/99.2 | 67.0/91.1 | 60.9/86.6 | 14.5/69.9 | 13.5/57.1 |   14.7/25.2    |   11.6/18.6    |
| **RAISE (no rule anno.)** | 54.5/67.7 | 30.2/56.6 | 47.9/80.8 | 87.0/94.9 | 96.9/99.2 | 56.9/83.9 |   30.4/30.5    |   32.0/27.8    |
|           ALANS           | 74.4/78.5 | 69.1/72.3 | 72.2/79.2 | 73.3/79.6 | 76.3/85.9 | 74.9/79.9 |   80.2/79.5    |   75.0/72.9    |
|           RAISE           | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |

The experimental results show that, the average accuracy of unsupervised RAISE is between the unsupervised arbitrary-generation baselines and the generative RPM solvers trained with full rule annotations.

Reviewer comment (Novelty):
Do you have any suggestions for what one can learn from your model that can be generalized away from the RPM setting? Any specific insights / technical novelty compared to previous works?

Author response:
**Q1. How to generalize RAISE away from the RPM setting?**

We provide three examples to illustrate the way to generalize RAISE away from the RPM setting.

1. RPM can be regarded as an **analogical reasoning task**. The first two rows are reference examples following specific rules, and models need to apply the rules to the third row to produce predictions (the third row is the problem panel where the first two images are queries, and the third image is the solution). RAISE can solve other analogical reasoning tasks by concatenating reference examples and problem panels into RPM-like matrices. The reasoning process of RAISE supports learning latent concepts and atomic rules on matrices of different shapes and generating solutions for given queries.

2. The **odd-one-out** experiment is another example. In this experiment, we leverage the arbitrary-position generation ability of RAISE to find the image that breaks the panel rules. We can set each image on the panel as the target image, generate the prediction of targets using RAISE, and find the one having the largest prediction error as the rule-breaking image.

3. Regarding image sequences as one-row matrices, we can use RAISE to **generate the missing frames in sequences**. For example, we can represent object appearances and positions in latent concepts, and different patterns of movement in atomic rules. RAISE can parse the movement pattern based on the given frames, and then apply it to the latent concept of position to generate the missing frames. 


**Q2.  Any specific insights / technical novelty compared to previous works?**

The novelty of this paper lies in the definition of rules and how we realize atomic rules in the latent space via learnable networks. RAISE defines the **overall rules on the panel** instead of mapping the context images at eight positions to the bottom-right one or predicting the third image from the previous two images in a row. Therefore, RAISE **can predict answers at arbitrary and even multiple positions** conditioned on the given context and solve non-RPM reasoning tasks like odd-one-out.

Reviewer comment (Reproducibility):
Most decisions were extremely clear, but it was not that well explained how candidate answer selection was performed (Section 4.1 and 4.2).

Author response:
**Q3.  Details about candidate answer selection**

RAISE computes distances between the generated target latent concepts and the latent concepts of candidates, and selects the candidate with the smallest distance.

Reviewer comment (Evaluation):
Do you generate a sample x_t and compare to the x_candidates in pixel space?

Author response:
**Q3.  Details about candidate answer selection**

RAISE computes distances between the generated target latent concepts and the latent concepts of candidates, and selects the candidate with the smallest distance.

Reviewer comment (Reproducibility):
Do you generate a z_t and compare g^enc(x_candidates)?

Author response:
**Q3.  Details about candidate answer selection**

RAISE computes distances between the generated target latent concepts and the latent concepts of candidates, and selects the candidate with the smallest distance.

Reviewer comment (Evaluation):
Do you instead do a likelihood test?

Author response:
**Q3.  Details about candidate answer selection**

RAISE computes distances between the generated target latent concepts and the latent concepts of candidates, and selects the candidate with the smallest distance.

Reviewer comment (Experiments):
Finding values for C and K required going into the Appendix (C=8, K=4), and their choice wasn’t discussed.

Author response:
**Q4. Choice of $C$ and $K$**

> How sensitive is the model to varying C or K? How were these chosen? How adapted to the number of attributes and real rule numbers do they have to be? I’m aware they are different, but how sensitive is it e.g. can you use K=100?

To test the sensitivity of RAISE to $C$ and $K$, we train RAISE by varying $C$ ($C=4, 8, 16$) and $K$ ($K=4, 25$) on the original configuration. The results are given as follows.

| Configs |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|  $C=4$  | 59.5/78.6 | 98.8/99.7 | 42.2/79.1 | 34.4/62.0 | 51.3/85.7 | 37.0/65.1 |   72.6/80.4    |   80.2/78.0    |
|  $C=8$  | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |
| $C=16$  | 90.6/93.5 | 98.9/99.7 | 99.2/99.8 | 99.5/99.9 | 98.1/99.3 | 84.5/92.7 |   74.7/83.3    |   79.3/79.7    |

| Configs |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|  $K=4$  | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |
| $K=25$  | 90.4/92.8 | 97.7/99.1 | 98.9/99.5 | 99.4/99.8 | 98.2/99.6 | 86.5/94.3 |   75.7/82.2    |   76.1/75.2    |

RAISE is **insensitive** when increasing $C$ because it automatically generates some redundant latent concepts. When $C$ is too small to encode all changing attributes, the accuracy of RAISE will decline significantly (e.g., the accuracy of *L-R*, *U-D*, *O-IC*, and *O-IG* when $C=4$). To choose the proper $C$, we can first set a large number for $C$ and reduce it until the number of redundant latent concepts is reasonable. The results show that setting a large $K$ (here we set $K=25$ due to the limitation of devices) has **no significant influence** on the accuracy. But in general, we choose the hyperparameter $K$ by counting the unique labels in rule annotations.

Reviewer comment (Experiments):
How sensitive is the model to varying C or K?

Author response:
**Q4. Choice of $C$ and $K$**

> How sensitive is the model to varying C or K? How were these chosen? How adapted to the number of attributes and real rule numbers do they have to be? I’m aware they are different, but how sensitive is it e.g. can you use K=100?

To test the sensitivity of RAISE to $C$ and $K$, we train RAISE by varying $C$ ($C=4, 8, 16$) and $K$ ($K=4, 25$) on the original configuration. The results are given as follows.

| Configs |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|  $C=4$  | 59.5/78.6 | 98.8/99.7 | 42.2/79.1 | 34.4/62.0 | 51.3/85.7 | 37.0/65.1 |   72.6/80.4    |   80.2/78.0    |
|  $C=8$  | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |
| $C=16$  | 90.6/93.5 | 98.9/99.7 | 99.2/99.8 | 99.5/99.9 | 98.1/99.3 | 84.5/92.7 |   74.7/83.3    |   79.3/79.7    |

| Configs |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|  $K=4$  | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |
| $K=25$  | 90.4/92.8 | 97.7/99.1 | 98.9/99.5 | 99.4/99.8 | 98.2/99.6 | 86.5/94.3 |   75.7/82.2    |   76.1/75.2    |

RAISE is **insensitive** when increasing $C$ because it automatically generates some redundant latent concepts. When $C$ is too small to encode all changing attributes, the accuracy of RAISE will decline significantly (e.g., the accuracy of *L-R*, *U-D*, *O-IC*, and *O-IG* when $C=4$). To choose the proper $C$, we can first set a large number for $C$ and reduce it until the number of redundant latent concepts is reasonable. The results show that setting a large $K$ (here we set $K=25$ due to the limitation of devices) has **no significant influence** on the accuracy. But in general, we choose the hyperparameter $K$ by counting the unique labels in rule annotations.

Reviewer comment (Experiments):
How were C and K chosen? How adapted to the number of attributes and real rule numbers do they have to be? I’m aware they are different, but how sensitive is it e.g. can you use K=100?

Author response:
**Q4. Choice of $C$ and $K$**

> How sensitive is the model to varying C or K? How were these chosen? How adapted to the number of attributes and real rule numbers do they have to be? I’m aware they are different, but how sensitive is it e.g. can you use K=100?

To test the sensitivity of RAISE to $C$ and $K$, we train RAISE by varying $C$ ($C=4, 8, 16$) and $K$ ($K=4, 25$) on the original configuration. The results are given as follows.

| Configs |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|  $C=4$  | 59.5/78.6 | 98.8/99.7 | 42.2/79.1 | 34.4/62.0 | 51.3/85.7 | 37.0/65.1 |   72.6/80.4    |   80.2/78.0    |
|  $C=8$  | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |
| $C=16$  | 90.6/93.5 | 98.9/99.7 | 99.2/99.8 | 99.5/99.9 | 98.1/99.3 | 84.5/92.7 |   74.7/83.3    |   79.3/79.7    |

| Configs |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|  $K=4$  | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |
| $K=25$  | 90.4/92.8 | 97.7/99.1 | 98.9/99.5 | 99.4/99.8 | 98.2/99.6 | 86.5/94.3 |   75.7/82.2    |   76.1/75.2    |

RAISE is **insensitive** when increasing $C$ because it automatically generates some redundant latent concepts. When $C$ is too small to encode all changing attributes, the accuracy of RAISE will decline significantly (e.g., the accuracy of *L-R*, *U-D*, *O-IC*, and *O-IG* when $C=4$). To choose the proper $C$, we can first set a large number for $C$ and reduce it until the number of redundant latent concepts is reasonable. The results show that setting a large $K$ (here we set $K=25$ due to the limitation of devices) has **no significant influence** on the accuracy. But in general, we choose the hyperparameter $K$ by counting the unique labels in rule annotations.

Reviewer comment (Reproducibility):
The Transformer baseline lacked details, even in Appendix C.3.

Author response:
**Q5. Details about the Transformer baseline**

> 1. Which encoder did you use? Was it a discrete representation? Transformers behave much better on VQ-like latents.
>
> 2. I somehow expected it to do better, e.g. if attributes were provided instead of pixels, isn’t a Transformer a pretty strong baseline?

The Transformer baseline uses the same encoder and decoder structures as RAISE. We conduct **additional experiments** to evaluate the Transformer baseline with discrete representations. The results are given as follows.

|        Models        |    Average    |      Center       |        L-R        |        U-D        |       O-IC        |       O-IG        | 2$\times$2Grid | 3$\times$3Grid |
| :------------------: | :-----------: | :---------------: | :---------------: | :---------------: | :---------------: | :---------------: | :------------: | :------------: |
|     Transformer      | 40.1/**64.0** | **98.4**/**99.2** |     67.0/91.1     |     60.9/86.6     | **14.5**/**69.9** | **13.5**/**57.1** | 14.7/**25.2**  | 11.6/**18.6**  |
| Transformer-discrete | **44.2**/51.1 |     97.6/99.0     | **85.1**/**96.0** | **91.1**/**97.6** |     3.2/13.6      |     4.7/12.2      | **15.0**/22.2  | **12.7**/17.1  |

We take the vector quantization method of VQ-VAE [1] to discretize the representations. We set the size of the discrete latent space as 512, the dimensionality of each latent embedding vector as 32, and the weight of commitment loss as 0.1. The outputs of the encoder are split into 8 vectors of the length 32 and replaced by looking up the nearest embeddings in the latent space. The experimental results show that **the discrete representations improve the performance on *L-R* and *U-D* but decrease the accuracy on *O-IC* and *O-IG* with in-out layouts**.

We think the annotation of attribute values can be **too strong supervision** for model training. Though powerful RPM solvers like ALANS and PrAE leverage meta information about attributes to design visual representations, their training does not require any annotations of attribute values. We consider the Transformer as a baseline rather than a powerful solver because it is a black-box model and can hardly produce explicit representations of underlying rules on RPMs.

[1] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." *Advances in neural information processing systems* 30 (2017).

Reviewer comment (Experiments):
Which encoder did you use for the Transformer baseline? Was it a discrete representation? Transformers behave much better on VQ-like latents.

Author response:
**Q5. Details about the Transformer baseline**

> 1. Which encoder did you use? Was it a discrete representation? Transformers behave much better on VQ-like latents.
>
> 2. I somehow expected it to do better, e.g. if attributes were provided instead of pixels, isn’t a Transformer a pretty strong baseline?

The Transformer baseline uses the same encoder and decoder structures as RAISE. We conduct **additional experiments** to evaluate the Transformer baseline with discrete representations. The results are given as follows.

|        Models        |    Average    |      Center       |        L-R        |        U-D        |       O-IC        |       O-IG        | 2$\times$2Grid | 3$\times$3Grid |
| :------------------: | :-----------: | :---------------: | :---------------: | :---------------: | :---------------: | :---------------: | :------------: | :------------: |
|     Transformer      | 40.1/**64.0** | **98.4**/**99.2** |     67.0/91.1     |     60.9/86.6     | **14.5**/**69.9** | **13.5**/**57.1** | 14.7/**25.2**  | 11.6/**18.6**  |
| Transformer-discrete | **44.2**/51.1 |     97.6/99.0     | **85.1**/**96.0** | **91.1**/**97.6** |     3.2/13.6      |     4.7/12.2      | **15.0**/22.2  | **12.7**/17.1  |

We take the vector quantization method of VQ-VAE [1] to discretize the representations. We set the size of the discrete latent space as 512, the dimensionality of each latent embedding vector as 32, and the weight of commitment loss as 0.1. The outputs of the encoder are split into 8 vectors of the length 32 and replaced by looking up the nearest embeddings in the latent space. The experimental results show that **the discrete representations improve the performance on *L-R* and *U-D* but decrease the accuracy on *O-IC* and *O-IG* with in-out layouts**.

We think the annotation of attribute values can be **too strong supervision** for model training. Though powerful RPM solvers like ALANS and PrAE leverage meta information about attributes to design visual representations, their training does not require any annotations of attribute values. We consider the Transformer as a baseline rather than a powerful solver because it is a black-box model and can hardly produce explicit representations of underlying rules on RPMs.

[1] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." *Advances in neural information processing systems* 30 (2017).

Reviewer comment (Experiments):
I somehow expected the Transformer baseline to do better, e.g. if attributes were provided instead of pixels, isn’t a Transformer a pretty strong baseline?

Author response:
**Q5. Details about the Transformer baseline**

> 1. Which encoder did you use? Was it a discrete representation? Transformers behave much better on VQ-like latents.
>
> 2. I somehow expected it to do better, e.g. if attributes were provided instead of pixels, isn’t a Transformer a pretty strong baseline?

The Transformer baseline uses the same encoder and decoder structures as RAISE. We conduct **additional experiments** to evaluate the Transformer baseline with discrete representations. The results are given as follows.

|        Models        |    Average    |      Center       |        L-R        |        U-D        |       O-IC        |       O-IG        | 2$\times$2Grid | 3$\times$3Grid |
| :------------------: | :-----------: | :---------------: | :---------------: | :---------------: | :---------------: | :---------------: | :------------: | :------------: |
|     Transformer      | 40.1/**64.0** | **98.4**/**99.2** |     67.0/91.1     |     60.9/86.6     | **14.5**/**69.9** | **13.5**/**57.1** | 14.7/**25.2**  | 11.6/**18.6**  |
| Transformer-discrete | **44.2**/51.1 |     97.6/99.0     | **85.1**/**96.0** | **91.1**/**97.6** |     3.2/13.6      |     4.7/12.2      | **15.0**/22.2  | **12.7**/17.1  |

We take the vector quantization method of VQ-VAE [1] to discretize the representations. We set the size of the discrete latent space as 512, the dimensionality of each latent embedding vector as 32, and the weight of commitment loss as 0.1. The outputs of the encoder are split into 8 vectors of the length 32 and replaced by looking up the nearest embeddings in the latent space. The experimental results show that **the discrete representations improve the performance on *L-R* and *U-D* but decrease the accuracy on *O-IC* and *O-IG* with in-out layouts**.

We think the annotation of attribute values can be **too strong supervision** for model training. Though powerful RPM solvers like ALANS and PrAE leverage meta information about attributes to design visual representations, their training does not require any annotations of attribute values. We consider the Transformer as a baseline rather than a powerful solver because it is a black-box model and can hardly produce explicit representations of underlying rules on RPMs.

[1] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." *Advances in neural information processing systems* 30 (2017).

Reviewer comment (Experiments):
How needed was the rule supervision?

Author response:
**Q6. How needed was the rule supervision? Do you have numbers when you drop this to 0?**

Without the supervision of rule annotations, the accuracy of RAISE is

|          Models           |  Average  |  Center   |    L-R    |    U-D    |   O-IC    |   O-IG    | 2$\times$2Grid | 3$\times$3Grid |
| :-----------------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :------------: | :------------: |
|        Transformer        | 40.1/64.0 | 98.4/99.2 | 67.0/91.1 | 60.9/86.6 | 14.5/69.9 | 13.5/57.1 |   14.7/25.2    |   11.6/18.6    |
| **RAISE (no rule anno.)** | 54.5/67.7 | 30.2/56.6 | 47.9/80.8 | 87.0/94.9 | 96.9/99.2 | 56.9/83.9 |   30.4/30.5    |   32.0/27.8    |
|           ALANS           | 74.4/78.5 | 69.1/72.3 | 72.2/79.2 | 73.3/79.6 | 76.3/85.9 | 74.9/79.9 |   80.2/79.5    |   75.0/72.9    |
|           RAISE           | 90.0/92.1 | 99.2/99.8 | 98.5/99.6 | 99.3/99.9 | 97.6/99.6 | 89.3/96.0 |   68.2/71.3    |   77.7/78.7    |

The experimental results show that, the average accuracy of unsupervised RAISE is between the unsupervised arbitrary-generation baselines and the generative RPM solvers trained with full rule annotations.

Reviewer comment (Writing):
It would have helped to explicitly write that r^c takes values between 1 and K.

Author response:
**Q7. Typos**

> 1. It would have helped to explicitly write that r^c takes values between 1 and K.
> 2. \mu_T^c -> \mu_t^c in (4)
> 3. z_T^c -> z_t^c in (6)

Thanks for pointing out the typos. We will fix them in the revised version. We would like to clarify that, 2 and 3 are not typos. Here we use $T$ to index all target latent concepts.

Reviewer comment (Writing):
\mu_T^c -> \mu_t^c in (4).

Author response:
**Q7. Typos**

> 1. It would have helped to explicitly write that r^c takes values between 1 and K.
> 2. \mu_T^c -> \mu_t^c in (4)
> 3. z_T^c -> z_t^c in (6)

Thanks for pointing out the typos. We will fix them in the revised version. We would like to clarify that, 2 and 3 are not typos. Here we use $T$ to index all target latent concepts.

Reviewer comment (Writing):
z_T^c -> z_t^c in (6).

Author response:
**Q7. Typos**

> 1. It would have helped to explicitly write that r^c takes values between 1 and K.
> 2. \mu_T^c -> \mu_t^c in (4)
> 3. z_T^c -> z_t^c in (6)

Thanks for pointing out the typos. We will fix them in the revised version. We would like to clarify that, 2 and 3 are not typos. Here we use $T$ to index all target latent concepts.

Reviewer comment (Experiments):
My chief concern with this paper, and the overall idea behind structuring latent representations and re-combining them at inference, is how scalable is it beyond toy problems like RPMs? The bottleneck with DLVMs like RAISE is two-fold: learnability (e.g. authors re-use encoder parameters between generative and inference process) and reliance on human knowledge to design the graphical model. RPMs have a very limited rule set, and the visual abstraction process is vastly simpler than human scenes. These reasons why existing concept learning methods that rely on similar modelling and inference have not been able to scale to harder visual reasoning problems are yet to be solved.

Author response:
If we can design general inference and generative processes, DLVMs can potentially solve different tasks in various scenarios. RAISE achieves the arbitrary-position generation ability by decomposing high-dimensional images into low-dimensional latent concepts and learning atomic rules shared among latent concepts. This design allows RAISE to be potentially applied to different real-world scenarios.

For example, when **predicting missing frames in surveillance videos**, latent concepts can represent object appearances (e.g., the appearance of a car) and spatial parameters (e.g., the position of the car), while atomic rules can encode different patterns of movement (e.g., turn left, stop, etc.). Through the given frames, RAISE can analyze the most likely movement pattern of the car, and then apply it to the latent concept of position to generate the missing frames. In this framework, latent concepts can also represent objects in scenes, and atomic rules can represent the actions learned from observation. We can apply the atomic rules to latent concepts (objects) parsed from new scenes and **predict the results after applying the action**.

Reviewer comment (Theory):
In Sec 3.1 "Previous studies have emphasized the role of abstract object representations in the abstract reasoning of infants, which is similar to the idea of object-centric representation learners that decompose complex scenes into object representations. Both views reflect the compositionality of human cognition (Lake et al., 2011)." I don't know if the human analogy to concept learning should drive artificial concept learning agents, particularly because the underlying constraints (inductive biases?) on human and artificial systems are vastly different. A much more recent work from Lake et al [2] shows that neural network based reasoning combined with a meta learning objective can solve similar systematic generalization problems. I would also refer the authors to the discussion section of the paper for a much more nuanced discussion on Bayesian vs Neural approaches to learning systematicity. For a similar treatment of systematic reasoning by learning the underlying rule structure and routing at inference but from a purely neural treatment, I would refer the authors to [3] which also reports strong results on RPM style reasoning tasks.

Author response:
We thank the reviewer for providing the insightful works on systematic generalization. We would like to discuss Bayesian and Neural concept learning in three folds.

1. **The learning objective.** As mentioned by the reviewer, MLC uses meta-learning objectives to solve systematic generalization problems. In the discussion of MLC, it is mentioned that the hierarchical Bayesian modeling can be explained from the perspective of meta-learning. Grant et al [1] also indicate the connection between meta-learning and hierarchical Bayesian models. From this perspective, the global knowledge of atomic rules in RAISE can be regarded as a kind of global latent variable in hierarchical modeling. Although RAISE and MLC have different thoughts on model design, there may be close connections between their learning objectives.

2. **Interpretability of latent variables.** Bayesian and Neural methods can define basic modules in the learning processes, e.g., Functions in Neural Interpreters and atomic rules in RAISE. In Bayesian methods, we usually take interpretable latent variables in the learning process, e.g., using categorical random variables to indicate the types of the selected rules. While Neural Interpreters route inputs to different Functions by calculating specific scores. We can define more latent variables with specific meanings in DLVMs, e.g., RAISE defines a set of latent concepts to capture attributes (object size, color, etc.). In this way, visual scenes are decomposed into a simple set of latent variables, which may reduce the complexity of abstract reasoning and enable systematic generalization on attribute-rule combinations.

3. **Solving multi-solution problems.**  In generative reasoning tasks, there can be multiple solutions for the given context of a problem. Bayesian methods can handle multi-solution problems by sampling latent variables from distributions (as shown in Figure 3, RAISE can produce results that are different from the original sample but still follow the correct rules). The experimental results of RAISE indicate that DLVMs can potentially solve generative RPM problems with multiple solutions in less noisy data.

[1] Grant, Erin, et al. "Recasting Gradient-Based Meta-Learning as Hierarchical Bayes." *International Conference on Learning Representations*. 2018.