From: NeurIPS.cc/2021/Workshop/SVRHM/Program_Chairs
title: Paper Decision
decision: Accept (Poster)

From: NeurIPS.cc/2021/Workshop/SVRHM/Paper14/Reviewer_DEP2
title: Interesting paper with some methodological issues
review: 
This paper proposes a model for abstract visual reasoning. The proposed method, in accordance with previous work, emphasizes the role of abstraction and analogy in generalization. NSM relies on a theoretical framework for analogy, "Structure Mapping Theory", and uses two stages which extract the relational structure and employ it for analogical reasoning on different domains. The model is tested on a benchmark of Raven's Progressive Matrices and compared to other baselines.

## Strengths:

- Interesting perspective on abstraction and analogies that is based on a solid theoretical framework and inspired from human cognition.
- The paper presents thorough generalization experiments and evaluations.
- Many model analysis experiments are presented which brings insight on the model's inner workings.
- The paper is thorough and detailed. All model details are provided.
- Authors report all the experiments that they conducted, including ones that are not conclusive or supportive of their hypothesis.

## Weaknesses:

Major:
- The objective of the paper is learning a disentanglement of structure from semantics in order to simulate human reasoning capabilities. Among the ways of achieving this objective is direct supervision over domain and structural elements, unlike the [Hill et al.]. Thus, in practice, a fair comparison would involve each reasoning model trained with and without supervision. The issue is that only NSM is trained only with supervision at predicting the relationships while other baselines are trained only without supervision. This gives NSM an advantage over other models and makes the comparison unfair.
- The choice of Unary and Binary modules is well explained. However the layout-relationship association choices are not properly motivated.  
- Experiments in the SI show that it's not necessary to use different layouts for different relationship types. Since both layouts perform equally well on all relations, even if the layout prediction model worked properly, its choice would be inconsequential.
- Using neural modules in the analogy inference engine is a logical choice that's well motivated. However, its advantage over a simpler architecture isn't shown experimentally. For example, the visual relationship encoder experiments show that the CNN-LSTM architecture is fully capable of predicting the correct relationships, so it should be the a baseline for the Analogy Inference Engine as well.
- Not all prior PGM methods are covered (recent baselines include SCL [4], HriNet [3], CoPINet [2], LEN [1])  

Minor:
- Although the paper is thorough, the writing is difficult to follow in certain paragraphs.
- No code is provided for reproducing the results.

[1] Zheng, Kecheng, Zheng-Jun Zha, and Wei Wei. "Abstract reasoning with distracting features." Advances in Neural Information Processing Systems 32 (2019): 5842-5853.
[2] Zhang, Chi, et al. "Learning perceptual inference by contrasting." arXiv preprint arXiv:1912.00086 (2019).
[3] Hu, Sheng, et al. "Hierarchical rule induction network for abstract visual reasoning." arXiv preprint arXiv:2002.06838 2.4 (2020).
[4] Wu, Yuhuai, et al. "The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning." arXiv preprint arXiv:2007.04212 (2020).
rating: 5: Marginally below acceptance threshold
confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: NeurIPS.cc/2021/Workshop/SVRHM/Paper14/Reviewer_KjFH
title: Good contribution 
review: This paper examines a two-stage Neural Structure Mapping (NSM) framework to learn abstract visual analogies. The framework uses the two-step method similar to how humans are supposed to reason about analogical problems. The paper is clearly written and all details about the methodology and experiments are presented either in the text or in the appendices. The paper provides original work in high quality. I have some clarifications questions: 
•	Some of the related work could be given in the text rather than appendix and details of the method could be in the appendix. 
•	Additional information (even a short one) can be given for NSM and NSM full context in the text. What exactly did the authors change in the second context? 
•	In the Raven’s Progressive Matrices, the difficulty of each trial is different. Would structural mapping be more important in more difficult trials in drawing correct analogy? Although test accuracy results are not low and the engine generalizes systematically, it would still be good to know whether the progression changes by the trial difficulty. 
•	On p. 10 in the problem setup section, the details of the domains are given. Does the engine work similarly for different novel domain transfers (i.e., from line to shape vs. shape to line)? 

rating: 8: Top 50% of accepted papers, clear accept
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: NeurIPS.cc/2021/Workshop/SVRHM/Paper14/Reviewer_Yqb1
title: A 2-stage model for analogical reasoning
review: The paper proposes a new 2-stage architecture for learning abstract visual analogies (Raven Progressive Matrices) according to the Structure Mapping Theory. The first stage is a visual relationship encoder that extracts object attributes and the relation expressed by the first image triplet. Learning the analogy requires transferring the same relation to a new triplet in another object/attribute domain, where the third image is missing and must be identified among 4 candidates. This is done by the second stage, the Analogy Inference Engine. The key insight in the paper is that this Engine can reconfigure its internal structure (using neural module nets) based on the exact relation computed in stage 1. In particular, there are 2 possible configurations depending on whether the relation is unary (progression) or binary (AND, OR, XOR). 

Pros:
* The authors show that this model outperforms previous state-of-the-art models. 
* The topic is important, and relevant to cognitive science (Structure Mapping Theory is meant to account for human analogical reasoning). It will likely produce fruitful discussions at the workshop.
* The experiments are thorough and provide a complete picture of the model behavior

Cons:
* The 5-page format is particularly problematic for this paper, as much of the relevant information is missing in the main text. Personally, I could only understand the model and tasks after reading the appendix.
* While the model performs well, critical tests in the appendix reveal that this may not be due to the adaptive nature of the Analogy Inference Engine, or to the mapping of the 2 possible configurations with the unary/binary identified relations. This casts doubt on the paper’s motivation.
* My (possibly simplistic) understanding of the model and tasks suggests that it should be possible to perform the analogy based only on the visual relation encoder (Stage 1). Once the relation in the first triplet is identified, this encoder can also compare the 4 candidate images as part of the second triplet, and pick the one that assigns the highest likelihood to that same relation. Maybe this is what the authors compute as part of their “Full-context” model (described in the Appendix), but if that’s true, the tables and Figure 3 suggest that this simple model is more efficient than the full 2-stage NSM model. This again casts doubt on the paper’s main motivation.

rating: 6: Marginally above acceptance threshold
confidence: 2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper

From: NeurIPS.cc/2021/Workshop/SVRHM/Paper14/Reviewer_tW4F
title: Two-stage approach for Abstract Visual Analogy Learning
review: Note for the program chairs: This paper is also submitted to ICLR 2022 ([link](https://openreview.net/forum?id=By5Uwd_xzNF)).

This paper proposes a two-stage solution for abstract visual analogy learning. In the first stage, the visual relationship encoder predicts a relationship triplet {object, attribute, relation}. The second stage consists of a neural module network (NMN), which is called as the analogy inference engine, and this modular network selects the best candidate answer by adapting its layout depending on the relationship triplet predicted in the first stage. The main contribution of this work is to use a dynamic reasoning mechanism for inferring abstract visual analogies. I consider the main contribution neither as a strong contribution nor a weak contribution.

The authors proposed NSM by addressing the issues between lines 42-45. Even though the authors claimed constructing datasets that exploit the structure mapping prior is not always feasible, they still experimented with the same dataset only. I think their approach is still useful, it gives a huge performance boost, but the experiments with single dataset/setting do not support this claim.

It’s really hard to interpret the results table (Table 1). The top right cell denotes that the numbers are the best accuracy values achieved on the test set with contrasting or normal training strategy. However, Contrasting / Normal columns also exist (maybe one stands for training, the other stands for testing, it's not clear). ResNet, ResNet-Parallel, and WReN baselines lack citations.

Multi-task term in this paper is also misleading. I’d consider classifying different labels at the same time as multi-label classification. They argued that their approach is not multi-label classification, but using different classification heads is not multi-task learning. The authors performed experiments with both strategies, and the difference between them is insignificant.

In lines between 133-136, the authors claimed that their visual relationship encoder module performs the layout prediction nearly perfect with accuracy of 98.42%-99.9% (minimum and maximum values). However, I couldn’t find these exact numbers on the referred table (Table 7). The closest results to these numbers represent the results for the training split. The test results are much worse on the extrapolation (99%+ vs. \~60%), novel domain (shape color) (\~98% vs. \~80%), novel domain (line type) (\~95% vs \~75%) splits. The high performance on the training set points out that the model is capable of learning / memorizing a mapping between the input scenes and the associated relationships. The gap between the training and test splits points out that the model fails in terms of systematic generalization.

I think the authors should cite the following systematic generalization papers,
- https://proceedings.neurips.cc/paper/2020/file/e5a90182cc81e12ab5e72d66e0b46fe3-Paper.pdf
- https://openreview.net/forum?id=bKBhQhPeKaF

As an aesthetic comment, the texts are really small in the tables, and it increases the difficulty of reading.

rating: 4: Ok but not good enough - rejection
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct