Paper: Deep Neural Networks Can Learn Generalizable Same-Different Visual Relations (ICLR-2024)

Reviewer comment (Experiments):
In the human-visual system, two objects that are considered the same are usually based on some specific semantic and/or attribute similarities, rather than counting how many pixels that exactly have the same values. It means the dataset/task should define same-different visual relations by more criteria, such as geometry, texture, color, categories, identities, and etc. Moreover, the dataset should consider more visual distortions in real scenarios, such as cluttered background, color jittering, slight/moderate shape distortion, or allowing overlapping between objects. Sec. 4.2 tries to dissociate color, texture and shape, but more test scenarios can be included.

Author response:
We would like to thank the reviewer for their comments on our work. In particular, the reviewer's comment encouraging us to go “beyond comparing objects at a pixel level” was very helpful in guiding our revisions and in inspiring a new experiment, which we added to the article. We hope the reviewer will find our work improved.

To address your main points:

* We agree that same-different in real-world environments also requires invariance to factors like perspective, overlap, and background noise. To address this point, we have added a **new experiment** (3.3. Out-of-Distribution Generalization to Photorealistic Stimuli) testing how well our models generalize to highly realistic images that include object rotation, perspective (which distorts shape), object overlap, and textured/cluttered backgrounds. We do *not* fine-tune or train any of our models on these more realistic images, only using them as an additional evaluation (much like how we evaluated our models on Puebla and Bowers’ datasets in Appendix A.4).  Surprisingly, our CLIP ViT models have a median test accuracy of up to 90% on these realistic stimuli  (0.98 AUC-ROC), even when fine-tuned only on very simple synthetic 2D stimuli. This suggests that models actually learned a much more “human-like” representation of same-different than they needed to during fine-tuning, enabling them to further generalize to photorealistic stimuli under very different viewing conditions. This hopefully also addresses your concern of saturated prediction results due to carefully constructed testing datasets: generalization success is not actually limited to 64x64 non-overlapping items on a white background.

Reviewer comment (Experiments):
Sec. 3.2: As what I have mentioned, the OOD generalization in this paper may come from how well the shape is extracted. The closeness of stimuli may not be a reliable cue to better extraction of shapes, and thus is not well correlated with OOD generalization. The authors apply another dataset containing patches with random noise to prove that closeness of stimuli is not a perfect correlate of OOD generalization. But classifying this dataset relies more on texture similarity, thus it is reasonable that models fine-tuned on this dataset exhibit weaker generalization. I am interested in another dataset combining SQU with noises filling in each object. Possibly this dataset has more close stimuli, but also a higher OOD generalization.

Author response:
Your idea of a dataset combining SQU with noise filling in each object is interesting because it would combine the “closeness” idea from cosine similarity results with a dataset only solvable by shape. Our guess as to why the experiments in Section A.7 failed is because there was no incentive for the model to learn a global shape-based comparison, with the task being solvable even if a subset of pixels were sampled and compared. We worry that filling in SQU with noise would simply result in the same issue.

Reviewer comment (Novelty):
It is hard to tell how much the paper can contribute to the community. In my opinion, this work is an empirical study. For this category of works, the criteria are usually a) how many new observations are found, how surprising they are, and how useful they are for future works; b) whether the study is systematic and convincing; c) whether a new perspective is proposed or a new methodology is used to study the problem. I think this paper focuses more on a) and b). I am not sure how novel and important the findings introduced in this paper are, since the authors study a network architecture (ViT) which are actually already widely used in VLM models and tasks that require relation understanding like VQA. **I suggest the authors should clarify the generality and importance of their findings if only same-different relation is studied, or try tasks beyond same-different relation.**

Author response:
We thank the reviewer for their thoughtful comments on our work. We view learning the same-different relation as a case study for the larger problem of whether neural networks have the capacity to learn abstract relations. Since prior work has claimed that NNs are not capable of acquiring abstract same-different relations, our work centers around a question of *existence,* and whether we are able to find *any* neural network that is able to solve the same-different task in a generalizable way. 

We agree that ViT models performing well on tasks like VQA may already suggest that they have the capacity to handle certain kinds of abstract relations. However, this is only *implicit* evidence for relational reasoning in ViTs. Our experiments strip away confounding variables that may be present in more realistic tasks in order to explicitly test models on OOD generalization for the same-different task. Only some ViT models are able to generalize well (ViT pretrained with CLIP), while others fail (randomly-initialized ViT and ImageNet ViT, as seen in Table 6 and Table 8). 

Notably, with regards to the reviewer’s questions regarding generality and importance, we update the article with a **new section** (3.3. Out-of-Distribution Generalization to Photorealistic Stimuli) showing that CLIP ViT models can generalize to 3D objects placed in a realistic scene with up to 90% median test accuracy (0.98 AUC-ROC) when fine-tuned *only* on our toy 2D stimuli. This is a surprising result, as “same” in our fine-tuning datasets was defined based on pixel-level similarity, making generalization risky when transferring across differences in dimension, depth, orientation, lighting, occlusion, etc. Remarkably, we find that the best models retain strong performance. We hope that this new result, combined with the above clarification regarding the significance of our findings, will improve the reviewer’s view of our paper. 

Regarding your criteria for empirical studies: apart from our surprising results (your point (a)), the novelty of our approach to this problem (your point (c)) also comes from our detailed analysis of the impact of *dataset distribution* on generalization (Section 4). This is a factor that has not yet been thoroughly explored, despite clearly having a large impact on how models learn to solve a given task. As our results show, models will learn qualitatively different “solutions” to the same-different task depending on things like the presence of *color* variation in their training data.

Reviewer comment (Writing):
In Appendix B.1, "Because training datasets are constructed by sampling random objects, the exact objects used between the original, grayscale, and masked datasets are not the same" is a little confusing to me. Given the description above this claim, it seems that the authors can convert the original datasets to grayscale or masked versions with the same objects.

Author response:
Here are our answers to your questions:
2. In order to construct the Grayscale Shapes and Masked Shapes datasets, we first created grayscale and graymasked versions of all the objects in the Color Shapes dataset, then randomly sampled objects that would actually be included in the train/val/test data. As all of our datasets contained 1600 unique objects, we sampled 1600 objects out of 1793 unique Shapes objects we had access to, meaning that one specific object may have made it into the Grayscale Shapes dataset, but not the Color Shapes dataset (for example). We were not worried about test set contamination as shapes were modified across datasets.

Reviewer comment (Experiments):
As ViT takes image patches as inputs, it natively "segments" objects in the image. Have the authors tried to change object sizes in the image or experiment with other transformer architectures like Swin-Transformer?

Author response:
Here are our answers to your questions:
3. This was an intuition we had as well, but it is important to point out that even though objects are not actually aligned to ViT image patches in any of our main experiments, models still succeed on the task. In terms of object size, two of our experiments indirectly address this: the new section we added to the PDF testing on photorealistic images (Section 3.3) involves changes in object size due to distance from the camera, and Appendix A.4 involves testing on objects of different sizes. CLIP ViT is robust to these changes in object size in both cases. We also **added experiments to the appendix** (Appendix E) showing that performance does not seem to improve when objects are aligned with ViT image patches during fine-tuning. While it would certainly be interesting to see how Swin-Transformer would compare to our models, we leave this investigation to future work.