Paper: Does resistance to style-transfer equal Shape Bias? Evaluating shape bias by distorted shape (ICLR-2024)

Reviewer comment (Experiments):
In the experiment section, the authors only compared Resnet and ViT, the limited number of network architectures may make the conclusion less persuasive.

Author response:
**About the limited number of network architectures**: We extend the scope of our benchmarking model, please refer to our general response and section 4.1 and figure 5 in the paper for more details.

Reviewer comment (Experiments):
The proposed ways of generating shape distortion is based on Neural Style Transfer, does the specific ways of style transfer algorithm used matter? Or could other style transfer/shape distortion methods achieve the same performance?

Author response:
**About other Style transfer Methods:**  
Two major issues that limit the method we used to generate shape-distorted images is whether they can generate the image which randomly shuffles the local components of the original image, and how much does it take to generate the images. Methods like AdaIN[1] or pix2pix [2] are popular methods to do the style transfer,  however it’s hard for them to do the texture synthesis task where there is no content constraint.  Another practical issue is that many methods using GAN to generate the texture are too slow to produce enough images for either training or testing. 

As shown in section 4.2.1, we use an approximate method to generate the shape distorted images, and in the small scale evaluation, it’s as good as the original method. During our experiment, the detailed configuration of generation won’t have a significant influence on the result. Based on our experiment, we think the methods that are practical to be used to generate the image wouldn’t influence the performance. 


*[1] Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision (pp. 1501-1510).*

*[2] Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125-1134).*

Reviewer comment (Experiments):
The distorted images have similar textures but different global shape with the originally images. Would contrastive learning help improve the performance in this situation?

Author response:
**About contrastive learning:** We did use contrastive learning at the beginning as a possible way of improvement. Compared with traditional contrastive learning methods, we added the shape-distorted variation as a part of the negative pairs to train the model. We performed a small scale test on ResNet18 trained on Imagenette (a 10-class version of ImageNet), the baseline (supervised learning) achieved **39.4%** on DiST and the contrastive learning with additional shape distorted negative pairs achieved **34.3%** on DiST. We think the low performance of contrastive learning is due to the following reasons: 

Consider the contrastive loss  (openreview fail to render the whole formulas so we sepeate it):  
$-\sum_{i \in I} \log \frac{\exp ({z}_i \cdot {z}_j / \tau)}{Neg}$

$Neg=\sum_{a \in A(i)} \exp ({z}_i \cdot {z}_a / \tau)$


where $j$ is the positive pair for $i$ and for other $a \in A(i)$ are the negative pair. If there are $N$ images in total, there would be $N$ augmented images and $N$ shape-distorted images, meaning that there would be 1 positive pair and $3N-2$ negative pairs for each image. However, the there is only 1 negative pair that would contribute to model performance on DiST, which is the shape-distorted variation of the original image. Even if we don't use contrastive learning, it’s easy for model to distinguish the image with images in other classes and their shape-distorted variations. The only one that would be challenging for the model is the  shape-distorted variation of that image itself. While this negative pair only  occupies 1/(3N-1) of the negative pairs. Even if the model is not optimizing towards the way we want, to separate the representation of the original image with its shape-distorted variations, the contrastive loss can still be optimized well. 

We think that the key signal is too weak by using the contrastive loss, and explicitly setting the shape-distorted variation as the other class would provide a strong signal for the model to separate the original image and its shape-distorted variations.

Reviewer comment (Novelty):
It is clear from simple observation that texture and shape can both change, and that neural nets are currently configured to rely primarily on texture. So the paper is not saying too much in that regard.

Author response:
**neural nets are currently configured to rely primarily on texture**: 
It has been discovered years ago that **normally trained** neural networks are biased towards texture. While our paper wants to emphasize another crucial aspect: the current way to evaluate the shape bias of the model can only show how robust it is against the change of the style, models that achieve high performance on this evaluation are believed to have a “shape bias”. However, as shown in Figure 1 left, even **the model with “shape bias”** through style augmentation, still heavily focuses on the local feature. According to our knowledge, few research in deep learning have paid attention to the definition of “shape bias” but simply evaluate through those style-transferred images. This could cause some confusion when trying to understand the bias of the model. We want to use DiST to show how the model is sensitive to the global shape of the image,  which is missing in the previous metrics. And it would benefit the model in particular application (as shown in general response 3).

Reviewer comment (Experiments):
The dataset introduced comprises an unconvincing collection of images. These include images that have been assembled from sub-images of the object at different scales and points of view, peeled fruit (and peeled footballs). The reason this is not convincing is that the shape changes within a class will rarely if ever be expressed in such a way. Far more common would be simple geometric distortions. As an example, a "man" could be distorted into a "strong man" by increasing body and muscle size compared to the head.

Author response:
**Why not use some kind of warping to resist changes in shape? (Inter-class warps should be distinct from intra-class warps). What is your justification for using images made of pieces of photos of an object? (These are not shape distorted, they are mosaics of regularly shaped parts)**

It appears there may be a slight misunderstanding regarding the concept of "shape distortion" as used in our study. Our purpose of using texture synthesis is to create an image that has the same texture details as the original image, but the arrangement of local components in the image is disrupted, leading to a different global shape.  Like the examples shown in figure 2, the components of the owl are separated and shuffled, even if the local component would remain relatively the same (like the eyes), it would be hard to regard it as an owl due to its weird arrangement. That’s also the reason why styled augmented models still fail on this task, since it is not aware of the global shape, but only focuses on the eye of the owl, which is the same in the generated result.

Reviewer comment (Experiments):
Why not use some kind of warping to resist changes in shape? (Inter-class warps should be distinct from intra-class warps).

Author response:
**Why not use some kind of warping to resist changes in shape? (Inter-class warps should be distinct from intra-class warps). What is your justification for using images made of pieces of photos of an object? (These are not shape distorted, they are mosaics of regularly shaped parts)**

It appears there may be a slight misunderstanding regarding the concept of "shape distortion" as used in our study. Our purpose of using texture synthesis is to create an image that has the same texture details as the original image, but the arrangement of local components in the image is disrupted, leading to a different global shape.  Like the examples shown in figure 2, the components of the owl are separated and shuffled, even if the local component would remain relatively the same (like the eyes), it would be hard to regard it as an owl due to its weird arrangement. That’s also the reason why styled augmented models still fail on this task, since it is not aware of the global shape, but only focuses on the eye of the owl, which is the same in the generated result.

Reviewer comment (Experiments):
What is your justification for using images made of pieces of photos of an object? (These are not shape distorted, they are mosaics of regularly shaped parts).

Author response:
**Why not use some kind of warping to resist changes in shape? (Inter-class warps should be distinct from intra-class warps). What is your justification for using images made of pieces of photos of an object? (These are not shape distorted, they are mosaics of regularly shaped parts)**

It appears there may be a slight misunderstanding regarding the concept of "shape distortion" as used in our study. Our purpose of using texture synthesis is to create an image that has the same texture details as the original image, but the arrangement of local components in the image is disrupted, leading to a different global shape.  Like the examples shown in figure 2, the components of the owl are separated and shuffled, even if the local component would remain relatively the same (like the eyes), it would be hard to regard it as an owl due to its weird arrangement. That’s also the reason why styled augmented models still fail on this task, since it is not aware of the global shape, but only focuses on the eye of the owl, which is the same in the generated result.

Reviewer comment (Experiments):
People can draw things—say a face—in all sorts of shapes. Eyes can be round, lines, crosses, diamonds and many more shapes. Why not include such examples in your dataset (these examples are real and common in art)?

Author response:
**People can draw things - say a face - in all sorts of shapes. Eyes can be round, lines, crosses, diamonds and many more shapes. Why not include such examples in your dataset (these examples are real and common in art)**

The term “shape” we used in the paper is more related to the global arrangement of local components, which we called “global shape”. Although the small local component can be drawn using different shapes (circle, diamonds), this does not represent the “global shape” of the object. And it might still be within the scope of “texture” if we consider the model that heavily focuses on the local texture. For example, changing a round eye into a square eye, but not changing the position of the eye, would not influence the global shape, and can be regarded as a change of local texture. Since we know that models, even with style augmentation, still focus on those local textures, we want to make sure that the image in our test has exactly the same texture as the original images, the scope of this “texture” might also cover what you consider as the shape. Given those considerations, using human drawing might not be a very good option because it’s hard for us to control those texture details. Otherwise we are not able to ensure that model can’t distinguish between those images is due to different local texture, or different global shape (i.e. arrangement of local component )