Original: M INIMAL I MAGES IN D EEP N EURAL N ETWORKS : F RAGILE O BJECT R ECOGNITION IN N ATURAL I MAGES
Revised: M INIMAL I MAGES IN D EEP N EURAL N ETWORKS : F RAGILE O BJECT R ECOGNITION IN N ATURAL I MAGES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The human ability to recognize objects is impaired when the object is not shown in full. "
Revised: The human ability to recognize objects is impaired when the object is not shown in full. "

Original: Minimal images" are the smallest regions of an image that remain recognizable for humans.
Revised: Minimal images" are the smallest regions of an image that remain recognizable for humans.

Original: Ullman et al.
Revised: Ullman et al.

Original: (2016) show that a slight modiﬁcation of the location and size of the visible region of the minimal image produces a sharp drop in human recognition accuracy.
Revised: (2016) show that a slight modiﬁcation of the location and size of the visible region of the minimal image produces a sharp drop in human recognition accuracy.

Original: In this paper, we demonstrate that such drops in accuracy due to changes of the visible region are a common phenomenon between humans and existing state-of-the-art deep neural networks (DNNs), and are much more prominent in DNNs.
Revised: In this paper, we demonstrate that such drops in accuracy due to changes of the visible region are a common phenomenon between humans and existing state-of-the-art deep neural networks (DNNs), and are much more prominent in DNNs.

Original: We found many cases where DNNs classiﬁed one region correctly and the other incorrectly, though they only differed by one row or column of pixels, and were often bigger than the average human minimal image size.
Revised: We found many cases where DNNs classiﬁed one region correctly and the other incorrectly, though they only differed by one row or column of pixels, and were often bigger than the average human minimal image size.

Original: We show that this phenomenon is independent from previous works that have reported lack of invariance to minor modiﬁcations in object location in DNNs.
Revised: We show that this phenomenon is independent from previous works that have reported lack of invariance to minor modiﬁcations in object location in DNNs.

Original: Our results thus reveal a new failure mode of DNNs that also affects humans to a much lesser degree.
Revised: Our results thus reveal a new failure mode of DNNs that also affects humans to a much lesser degree.

Original: They expose how fragile DNN recognition ability is for natural images even without adversarial patterns being introduced.
Revised: They expose how fragile DNN recognition ability is for natural images even without adversarial patterns being introduced.

Original: Bringing the robustness of
Revised: Bringing the robustness of

Original: DNNs in natural images to the human level remains an open challenge for the community.
Revised: DNNs in natural images to the human level remains an open challenge for the community.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) have reached tremendous success in recognizing and localizing objects in images.
Revised: Deep neural networks (DNNs) have reached tremendous success in recognizing and localizing objects in images.

Original: The fundamental approach that led to DNNs consists of building artiﬁcial systems based on the brain and human vision.
Revised: The fundamental approach that led to DNNs consists of building artiﬁcial systems based on the brain and human vision.

Original: Yet in many important aspects, the capabilities of DNNs are inferior to those of human vision.
Revised: Yet in many important aspects, the capabilities of DNNs are inferior to those of human vision.

Original: A promising strand of research is to investigate the similarities and differences, and by bridging the gaps, further improve DNNs (Hassabis et al., 2017).
Revised: A promising strand of research is to investigate the similarities and differences, and by bridging the gaps, further improve DNNs (Hassabis et al., 2017).

Original: Studying cognitive biases and optical illusions is particularly revealing of the function of a visual system, whether natural or artiﬁcial.
Revised: Studying cognitive biases and optical illusions is particularly revealing of the function of a visual system, whether natural or artiﬁcial.

Original: Ullman et al.
Revised: Ullman et al.

Original: (2016) present such a striking phenomenon of human vision, called "minimal images" (Ullman et al., 2016;
Revised: (2016) present such a striking phenomenon of human vision, called "minimal images" (Ullman et al., 2016;

Original: Ben-Yosef et al., 2018; Ben-Yosef &
Revised: Ben-Yosef et al., 2018; Ben-Yosef &

Original: Ullman, 2018).
Revised: Ullman, 2018).

Original: Minimal images are small regions of an image ( e.g. 10x10 pixels) in which only a part of an object is observed, and a slight adjustment of the visible area produces a sharp drop in human recognition accuracy.
Revised: Minimal images are small regions of an image ( e.g. 10x10 pixels) in which only a part of an object is observed, and a slight adjustment of the visible area produces a sharp drop in human recognition accuracy.

Original: Figure 1 provides examples of human minimal images.
Revised: Figure 1 provides examples of human minimal images.

Original: Ullman et al.
Revised: Ullman et al.

Original: (2016) show that DNNs are unable to recognize human minimal images, and the DNN drop in accuracy for these minimal images is gradual rather than sharp.
Revised: (2016) show that DNNs are unable to recognize human minimal images, and the DNN drop in accuracy for these minimal images is gradual rather than sharp.

Original: This begs the question of whether the sharp drop in accuracy for minimal images is a phenomenon exclusive to human vision, or there exist distinct but analogous images that produce a sharp drop in DNN accuracy.
Revised: This begs the question of whether the sharp drop in accuracy for minimal images is a phenomenon exclusive to human vision, or there exist distinct but analogous images that produce a sharp drop in DNN accuracy.

Original: In this paper, we provide evidence for the latter hypothesis by showing that there is a large set of analogs to minimal images that affect DNNs.
Revised: In this paper, we provide evidence for the latter hypothesis by showing that there is a large set of analogs to minimal images that affect DNNs.

Original: These are different from the minimal images for humans in several aspects, namely region size and location, and frequency and sharpness of the drop in accuracy.
Revised: These are different from the minimal images for humans in several aspects, namely region size and location, and frequency and sharpness of the drop in accuracy.

Original: We ﬁnd that a slight adjustment of a one-pixel shift or two-pixel shrink of the visible  image region produces a drop in DNN recognition accuracy in many image regions.
Revised: We ﬁnd that a slight adjustment of a one-pixel shift or two-pixel shrink of the visible [Figure] [Figure] image region produces a drop in DNN recognition accuracy in many image regions.

Original: Figure 2 shows several examples of minimal image analogs of state-of-the-art DNNs.
Revised: Figure 2 shows several examples of minimal image analogs of state-of-the-art DNNs.

Original: The adjustments of the visible area that affect DNNs are almost indistinguishable to humans and can occur in larger regions than the “human minimal” region.
Revised: The adjustments of the visible area that affect DNNs are almost indistinguishable to humans and can occur in larger regions than the “human minimal” region.

Original: To describe this phenomenon we introduce fragile recognition images (FRIs) for DNNs, which are more general than minimal images:[Figure]
Revised: To describe this phenomenon we introduce fragile recognition images (FRIs) for DNNs, which are more general than minimal images:

Original: [Figure] Fragile Recognition Image (FRI): A fragile recognition image is a region of an image for which a slight change of the region’s size or location in the image produces a large change in DNN recognition output.
Revised: Fragile Recognition Image (FRI): A fragile recognition image is a region of an image for which a slight change of the region’s size or location in the image produces a large change in DNN recognition output.

Original: This deﬁnition is more general than the deﬁnition of minimal images for humans, as the latter is included in the deﬁnition for DNNs.
Revised: This deﬁnition is more general than the deﬁnition of minimal images for humans, as the latter is included in the deﬁnition for DNNs.

Original: Minimal images are the case of fragile recognition in which the slight change is a reduction in the size of the region, and the minimal image is one of the smallest possible FRIs.
Revised: Minimal images are the case of fragile recognition in which the slight change is a reduction in the size of the region, and the minimal image is one of the smallest possible FRIs.

Original: In human vision, the more general deﬁnition of fragile recognition that we are introducing here is not useful because human minimal images appear only when the visible area of the object is small. 
Revised: In human vision, the more general deﬁnition of fragile recognition that we are introducing here is not useful because human minimal images appear only when the visible area of the object is small. [Figure]

Original: Since FRIs are hardly distinguishable to humans but cause DNNs to fail, there is a connection with so-called adversarial examples presented by Szegedy et al.
Revised: Since FRIs are hardly distinguishable to humans but cause DNNs to fail, there is a connection with so-called adversarial examples presented by Szegedy et al.

Original: Adversarial examples are images with small synthetic perturbations that are imperceptible to humans, but produce a sharp drop in
Revised: Adversarial examples are images with small synthetic perturbations that are imperceptible to humans, but produce a sharp drop in

Original: DNN recognition accuracy.
Revised: DNN recognition accuracy.

Original: There are several types, eg .
Revised: There are several types, eg .

Original: (Goodfellow et al., 2014; Liu et al., 2016), and the strategies to alleviate them are not able to fully protect DNNs (Tramèr et al., 2017; Madry et al., 2017).
Revised: (Goodfellow et al., 2014; Liu et al., 2016), and the strategies to alleviate them are not able to fully protect DNNs (Tramèr et al., 2017; Madry et al., 2017).

Original: Furthermore, humans may suffer from adversarial attacks; human recognition ability is shown to be impaired by perturbations under rapid image presentation (Elsayed et al., 2018).
Revised: Furthermore, humans may suffer from adversarial attacks; human recognition ability is shown to be impaired by perturbations under rapid image presentation (Elsayed et al., 2018).

Original: Unlike[Figure] these adversarial examples, fragile recognition arise in natural images without introducing synthetic perturbation.
Revised: Unlike these adversarial examples, fragile recognition arise in natural images without introducing synthetic perturbation.

Original: This causes new concerns for use of DNNs in computer vision applications.
Revised: This causes new concerns for use of DNNs in computer vision applications.

Original: We evaluate FRIs in ImageNet (Deng et al., 2009) for state-of-the-art DNNs, speciﬁcally VGG16 (Simonyan & Zisserman, 2015), Inception (Szegedy et al., 2014), and ResNet (He et al., 2016).
Revised: We evaluate FRIs in ImageNet (Deng et al., 2009) for state-of-the-art DNNs, speciﬁcally VGG16 (Simonyan & Zisserman, 2015), Inception (Szegedy et al., 2014), and ResNet (He et al., 2016).

Original: Results show that FRIs are abundant and can occur for any region size.
Revised: Results show that FRIs are abundant and can occur for any region size.

Original: Furthermore, we investigate whether fragile recognition is related to the lack of invariance to small changes in the object location, which has been recently reported in the literature to affect DNNs (Engstrom et al., 2017; Azulay &
Revised: Furthermore, we investigate whether fragile recognition is related to the lack of invariance to small changes in the object location, which has been recently reported in the literature to affect DNNs (Engstrom et al., 2017; Azulay &

Original: Weiss, 2018).
Revised: Weiss, 2018).

Original: Our results demonstrate that fragile recognition is independent from this phenomenon: bigger pooling regions reduce most of the lack of invariance to changes in object location, while pooling only marginally reduces fragile recognition.
Revised: Our results demonstrate that fragile recognition is independent from this phenomenon: bigger pooling regions reduce most of the lack of invariance to changes in object location, while pooling only marginally reduces fragile recognition.

Original: We also show that known strategies to increase network generalization, i .
Revised: We also show that known strategies to increase network generalization, i .

Original: e .
Revised: e .

Original: adding regularization and data augmentation, reduce the number of FRIs but still leave far more than humans have.
Revised: adding regularization and data augmentation, reduce the number of FRIs but still leave far more than humans have.

Original: These results highlight how much more fragile current
Revised: These results highlight how much more fragile current

Original: DNN recognition ability is than human vision.
Revised: DNN recognition ability is than human vision.

Original:  M ETHODS : E XTRACTING F RAGILE R ECOGNITION I MAGES
Revised:  M ETHODS : E XTRACTING F RAGILE R ECOGNITION I MAGES

Original: In this Section we introduce the method for extracting FRIs for DNNs.
Revised: In this Section we introduce the method for extracting FRIs for DNNs.

Original: The method for extracting human minimal images employs a tree search strategy (Ullman et al., 2016).
Revised: The method for extracting human minimal images employs a tree search strategy (Ullman et al., 2016).

Original: The full object is presented to human subjects and they are asked to recognize it.
Revised: The full object is presented to human subjects and they are asked to recognize it.

Original: If at least 50% of subjects recognize it correctly, smaller crops of the object, called descendants, are tested.
Revised: If at least 50% of subjects recognize it correctly, smaller crops of the object, called descendants, are tested.

Original: If any of the smaller region is recognizable, the crop size is further reduced and tested.
Revised: If any of the smaller region is recognizable, the crop size is further reduced and tested.

Original: Once an image crop is found such that it is recognizable and all of its descendants are not recognizable, it is considered a human minimal image.
Revised: Once an image crop is found such that it is recognizable and all of its descendants are not recognizable, it is considered a human minimal image.

Original: Our FRI extraction method for DNNs relies on an exhaustive grid search.
Revised: Our FRI extraction method for DNNs relies on an exhaustive grid search.

Original: This is possible in DNNs due to parallelization across multiple GPUs, and would be prohibitively time-consuming to replicate with humans.
Revised: This is possible in DNNs due to parallelization across multiple GPUs, and would be prohibitively time-consuming to replicate with humans.

Original: The grid search consists on a two-step process: ﬁrst, every possible square region is classiﬁed by the DNN and the correctness is annotated in the correctness map .
Revised: The grid search consists on a two-step process: ﬁrst, every possible square region is classiﬁed by the DNN and the correctness is annotated in the correctness map .

Original: From the correctness map, each region’s correctness is compared with the region’s slightly changed location or size in order to determine if there has been a change of the correctness.
Revised: From the correctness map, each region’s correctness is compared with the region’s slightly changed location or size in order to determine if there has been a change of the correctness.

Original: FRIs are regions that are classiﬁed correctly and a small change causes failure, as well as regions that are classiﬁed incorrectly and a small change causes success.
Revised: FRIs are regions that are classiﬁed correctly and a small change causes failure, as well as regions that are classiﬁed incorrectly and a small change causes success.

Original: The two step process to detect FRIs is summarized in Figure 3.
Revised: The two step process to detect FRIs is summarized in Figure 3.

Original: We now detail the two steps:
Revised: We now detail the two steps:

Original: Correctness map generation.
Revised: Correctness map generation.

Original: An exhaustive grid search is performed to see if each possible square image region of a ﬁxed size is classiﬁed correctly by a given DNN.
Revised: An exhaustive grid search is performed to see if each possible square image region of a ﬁxed size is classiﬁed correctly by a given DNN.

Original: After extracting the region from the image, the region is resized to be of the size required by the network. 
Revised: After extracting the region from the image, the region is resized to be of the size required by the network. The region

Original: The region size is parametrized by P , which is deﬁned as the proportion of the image occupied by the region, i 
Revised: [Figure] size is parametrized by P , which is deﬁned as the proportion of the image occupied by the region, i .

Original: e .
Revised: e .

Original: P = S/ min( h, w ) where h and w are the height and width of the input image, and S is the region’s side length.
Revised: P = S/ min( h, w ) where h and w are the height and width of the input image, and S is the region’s side length.

Original: The results are arranged in a map such that a given map pixel contains the binary correctness for the square region centered at the corresponding pixel in the original image.
Revised: The results are arranged in a map such that a given map pixel contains the binary correctness for the square region centered at the corresponding pixel in the original image.

Original: The resulting map is of dimension ( h − S ) ×
Revised: The resulting map is of dimension ( h − S ) ×

Original: ( w − S ) due to padding loss.
Revised: ( w − S ) due to padding loss.

Original: The ﬁrst two panels of Figure 3 show a visualization of the correctness map generation process.
Revised: The ﬁrst two panels of Figure 3 show a visualization of the correctness map generation process.

Original: [Figure]

Original: Fragile recognition image (FRI) extraction from correctness maps.
Revised: Fragile recognition image (FRI) extraction from correctness maps.

Original: We deﬁne different variations of FRIs, depending if they are based on changes on the location or size of the image region, and on how strict we are when evaluating the changes in the correctness of DNN:
Revised: We deﬁne different variations of FRIs, depending if they are based on changes on the location or size of the image region, and on how strict we are when evaluating the changes in the correctness of DNN:

Original: – Shift or Shrink : we deﬁne two types of "small changes" of the region that affect DNN correctness.
Revised: – Shift or Shrink : we deﬁne two types of "small changes" of the region that affect DNN correctness.

Original: "Shift" is a one-pixel translation of the region location; "shrink" is a two-pixel reduction of the region side length within the region’s original boundaries, in a ﬁxed full image size.
Revised: "Shift" is a one-pixel translation of the region location; "shrink" is a two-pixel reduction of the region side length within the region’s original boundaries, in a ﬁxed full image size.

Original: – Strict or Loose : the visible region can be shifted in various directions, or shrunk in different ways while remaining within its original boundaries.
Revised: – Strict or Loose : the visible region can be shifted in various directions, or shrunk in different ways while remaining within its original boundaries.

Original: "Loose" FRIs are regions such that there exists a small change that ﬂip network correctness.
Revised: "Loose" FRIs are regions such that there exists a small change that ﬂip network correctness.

Original: "Strict" FRIs are regions such that network correctness is ﬂipped for all small changes.
Revised: "Strict" FRIs are regions such that network correctness is ﬂipped for all small changes.

Original: These deﬁnitions yield four fragile recognition types: loose shift, loose shrink, strict shift, and strict shrink.
Revised: These deﬁnitions yield four fragile recognition types: loose shift, loose shrink, strict shift, and strict shrink.

Original: Note that strict shrink is the most analogous to human minimal images.
Revised: Note that strict shrink is the most analogous to human minimal images.

Original: The correctness maps are used to detect fragile recognition due to shifts by comparing neighbouring pixels in a correctness map, and due to shrinks by comparing correctness maps at two slightly different region sizes.
Revised: The correctness maps are used to detect fragile recognition due to shifts by comparing neighbouring pixels in a correctness map, and due to shrinks by comparing correctness maps at two slightly different region sizes.

Original: We use fragile recognition image (FRI) maps to visually represent the result of the grid search that extracts FRIs.
Revised: We use fragile recognition image (FRI) maps to visually represent the result of the grid search that extracts FRIs.

Original: As shown in the FRI map of Figure 3, each pixel of the map indicates whether the corresponding window in the original image is an FRI.
Revised: As shown in the FRI map of Figure 3, each pixel of the map indicates whether the corresponding window in the original image is an FRI.

Original: The second and third panel show a visualization of the FRI map generation process.
Revised: The second and third panel show a visualization of the FRI map generation process.

Original: Figure 4 shows an ImageNet image and each of its
Revised: Figure 4 shows an ImageNet image and each of its

Original: FRI maps.
Revised: FRI maps.

Original:  F RAGILE R ECOGNITION I MAGES OF S TATE - OF - THE - ART DNN S
Revised:  F RAGILE R ECOGNITION I MAGES OF S TATE - OF - THE - ART DNN S

Original: In this Section, we will ﬁrst discuss occurrence of fragile recognition for state-of-the-art object recognition models through results based on ImageNet (Deng et al., 2009).
Revised: In this Section, we will ﬁrst discuss occurrence of fragile recognition for state-of-the-art object recognition models through results based on ImageNet (Deng et al., 2009).

Original: Then, we analyze if data augmentation and regularization help reduce fragile recognition through experiments in CIFARKrizhevsky (2009). 
Revised: Then, we analyze if data augmentation and regularization help reduce fragile recognition through experiments in CIFARKrizhevsky (2009). [Figure]

Original: 3. F RAGILE RECOGNITION IMAGES FOR STATE - OF - THE - ART DNN S IN I MAGE N ET
Revised: 3. F RAGILE RECOGNITION IMAGES FOR STATE - OF - THE - ART DNN S IN I MAGE N ET

Original: The following experiments are performed on 500 images, randomly sampled from ImageNet’s validation set, which consists of 50 , 000 images and ground-truth object bound-in boxes (Deng et al., 2009).
Revised: The following experiments are performed on 500 images, randomly sampled from ImageNet’s validation set, which consists of 50 , 000 images and ground-truth object bound-in boxes (Deng et al., 2009).

Original: The images are sampled from 10 supercategories (dog, snake, monkey, ﬁsh, vegetable, musical[Figure] instrument, boat, land vehicle, drinks, furniture), each of which covers a set of ImageNet categories.
Revised: The images are sampled from 10 supercategories (dog, snake, monkey, ﬁsh, vegetable, musical instrument, boat, land vehicle, drinks, furniture), each of which covers a set of ImageNet categories.

Original: Correctness is measured as top- 5 accuracy, which is commonly used in ImageNet.
Revised: Correctness is measured as top- 5 accuracy, which is commonly used in ImageNet.

Original: FRIs are extracted for three architectures: VGG-16 (Simonyan & Zisserman, 2015), Inception (Szegedy et al., 2014), and ResNet (He et al., 2016).
Revised: FRIs are extracted for three architectures: VGG-16 (Simonyan & Zisserman, 2015), Inception (Szegedy et al., 2014), and ResNet (He et al., 2016).

Original: Experiments are run in eight K80 NVIDIA GPUs.
Revised: Experiments are run in eight K80 NVIDIA GPUs.

Original: The exhaustive grid search takes about 5 minutes for one image using all eight GPUs.
Revised: The exhaustive grid search takes about 5 minutes for one image using all eight GPUs.

Original: In the following paragraphs, we report the results of the experiments.
Revised: In the following paragraphs, we report the results of the experiments.

Original: We evaluate how much a DNN is affected by fragile recognition by quantifying the proportion of possible image regions that are FRIs, i .
Revised: We evaluate how much a DNN is affected by fragile recognition by quantifying the proportion of possible image regions that are FRIs, i .

Original: e .
Revised: e .

Original: we quantify the amount of regions affected by a shift or shrink.
Revised: we quantify the amount of regions affected by a shift or shrink.

Original: Recall that we consider a region to be an FRI when the classiﬁcation changes from correct to incorrect, and also from incorrect to correct.
Revised: Recall that we consider a region to be an FRI when the classiﬁcation changes from correct to incorrect, and also from incorrect to correct.

Original: As expected, we found that both of these cases are approximately equally probable under all tested conditions.
Revised: As expected, we found that both of these cases are approximately equally probable under all tested conditions.

Original: In Figure 5, we show the percentage of shrink FRIs in the image for a given region size, P ; in
Revised: In Figure 5, we show the percentage of shrink FRIs in the image for a given region size, P ; in

Original: Figure A.1 we show the same for shift FRIs.
Revised: Figure A.1 we show the same for shift FRIs.

Original: All networks are signiﬁcantly affected by FRIs, with
Revised: All networks are signiﬁcantly affected by FRIs, with

Original: ResNet being the most: almost one out of 50 regions of a size that covers almost the entire image ( P = 80% ) can affect ResNet.
Revised: ResNet being the most: almost one out of 50 regions of a size that covers almost the entire image ( P = 80% ) can affect ResNet.

Original: When the size of the image region is P = 20% , FRIs are even more frequent ( 8 times more), i .
Revised: When the size of the image region is P = 20% , FRIs are even more frequent ( 8 times more), i .

Original: e .
Revised: e .

Original: almost two out of 25 regions in the image is a loose FRI for ResNet.
Revised: almost two out of 25 regions in the image is a loose FRI for ResNet.

Original: Comparing Figure 5a and Figure 5b, we see that the proportion of strict FRIs is less than that of loose FRIs because of the more stringent deﬁnition.
Revised: Comparing Figure 5a and Figure 5b, we see that the proportion of strict FRIs is less than that of loose FRIs because of the more stringent deﬁnition.

Original: The results show that there are many regions in an image for which the network is very sensitive to slight changes in the region, eg . one out of 250 image regions of size P = 20% will be misclassiﬁed by ResNet when the region is slightly shrunk.
Revised: The results show that there are many regions in an image for which the network is very sensitive to slight changes in the region, eg . one out of 250 image regions of size P = 20% will be misclassiﬁed by ResNet when the region is slightly shrunk.

Original: Recall that strict shrink FRIs are the equivalent case of minimal images in humans.
Revised: Recall that strict shrink FRIs are the equivalent case of minimal images in humans.

Original: Thus, these results demonstrate the existence of many minimal images analogs in DNNs.
Revised: Thus, these results demonstrate the existence of many minimal images analogs in DNNs.

Original: Note that smaller FRIs (lower P ) are much more frequent than larger ones, as observed across the different network architectures and types of FRIs.
Revised: Note that smaller FRIs (lower P ) are much more frequent than larger ones, as observed across the different network architectures and types of FRIs.

Original: This trend is expected: one-pixel shifts and two-pixel shrinks are proportionally larger changes for smaller regions, and larger regions generally allow for more high-level features to be included.
Revised: This trend is expected: one-pixel shifts and two-pixel shrinks are proportionally larger changes for smaller regions, and larger regions generally allow for more high-level features to be included.

Original: We veriﬁed that FRIs are not an artifact of the algorithm that resizes the region to the size required by the DNN ( 224 × 224 pixels for VGG-16).
Revised: We veriﬁed that FRIs are not an artifact of the algorithm that resizes the region to the size required by the DNN ( 224 × 224 pixels for VGG-16).

Original: We took regions of side length 224 and removed any resizing before input, and we observed that this procedure produces the same results we reported.
Revised: We took regions of side length 224 and removed any resizing before input, and we observed that this procedure produces the same results we reported.

Original: Besides the qualitative examples in Figure 1 and Figure 2, we display a more varied set of qualitative examples in Figure A.10.
Revised: Besides the qualitative examples in Figure 1 and Figure 2, we display a more varied set of qualitative examples in Figure A.10.

Original: We observe that FRIs are usually located within object boundaries butcan also be found in the background.
Revised: We observe that FRIs are usually located within object boundaries but

Original: This is because DNNs are able to recognize regions that only contain background (Zhu et al., 2016), as they have been shown to exploit dataset biases.

Original: In Figure A.9 we display the DNN’s output conﬁdence in the true class for individual regions in map form.

Original: We see that sharp drops in conﬁdence are frequent within an image, providing a basis for minimal images.

Original: Finally, in Figure A.11 we show qualitative examples of the activation maps at different layers of the DNN of the correctly classiﬁed crop and its shifted version.

Original: These examples

Original: Region size (pixels)       %
Revised: Region size (pixels)       %

Original: Loo s e s h i ft
Revised: Loo s e s h i ft

Original: F R I s
Revised: F R I s

Original: Loose shift decrease with more augmentation
Revised: Loose shift [Figure] more augmentation

Original: Sizes at Training 28 pix.
Revised: Sizes at Training 28 pix.

Original: 24 pix.
Revised: 24 pix.

Original: 20 pix.
Revised: 20 pix.

Original: 16 pix.
Revised: 16 pix.

Original: 12 pix.
Revised: 12 pix.

Original: Region size (pixels)       %
Revised: Region size (pixels)       %

Original: Loo s e s h i ft
Revised: Loo s e s h i ft

Original: F R I s
Revised: F R I s

Original: Loose shift decrease with regularizers Regularizer Non Regularized
Revised: [Figure]

Original: Distortion Augment.
Revised: Distortion Augment

Original: Dropout
Revised: [Figure]

Original: Weight Decay All Regularizers
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: can also be found in the background.

Revised: This is because DNNs are able to recognize regions that only contain background (Zhu et al., 2016), as they have been shown to exploit dataset biases.

Revised: In Figure A.9 we display the DNN’s output conﬁdence in the true class for individual regions in map form.

Revised: We see that sharp drops in conﬁdence are frequent within an image, providing a basis for minimal images.

Revised: Finally, in Figure A.11 we show qualitative examples of the activation maps at different layers of the DNN of the correctly classiﬁed crop and its shifted version.

Original:  show what we have observed in all cases: the activation maps are imperceptibly similar at the ﬁrst layers but are clearly different at the last layers.
Revised: These examples show what we have observed in all cases: the activation maps are imperceptibly similar at the ﬁrst layers but are clearly different at the last layers.

Original: Finally, we show qualitative examples to illustrate that FRIs are a concern for computer vision systems based on similar DNNs architectures.
Revised: Finally, we show qualitative examples to illustrate that FRIs are a concern for computer vision systems based on similar DNNs architectures.

Original: We conducted a test on detection algorithms, to validate that
Revised: We conducted a test on detection algorithms, to validate that

Original: FRIs dramatically affect both the location and the label of the detected objects.
Revised: FRIs dramatically affect both the location and the label of the detected objects.

Original: In Figure A.12 we show FRIs for the widely used object detector called “YOLO” (Redmon & Farhadi, 2017).
Revised: In Figure A.12 we show FRIs for the widely used object detector called “YOLO” (Redmon & Farhadi, 2017).

Original: 3. F RAGILE RECOGNITION WITH DATA AUGMENTATION AND REGULARIZATION
Revised: 3. F RAGILE RECOGNITION WITH DATA AUGMENTATION AND REGULARIZATION

Original: We now investigate if data augmentation and regularization help alleviate fragile recognition.
Revised: We now investigate if data augmentation and regularization help alleviate fragile recognition.

Original: In this experiment we use the CIFAR-10 dataset (Krizhevsky, 2009), which contains 10 object categories, 50 , 000 training images and 10 , 000 testing images of size 32 × 32 pixels.
Revised: In this experiment we use the CIFAR-10 dataset (Krizhevsky, 2009), which contains 10 object categories, 50 , 000 training images and 10 , 000 testing images of size 32 × 32 pixels.

Original: The evaluation criteria is top- 1 accuracy.
Revised: The evaluation criteria is top- 1 accuracy.

Original: We use CIFAR-10 for convenience and without loss of generality, since the data augmentation and regularization we test are used in the models tested in ImageNet.
Revised: We use CIFAR-10 for convenience and without loss of generality, since the data augmentation and regularization we test are used in the models tested in ImageNet.

Original: We reproduce the AlexNet version for CIFAR-10 introduced by Zhang et al.
Revised: We reproduce the AlexNet version for CIFAR-10 introduced by Zhang et al.

Original: (2016), which consists of two convolutional-pooling-normalization layers followed by two fully connected layers.
Revised: (2016), which consists of two convolutional-pooling-normalization layers followed by two fully connected layers.

Original: In the following, all regularizers and data augmentation are turned off unless stated otherwise.
Revised: In the following, all regularizers and data augmentation are turned off unless stated otherwise.

Original: We will analyze the impact of each to fragile recognition.
Revised: We will analyze the impact of each to fragile recognition.

Original: For data augmentation, we augment the training dataset with FRI regions of at least a given size.
Revised: For data augmentation, we augment the training dataset with FRI regions of at least a given size.

Original: For regularizers, we add weight decay, dropout, and distortions (e.g. reﬂections, altered brightness, altered contrast).
Revised: For regularizers, we add weight decay, dropout, and distortions (e.g. reﬂections, altered brightness, altered contrast).

Original: Both the data augmentation and the regularizers improve the accuracy of the DNN for the different crop sizes (Figure A.2).
Revised: Both the data augmentation and the regularizers improve the accuracy of the DNN for the different crop sizes (Figure A.2).

Original: Figure 6 shows results for loose shift FRIs (see Figure A. for shrink FRIs); both data augmentation and regularization have a clear impact on FRI occurrence in all cases.
Revised: Figure 6 shows results for loose shift FRIs (see Figure A. for shrink FRIs); both data augmentation and regularization have a clear impact on FRI occurrence in all cases.

Original: The largest general improvement comes for the 12-pixel region size, as data augmentation and regularization both lead to decrease of more than 12% of FRIs.
Revised: The largest general improvement comes for the 12-pixel region size, as data augmentation and regularization both lead to decrease of more than 12% of FRIs.

Original: For regularizers speciﬁcally, dropout provides the most individual improvement.
Revised: For regularizers speciﬁcally, dropout provides the most individual improvement.

Original: However, all of these generalization efforts still allow a high failure rate.
Revised: However, all of these generalization efforts still allow a high failure rate.

Original: They are also unable to bring the FRI effect to the human level, as the DNN
Revised: They are also unable to bring the FRI effect to the human level, as the DNN

Original: FRIs are indistinguishable for humans.
Revised: FRIs are indistinguishable for humans.

Original: This is further discussed in Section 5.
Revised: This is further discussed in Section 5.

Original: In ImageNet, smaller region sizes lead to higher FRI occurrence.
Revised: In ImageNet, smaller region sizes lead to higher FRI occurrence.

Original: In CIFAR-10, FRI occurrence in very small regions (smaller than 20 pixels) decreases with size.
Revised: In CIFAR-10, FRI occurrence in very small regions (smaller than 20 pixels) decreases with size.

Original: This is because 12- and 16-pixel regions are prohibitively small, so the number of correctly classiﬁed regions is severely reduced; consequently, there are fewer opportunities for FRIs to occur at all. 
Revised: This is because 12- and 16-pixel regions are prohibitively small, so the number of correctly classiﬁed regions is severely reduced; consequently, there are fewer opportunities for FRIs to occur at all. [Figure]

Original:  F RAGILE RECOGNITION IS NOT LACK OF O BJECT L OCATION INVARIANCE
Revised:  F RAGILE RECOGNITION IS NOT LACK OF O BJECT L OCATION INVARIANCE

Original: We now focus on the relationship of fragile recognition with recent works that show that DNNs are affected by small changes of the object location, scale and orientation (Engstrom et al., 2017; Azulay & Weiss, 2018).
Revised: We now focus on the relationship of fragile recognition with recent works that show that DNNs are affected by small changes of the object location, scale and orientation (Engstrom et al., 2017; Azulay & Weiss, 2018).

Original: The phenomenon described in previous works is referred to as lack of invariance[Figure] due to afﬁne transformations of the object.
Revised: The phenomenon described in previous works is referred to as lack of invariance due to afﬁne transformations of the object.

Original: Here, we focus only on location changes, as it will allow to distinguish FRIs and these previous works.
Revised: Here, we focus only on location changes, as it will allow to distinguish FRIs and these previous works.

Original: The procedure to evaluate location invariance is the following, quoting from Azulay & Weiss (2018): “we embed the original image in a larger image and shift it in the image plane (while ﬁlling in the rest of the image with a simple inpainting procedure)”.
Revised: The procedure to evaluate location invariance is the following, quoting from Azulay & Weiss (2018): “we embed the original image in a larger image and shift it in the image plane (while ﬁlling in the rest of the image with a simple inpainting procedure)”.

Original: The embedding of the image can also be done with a black empty background as in (Engstrom et al., 2017), or using videos in which the background is static and only the object is moving (Azulay & Weiss, 2018).
Revised: The embedding of the image can also be done with a black empty background as in (Engstrom et al., 2017), or using videos in which the background is static and only the object is moving (Azulay & Weiss, 2018).

Original: In fragile recognition, both the object and the background, i .
Revised: In fragile recognition, both the object and the background, i .

Original: e .
Revised: e .

Original: the entire image, change due to the shift or shrink of the image region.
Revised: the entire image, change due to the shift or shrink of the image region.

Original: In previous works, only the object changes location.
Revised: In previous works, only the object changes location.

Original: We introduce an experiment that reveals that this subtle difference makes fragile recognition an independent and more complex phenomenon than lack of location invariance.
Revised: We introduce an experiment that reveals that this subtle difference makes fragile recognition an independent and more complex phenomenon than lack of location invariance.

Original: Pooling induces location invariance.
Revised: Pooling induces location invariance.

Original: DNN architectures with large pooling regions are known to induce invariance to the object location, cf .
Revised: DNN architectures with large pooling regions are known to induce invariance to the object location, cf .

Original: (Anselmi et al., 2015).
Revised: (Anselmi et al., 2015).

Original: A pooling layer operates independently for each feature by extracting the maximum activation across the spatial dimensions.
Revised: A pooling layer operates independently for each feature by extracting the maximum activation across the spatial dimensions.

Original: For a pooling region of the same size as the full image, the network response is invariant to the object position within the image frame.
Revised: For a pooling region of the same size as the full image, the network response is invariant to the object position within the image frame.

Original: This is under the assumption that there is enough spatial resolution in the responses in order to guarantee that the responses after the shift do not vary except for the shift.
Revised: This is under the assumption that there is enough spatial resolution in the responses in order to guarantee that the responses after the shift do not vary except for the shift.

Original: See (Azulay & Weiss, 2018) for a detailed explanation.
Revised: See (Azulay & Weiss, 2018) for a detailed explanation.

Original: Note that this assumption is not fulﬁlled in FRIs, as a shift of the image region introduces and removes patterns in the borders of the image, which can produce changes in the responses after pooling.
Revised: Note that this assumption is not fulﬁlled in FRIs, as a shift of the image region introduces and removes patterns in the borders of the image, which can produce changes in the responses after pooling.

Original: We now show this experimentally.
Revised: We now show this experimentally.

Original: Experimental results.
Revised: Experimental results.

Original: We evaluate the network trained on CIFAR-10 and introduced in the previous section with different pooling region sizes for the second pooling layer.
Revised: We evaluate the network trained on CIFAR-10 and introduced in the previous section with different pooling region sizes for the second pooling layer.

Original: These sizes range from three pixels to the entire image; theoretically, this should produce full location invariance.
Revised: These sizes range from three pixels to the entire image; theoretically, this should produce full location invariance.

Original: In order to guarantee sufﬁcient spatial resolution to obtain location invariance, we use a stride of one pixel for the convolutional layers and add padding.
Revised: In order to guarantee sufﬁcient spatial resolution to obtain location invariance, we use a stride of one pixel for the convolutional layers and add padding.

Original: Adding larger pooling regions does not reduce network accuracy (see Figure A.5a), but as we show next, it massively reduces the lack of location invariance.
Revised: Adding larger pooling regions does not reduce network accuracy (see Figure A.5a), but as we show next, it massively reduces the lack of location invariance.

Original: We evaluate the DNN by embedding the object in a black background at all possible locations, and we quantify for how many locations a one-pixel shift of the object location produces a change in the output of the DNN (similar to the method in Section 2).
Revised: We evaluate the DNN by embedding the object in a black background at all possible locations, and we quantify for how many locations a one-pixel shift of the object location produces a change in the output of the DNN (similar to the method in Section 2).

Original: In this experiment we evaluate changes to the output label and not correctness, in order to accurately match the deﬁnition of invariance.
Revised: In this experiment we evaluate changes to the output label and not correctness, in order to accurately match the deﬁnition of invariance.

Original: Figure 7a shows that as expected, the lack of invariance is dramatically reduced when using large pooling regions.
Revised: Figure 7a shows that as expected, the lack of invariance is dramatically reduced when using large pooling regions.

Original: For embedded 12-pixel images, 75% of the one-pixel shifts affect the network with small pooling regions, while only 10% of the one-pixel shifts affect the output of the network.
Revised: For embedded 12-pixel images, 75% of the one-pixel shifts affect the network with small pooling regions, while only 10% of the one-pixel shifts affect the output of the network.

Original: Pooling is not completely location invariant due to boundary effects near the perimeter of the image.
Revised: Pooling is not completely location invariant due to boundary effects near the perimeter of the image.

Original: We show qualitative examples of this reduction in Figure A.7.
Revised: We show qualitative examples of this reduction in Figure A.7.

Original: In Figure 7b, we see that increasing the pooling size only slightly decreases FRIs.
Revised: In Figure 7b, we see that increasing the pooling size only slightly decreases FRIs.

Original: The region size with a larger decrease is 12 pixels (note that the embedding size can not be directly compared with the region size of FRIs).
Revised: The region size with a larger decrease is 12 pixels (note that the embedding size can not be directly compared with the region size of FRIs).

Original: For this region size, there are approximately 10% less FRIs, but a signiﬁcant amount still remain (about 65% of the regions are FRIs).
Revised: For this region size, there are approximately 10% less FRIs, but a signiﬁcant amount still remain (about 65% of the regions are FRIs).

Original: Since the pooling mechanisms that largely reduced the lack of invariance are not effective for fragile recognition, we can conclude that FRIs are a more complex phenomenon than lack of location invariance.
Revised: Since the pooling mechanisms that largely reduced the lack of invariance are not effective for fragile recognition, we can conclude that FRIs are a more complex phenomenon than lack of location invariance.

Original: Note that for small region sizes the amount of FRIs increases, which is different from what we observe in the data augmentation and regularization experiment in the previous section.
Revised: Note that for small region sizes the amount of FRIs increases, which is different from what we observe in the data augmentation and regularization experiment in the previous section.

Original: This is because in Figure 7b we report change in the output label rather than change in the correctness as in previous experiments.
Revised: This is because in Figure 7b we report change in the output label rather than change in the correctness as in previous experiments.

Original: See Figure A.5b for the effect of pooling on FRI occurrence, which is in accordance to previous results.
Revised: See Figure A.5b for the effect of pooling on FRI occurrence, which is in accordance to previous results.

Original: Finally, we control that the differences we observe between lack of invariance and FRIs are not caused by use of zero-padding in one case and not the other.
Revised: Finally, we control that the differences we observe between lack of invariance and FRIs are not caused by use of zero-padding in one case and not the other.

Original: In Figure A.6, we evaluate FRIs with zero-padding instead of up-scaling the cropped region.
Revised: In Figure A.6, we evaluate FRIs with zero-padding instead of up-scaling the cropped region.

Original: The results support the same conclusions as for FRIs with up-scaling.
Revised: The results support the same conclusions as for FRIs with up-scaling.

Original:  C OMPARING FRAGILE RECOGNITION IN H UMANS AND DNN S
Revised:  C OMPARING FRAGILE RECOGNITION IN H UMANS AND DNN S

Original: In this section, we further compare the FRIs found for DNNs with the human minimal images found by Ullman et al.
Revised: In this section, we further compare the FRIs found for DNNs with the human minimal images found by Ullman et al.

Original: Recall that minimal images are equivalent to strict shrink FRIs of small size.
Revised: Recall that minimal images are equivalent to strict shrink FRIs of small size.

Original: Both humans and DNNs are susceptible to small image changes, but these two sets of fragile images are not necessarily the same in humans and in DNNs.
Revised: Both humans and DNNs are susceptible to small image changes, but these two sets of fragile images are not necessarily the same in humans and in DNNs.

Original: As shown by Ullman et al.
Revised: As shown by Ullman et al.

Original: ; Ben-Yosef et al.
Revised: ; Ben-Yosef et al.

Original: (2018), when DNNs are trained and tested on human minimal images, i .
Revised: (2018), when DNNs are trained and tested on human minimal images, i .

Original: e . images that humans can still recognize, DNNs are unable to recognize the objects.
Revised: e . images that humans can still recognize, DNNs are unable to recognize the objects.

Original: Here we show the converse, namely that when humans are tested on DNN fragile recognition images, they do not exhibit the same fragile response.
Revised: Here we show the converse, namely that when humans are tested on DNN fragile recognition images, they do not exhibit the same fragile response.

Original: This can be seen in our qualitative examples: DNN fragile recognition images and their incorrectly-classiﬁed counterparts are difﬁcult to distinguish.
Revised: This can be seen in our qualitative examples: DNN fragile recognition images and their incorrectly-classiﬁed counterparts are difﬁcult to distinguish.

Original: To verify this, we randomly sample 40 DNN shrink FRIs generated from ImageNet.
Revised: To verify this, we randomly sample 40 DNN shrink FRIs generated from ImageNet.

Original: We present the correctly-classiﬁed FRI and the slightly changed image (both resized to 100 × 100 pixels) to separate groups of 20 subjects in Mechanical Turk (Buhrmester et al., 2011).
Revised: We present the correctly-classiﬁed FRI and the slightly changed image (both resized to 100 × 100 pixels) to separate groups of 20 subjects in Mechanical Turk (Buhrmester et al., 2011).

Original: The subjects annotate the images using one of our 10 supercategory labels.
Revised: The subjects annotate the images using one of our 10 supercategory labels.

Original: The results show a small gap in correct recognition: on average, success rates among humans for DNN fragile recognition images and incorrect counterparts differ by 14 . 5% .
Revised: The results show a small gap in correct recognition: on average, success rates among humans for DNN fragile recognition images and incorrect counterparts differ by 14 . 5% .

Original: DNNs experience an average gap in conﬁdence of 56% for the same pairs.
Revised: DNNs experience an average gap in conﬁdence of 56% for the same pairs.

Original: Furthermore, we directly compare human and DNN minimal images of the same image.
Revised: Furthermore, we directly compare human and DNN minimal images of the same image.

Original: We use six of the original images from Ullman et al.
Revised: We use six of the original images from Ullman et al.

Original: (2016) that were used to ﬁnd human minimal images.
Revised: (2016) that were used to ﬁnd human minimal images.

Original: We identify the DNN FRIs for these.
Revised: We identify the DNN FRIs for these.

Original: Comparing DNN FRIs and human minimal images of the same region size, P , we ﬁnd two additional differences: ﬁrst, DNNs have more FRIs ( 5 . 3 minimal images per image for humans on average, 13 . 4 for DNNs).
Revised: Comparing DNN FRIs and human minimal images of the same region size, P , we ﬁnd two additional differences: ﬁrst, DNNs have more FRIs ( 5 . 3 minimal images per image for humans on average, 13 . 4 for DNNs).

Original: Second, DNN FRIs differ in location within the object region (Intersection Over Union between human minimal image maps and DNN FRI maps was small, < 6% for all tested images).
Revised: Second, DNN FRIs differ in location within the object region (Intersection Over Union between human minimal image maps and DNN FRI maps was small, < 6% for all tested images).

Original: For example, while human minimal images are centered in meaningful object parts (e.g. the eagle head and wing in Figure A.8 row 1, column 1), DNN FRIs contain mostly a background set of pixels (see Figure A.8 row 1, column 2).
Revised: For example, while human minimal images are centered in meaningful object parts (e.g. the eagle head and wing in Figure A.8 row 1, column 1), DNN FRIs contain mostly a background set of pixels (see Figure A.8 row 1, column 2).

Original: Finally, we control that current DNNs architectures that are based on human vision are also severely affected by FRIs.
Revised: Finally, we control that current DNNs architectures that are based on human vision are also severely affected by FRIs.

Original: This is shown in Figure A.4 for the recently introduced multi-scale eccentricity dependent DNN that mimics these aspects of human vision (Chen et al., 2017).
Revised: This is shown in Figure A.4 for the recently introduced multi-scale eccentricity dependent DNN that mimics these aspects of human vision (Chen et al., 2017).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have demonstrated that both humans and DNNs are affected by slight changes of the visible region.
Revised: We have demonstrated that both humans and DNNs are affected by slight changes of the visible region.

Original: Since the human visual system also exhibits similar fragile recognition behavior, this might be expected or even desired.
Revised: Since the human visual system also exhibits similar fragile recognition behavior, this might be expected or even desired.

Original: Our results have revealed that the fragile recognition level in humans is fundamentally different from the one in DNNs in terms of size, position and frequency.
Revised: Our results have revealed that the fragile recognition level in humans is fundamentally different from the one in DNNs in terms of size, position and frequency.

Original: Furthermore, data augmentation, regularization and larger-size pooling regions alleviate fragile recongition in
Revised: Furthermore, data augmentation, regularization and larger-size pooling regions alleviate fragile recongition in

Original: DNNs, but are not sufﬁcient to close the gap with humans.
Revised: DNNs, but are not sufﬁcient to close the gap with humans.

Original: Thus, making DNNs robust like humans remains a challenge for the community.
Revised: Thus, making DNNs robust like humans remains a challenge for the community.

Original: Finally, we have shown that fragile recognition is a more complex phenomenon than object location invariance, which exposes new concerns of the recognition ability of DNNs in natural images, even without adversarial patterns being introduced.
Revised: Finally, we have shown that fragile recognition is a more complex phenomenon than object location invariance, which exposes new concerns of the recognition ability of DNNs in natural images, even without adversarial patterns being introduced A CKNOWLEDGMENTS

Revised: We are grateful to Tomaso Poggio and Shimon Ullman for helpful feedback and discussions.

Revised: This work is supported by the National Science Foundation Science and Technology Center Award

Revised: CCF-123121, and the MIT-IBM Brain-Inspired Multimedia Comprehension project.