Paper: Improved Efficiency Based on Learned Saccade and Continuous Scene Reconstruction From Foveated Visual Sampling (ICLR-2024)

Reviewer comment (Writing):
I am confused how the image information from the sequential glimpses is passed and integrated in the predictive reconstruction model. Much more space is spent on the background to the hybrid loss function than actually making explicit how the sequential image information is used to improve reconstruction.

Author response:
A: Thanks for pointing that out. We added Appendix A.1 in the revised paper to provide more explanations about the predictive reconstruction model. The model has three-layers of ConvLSTM and one convolutional layer. It predicts the reconstructed image using the current foveal-peripheral view sampled at current glimpse, and the history information that is carried in the hidden states.  The first two layers of the ConvLSTM gradually increase the depth of feature maps, while the last layer compresses all the generated feature maps into the reconstructed image.

Reviewer comment (Evaluation):
The abstract states "our model reduces the required input pixels by over 90% per frame while maintaining the same level of performance in image recognition as with the original images." I don't understand where to find support for this claim in the results. For example, in Figure 3, all subsampled models perform worse than the original. The data in Figure 4 are coming closest to the original; is this what is meant?

Author response:
A: We apologize for not making it clear. You are correct, when we say “our model reduces the required input pixels by over 90% per frame while maintaining the same level of performance”, we mean the results in Figure 4. Figure 3 shows the results only for image reconstruction without trained saccade, while Figure 4 shows the results with trained saccade. Because we will sample 1/16 image at the foveal center and sample 2% pixels as peripheral vision, the number of pixels sampled at each time step is 8.25% of the original image, which corresponds to 90% of pixel reduction per frame.

Reviewer comment (Experiments):
Please clarify whether the experiments in Figure 3 are conducted with the trained saccade control model (which one?).

Author response:
A: Experiments in Figure 3 are conducted without trained saccade. We have modified the figure caption and added this information.

Reviewer comment (Writing):
There are now known to be three types of photosensitive cells: rods, cones and intrinsically-photosensitive ganglion cells [1, 8].

Author response:
A: Thanks for pointing this out. We have rewritten the relevant sentences in section 2.1 to emphasize retinal image capture and formation. The extent to which ipRGCs participate is somewhat controversial and not central to the focus of this paper.

Reviewer comment (Writing):
Heading 3.1 "Periphrl".

Author response:
A: Thank you for pointing this out. We have corrected this typo.

Reviewer comment (Evaluation):
I would like to see how the hybrid reconstruction loss changes over timestep, and not just classification accuracy.

Author response:
A: Three plots were added in Appendix A.2 to illustrate how hybrid loss, SSIM, and MSE changes with the number of glimpses on the ImageNet dataset. Data in plots were generated from three different foveal-peripheral settings with controlled saccade. The size of the fovea region is fixed to 1/16 of the original image, and the peripheral sampling proportion varies from 1% to 2%. As we can see, the hybrid loss and MSE reduces and SSIM increases with the number of glimpses.

Reviewer comment (Novelty):
While the task is novel, it lacks a convincing real-world application, as it simulates the process of multiple eye samplings without addressing practical problems.

Author response:
A: While our motivation was discussed in the abstract section, we acknowledge the oversight in not highlighting it extensively later in the paper, constrained by space limitations. Our goal is to build a pixel efficient visual system. Our approach decreases the volume of incoming pixels (from the camera) while maintaining the performance of backend visual applications and simplifies camera design and cost, increases frame rate, and reduces the energy and latency for data transmission between the camera and processor. Furthermore, previous research indicated that the texture-based foveal representation also offers higher adversarial and occlusion robustness. The integrated saccade controller and predictive reconstruction model allow us to achieve an accuracy comparable to the original image without retraining or fine tuning the functional backend, such as an image classifier.

Reviewer comment (Experiments):
The experimental comparisons are not entirely fair. The uniform control group uses an 8% sampling probability, while the 1/16+2% group differs by 0.25%, indicating an unequal amount of information that might affect performance.

Author response:
A: Thank you so much for pointing out this discrepancy. We have increased the sampling probability of the uniform control group to 8.25% and updated Figure 3 in the revised paper. We note that the new results are very similar to the old ones.

Reviewer comment (Evaluation):
Using classification model metrics to assess the quality of reconstruction is questionable, as classification tasks do not focus on texture details. If this method was to downsample the original image with the same number of sampled pixels, how much better is the method in terms of performance compared to this?

Author response:
A: Thanks for raising this concern. The goal of our work is not to just reconstruct the image, but to use fewer camera pixels to achieve a good performance in the functional backend. Image classification is just an example of backend application. Our framework can support other backend applications, such as object detection, object tracking, etc. We train the saccade model to maximize the performance of the targeted backend application.

Notably, using a saccade model trained with the feedback from an image classifier, the reconstructed images (Table 2, row 3~8) have a lower structural similarity (SSIM) to the original image, but a higher accuracy in classification. However, using a saccade model trained directly to maximize the SSIM of the reconstruction (row 2), the reconstructed images have a higher SSIM but a lower classification accuracy.  In other words, the saccade model should be trained using the performance of  targeted backend applications as a feedback.

A: In Figure 3, the data points “uniform, no conn” and “uniform” give the performance of this case.  The “uniform, no conn” down samples images to 8.25% of their original size without reconstruction; while “uniform” reconstruct the images after downsampling. As we can see, the classification accuracy of downsampled images without reconstruction is less than 10%. The accuracy increases to 75.3% and 57% for GTSRB and ImageNet dataset respectively after reconstruction. However, it is still lower than the accuracy (88.9% and 62.6%) of the reconstructed foveal-peripheral view that has the same number of pixels (the red data points in Figure 3). With a trained saccade model, the classification accuracy increases to 91.8% and 73.7% for GTSRB and ImageNet respectively.

Reviewer comment (Evaluation):
Although the paper addresses the trade-off between image quality and computational efficiency, it would be valuable to provide insights into the computational overhead of implementing the proposed model, particularly in terms of hardware and energy requirements.

Author response:
We appreciate the valuable feedback from the reviewer. A revision paper has been uploaded. Below is our response. The original comments are copied followed by our answers.

>Q1:  Can you provide more details about the computation overhead of your model and possible complications in real world applications?

A: . By leveraging foveal-peripheral representation, we can reduce the amount of required pixels by 60-70%. (Please see Table 2 for the pixel usage and test accuracy). This reduction can help to simplify the camera design, increase the frame rate, and reduce the energy for data transmission between camera and processor. The computation overhead is the extra step of image reconstruction and saccade control before the backend applications such as image classifying. The reconstruction model and saccade control model totally have 24.5M trainable parameters, which is about half of the size of ResNet101. In terms of computation complexity, in each glimpse, the reconstruction and saccade model totally have 4.3G floating point operations (FLOPs) (0.7G FLOPs for reconstruction and 3.6G FLOPs for saccade control). We did not include the computation complexity of the critic network of the saccade controller, as it is not needed in the execution time. As a reference, ResNet101 has 15.7G FLOPs per image.

During real-world applications, the foveal-peripheral view will first be sent to the reconstruction model. The hidden state of the reconstruction model will be sent to the saccade controller for saccade action and the reconstructed image will be sent to the backend model for further classification. A potential complication in real world application is that multiple glimpses (3.5 ~ 7 glimpses on average) may be needed for the backend model to reach a performance comparable to the original image. (Please note that the pixel usage reported in Table 1 is the total number of pixels used by all glimpses.) However, in a real world environment, the scene change is usually slower than the camera frame rate and the forward propagation time of the reconstruction model and saccade controller. The latency of multiple saccade movements can be hidden.

Reviewer comment (Novelty):
The paper totally fails to mention a whole branch of literature in saccade modeling. See for example [1], [2], or [3]. In particular, [2] also uses reconstruction as a guiding task. It seems true that none of the mentioned approaches focused on performance in terms of image reconstruction, but I think it is relevant to at least position the current contribution compared to those. I imagine, some of these saccade models could potentially be used in the same framework proposed by the authors here.

Author response:
>Q2: Can you better frame your contribution, and compare it to the literature in saccade modeling?
A: Thank you for pointing out prior works ( [1][2][3]) in saccade model and scanpath generation. We have cited them in the revised paper. 

Among these papers, reference [1] splits images into central path and peripheral path for saliency extraction and combines them to form a conspicuity map. The fovea center is picked based on the bottom up saliency map. This, to some degree, is similar to “Lukanov et al. (2021)” (cited in section 2.2 of our paper). The limitation of saliency based saccade is that it ignores requirements from high level applications. “Lukanov et al. (2021)” mitigated this limitation by combining the saliency information with a top-down feedback, which is the activation map from a high level classifier. This again has a problem. At the beginning of the visual process, with very few glimpses, the classification is likely to be wrong, which will lead to a wrong position for the next saccade movement. Furthermore, they trained the saccade model and the classifier together. The mutual dependency on each other makes the training difficult. While our approach selects the next saccade movement based on the hidden state of the reconstruction model, which has been pre-trained. The reward to train the saccade policy comes from an existing classifier working on reconstructed images. By training these models separately, our flow is simpler and achieves better results.

As explained above, image reconstruction is not our final goal, it is an intermediate step to facilitate the training of the saccade controller using the feedback from an existing upper level application model. Although reference [2] also considers image reconstruction, they perform it as a final goal (i.e., upper level application.) Furthermore, the input image of [2] is a foveation-blur representation generated by a Gaussian filter. Technically speaking, foveation-blur representation of the image is not sparse. Every pixel in the image still has a non-zero value, hence they still have to be transmitted and processed. We did not use the foveation-blur representation in our work because our goal is to reduce the amount of input data from the camera.

Reference [3] generates a visual scanpath for a given image using generative adversarial networks (GAN). They do not consider the foveation effect. The GAN needs the original image with full resolution to generate the scanpath. Furthermore, to train this GAN, recorded human eye movement is needed as labeled data. Therefore, the training process will be very expensive.

[1] Wloka, C., Kotseruba, I., & Tsotsos, J. K. (2018). Active fixation control to predict saccade sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3184-3193). 

[2] Schwinn, L., Precup, D., Eskofier, B., & Zanca, D. (2022). Behind the Machine’s Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attention. Transactions on Machine Learning Research. 

[3] Assens, M., Giro-i-Nieto, X., McGuinness, K., & O'Connor, N. E. (2018). PathGAN: Visual scanpath prediction with generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops (pp. 0-0).

Reviewer comment (Reproducibility):
Can you provide more details about the computation overhead of your model and possible complications in real world applications?

Author response:
We appreciate the valuable feedback from the reviewer. A revision paper has been uploaded. Below is our response. The original comments are copied followed by our answers.

>Q1:  Can you provide more details about the computation overhead of your model and possible complications in real world applications?

A: . By leveraging foveal-peripheral representation, we can reduce the amount of required pixels by 60-70%. (Please see Table 2 for the pixel usage and test accuracy). This reduction can help to simplify the camera design, increase the frame rate, and reduce the energy for data transmission between camera and processor. The computation overhead is the extra step of image reconstruction and saccade control before the backend applications such as image classifying. The reconstruction model and saccade control model totally have 24.5M trainable parameters, which is about half of the size of ResNet101. In terms of computation complexity, in each glimpse, the reconstruction and saccade model totally have 4.3G floating point operations (FLOPs) (0.7G FLOPs for reconstruction and 3.6G FLOPs for saccade control). We did not include the computation complexity of the critic network of the saccade controller, as it is not needed in the execution time. As a reference, ResNet101 has 15.7G FLOPs per image.

During real-world applications, the foveal-peripheral view will first be sent to the reconstruction model. The hidden state of the reconstruction model will be sent to the saccade controller for saccade action and the reconstructed image will be sent to the backend model for further classification. A potential complication in real world application is that multiple glimpses (3.5 ~ 7 glimpses on average) may be needed for the backend model to reach a performance comparable to the original image. (Please note that the pixel usage reported in Table 1 is the total number of pixels used by all glimpses.) However, in a real world environment, the scene change is usually slower than the camera frame rate and the forward propagation time of the reconstruction model and saccade controller. The latency of multiple saccade movements can be hidden.

Reviewer comment (Novelty):
Can you better frame your contribution, and compare it to the literature in saccade modeling?

Author response:
>Q2: Can you better frame your contribution, and compare it to the literature in saccade modeling?
A: Thank you for pointing out prior works ( [1][2][3]) in saccade model and scanpath generation. We have cited them in the revised paper. 

Among these papers, reference [1] splits images into central path and peripheral path for saliency extraction and combines them to form a conspicuity map. The fovea center is picked based on the bottom up saliency map. This, to some degree, is similar to “Lukanov et al. (2021)” (cited in section 2.2 of our paper). The limitation of saliency based saccade is that it ignores requirements from high level applications. “Lukanov et al. (2021)” mitigated this limitation by combining the saliency information with a top-down feedback, which is the activation map from a high level classifier. This again has a problem. At the beginning of the visual process, with very few glimpses, the classification is likely to be wrong, which will lead to a wrong position for the next saccade movement. Furthermore, they trained the saccade model and the classifier together. The mutual dependency on each other makes the training difficult. While our approach selects the next saccade movement based on the hidden state of the reconstruction model, which has been pre-trained. The reward to train the saccade policy comes from an existing classifier working on reconstructed images. By training these models separately, our flow is simpler and achieves better results.

As explained above, image reconstruction is not our final goal, it is an intermediate step to facilitate the training of the saccade controller using the feedback from an existing upper level application model. Although reference [2] also considers image reconstruction, they perform it as a final goal (i.e., upper level application.) Furthermore, the input image of [2] is a foveation-blur representation generated by a Gaussian filter. Technically speaking, foveation-blur representation of the image is not sparse. Every pixel in the image still has a non-zero value, hence they still have to be transmitted and processed. We did not use the foveation-blur representation in our work because our goal is to reduce the amount of input data from the camera.

Reference [3] generates a visual scanpath for a given image using generative adversarial networks (GAN). They do not consider the foveation effect. The GAN needs the original image with full resolution to generate the scanpath. Furthermore, to train this GAN, recorded human eye movement is needed as labeled data. Therefore, the training process will be very expensive.

[1] Wloka, C., Kotseruba, I., & Tsotsos, J. K. (2018). Active fixation control to predict saccade sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3184-3193). 

[2] Schwinn, L., Precup, D., Eskofier, B., & Zanca, D. (2022). Behind the Machine’s Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attention. Transactions on Machine Learning Research. 

[3] Assens, M., Giro-i-Nieto, X., McGuinness, K., & O'Connor, N. E. (2018). PathGAN: Visual scanpath prediction with generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops (pp. 0-0).

Reviewer comment (Experiments):
I am confused on how the system is trained to do reconstruction. Is it doing the reconstruction from the same image and "testing on the training set"? Otherwise, I am surprised the first auto-completion of the image is surprisingly quite well without any prior knowledge of the underlying geometry of the visual stimulus.

Author response:
A:  Thank you for raising this question. The reconstruction model is trained and tested on traning set and testing set respectively. It is not “testing on the training set”. Although the model has no prior knowledge of the geometry of the object, it does receive inputs from the peripheral region. The peripheral sampling portion is 2% for the Monalisa image shown in Figure 2 and 5% for the example images in the Appendix in the original paper. Although the peripheral input is too sparse for humans to recognize any geometry information, the reconstruction model is capable of roughly completing the image by filling up the gaps. In the revised paper, we added more examples to the Appendix A.3 to showcase the reconstruction. In those examples, the peripheral sampling portion varies from 1% to 2% to 5%. As we can see, even with a 2% sampling portion, the model can already roughly detect the shape of the object after the 1st glimpse. The examples also include the reconstruction of Monalisa, which is not part of the imageNet data set, as evidence that our testing data is different from the training data.

Reviewer comment (Novelty):
There are a set of missing papers that the authors should add and/or discuss in this work. While none of these papers directly attack the problem of using foveation as a tool for reconstruction, many of such works discuss the complementary theory of foveation having a representational goal in addition to purely optimizing for metabolic cost, thus limiting the impact of the authors through this paper.

Author response:
A: Thank you so much for pointing out these references. We have cited them in our revised version. Due to the space limit, we are not able to discuss and compare to each one of them in the paper. We need to point out that, in general, these works have a different focus compared to ours. 

Thank you for the  references [1][5][6][7][8]. Indeed they provide useful background in showing that the visual crowding effect in the texture-based fovea representation of the primate retina provides adversarial robustness and efficiency. This is another reason why it is important to study foveal-peripheral visual systems. We have mentioned this in section 1 in the revised paper. However, the goal of these papers is to demonstrate a property of foveated images, while our goal is to build a processing pipeline and saccade mechanism for the foveated images.

Reference [2] shares some similarity with (Jaramillo-Avila & Anderson, 2019) cited in section 2.2 of our paper, in the sense that both works trained a backend model for a specific application with foveal-peripheral input. As we mentioned in the paper, the issue with training a new backend model for the foveal-peripheral input is the need for a large amount of labeled data in order to reach the state-of-art performance. By using unsupervised training to reconstruct the image, we can leverage the existing backend model without re-train them.

Reference [3] is somewhat similar to (Chen, 2021) that we cited in section 2.2. Both papers take several decision steps to find the location of the foveal center and, for each image, only one foveal center is identified. Furthermore, the goal of [3] is to locate the target as quickly as possible. 
 
[1]Deza & Konkle. ArXiv, 2021. Emergent Properties of Foveated Perceptual Systems.

[2]Wang & Cottrell. Journal of Vision, 2017. Central and peripheral vision for scene recognition: A neurocomputational modeling exploration.

[3]Cheung, Weiss & Olshausen. ICLR 2017. Emergence of foveal image sampling from learning to attend in visual scenes

[4]Wang, Bovik, Sheik & Simoncelli. IEEE TIP, 2004. Image quality assessment: from error visibility to structural similarity (SSIM).
 interesting, and include

[5]Gant, Banburski & Deza. SVRHM, 2022. Evaluating the adversarial robustness of a foveated texture transform module in a CNN.

[6]Reddy, Banburski, Pant & Poggio. NeurIPS 2020. Biologically inspired mechanisms for adversarial robustness

[7]Wang, Mayo, Deza, Barbu & Conwell. SVRHM, 2021. On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation

[8]Harrington & Deza. ICLR, 2022. Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks

Reviewer comment (Novelty):
Key missing critical references:  
- Deza & Konkle. ArXiv, 2021. Emergent Properties of Foveated Perceptual Systems.  
- Wang & Cottrell. Journal of Vision, 2017. Central and peripheral vision for scene recognition: A neurocomputational modeling exploration.  
- Cheung, Weiss & Olshausen. ICLR 2017. Emergence of foveal image sampling from learning to attend in visual scenes.

Author response:
A: Thank you so much for pointing out these references. We have cited them in our revised version. Due to the space limit, we are not able to discuss and compare to each one of them in the paper. We need to point out that, in general, these works have a different focus compared to ours. 

Thank you for the  references [1][5][6][7][8]. Indeed they provide useful background in showing that the visual crowding effect in the texture-based fovea representation of the primate retina provides adversarial robustness and efficiency. This is another reason why it is important to study foveal-peripheral visual systems. We have mentioned this in section 1 in the revised paper. However, the goal of these papers is to demonstrate a property of foveated images, while our goal is to build a processing pipeline and saccade mechanism for the foveated images.

Reference [2] shares some similarity with (Jaramillo-Avila & Anderson, 2019) cited in section 2.2 of our paper, in the sense that both works trained a backend model for a specific application with foveal-peripheral input. As we mentioned in the paper, the issue with training a new backend model for the foveal-peripheral input is the need for a large amount of labeled data in order to reach the state-of-art performance. By using unsupervised training to reconstruct the image, we can leverage the existing backend model without re-train them.

Reference [3] is somewhat similar to (Chen, 2021) that we cited in section 2.2. Both papers take several decision steps to find the location of the foveal center and, for each image, only one foveal center is identified. Furthermore, the goal of [3] is to locate the target as quickly as possible. 
 
[1]Deza & Konkle. ArXiv, 2021. Emergent Properties of Foveated Perceptual Systems.

[2]Wang & Cottrell. Journal of Vision, 2017. Central and peripheral vision for scene recognition: A neurocomputational modeling exploration.

[3]Cheung, Weiss & Olshausen. ICLR 2017. Emergence of foveal image sampling from learning to attend in visual scenes

[4]Wang, Bovik, Sheik & Simoncelli. IEEE TIP, 2004. Image quality assessment: from error visibility to structural similarity (SSIM).
 interesting, and include

[5]Gant, Banburski & Deza. SVRHM, 2022. Evaluating the adversarial robustness of a foveated texture transform module in a CNN.

[6]Reddy, Banburski, Pant & Poggio. NeurIPS 2020. Biologically inspired mechanisms for adversarial robustness

[7]Wang, Mayo, Deza, Barbu & Conwell. SVRHM, 2021. On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation

[8]Harrington & Deza. ICLR, 2022. Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks

Reviewer comment (Novelty):
Secondary, but also important missing references:  
- Gant, Banburski & Deza. SVRHM, 2022. Evaluating the adversarial robustness of a foveated texture transform module in a CNN.  
- Reddy, Banburski, Pant & Poggio. NeurIPS 2020. Biologically inspired mechanisms for adversarial robustness.  
- Wang, Mayo, Deza, Barbu & Conwell. SVRHM, 2021. On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation.  
- Harrington & Deza. ICLR, 2022. Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks.

Author response:
A: Thank you so much for pointing out these references. We have cited them in our revised version. Due to the space limit, we are not able to discuss and compare to each one of them in the paper. We need to point out that, in general, these works have a different focus compared to ours. 

Thank you for the  references [1][5][6][7][8]. Indeed they provide useful background in showing that the visual crowding effect in the texture-based fovea representation of the primate retina provides adversarial robustness and efficiency. This is another reason why it is important to study foveal-peripheral visual systems. We have mentioned this in section 1 in the revised paper. However, the goal of these papers is to demonstrate a property of foveated images, while our goal is to build a processing pipeline and saccade mechanism for the foveated images.

Reference [2] shares some similarity with (Jaramillo-Avila & Anderson, 2019) cited in section 2.2 of our paper, in the sense that both works trained a backend model for a specific application with foveal-peripheral input. As we mentioned in the paper, the issue with training a new backend model for the foveal-peripheral input is the need for a large amount of labeled data in order to reach the state-of-art performance. By using unsupervised training to reconstruct the image, we can leverage the existing backend model without re-train them.

Reference [3] is somewhat similar to (Chen, 2021) that we cited in section 2.2. Both papers take several decision steps to find the location of the foveal center and, for each image, only one foveal center is identified. Furthermore, the goal of [3] is to locate the target as quickly as possible. 
 
[1]Deza & Konkle. ArXiv, 2021. Emergent Properties of Foveated Perceptual Systems.

[2]Wang & Cottrell. Journal of Vision, 2017. Central and peripheral vision for scene recognition: A neurocomputational modeling exploration.

[3]Cheung, Weiss & Olshausen. ICLR 2017. Emergence of foveal image sampling from learning to attend in visual scenes

[4]Wang, Bovik, Sheik & Simoncelli. IEEE TIP, 2004. Image quality assessment: from error visibility to structural similarity (SSIM).
 interesting, and include

[5]Gant, Banburski & Deza. SVRHM, 2022. Evaluating the adversarial robustness of a foveated texture transform module in a CNN.

[6]Reddy, Banburski, Pant & Poggio. NeurIPS 2020. Biologically inspired mechanisms for adversarial robustness

[7]Wang, Mayo, Deza, Barbu & Conwell. SVRHM, 2021. On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation

[8]Harrington & Deza. ICLR, 2022. Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks

Reviewer comment (Novelty):
Additional missing references include the original SSIM paper:  
- Wang, Bovik, Sheik & Simoncelli. IEEE TIP, 2004. Image quality assessment: from error visibility to structural similarity (SSIM).

Author response:
A: Thank you so much for pointing out these references. We have cited them in our revised version. Due to the space limit, we are not able to discuss and compare to each one of them in the paper. We need to point out that, in general, these works have a different focus compared to ours. 

Thank you for the  references [1][5][6][7][8]. Indeed they provide useful background in showing that the visual crowding effect in the texture-based fovea representation of the primate retina provides adversarial robustness and efficiency. This is another reason why it is important to study foveal-peripheral visual systems. We have mentioned this in section 1 in the revised paper. However, the goal of these papers is to demonstrate a property of foveated images, while our goal is to build a processing pipeline and saccade mechanism for the foveated images.

Reference [2] shares some similarity with (Jaramillo-Avila & Anderson, 2019) cited in section 2.2 of our paper, in the sense that both works trained a backend model for a specific application with foveal-peripheral input. As we mentioned in the paper, the issue with training a new backend model for the foveal-peripheral input is the need for a large amount of labeled data in order to reach the state-of-art performance. By using unsupervised training to reconstruct the image, we can leverage the existing backend model without re-train them.

Reference [3] is somewhat similar to (Chen, 2021) that we cited in section 2.2. Both papers take several decision steps to find the location of the foveal center and, for each image, only one foveal center is identified. Furthermore, the goal of [3] is to locate the target as quickly as possible. 
 
[1]Deza & Konkle. ArXiv, 2021. Emergent Properties of Foveated Perceptual Systems.

[2]Wang & Cottrell. Journal of Vision, 2017. Central and peripheral vision for scene recognition: A neurocomputational modeling exploration.

[3]Cheung, Weiss & Olshausen. ICLR 2017. Emergence of foveal image sampling from learning to attend in visual scenes

[4]Wang, Bovik, Sheik & Simoncelli. IEEE TIP, 2004. Image quality assessment: from error visibility to structural similarity (SSIM).
 interesting, and include

[5]Gant, Banburski & Deza. SVRHM, 2022. Evaluating the adversarial robustness of a foveated texture transform module in a CNN.

[6]Reddy, Banburski, Pant & Poggio. NeurIPS 2020. Biologically inspired mechanisms for adversarial robustness

[7]Wang, Mayo, Deza, Barbu & Conwell. SVRHM, 2021. On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation

[8]Harrington & Deza. ICLR, 2022. Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks

Reviewer comment (Writing):
I am struggling to know what $t_0$ is. Is it a blank image? Is it a corrupted image? Is it only a fraction/glimpse of an image?

Author response:
A: We couldn’t find where variable t0 is mentioned in the paper except in equation (4), where we tried to sum the gradient from t=0 to T-1. We must apologize that this is a typo in the equation, the summation should start from t=1 to T as the following:

\begin{equation}
    \bigtriangledown_{\theta_p} J(\theta_p) =E[\sum_{t=1}^{T}\bigtriangledown_{\theta_p}log{\pi\theta_p}(a_{t}|s_{1:t})A(s_{1:t},a_t)],
\end{equation}

If the question is about the first input at time t=0, then the first input is the foveal-peripheral view of the image with a randomly chosen fovea center. We have added this information in the revised paper.