Paper: COCO-Periph: Bridging the Gap Between Human and Machine Perception in the Periphery (ICLR-2024)

Reviewer comment (Novelty):
The paper introduces a new dataset called COCO-Periph. However, it is unclear how this dataset significantly contributes to the existing knowledge. Given that the primary focus is on modeling peripheral vision in DNNs, the dataset itself does not seem to offer substantial value to the research community. A better approach would be to release the code for the community to experiment with, rather than creating a dataset that doesn't seem to add much to the field.

Author response:
> "the dataset itself does not seem to offer substantial value to the research community"

We agree that code would be useful for further experimentation and plan to release it alongside the dataset. That said, we believe a dataset is also a valuable contribution because the peripheral vision model we would like to make available to the community takes ~5-6 hours per image to render. The entire dataset took over $6$ months to create. The procedure also depends on choosing a fixation point beforehand. To render multiple fixations at large scale during training is thus intractable. Furthermore, by providing a dataset, we are standardizing evaluation for future work. If we share only code, everyone will generate slightly different versions of the dataset, making it difficult to evaluate progress.  We have made these points more clear in the introduction.

Reviewer comment (Novelty):
If we interpret peripheral image transformation via TTM as a form of image perturbation, results are unsurprising. The observation that models perform poorly on shifted domains (in this case, TTM generated images) and become more robust after training on the new domain (COCO-Periph) is not surprising. The results are, in essence, not only predictable but do not provide an insight on peripheral vision. To be clear: the exact same results could have been obtained with ANY image transformation resulting in a domain shift.

Author response:
> "If we interpret peripheral image transformation via TTM as a form of image perturbation, results are unsurprising"

We appreciate the reviewer eW1M’s insight about domain shift, but as noted by other reviewers, our results are interesting in the broader context of human vision. When humans view TTM images, their performance is highly correlated with original images -- despite never having “trained” on this exact image perturbation. If a DNN learned the same representational properties as human vision during training, then arguably they too should be able to handle this transformation. Yet, we observe that DNNs do not perform well under this transformation. As an experiment, we explore simply fine-tuning and training models on COCO-Periph to understand this behavior. We find that these models still exhibit drops in performance greater than humans do, suggesting the behavior is not just about domain shift. In the future, we ideally will have new models that do not require extensive fine-tuning -- just as humans do not require this. With our dataset and human experiment, our long-term goal is to build new methods that do perform like humans. We see COCO-Periph as a stepping-stone toward that goal, informing how to build representations robust to a peripheral transformation. We have added some of these points in the Discussion.

Reviewer comment (Evaluation):
The paper fails to provide a clear explanation for why the proposed model, even after fine-tuning on the entire dataset, still performs significantly worse than DINO. This raises questions about the efficacy of the proposed model and whether it addresses the problem of peripheral vision adequately.

Author response:
> "The paper fails to provide a clear explanation for why the proposed model, even after fine-tuning on the entire dataset, still performs significantly worse than DINO"
>"Figure 5, which assesses the model's resilience, is presented misleadingly. It only demonstrates the model's performance across various eccentricities, which is expected, given that the model was trained on the same data ..."

First, we would like to emphasize that our main contribution is a dataset and human experiment, not a new proposed model. We include a fine-tuning experiment as an additional evaluation, but our larger goal is to uncover methods that do not require extensive optimization on TTM to perform well on this OOD transformation like humans do.

We appreciate reviewer eW1M’s request for per eccentricity psychophysics performance in Figure 5. We agree that this is useful information, and we have added a subplot of these values in Figure 5. We originally chose to report an across eccentricity value ($\mu$) because it is standard practice in psychophysics studies to fit a curve across stimulus conditions and report a critical value where performance deteriorates to 75% accuracy (Strasburger, 2020, Figure 12), and we aimed to make an apples to apples comparison between humans and models. 

Regarding the results themselves, we agree it is puzzling that the fine-tuned model out-performs DINO when it under-performs on AP. We believe this can be explained by the nature of the TTM transform and psychophysics task we propose. By design, TTM perturbs the true object location -- especially for far eccentricities. When we report AP, we do not account for this and rely on the original COCO bounding boxes. We acknowledge that this is a limitation, and so we do account for it in the psychophysics task by padding the ground truth bounding boxes (see Section 6 and Appendix A.4).

In addition, the psychophysics task itself only accounts for object localization and does not account for recognition performance like AP does. To test if this played a role in our results, we evaluated recognition alone for the RCNN in Figure 6b. Here, we see that the fine-tuning has a less noticeable improvement in performance; training from scratch even reduces performance. This indicates that object localization, rather than recognition, drives the performance boost we see in Figure 5. Indeed, peripheral vision is thought to primarily play a role in guiding where to look next rather than in recognizing object identity. For this reason, we evaluate models against humans on this metric and expect to see more improvements in this area when mimicking peripheral vision.

Reviewer comment (Experiments):
The paper includes an experiment comparing humans presented with objects at different eccentricities with a model presented with a uniform peripheral transformation obtained with TTM. This comparison is not fair. This essentially tests the model on out-of-distribution data, making the poor performance of the models obvious. In no way do I see a support for any conclusions regarding properties of the peripheral vision.

Author response:
> "The paper includes an experiment comparing humans presented with objects at different eccentricities with a model presented with a uniform peripheral transformation obtained with TTM. This comparison is not fair. This essentially tests the model on out-of-distribution data, making the poor performance of the models obvious. In no way I see a support for any conclusions regarding properties of the peripheral vision."

Based on our results and prior experiments with TTM, we again propose that it is fair to test models on TTM transformed images because humans can effectively perform tasks with them despite never having trained on them. Studying DNN performance on such OOD data is thus critical, and we show that DNNs struggle in this regard. Prior work has established for multiple tasks including object detection that human performance on original and TTM transformations strongly match one another (Ehinger & Rosenholtz). This begs the question, what features do humans rely on to perform visual tasks in their periphery? Should these be learned or hard-coded in DNNs? By showing DNNs these transformed images and comparing performance to humans, our experiment helps the community answer these questions.

Reviewer comment (Novelty):
The paper lacks a clear explanation of the practical importance of a peripheral vision model. The only advantage seems to be an increase in robustness for very related image transformations.

Author response:
> "The paper lacks a clear explanation of the practical importance of a peripheral vision model. The only advantage seems to be an increase in robustness for very related image transformations…"

We thank reviewer eW1M’s for requesting a more clear explanation for why peripheral vision is useful to model. We acknowledge that our results do not demonstrate a concrete performance boost on standard benchmarks. However, we argue that peripheral vision is useful to model because peripheral vision plays a key role in guiding visual attention in humans, enabling fast, efficient detection of features and motion over a wide visual field By modeling peripheral vision, we can leverage the human visual system as a feature selector. This has already been shown to improve performance in object recognition (Pramod 2022, Jonnalagadda, 2021, Min 2022) . Peripheral vision, however, has not been modeled in natural scene detection tasks where it arguably plays a much bigger role than recognition. We agree that we do not find a peripheral vision transformation alone to be sufficient to improve baseline original COCO performance. We believe this is an interesting contribution suggesting modeling peripheral vision is more beneficial for some tasks than others. We have included some of these points in the discussion.  

> "Can you provide more context on potential real-world applications or implications of these results?"

Please refer to our response to weakness #6.
* Peripheral vision guides human attention. It is already useful for recognition and could potentially aid performance in scene related tasks detection where peripheral vision plays a stronger role in performance 
* By modeling peripheral vision, we could predict human visual behavior. This has applications in (1) driver safety where we could predict if a person sees a hazard, (2) content memorability where we could optimize images to capture attention, (3) UI/UX to create displays that easy to view, (4) foveated rendering, and (5) compression where peripheral vision could help models perform under reduced visual information. 
* Finally, peripheral vision underlies a significant part of human visual representations. If we can successfully model it and human vision more generally, we will have learned a representation that is robust to many of the transformations that humans are robust to. Rather than training on a large set of perturbations, we could simply mimic a human visual representation that has already learned the desired robustness.
* We have expanded on real-world applications in the Discussion.

Reviewer comment (Evaluation):
An alternative explanation for the results is that the TTM is an inadequate model of human peripheral vision. If the TTM discards more information than the human visual system actually preserves, humans would be expected to have better performance at larger eccentricities than DNNs trained on COCO-Periph. This difference would not be due to the model architecture, but rather because the COCO-Periph training set distorts images too aggressively. In such a case, improving machine peripheral vision might not require testing more advanced model architectures but instead finding a better representation for human peripheral vision.

Author response:
We thank reviewer ogr4 for the thoughtful and careful review of our paper, appreciation of the strengths and contributions of our work, and for many helpful suggestions that will improve the paper’s impact.

Firstly, we thank the reviewer for pointing us to the Perry & Geisler citations which stitch together blur images for computationally-tractable fixation, similar to what we propose using uniform TTM images. We have included these citations in the manuscript.

>“An alternative explanation for the results is that the TTM is an inadequate model of human peripheral vision”

We appreciate reviewer ogr4’s thoughtful alternative explanation for our results due to TTM throwing away too much information. Indeed, like all peripheral vision models,  TTM does indeed underpredict human performance for certain stimuli and situations. Among all the peripheral models currently available, we chose TTM to create our dataset because it has been validated on the widest range of tasks compared to any similar texture-based model. It is true that the object detection task tested here differs slightly from the object detection task tested in Ehinger & Rosenholtz 2016, with more hallmarks of a discrimination task. We note that models sharing pooled TTM statistics have been widely tested on discrimination with the metamer paradigm (Freeman & Simoncelli 2011, Wallis 2019). We agree with reviewer ogr4 that there are newer models of peripheral vision that each contribute their own improvements such as style-transfer based models like (Wallis 2019, Deza 2017)). However these models focus primarily on the constraint of metamerism rather than predicting task performance which we believe is important and interesting to consider when making comparisons to DNNs. In our work, although we use object detection as an exemplar, our goal is for COCO-Periph to be applied in many other visual tasks such as search which is labeled in COCO (Chen et al., 2021), as well as scene and image recognition which could be explored using the COCO things and stuff labels (Caesar et al., 2018). We appreciate that our language did not discuss the limitations of TTM and other related models, and we have updated our manuscript to address these points. 

In weighing the reviewer’s hypothesis against that presented in the paper, we acknowledge that the limitations of current state of the art peripheral vision models like TTM could contribute in part to our results. We see this as an important point to share with readers, and have incorporated this into the updated manuscript. We argue however, that the limitations of TTM cannot completely explain our results for the following reasons:

1. While TTM throwing away too much information could create a ceiling on model performance as compared to humans, our results also demonstrate the brittle falloff of performance for machine models as compared to the gradual falloff of performance in humans (Figure 3c). TTM uses the same statistical parameters at all eccentricities, modifying only pooling region size, which should not cause a sharp performance drop between low and high eccentricities. Indeed, we show when fine-tuning models on single eccentricities boosts performance at other eccentricities (Appendix Figure 22). Therefore, it is unlikely that this sharp performance drop off seen is attributable to TTM itself, and much more likely to be TTM-transforms being out of distribution to untrained models.

2. The differences in performance between the different model architectures tested (ie DINO, RCNN, DETR etc) cannot be explained by loss of information in TTM, as these models received the same TTM-transformed input images. 

3. Relatedly, the improvement in performance seen after training and fine-tuning models on COCO-Periph demonstrates a change in the ability of even the same model architecture to process peripherally-transformed images differently. Again, the baseline, fine-tuned, and trained models receive the same TTM-transformed input images. 

4. Also related, the improvement in corruption robustness after fine-tuning and training with COCO-Periph is orthogonal to any limitations of TTM as a peripheral vision model.

5. Furthermore, our results in comparing model performance in detection vs recognition are likewise orthogonal to any limitations of TTM as a peripheral vision model.

Reviewer comment (Theory):
The paper gives the impression that the TTM and related approaches have unequivocal support as a model of human peripheral vision through statements such as: "Humans viewing these transformed images perform visual tasks with an accuracy that well predicts human performance while fixating the original images"; "This model has been tested to well predict human performance on an extensive number of behavioral tasks, including peripheral object recognition, visual search, and a variety of scene perception tasks"; "rendered images that capture the amount of information available to humans at (5◦, 10◦, 15◦, 20◦)"; and "previous work demonstrated that TTM is a close match to human peripheral vision (Ehinger & Rosenholtz, 2016), so differences in performance cannot be explained by stimuli alone." However, other work [refs 3, 4, 5, 6] supports the alternative explanation that under some circumstances humans are more sensitive to image distortions than predicted by these texture models.

Author response:
We thank reviewer ogr4 for the thoughtful and careful review of our paper, appreciation of the strengths and contributions of our work, and for many helpful suggestions that will improve the paper’s impact.

Firstly, we thank the reviewer for pointing us to the Perry & Geisler citations which stitch together blur images for computationally-tractable fixation, similar to what we propose using uniform TTM images. We have included these citations in the manuscript.

>“An alternative explanation for the results is that the TTM is an inadequate model of human peripheral vision”

We appreciate reviewer ogr4’s thoughtful alternative explanation for our results due to TTM throwing away too much information. Indeed, like all peripheral vision models,  TTM does indeed underpredict human performance for certain stimuli and situations. Among all the peripheral models currently available, we chose TTM to create our dataset because it has been validated on the widest range of tasks compared to any similar texture-based model. It is true that the object detection task tested here differs slightly from the object detection task tested in Ehinger & Rosenholtz 2016, with more hallmarks of a discrimination task. We note that models sharing pooled TTM statistics have been widely tested on discrimination with the metamer paradigm (Freeman & Simoncelli 2011, Wallis 2019). We agree with reviewer ogr4 that there are newer models of peripheral vision that each contribute their own improvements such as style-transfer based models like (Wallis 2019, Deza 2017)). However these models focus primarily on the constraint of metamerism rather than predicting task performance which we believe is important and interesting to consider when making comparisons to DNNs. In our work, although we use object detection as an exemplar, our goal is for COCO-Periph to be applied in many other visual tasks such as search which is labeled in COCO (Chen et al., 2021), as well as scene and image recognition which could be explored using the COCO things and stuff labels (Caesar et al., 2018). We appreciate that our language did not discuss the limitations of TTM and other related models, and we have updated our manuscript to address these points. 

In weighing the reviewer’s hypothesis against that presented in the paper, we acknowledge that the limitations of current state of the art peripheral vision models like TTM could contribute in part to our results. We see this as an important point to share with readers, and have incorporated this into the updated manuscript. We argue however, that the limitations of TTM cannot completely explain our results for the following reasons:

1. While TTM throwing away too much information could create a ceiling on model performance as compared to humans, our results also demonstrate the brittle falloff of performance for machine models as compared to the gradual falloff of performance in humans (Figure 3c). TTM uses the same statistical parameters at all eccentricities, modifying only pooling region size, which should not cause a sharp performance drop between low and high eccentricities. Indeed, we show when fine-tuning models on single eccentricities boosts performance at other eccentricities (Appendix Figure 22). Therefore, it is unlikely that this sharp performance drop off seen is attributable to TTM itself, and much more likely to be TTM-transforms being out of distribution to untrained models.

2. The differences in performance between the different model architectures tested (ie DINO, RCNN, DETR etc) cannot be explained by loss of information in TTM, as these models received the same TTM-transformed input images. 

3. Relatedly, the improvement in performance seen after training and fine-tuning models on COCO-Periph demonstrates a change in the ability of even the same model architecture to process peripherally-transformed images differently. Again, the baseline, fine-tuned, and trained models receive the same TTM-transformed input images. 

4. Also related, the improvement in corruption robustness after fine-tuning and training with COCO-Periph is orthogonal to any limitations of TTM as a peripheral vision model.

5. Furthermore, our results in comparing model performance in detection vs recognition are likewise orthogonal to any limitations of TTM as a peripheral vision model.

Reviewer comment (Experiments):
While TTM may correlate well with human performance for certain tasks (e.g. visual search, scene gist recognition), object detection as tested here may not be one of them. It is generally considered that crowding is about discrimination, not detection. If an object discrimination task had been used instead, the correlation between human and TTM-trained-DNN performance might be higher. Thus, it remains uncertain whether the "gap in performance between humans and computer vision DNNs in the periphery" should be addressed via more advanced DNN architectures and COCO-Periph training, or whether the adequacy of TTM for this purpose is itself questionable.

Author response:
We thank reviewer ogr4 for the thoughtful and careful review of our paper, appreciation of the strengths and contributions of our work, and for many helpful suggestions that will improve the paper’s impact.

Firstly, we thank the reviewer for pointing us to the Perry & Geisler citations which stitch together blur images for computationally-tractable fixation, similar to what we propose using uniform TTM images. We have included these citations in the manuscript.

>“An alternative explanation for the results is that the TTM is an inadequate model of human peripheral vision”

We appreciate reviewer ogr4’s thoughtful alternative explanation for our results due to TTM throwing away too much information. Indeed, like all peripheral vision models,  TTM does indeed underpredict human performance for certain stimuli and situations. Among all the peripheral models currently available, we chose TTM to create our dataset because it has been validated on the widest range of tasks compared to any similar texture-based model. It is true that the object detection task tested here differs slightly from the object detection task tested in Ehinger & Rosenholtz 2016, with more hallmarks of a discrimination task. We note that models sharing pooled TTM statistics have been widely tested on discrimination with the metamer paradigm (Freeman & Simoncelli 2011, Wallis 2019). We agree with reviewer ogr4 that there are newer models of peripheral vision that each contribute their own improvements such as style-transfer based models like (Wallis 2019, Deza 2017)). However these models focus primarily on the constraint of metamerism rather than predicting task performance which we believe is important and interesting to consider when making comparisons to DNNs. In our work, although we use object detection as an exemplar, our goal is for COCO-Periph to be applied in many other visual tasks such as search which is labeled in COCO (Chen et al., 2021), as well as scene and image recognition which could be explored using the COCO things and stuff labels (Caesar et al., 2018). We appreciate that our language did not discuss the limitations of TTM and other related models, and we have updated our manuscript to address these points. 

In weighing the reviewer’s hypothesis against that presented in the paper, we acknowledge that the limitations of current state of the art peripheral vision models like TTM could contribute in part to our results. We see this as an important point to share with readers, and have incorporated this into the updated manuscript. We argue however, that the limitations of TTM cannot completely explain our results for the following reasons:

1. While TTM throwing away too much information could create a ceiling on model performance as compared to humans, our results also demonstrate the brittle falloff of performance for machine models as compared to the gradual falloff of performance in humans (Figure 3c). TTM uses the same statistical parameters at all eccentricities, modifying only pooling region size, which should not cause a sharp performance drop between low and high eccentricities. Indeed, we show when fine-tuning models on single eccentricities boosts performance at other eccentricities (Appendix Figure 22). Therefore, it is unlikely that this sharp performance drop off seen is attributable to TTM itself, and much more likely to be TTM-transforms being out of distribution to untrained models.

2. The differences in performance between the different model architectures tested (ie DINO, RCNN, DETR etc) cannot be explained by loss of information in TTM, as these models received the same TTM-transformed input images. 

3. Relatedly, the improvement in performance seen after training and fine-tuning models on COCO-Periph demonstrates a change in the ability of even the same model architecture to process peripherally-transformed images differently. Again, the baseline, fine-tuned, and trained models receive the same TTM-transformed input images. 

4. Also related, the improvement in corruption robustness after fine-tuning and training with COCO-Periph is orthogonal to any limitations of TTM as a peripheral vision model.

5. Furthermore, our results in comparing model performance in detection vs recognition are likewise orthogonal to any limitations of TTM as a peripheral vision model.

Reviewer comment (Reproducibility):
The paper lacks a comprehensive description of what the Texture Tiling Model (TTM) computes, which hinders a clear understanding of its workings. The reference given for the original TTM (Rosenholtz et al., 2012a) also does not contain a sufficiently detailed description to re-implement the model. It is unclear whether the authors intend to release the TTM code, since they mention releasing the COCO-Periph dataset "along with all analysis code" without specifying if this includes TTM. For deeper understanding of TTM's summary statistics, reference to Portilla and Simoncelli (2000) is necessary, but this is absent from the paper and should be included.

Author response:
>“A more comprehensive description of the TTM should be provided”

We are thankful to the reviewer for asking for clarification regarding the release of our code, and for a more comprehensive description of TTM in our paper. Indeed, as authors we strongly value reproducibility in our work and aim to support this wherever possible. We have updated the manuscript to clarify that we will release both our dataset, as well as the code for the uniform TTM model used to create the dataset, in addition to the machine psychophysics model. Furthermore, we have supplemented the citation to TTM with a citation to the original TTM codebase (https://dspace.mit.edu/handle/1721.1/124819), as we aim to ensure full transparency. We especially appreciate the note on adding the reference to Portilla and Simoncelli, which we have now included in the paper.

Reviewer comment (Evaluation):
There is an absence of uncertainty indications in the data plots, which is a noticeable shortcoming. Accuracies and psychometric function parameter estimates (Figures 3c, 5, 6, etc.) should include uncertainty measures (e.g. confidence intervals). It is unclear whether psychometric function parameters are estimated from pooled human data or from each participant individually, making it hard to assess the data's precision.

Author response:
>“Data analysis and quantification”

We appreciate the reviewer asking for clarification regarding how data is pooled in figures 3c, 5, and 6 - we have added explanations for this in each of these figure captions. To clarify, figure 3 shows data for all human subjects and models for a single image. Figure 5 is not a fit to only the data from figure 3, but averages data over all human subjects over all images. We thank the reviewer for pointing out the absence of error bars for our critical eccentricity (\mu) fits in figures 3c, 5, and 6. For figure 3c which shows results for an individual image, this is indeed possible for the human data, and we have included these, as well as explanation in the figure caption. As figures 5 and 6 show data averaged over all human subjects and images, with different images having different inherent difficulties (object size, amount of crowding, contrast, etc - see Figure 7), error bars in these plots would not reflect the overall variability in performance in a meaningful way. 

Regarding reporting uncertainty with error bars for the model data, we note that measuring uncertainty even for a single image (figure 3c) is not possible for the models given the machine psychophysics, which averages the simulated response of a single model over 100 trials (100 possible pairs of 10 present, 10 absent mongrels) at each eccentricity to give each accuracy value. There is no sense of multiple ‘individuals’ of the same model, and models are deterministic, always giving the same response for a given present/absent pair. We previously considered splitting the 100 trials into a simulated 10 accuracy values each with 10 trials to allow for the inclusion of error analysis, but found that calculating an accuracy value over only 10 trials gave too little accuracy granularity in the reported accuracy values (i.e. there would only be 10 possible accuracy values).

Reviewer comment (Evaluation):
The step size of TTM eccentricities might be insufficient for robustly constraining model μ values. Figure 3c shows multiple nearly equally plausible psychometric curves that could be drawn for DINO FocalNet, potentially altering μ estimates. These issues emphasize the need to disclose parameter estimate uncertainties, without which the improvement gained from COCO-Periph training or fine-tuning cannot be properly evaluated.

Author response:
Relatedly, we share the reviewer’s desire to evaluate these models at a wider range of eccentricities and with smaller step-sizes. Put simply, the computational requirements of creating these mongrel images (1-2 hours per image for the entire 328,000 MS-COCO dataset) limits the number of eccentricities we can feasibly explore.

To address these above concerns about psychometric function fits specifically, and along the lines of the suggestions by reviewer eW1M, we have included a plot reporting averaged accuracy results for each model, before the psychometric fit (Figure 5). This demonstrates the falloff in performance in machines happening more sharply and at earlier eccentricities as compared to humans.