E MERGENT P ROPERTIES OF F OVEATED P ERCEPTUAL S YSTEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce foveated perceptual systems – a hybrid architecture inspired by human vision, to explore the role of a texture-based foveation stage on the nature and robustness of subsequently learned visual representation in machines.
Speciﬁcally, these two-stage perceptual systems ﬁrst foveate an image, inducing a texture-like encoding of peripheral information – mimicking the effects of visual crowding – which is then relayed through a convolutional neural network (CNN) trained to perform scene categorization.
We ﬁnd that these foveated perceptual systems learn a visual representation that is distinct from their non-foveated counterpart through experiments that probe: 1) i.i.d and o.o.d generalization; 2) robustness to occlusion; 3) a center image bias; and 4) high spatial frequency sensitivity.
In addition, we examined the impact of this foveation transform with respect to two additional models derived with a rate-distortion optimization procedure to compute matched-resource systems: a lower resolution non-foveated system, and a foveated system with adaptive Gaussian blurring.
The properties of greater i.i.d generalization, high spatial frequency sensitivity, and robustness to occlusion emerged exclusively in our foveated texture-based models, independent of network architecture and learning dynamics. Altogether, these results demonstrate that foveation – via peripheral texture-based computations – yields a distinct and robust representational format of scene information relative to standard machine vision approaches, and also provides symbiotic computational support that texture-based peripheral encoding has important representational consequences for processing in the human visual system.
I NTRODUCTION
In the human visual system, incoming light is sampled with different resolution across the retinal area, a stark contrast to machines that perceive images at uniform resolution.
One account for the nature of this foveated (spatially-varying) array in humans is related purely to sensory efﬁciency (biophysical constraints)
(Land & Nilsson, 2012; Eckstein, 2011), e.g., there is only a ﬁnite amount of retinal ganglion cells (RGC) that can relay information from the retina to the LGN constrained by the ﬂexibility and thickness of the optic nerve.
Thus it is “more efﬁcient” to have a moveable high-acuity fovea, rather than a non-moveable uniform resolution retina when given a limited number of photoreceptors as suggested in Akbas & Eckstein (2017).
Machines, however do not have such wiring/resource constraints – and with their already proven success in computer vision (LeCun et al., 2015) – this raises the question if a foveated inductive bias is even necessary for vision at all.
However, it is also possible that foveation plays a functional role at the representational level , which can confer perceptual advantages as has been explored in humans .
This idea has remained elusive in computer vision, but popular in vision science, and has been explored both psychophysically (Loschky et al., 2019) and computationally (Poggio et al., 2014; Cheung et al., 2017; Han et al., 2020).
There are several symbiotic examples arguing for the functional advantages of foveation in humans, via functional advantages in machine vision systems. For example, in the work of Pramod et al.
(2018), blurring the image in the periphery gave an increase in object recognition performance of computer vision systems by reducing their false positive rate. In Wu et al.
(2018)’s GistNet, directly introducing a dual-stream foveal-peripheral pathway in a neural network boosted object detection performance via scene gist and contextual cueing.
Relatedly, the most well known example of work that has directly shown the advantage of peripheral vision for scene processing in humans is Wang & Cottrell
[Figure]
Importantly, none of these studies introducing the notion of texture representation in the periphery – a key property of peripheral computation as posed in Rosenholtz (2016).
Testing whether this texture-based coding of the visual periphery is functional in any perceptual system is still an open question.
Here we address this question directly.
Speciﬁcally, we introduce foveated perceptual systems : these are two-stage (hybrid) systems that have a texture-based foveation stage followed by a deep convolutional neural network.
In particular, we will mimic foveation over images using a transform that simulates visual crowding (Levi, 2011; Pelli, 2008; Doerig et al., 2019b;a) in the periphery as shown in Figure 1 (Deza et al., 2019), rather than Gaussian blurring (Pramod et al., 2018;
Wang & Cottrell, 2017) or compression (Patney et al., 2016; Kaplanyan et al., 2019).
These rendered images are akin to visual metamers , capturing image statistics known to be preserved in human peripheral vision, and resembling texture computation at the stage of area V2, as argued in Freeman & Simoncelli (2011); Rosenholtz (2016); Wallis et al.
Thus, our strategy in this paper is to compare these hybrid models’ perceptual biases to their non-foveated counterpart through a set of experiments: generalization, robustness to occlusion, image-region bias and spatial frequency sensitivity.
A difference from Wang & Cottrell (2017) is that our goal is not to implement foveation with adaptive gaussian blurring to ﬁt known results to data (Larson & Loschky, 2009); but rather to explore the emergent representational consequences on scene representation following texture-based foveation linked to area V2 representation (Ziemba et al., 2016).
While it is certainly possible that in these machine vision systems that only need to categorize scenes, there may be little to no beneﬁt of this texture-based computation; the logic of our approach however, is that any beneﬁts or relevant differences between these systems can shed light into both the importance of texture-based peripheral computation in humans, and could suggest a new inductive bias for advanced machine perception.
F OVEATED P ERCEPTUAL S YSTEMS
We deﬁne perceptual systems as two-stage with a foveation transform (stage 1, f ( ◦ ) :
R D → R D ), that is relayed to a deep convolutional neural network (stage 2, g ( ◦ ) :
R D → R d ).
Note that the ﬁrst transform stage is a ﬁxed operation over the input image, while the second stage has learnable parameters.
In general, the perceptual system S ( ◦ ) , with retinal image input I : R D is deﬁned as:
[Equation]
Such two stage models have been growing in popularity, and the reasons these models (including our own) are designed to not be fully end-to-end differentiable is mainly to force one type of computation into the ﬁrst-stage of a system such that the second-stage g ( ◦ ) must ﬁgure out how to capitalize on such forced transformation and thus assess if f ( ◦ ) has a representational goal.
For
[Figure]
t i
[Figure]
n
[Figure]
t
S t a n d a r d - N e t
[Figure]
example, Parthasarathy & Simoncelli (2020) successfully imposed V1-like computation in stageto explore the learned role of texture representation in later stages with a self-supervised objective, and Dapello et al.
(2020) found that ﬁxing V1-like computation also at stage 1 aided adversarial robustness.
At a higher level, our objective is similar where we would like to force a texture-based peripheral coding mechanism (V2) at the ﬁrst stage to check if the perceptual system through g ( ◦ ) will learn to pick-up on this newly made representation and make ‘good’ use of it potentially shedding light on the functionality hypothesis for machines and humans.
This leads us to further motivate our choice for a single-stream foveated-peripheral pathway (vs dual-stream Wang & Cottrell (2017); Wu et al.
(2018)), as would like to focus on how g ( ◦ ) will differ purely based on the learning dynamics driven by SGD rather than architectural constraints (Deza et al., 2020) for the newly transformed input space ( foveated : f ∗ vs non-foveated : f 0 ) to perform a 20-way scene categorization task.
2. S TAGE 1: F OVEATION T RANSFORM ( f )
To model the computations of a foveated visual system, we employed the recently proposed Metamer model of Deza et al.
(henceforth Foveation Transform ).
This model is inspired by the metamer synthesis model of Freeman & Simoncelli (2011), where image metamers are constructed by locally matching texture statistics (Portilla & Simoncelli, 2000; Balas et al., 2009) in the visual periphery.
Analogously, the Deza et al.
(2019) Foveation Transform uses a foveated feed-forward style transfer (Huang & Belongie, 2017) network to latently perturb the image in the direction of its locally matched texture (see Figure 1).
When calibrating the distortions correctly, the resulting procedure leads to a visual metamer (for a human – now forced onto the machine), which is a carefully perturbed image perceptually indistinguishable from its reference image (Feather et al., 2019).
In summary, f : R D → R D is nothing more than a convolutional auto-encoder that is non-foveated when the latent space is un-perturbed: f 0 ( I ) =
D ( E ( I )) , but foveated ( ◦ Σ ) when the latent space is perturbed via localized style transfer: f ∗ ( I ) =
D ( E Σ ( I )) , for a given encoder-decoder ( E , D ) pair.
2. S TAGE 2: C ONVOLUTIONAL N EURAL N ETWORK BACKBONE ( g )
The foveated images (stage 1) are passed into a standard neural network architecture.
Here we tested two different base architectures: AlexNet (Krizhevsky et al., 2012), and ResNet18 (He et al., 2016).
The goal of running these experiments on two different architectures is to let us examine the consequences of foveation that are robust to these different network architectures.
Further, this CNN backbone ( g : R D → R d ) should not be viewed in the traditional way of an end-to-end input/output system where the input is the retinal image ( I ) , and the output is a one-hot vector encoding a d -classlabel in R d .
Rather, the CNN ( g ) acts as a proxy of higher stages of visual processing (as it receives input from f ), analogous to the 2-stage model of Lindsey et al.
C.
A d a - G a u ssN e t
M a t c h e d - N e t
Foveation Transform B. A.
Matched Resource w.r.t Foveation-Nets
F o v e a t e d
N o n - F o v e a t e d : :
[Figure]
2. C RITICAL M ANIPULATIONS : F OVEATED VS N ON -F OVEATED P ERCEPTUAL S YSTEMS
Now, we can deﬁne two perceptual systems as the focus of our experiments that must perform 20-way scene categorization: Foveation-Net , receives an image input, applies the foveated transform f ∗ ( ◦ ) , and later relays it through the CNN g ( ◦ ) ; and Standard-Net , that performs a non-foveated transform f 0 ( ◦ ) , where images are sent through the same convolutional auto-encoder D ( E ( I )) , but with the parameter that determines the degree of texture style transfer set to 0 .
Both of these systems are depicted in Figure 2 (A).
Anticipating the potential differences between Foveation-Nets and Standard-Nets, a natural next question arises: Will these differences be found with any type of spatially-varying processing, or are the effects speciﬁc to a type of foveation transformation (texture-based coding as linked to area V2 vs adaptive gaussian blurring given retinal constraints).
Thus, we created a matched-resource adaptive blurred foveation transform using a Rate-Distortion (RD) optimization framework inspired from Ballé et al.
Figure 3 (B) summarizes our goal and approach: we will have to ﬁnd a set of standard deviation of the guassian blurring kernel per retinal eccentricity (the ‘distortions’ D ), such that we can render a perceptually matched adaptive gaussian blurred image – in reference to the compressed image from Standard-Nets – that matches the perceptual transmission ‘rate’ R of Foveation-Nets via the SSIM perceptual metric (Wang et al., 2004) – also in reference to Standard-Nets.
[Figure]
Perceptual Systems trained on these images (which deﬁne a new function ˆ f ∗ ), provide a matched-resource control to FoveationNet that we dub Ada-Gauss-Net .
This same optimization procedure can be extended to the entire image to also develop a matched-resource control for a non-foveated visual system ( Matched-Net , ˆ f 0 : Figure 4), as one could argue that StandardNets may be at an unfair advantage to Foveation-Nets since their foveal regions are identical and the retinal sensors have not been properly re-distributed (Cheung et al., 2017) & Figure 5 (A).
Altogether these 4 perceptual systems help us answer three key questions: 1) Foveation-Net vs Standard-Nets will tell us how a texture-based foveation mechanism will compare to its undistorted perceptual upper-bound.
2) Foveation-Nets vs Ada-Gauss-Nets will tell us if any potentially interesting pattern of results is due to the type/stage of foveation.
This will help us measure the contributions of the crowding mechanism and texture computation vs adaptive gaussian blurring; 3) Foveation-Net vs Matched-Net will tell us how do these perceptual systems (one foveated, and the other one not) behave when allocated with a ﬁxed number of perceptual resources.
[Figure]
[Figure]
i.i.d Generalization o.o.d Generalization
[Figure]
E XPERIMENTS
In the following section we will discuss results found with regards to i.i.d. & o.o.d.
generalization, robustness to occlusion, central image bias and spatial frequency sensitivity.
All results shown in this section use AlexNet as g ( ◦ ) for the second stage of each perceptual system.
The same pattern of results are evident for ResNet18, – these extended results coupled with learning dynamics, as well as extra calculations and additional controls, can be accessed in the Supplementary Material.
3. G ENERALIZATION
We ﬁrst examined the generalization capacity for our 4 network types.
Given that Foveation-Nets receive distorted inputs in the periphery one could expect them to do worse compared to StandardNets (which operate over full resolution, untexturized images) and on par with Matched-Nets (which have with perceptually matched resources as Foveation-Nets).
On the other hand, it is possible that the texture-based image encoding could confer a functional advantage, in which case Foveation-Nets would do better. If this is indeed the case, we would also see an advantage of Foveation-Nets over
Ada-Gauss-Net ; but if their performance is the same, we can conclude that any type of spatiallyvarying processing operator can enhance scene categorization performance.
Additionally, a somewhat contrived but interesting experiment, we will also conduct is the out-of-distribution (o.o.d) case in which each perceptual system is shown stimuli type that they never encountered during training – in particular stimuli that is the input following the ﬁrst stage of the other perceptual systems.
Result 1: Figure 5(B.) shows that a foveated system with texture-based computation in the periphery (Foveation-Net) has greater i.i.d generalization than its matched resource non-foveated system (Matched-Net) and a foveated system with adaptive gaussian blurring (Ada-Gauss-Net); and also performs on par with an un-matched resource non-foveated system (Standard-Net) – an unexpected victory for Foveation-Nets.
For an o.o.d generalization task where systems are shown o.o.d images from another system, we ﬁnd that drops in performance for foveated perceptual systems are lower compared to their non-foveated counter-part (Foveation-Net ( ↑ ) vs Standard-Net ( ↓ ) ; Ada-GaussNet ( ↑ ) vs Matched-Net ( ↓ ) ); and that overall Ada-Gauss-Nets has greater o.o.d generalization compared to all other systems , while Foveation-Net struggles to cross-generalize to low pass spatially transformed images compared to Ada-Gauss-Net – a result that ties to the experiments of Section 3.4.
3. R OBUSTNESS TO O CCLUSION
We next examined how all perceptual systems could classify scene information under conditions of visual ﬁeld loss, either from the center part of the image (scotoma), or the periphery (glaucoma).
This manipulation lets us examine the degree to which the representations are relying on central vs peripheral information to classify scene categories.
For the foveal-occlusion conditions, we superimposed a central gray square on each image from the testing image set, with 8 levels of
[Figure]
T o p - 1 A cc u r a c y
[Figure]
a
T o p
[Figure]
cc u r a c y
[Figure]
[Figure] [Figure] [Figure]
increasing size.
For the peripheral-occlusion condition, we superimposed a gray box over the outter image boundaries, with area-matched levels of occlusion.
These images were then passed through all of the trained models to compute scene categorization performance.
Accuracy was measured at each level of occlusion and area under the curve was computed as an index of robustness to occlusion.
Figure 6 (top) shows a summary of these results.
Overall Foveation-Nets showed more robustness to central/scotoma occlusion to all other systems: un-matched non-foveated (Standard-Nets), matchedresource foveated system with adaptive gaussian blurring (Ada-Gauss-Net), and matched-resource non-foveated systems (Matched-Net).
This result is similar to peripheral/glaucoma occlusion with the exception of Foveation-Nets being on par with non-resource matched Standard-Net – which is still quite impressive.
To further explore these results, we visualized what the image inputs look like after the foveation stage, across all systems.
Figure 6 (B. left) reveals critical information:
Result 2a: The effect of the stage 1 crowding operation ( f ∗ ( ◦ )) of Foveation-Net exclusively reduces the occluded area in the periphery – a stark contrast compared to all other systems.
By ﬁlling in the image with texture statistics due to the foveation stage ( f ∗ ) – Foveation-Net (but not Ada-Gauss-Net) are able to boost classiﬁcation performance by using greater visual information and/or the peripheral texturized representations that Standard-Nets & Matched-Nets do not have access to.
But is this the main factor accounting for the higher robustness of the foveated systems?
In other words: is it driven by greater computable visual area that has been ﬁlled in from the foveation stage;
Or is the beneﬁt driven by the learned texturized representations in the subsequent deep convolutional neural network?
To examine this question, we occluded either the center or periphery of the images, but after the stage 1 computation (see Figure 6 (A., right)).
In this way, the area of occlusion was matched for the second stage classiﬁcation task.
Result 2b:
Across the matched-resource systems and matched-occluded area condition: FoveationNet has greater robustness to occlusion than Matched-Net and Ada-Gauss-Net.
This puzzling result for both scotomal and glaucomal occlusion, not only suggests that the learned representations in Foveation-Net is different than Ada-Gauss-Net, but that Foveation-Net has learned a locally distributed representation vs a globally distributed representation (Ada-Gauss-Net).
In addition as Foveation-Nets and Ada-Gauss-Nets only differ in peripheral computation, this suggests that a texture-based encoding mechanism provides a representational advantage than an adaptive blurred representation when information is unavailable, occluded or removed.
[Figure]
3. W INDOW C UE -C ONFLICT E XPERIMENT
It is possible that foveated systems weight visual information strongly in the foveal region than the peripheral region as hinted by our occlusion results (the different rate of decay for the accuracy curves in the Scotoma and Glaucoma conditions) – and due to the fact that the information in the periphery is texturized/blurred?
To assess such difference, we conducted an experiment where we created a windowed cue-conﬂict stimuli where we re-rendered our set of testing images with one image category in the fovea, and another one in the periphery.
All cue-conﬂicting images were paired with a different class ( ex: aquarium with badlands).
We then systematically varied the fovea-periphery visual area ratio & examined classiﬁcation accuracy for both the foveal and peripheral scenes (Figure 7).
Result 3: We found that Foveation-Nets and Ada-Gauss-Nets maintained higher accuracy for the foveal scene class than do Standard-Nets and Matched-Nets, as the amount of competing peripheral information increased – and vice versa for the peripheral scene class.
A qualitative way of seeing this foveal-bias is by checking the foveal/peripheral ratio where these two accuracy lines cross.
The more leftward the cross-over point ( ⊗ ) , the higher the foveal bias.
Thus, Foveation-Nets have learned to weigh information in the center of the image more when categorizing scenes – a similar ﬁnding to Wang & Cottrell (2017) indeed as Ada-Gauss-Net vs Matched-Net shows the exact same pattern of results.
Thus, these results indicate that the spatially varying computation from center to periphery (evident in both Foveation-Nets and Ada-Gauss-Nets) is mainly responsible for the development of a center image bias.
It is possible that one of the functions of any spatially-varying coding mechanisms in the visual ﬁeld is to enforce the perceptual system to attend on the foveal region – avoiding the shortcut of learning to attend the entire visual ﬁeld (Geirhos et al., 2020).
3. S PATIAL F REQUENCY S ENSITIVITY
We next examined differences of learned feature representations that are more reliant on low or high spatial frequency information, at the second stage of visual processing (post-foveation) across all systems.
To do so, we ﬁltered the testing image set at multiple levels to create both high pass and low pass frequency stimuli and assessed scene-classiﬁcation performance over these images for all models, as shown in Figure 8 (A.).
Low pass frequency stimuli ( F L ) were rendered by convolving a
Gaussian ﬁlter of standard deviation σ = [0 , 1 , 3 , 5 , 7 , 10 , 15 , 40] pixels on the foveation transform ( f 0 , ˆ f 0 , f ∗ , ˆ f ∗ ) outputs.
Similarly, the high pass stimuli ( F H ) was computed by subtracting the reference image from its low pass ﬁltered version with σ = [ ∞ , 3 , 1 . 5 , 1 , 0 . 7 , 0 . 55 , 0 . 45 , 0 . 4] pixels and adding a residual.
These are the same values used in the experiments of Geirhos et al.
Result 4: We found that Foveation-Nets and Standard-Nets were more sensitive to High Pass Frequency information, while Ada-Gauss-Nets and Matched-Nets were sensitive to Low Pass Frequency stimuli.
This suggests that foveation via adaptive gaussian blurring (Ada-Gauss-Nets) may implicitly contribute to scale-invariance as also shown in Poggio et al.
(2014); Cheung et al.
Han et al.
Additionally, an intriguing possibility is that Foveation-Nets’ peripheral texture representation may potentially support a biologically-plausible mechanism of a shape bias as argued in Figure 8.
[Figure]
D ISCUSSION
The present work was motivated by the broad question of the functional role of a texture-based peripheral representation. We speciﬁcally examined whether a texture-based peripheral encoding mechanism yielded any perceptual advantages and distinctive representational signatures in secondstage deep neural networks.
We found that when comparing Foveation-Nets to their matched-resource models that differed in computation: Ada-Gauss-Nets (foveated w/ adaptive gaussian blur) and Matched-Nets (non-foveated w/ uniform blur) – that peripheral texture encoding did lead to speciﬁc computational signatures, particularly in robustness to occlusion.
We also found that foveation (in general) seems to induce a focusing mechanism , servicing the foveal/central region, while the texture-based computation still preserves high spatial-frequency selectivity – this last result is likely due to weight-sharing that does not give room to low-spatial frequency tuned ﬁlters to naturally emerge as in Wang & Cottrell (2017).
The particular consequences of our foveation stage raises interesting future directions about what computational advantages could arise when trained on object categorization, as objects are typically centered in view and have different hierarchical/compositional priors than scenes (Zhou et al.
Deza et al.
in addition to different processing mechanisms (Renninger & Malik (2004)).
Speciﬁcally, one intriguing possibility is that our foveated representational signatures may induce more shape sensitivity for object recognition (rather than texture sensitivity; Geirhos et al.
Hermann et al.
and may amplify the perceptual differences we identiﬁed across foveated and non-foveated systems.
Finally, a future direction is investigating the effects of crowding to adversarial robustness.
Motivated by the recent work of Dapello et al.
(2020) which has shown promise of adversarial robustness via enforcing stochasticity and V1-like computation by matching the Nyquist sampling frequency of these ﬁlters (Serre et al., 2007)
in addition to a natural gamut of orientations and frequencies as studied in De Valois et al.
, it raises the question of how much we can further push for robustness in hybrid perceptual systems like these, drawing on even more biological mechanisms.
Both Luo et al. (2015) and Reddy et al.
(2020) have already taken steps in this direction by coupling ﬁxations with a spatially-varying retina. However, the computational impact of visual crowding and texture-based computation on adversarial robustness is an open question.  S UPPLEMENTARY M ATERIAL 5. D ESCRIPTION OF F OVEATED P ERCEPTUAL S YSTEMS Foveation-Net : We adjusted the parameters of the foveation transform to have stronger distortions in the periphery that can consequently amplify the differences between a foveated and non-foveated system.
This was done setting the rate of growth of the receptive ﬁeld size (scaling factor) s = 0 .
Thus, when foveation is on at the previous scaling factor, we will abbreviate f as f ∗ .
This value was used instead of s = 0 .
5 , given that experiments of Freeman & Simoncelli (2011);
Deza et al.
(2019) has shown that this scaling factor yields a match with physiology but only when human observers are psychophysically tested between pairs of synthesized/rendered image metamers.
Works of Wallis et al.
Deza et al.
(2019) have suggested that the when comparing a non-foveated reference image to it’s foveated crowded version, the scaling factor is actually much smaller than 0 .
We thus selected a smaller factor of s = 0 .
4 (that is still metameric to a human observer between synthesized pairs), as smaller scaling factors signiﬁcantly reduced the crowding effects.
Ultimately, the selection of this value is not critical in our studies as we are not making any comparative measurements to human psychophysical experiments (See Deza & Eckstein (2016) for an example where matching scaling factors is critical).
Standard-Net : We use the same foveation transform at the foveation stage for Standard-Net but set the scaling factor set to s = 0 .
In this way, any potential effects of the compression/expansion operations of the foveation stage in the image are matched between the Foveation-Nets and StandardNets.
Thus, the only difference after stage 1 is whether the image statistics were texturized in increasingly large pooling windows (Foveation-Nets), or not (Standard-Nets).
Extending our notation, no foveation in stage one will be abbreviated as f 0 .
Note however, that Standard-Nets are not matched-resource non-foveated controls – StandardNets only provide the control for comparing purely the effects of crowding (Foveation-Net) while potentially using more resources as both the foveal regions of both transforms remain intact, and a matched resource control should redistribute the retinal ganglion cell convergence uniformly as explored in Cheung et al. (2017).
In fact, the matched-resource control that is also non-foveated is
Matched-Net as described earlier in the paper, and more in detail as follows.
Matched-Net : Matched-Net provides a non-foveated resource matched control with respect to
Foveation-Net.
This perceptual system is essentially computed via ﬁnding the optimal standard deviation σ of the ﬁltering kernel G σ as shown in Figure 9.
{+This distortion image is computed via the convolution ( (cid:126) ) of the image f 
+}
( I ) with the low-pass ﬁlter: G σ .
Here, Wang et al.
(2004)’s
SSIM is our idea candidate as it will take into consideration the luminance, contrast and structural changes locally for the entire image and pool them together for an aggregate perceptual score (and also the rate R ) that is upper bounded by 1 and correlated with human perceptual judgments.
As
C. from Figure 9 adapted for Ada-Gauss-Nets.
B. The goal is to ﬁnd the equivalent ‘perceptual transmission rate’ for a given distortion σ to ﬁnd a matched-resource perceptual input for FoveationNets that is foveated but with adaptive gaussian blurring, i.e. we must ﬁnd the standard deviation of the gaussian blurring kernel which is computed over a set of eccentricity rings that have been windowed with cosine functions
C
The full
Distortion curves a retinal eccentricity.
SSIM operates on the luminance of the image, all validation images over which the RD curve (right) was computed were transformed to grayscale to ﬁnd the optimal standard deviation ( σ = 3 . 4737 ).
Ada-Gauss-Net : Is a foveated perceptual system that receives Rate-Distortion optimized images that have been blurred with different standard deviations of the gaussian kernel G σ as a function of retinal eccentricity.
We picked the same eccentricity rings (or pooling regions) as Foveation-Nets given that we did not want to include a potential effect that is driven by differences in receptive ﬁelds rather than differences in type of computation.
Figure 10 shows the full set of distortion strengths ( σ ) of each receptive ﬁeld ring to match the transmission rate of the Foveation Transform ( f ∗ ( ◦ )) that computes crowding.
There are other alternatives to potentially ﬁnd the set of standard deviation coefﬁcients that are not driven by a rate-distortion optimization procedure.
One possibility could have been to ﬁnd a mapping between pixels and degrees of visual angle as done in Pramod et al.
and derive the coefﬁcients by ﬁtting a contrast sensitivity function given the visual ﬁeld.
While this approach is appealing, the coefﬁcients for object recognition (ImageNet Russakovsky et al.
(2015)) can not be extended to scenes (Places Zhou et al.
In addition, the coupling of the RD-optimization with SSIM provides a perceptual guarantee to compare Ada-Gauss-Net with Foveation-Net or Matched-Net.
Foveation-Aug-Net (Supplementary Control): Eye-movements can be seen as a biologically motivated type of data-augmentation strategy.
More generally, the goal of this additional control is to test if eye-movements coupled with a texture-based foveated operator will vary the strength of our results in any possible dimension: generalization, robustness, center bias, spatial frequency sensitivity.
To test this, we created an enhanced version of Foveation-Net where the deep neural network at stagereceives a foveated image of a variable ﬁxation point from one of 9 points from a ﬁxed 3 × 3 grid.
A schematic of the type of inputs Foveation-Aug-Nets receives can be seen in Figure 11.
Statistical testing (paired t-test, n.s.) revealed no strong differences between Foveation-Nets and
Foveation-Aug-Nets, likely due to the fact that scene recognition can be done independent of ﬁxation point (Henderson & Hollingworth, 1999; Oliva & Torralba, 2001).
However there are nominal positives differences for Foveation-Aug-Nets over Foveation-Nets, and it is possible that the contribution of eye-movements in learning a new representation is redundant in scene recognition, and that re-computing these experiments for an object classiﬁcation task may yield stronger perceptual biases.
[Figure]
Standard-Aug-Net (Supplementary Control):
In analogy to the previous augmentation condition,
Standard-Aug-Net enhances our Standard-Net model with artiﬁcial data-augmentation regimes such as random cropping ( 0 .
7 − 1 . 0 of area), resizing, and a 0 .
5 chance of horizontally ﬂipping the image at training.
Standard-Aug-Nets was motivated by 2 main questions: 1) How will actual computer vision systems trained with common data-augmentations behave in reference to other perceptual systems?
; 2) Is an augmented non-foveated system asymptotically equal to a foveated system?
A schematic of the type of inputs Standard-Aug-Nets receives can be seen in Figure 11.
Several experiments as we will show through-out the Supplementary section, have shown remarkable perceptual similarities between Standard-Aug-Nets and Matched-Nets.
This should come at no surprise given that Standard-Aug-Nets received randomly cropped and up-sampled images (thus simulating a type of blur through loss of resolution in the up-sampling stage).
What is perhaps most interesting however is that random cropping, up-sampling and horizontal mirroring did not contribute to a stronger center bias or to a high spatial frequency sensitivity which is correlated to a shape-bias (See Figure 23 from the extended results of Spatial Frequency sensitivity).
These are ﬁndings that differ from the results of Hermann & Kornblith (2019); Hermann et al.
(2020) that suggest that data-augmentation enhances a shape bias for object recognition.
However, recall that scene recognition and object recognition processing mechanisms differ (Renninger & Malik, 2004;
Zhou et al., 2014; Deza et al., 2020) – so our ﬁndings may in fact be complimentary.
This is encouraging as it supports our initial conjectures of the impact of foveation on scenes vs objects, and that stronger pattern of results may emerge when training on such different image input distributions, as mentioned in the Discussion (Section 4).
5. R ESULTS ARE INDEPENDENT OF L EARNING D YNAMICS OF g ( ◦ )
To quantify the robustness of the results reported in the main body of the paper
[Figure]
we re-ran our experiments at different epochs during training of all 6 perceptual systems.
Critically
[Figure]
the 270-th epoch for AlexNet was chosen to be displayed in the main body of the paper given that it is the approximate epoch after which the validation loss begins to diverge from the training loss for at least one of the perceptual systems.
This promotes our choice of picking a relevant snapshot for AlexNet as two more points before and after the 270-th epoch, suggesting the 180-th (before 270) and 360-th epoch (after 270).
The same effect (of a diverging validation loss of at least one system) can be seen for epoch 120 of ResNet18.
Similarly, this encourages us to pick epoch 90 (before) and epoch(after) as other reference snapshots to evaluate the interaction of learning dynamics with our pattern of results
[Figure]
The collection of all the extended results for AlexNet + ResNet18 architectures at multiple epochs is shown in Section 5.6 (Generalization Extended), Section 5.7 (Robustness to Occlusion Extended),
Section 5.8 (Window Cue-Conﬂict Extended) and Section 5.9 (Spatial Frequency Senstitivity Extended).
5. M ETHODS
All models were trained to perform 20-way scene categorization.
The scene categories were selected from the Places2 dataset (Zhou et al., 2017), with 4500 images per category in the training set,per category for validation, and 250 per category for testing.
The categories included were: aquarium, badlands, bedroom, bridge, campus, corridor, forest path, highway, hospital, industrial area, japanese garden, kitchen, mansion, mountain, ocean, ofﬁce, restaurant, skyscraper, train interior, waterfall.
Samples of these scenes coupled with their foveation transforms for each perceptual system can be seen in Section 5.4.
Training: Convolutional neural networks of the stage 2 of each perceptual system were trained which resulted in 60 networks per architecture : 10 Foveation-Nets, 10 Standard-Nets, 10 Matched-Net, 10 Ada-Gauss-Net, 10 Foveation-Aug-Nets, 10 Standard-Aug-Nets; totalling 120 trained networks to compute relevant error bars shown in all ﬁgures (standard deviations, not standard errors) and to reduce effects of randomness driven by the particular network initialization.
All systems were paired such that their stage 2 architectures g ( ◦ ) started with the same random weight initialization prior to training, thus the randomness was purely imposed by SGD.
This procedure also facilitated our statistical testing via paired t-tests as the model systems began their visual diet with the same set of random weights, thus reasonably ‘pairing’ them up for statistical analysis when necessary.
Other parameters in training via backpropagation were: (learning rates:
η = 0 .
001 AlexNet, η = 0 .
ResNet18 – with no learning rate decay for both architectures), and batch size: 128 .
We trained the AlexNet architectures for each perceptual system ( g ( ◦ ))
up to 360 epochs and each ResNet architecture up to 180 epochs to study the learning dynamics that can be seen throughout the entire
Supplementary Material showing that the main effects and patterns of results reported in the paper are preserved throughout different epochs in training.
Validation : Validation images were used exclusively to: 1) visualize the convergence of the loss function for each perceptual system (See Figure 12); 2) compute quantities in our experiments that if otherwise may introduce biases.
Examples of these quantities are: the rate-distortion optimization curves for Matched-Nets and Ada-Gauss-Nets; the average channel intensity for each image across all perceptual systems as a sub-step to compute the residual in high-pass spatial frequency ﬁltering (See Section 5.9).
Testing: Foveation-Nets were only evaluated on center ﬁxation images at inference time.
StandardNets were tested on non-foveated images (the output of the foveation transform with scaling factor set to zero).
Matched-Nets were evaluated on uniformly blurred images that were gaussian ﬁltered with the same standard deviation coefﬁcient that they were trained on.
Ada-Gauss-Nets were tested on adaptive gaussian blurred images that held the same collection of standard deviation coefﬁcients over retinal eccentricities that they were trained on.
Standard-Aug-Nets were tested on the same images as Standard-Nets.
Foveation-Aug-Nets were tested only on center ﬁxation images (the same images as Foveation-Nets were tested on).
No data-augmentation or multi-crop averaging was performed at inference for any of the networks.
No data-augmentation was performed at training for each network either (except for Foveation-Aug-Nets + Standard-Aug-Nets).
All networks performed mean and standard contrast normalization both at training and testing with values set to mean: (0 . 485 , 0 . 456 , 0 . 406) , std: (0 . 229 , 0 .
5. F OVEATION T RANSFORM S AMPLE V ISUALIZATIONS
A q u a r i u m
B a d l a n d s
B e d r oo m
B r
i dg e
C a m p u s
Ada-Gauss-Net Matched-Net
Standard-Net
Standard-Aug-Net
Foveation-Net
Foveation-Aug-Net
C o
rr i
d o r
F o r e s t _ P a t h
H i g
h w a y
H o s p
i t a l
I n d u s t r i a l _ A r e a
Ada-Gauss-Net Matched-Net
Standard-Net
Standard-Aug-Net
Foveation-Net
Foveation-Aug-Net
J a p a n e s e _
G a r d e n
K i t c h e n
M a n s i o n
M o un t a i n
O c e a n
[Figure]
O ﬃ c e
R e s t a u r a n t
S k y s c
r a p e r
T r a i n _ I n t e r i o r
W a t e r f a ll
Ada-Gauss-Net
[Figure]
-Net
[Figure]
-Net
Standard-
[Figure]
Foveation
[Figure]
Foveation
[Figure]
5. D IFFERENCES ACROSS DIFFERENT F OVEATION MODELS AND R ELEVANT W ORK
There are currently 4 foveation models that implement texture-like computation in the peripheral ﬁeld of view as shown in Table 1.
We selected the Foveation Transform model given that it is computationally tractable to render a foveated image dataset (100’000) at a rate of 1 image/second (rather than hours Freeman & Simoncelli (2011) or minutes Wallis et al.
We did not use the highly accelerated model of Fridman et al.
(2017) (order of miliseconds, that was based on the
Texture-Tiling Model of Rosenholtz et al.
because it was: 1) Not psychophysically tested with human observers thus there is no guarantee of visual metamerism via the choice of texture statistics; 2) it does not provide an upper-bound computational baseline (similar to Standard-Net), when all the perturbations coefﬁcients are off, given that the system is closed and lossy; 3) the code was/is unavailable at the time.
However we think that re-running our experiments and testing them with other foveated models such as the before-mentioned is a direction of future work as we would be curious to see the replicability of our pattern of results across other texture-based peripheral models.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
There are several works that have used foveation to show a type of representational advantage over non-foveated systems.
Mainly Pramod et al.
(2018) with adaptive gaussian blur, and Wu et al.
(2018) with scene gist, that have been targeted towards a computational goal in increasing object recognition performance.
For scene recognition, only Wang & Cottrell (2017) has successfully modelled known behavioural results of Larson & Loschky (2009) via a dual-stream neural network that uses adaptive gaussian blurring and a log-polar transform.
One key difference however is that we are interested in exploring the effects of visual crowding that is a phenomena linked to area V2 in the primate ventral stream (rather than retinal as in Wang & Cottrell (2017) which resembles our control condition:
Ada-Gauss-Nets).
In general, we are taking a complimentary approach where we a priori do not know of a functional role of texture-based computation or prime ourselves to ﬁt our model to a reference behavioural result .
Thus we explore what functionality it may have in comparison to a non-foveated system (Matched-Nets, Standard-Nets) or a foveated system that only implements adaptive gaussian blurring (Ada-Gauss-Nets).
Table 2 highlights key similarities & differences between these papers and ours.
[Table]
5. G ENERALIZATION (E XTENDED )
[Figure]
5. R OBUSTNESS TO O CCLUSION (E XTENDED )
We decided to extend the robustness to occlusion analysis with 2 additional conditions: Left2Right and Top2Bottom, which are both occluding conditions where the stimuli is occluded with a gray patch from “left to right” and from “top to bottom” respectively.
Altogether we ﬁnd that the same pattern of results is sustained where Foveation-Nets are near the perceptual upper-bound provided by Standard-Nets.
Both matched-resource systems: Matched-Net (spatially uniform and lowerresolution) and Ada-Gauss-Net (spatially varying with adaptive gaussian blurring) are less robust to lateral, vertical, scotomal and peripheral occlusion than Foveation-Net (spatially-varying with texture-based peripheral encoding) across both AlexNet and ResNet18 architectures as g ( ◦ ) , and across different epochs (learning dynamics).
Furthermore the reported ﬁndings in the main body of the paper with regards to the impact of the visual crowding mechanism (exclusively f ∗ ( ◦ ) ) as a ﬁlling-in operator, is sustained across both AlexNet and ResNet18 architecture as g ( ◦ ) and number of epochs (learning dynamics).
These extended results are plotted in addition to the new controls
Foveation-Aug-Nets and Standard-Aug-Nets in Figures 18,19,20,21.
In these plots we once again show that the contribution of eye-movements in training (Foveation-Aug-Nets) provides marginal improvements in robustness to Foveation-Nets for both the pre and post foveation to occlusion conditions.
On the hand, Standard-Aug-Nets show a pattern of results very similar to a combination of Matched-Nets and Standard-Nets.
This should not be a surprise as Standard-Aug-Nets have been trained with multi-resolution images.
Somewhat surprisingly, Standard-Aug-Nets do not exceed the robustness of Standard-Nets, showing a particularly counter-intuitive limitation to data-augmentation.
[Figure]
-Nets, and Foveation-Aug-Nets being placed more leftwards than
Standard-Nets, Matched-Netd and Standard-Aug-Nets.
These results are independent of potential perceptual differences at testing time i.e. baseline.
For example, see AlexNet @ 360 epochs, or
ResNet18 @ 120 or 180 epochs, where the cross-over point for Standard-Nets, Matched-Nets and
Standard-Aug-Nets is still shifted more biased towards the right than Foveation-Nets, Ada-GaussNets and Foveation-Aug-Nets – implying a greated need for foveal area to arrive to the point of subjective equality (PSE).
A ﬁnal note on the interpretability of these results is that this foveal bias is being tested after the foveation transforms are computed (similar to our post-foveation occlusion experiments), such that no changes in area are driving the revealed biases, and thus the bias is driven purely by the learned representation.
[Figure]
S PATIAL F REQUENCY S ENSITIVITY (E XTENDED )
[Figure]
Images shown to each network were the respective images from each training-testing distribution pair.
Foveation-Nets were shown foveated images with a center ﬁxation, Standard-Net were shown nonfoveated images, Matched-Nets were shown matched-resource non-foveated (yet blurred) images, and
Ada-Gauss-Nets were shown foveated images with adaptive gaussian blurring.
Thus, both High Pass
Spatial Frequency and Low Pass Spatial Frequency experiments were conducted at the post-foveation stage to directly examined the learned representations of g ( ◦ ) : the second stage of each perceptual system.
The size of all shown images was 256 × 256 × 3 , thus the units of the gaussian ﬁlters speciﬁed from
Section 3.4 are in pixels.
For a given Gaussian ﬁltering operation G σ for a given standard deviation σ , low pass spatial frequency (LF) images were computed via:
[Equation]
for each channel C .
In the main body of the paper we used the summarized notation F L ( ◦ ) for low-pass frequency ﬁltering operator.
Similarly, High Pass Spatial Frequency (HF) image stimuli were computed via:
[Equation]
where mean C val (which we call the residual in the main body of the paper) is the average of image intensity over the held-out validation set for each channel C , a small extension from Geirhos et al.
(2019) as our image stimuli is in color vs grayscale.
In the main body of the paper we used the summarized notation F H ( ◦ ) for the high pass frequency ﬁltering operator.