The ventral visual stream is a series of cortical areas that represent spatial patterns, scenes and objects 1 . V1 is the earliest and most thoroughly characterized area. Individual V1 cells encode information about local orientation and spatial frequency 2 , and simple computational models can describe neural responses as a function of visual input 3 . Substantial progress has also been made in understanding later stages, such as inferotemporal cortex, where neurons exhibit complex object-selective responses 4 . However, the transformations between V1 and inferotemporal cortex remain a mystery. Several observations from physiology and theory can help to constrain the study of this problem. It has been shown that receptive field sizes increase along the ventral stream. Many models of visual pattern recognition [5] [6] [7] [8] [9] [10] have proposed that increases in spatial pooling provide invariance to geometric transformations (for example, changes in position or size). In addition, it is well established that receptive field sizes scale linearly with eccentricity in individual areas and that this rate of scaling is larger in each successive area along the ventral stream, providing a signature that distinguishes different areas [11] [12] [13] . We hypothesize that the increase in spatial pooling, both in successive ventral stream areas and with eccentricity, induces an irretrievable loss of information. Stimuli that differ only in terms of this lost information will yield identical population-level responses. If the human observer is unable to access the discarded information, such stimuli will be perceptually indistinguishable; thus, we refer to them as metamers. Visual metamers were crucial to one of the earliest and most successful endeavors in vision science: the elucidation of human trichromacy. Behavioral experiments predicted the loss of spectral information in cone photoreceptors 100 years before the physiological mechanisms were confirmed 14 . The concept of metamerism is not limited to trichromacy, however, and a number of studies have used it to understand aspects of pattern or texture vision [15] [16] [17] . We developed a population-level functional model for ventral stream computation beyond V1 that allowed us to synthesize and examine the perception of a new type of visual metamer. The first stage of the model decomposes an image with a population of oriented V1-like receptive fields. The second stage computes local averages of nonlinear combinations of these responses over regions that scale in size linearly with eccentricity, according to a scaling constant that we can vary parametrically. Given a photographic image, we synthesized distinct images with identical model responses and asked whether human observers could discriminate them. From these data, we estimated the scaling constant that yields metameric images and found that it was consistent with receptive field sizes in area V2, suggesting a functional account of representation in that area. Our model also provides an explanation for the phenomenon of visual crowding 18, 19 , in which humans fail to recognize peripherally presented objects surrounded by clutter. Crowding has been hypothesized to arise from compulsory pooling of peripheral information [20] [21] [22] [23] , and the development of our model was partly inspired by evidence that crowding is consistent with a representation based on local texture statistics 24 . Our model offers an instantiation of this hypothesis, providing a quantitative explanation for the spacing and eccentricity dependence of crowding effects, generalizing them to arbitrary photographic images and linking them to the underlying physiology of the ventral stream. Our model is motivated by known facts about cortical computation, human pattern vision and the functional organization of ventral stream receptive fields. The V1 representation uses a bank of oriented filters covering the visual field, at all orientations and spatial frequencies. 'Simple' cells encode a single phase at each position, whereas 'complex' cells combine pairs of filters with the same preferred position, orientation and scale, but with different phase 25 . The second stage of our model achieves selectivity for compound image features by computing products between particular pairs of V1 Jeremy Freeman 1 & Eero P Simoncelli 1-3 The human capacity to recognize complex visual patterns emerges in a sequence of brain areas known as the ventral stream, beginning with primary visual cortex (V1). We developed a population model for mid-ventral processing, in which nonlinear combinations of V1 responses are averaged in receptive fields that grow with eccentricity. To test the model, we generated novel forms of visual metamers, stimuli that differ physically but look the same. We developed a behavioral protocol that uses metameric stimuli to estimate the receptive field sizes in which the model features are represented. Because receptive field sizes change along the ventral stream, our behavioral results can identify the visual area corresponding to the representation. Measurements in human observers implicate visual area V2, providing a new functional account of neurons in this area. The model also explains deficits of peripheral vision known as crowding, and provides a quantitative framework for assessing the capabilities and limitations of everyday vision. 1 1 9 6 VOLUME 14 | NUMBER 9 | SEPTEMBER 2011 nature neurOSCIenCe a r t I C l e S responses (both simple and complex) and averaging these products over local regions, yielding local correlations. Correlations have been shown to capture important features of naturalistic texture images and have been used to explain some aspects of texture perception 17, 26, 27 . Correlations across orientations at different positions yield selectivity to angles and curved contours, as suggested by physiological studies of area V2 (refs. 28-32). Correlations across frequencies encode features with aligned phase or magnitude (for example, sharp edges or lines) 17, 33 , and correlations across positions capture periodicity. Finally, local correlations are compatible with models of cortical computation that propose hierarchical cascades of linear filtering, point nonlinearities and pooling [5] [6] [7] [8] [9] 25, 34, 35 (see Online Methods). We must specify the pooling regions over which pair-wise products of V1 responses are averaged. Receptive field sizes in the ventral stream grow approximately linearly with eccentricity, and the slope of this relationship (that is, the ratio of receptive field diameter to eccentricity) increases in successive areas (see Fig. 1 and Supplementary Analysis). In our model, pooling is performed by weighted averaging, with smoothly overlapping functions that grow in size linearly with eccentricity, parameterized with a single scaling constant (see Online Methods and Supplementary Fig. 1 ). If our model accurately describes the information captured (and discarded) at some stage of visual processing, and human observers cannot access the discarded information, then any two images that produce matching model responses should appear to be identical. To directly test this assertion, we examined perceptual discriminability of synthetic images that were as random as possible while producing identical model responses 17 . Model responses (Fig. 2a) were computed for a full-field photograph (for example, Fig. 2b) . Synthetic images were then generated by initializing them with samples of Gaussian white noise and iteratively adjusting them (using a variant of gradient descent) until they matched the model responses of the original image (see Online Methods). Synthetic images were identical to the original near the intended fixation point, where pooling regions were small, but features in the periphery were scrambled, and objects were grossly distorted and generally unrecognizable (Fig. 2c,d ). When generated with the correct scaling constant, and viewed with proper fixation, however, the two images appeared to be nearly identical to the original and to each other. To test the model more formally and to link it to a specific ventral stream area, we measured the perceptual discriminability of synthetic images as a function of the scaling constant used in their generation. If the model, with a particular choice of scaling constant, captures the information represented in some visual area, then model-generated stimuli will appear to be metameric. If the scaling constant is made larger, then the model will discard more information than the associated visual area and model-generated images will be readily distinguishable. If the model scaling is made smaller, then the model discards less information and the images will remain metameric. Thus, we sought the largest value of the scaling constant at which the stimuli appeared to be metameric. This critical scaling should correspond to the scaling of receptive field sizes in the area in which the information is lost. As a separate control for the validity of this procedure, we examined stimuli generated from a V1 model that computes pooled V1 complexcell responses 36 (that is, local spectral energy, see Supplementary  Fig. 2 ). The critical scaling estimated for these stimuli should match the receptive field sizes of area V1. As the mid-ventral model includes a larger and more complex set of responses than the V1 model, we know a priori that the critical scaling for the mid-ventral model will be as large, or larger, than for the V1 model, but we do not know by how much. For each model, we measured the ability of human observers to distinguish between synthetic images generated for a range of scaling constants ( Fig. 2e and Online Methods). All four observers exhibited monotonically increasing performance as a function of scaling constant (Fig. 3) . Chance performance (50%) indicates that the stimuli are metameric and, roughly speaking, the critical scaling is the value at which each curve first rises above chance. To obtain an objective estimate of the critical scaling values, we derived an observer model that used the same ventral stream representation as was used to generate the matched images. The inputs to the observer model were two images that were matched over region sizes specified by scaling s. If we assume that the observer computes responses to each of these images with receptive fields that grow in size according to a fixed (but unknown) critical scaling s 0 , then their ability to discriminate the two images will depend on the difference between the two sets of responses. We derived a closed-form expression for the dependency of this difference on s (see Online Methods). This expression is a function of the observer's scaling parameter, s 0, as well as a gain parameter, α 0 , which controls their overall performance. We used signal detection theory 37 to describe the probability of a correct answer and fit the parameters (s 0 , α 0 ) to the data of each subject by maximizing their likelihood. The observer model provided an excellent fit to individual observer data for both the V1 and mid-ventral experiments (Fig. 3) . Critical scaling values (s 0 ) were highly consistent across observers, with most Figure 1 Physiological measurements of receptive field size in macaque. (a) Receptive field size (diameter) as a function of the distance between the receptive field center and the fovea (eccentricity) for visual areas V1, V2 and V4. Data were adapted from refs. 11 and 12, the only studies to measure receptive fields in all three macaque ventral stream areas with comparable methods. The size-to-eccentricity relationship in each area is well described by a 'hinged' line (see Supplementary Analysis for details and an analysis of a larger set of ten physiological datasets). (b) Cartoon depiction of receptive fields with sizes based on physiological measurements. The fovea is at the center of each array. The size of each circle is proportional to its eccentricity, based on the corresponding scaling parameter (slope of the fitted line in a). At a given eccentricity, a larger scaling parameter implies larger receptive fields. In our model, we used overlapping pooling regions (linear weighting functions) that uniformly tiled the image and were separable and of constant size when expressed in polar angle and log eccentricity ( Supplementary Fig. 1 nature neurOSCIenCe VOLUME 14 | NUMBER 9 | SEPTEMBER 2011 1 1 9 7 a r t I C l e S of the between-subject variability being captured by differences in overall performance (α 0 ). As expected, the simpler V1 model required a smaller scaling to generate metameric images. Specifically, critical scaling values for the V1 model were 0.26 ± 0.05 (mean ± s.d.), whereas values for the mid-ventral model were roughly twice as large (0.48 ± 0.02). We then compared the psychophysically estimated scaling parameters to physiological estimates of receptive field size scaling in different cortical areas. Functional magnetic resonance imaging has been used to measure population receptive fields in humans by estimating the spatial extent of a stimulus that contributes to the hemodynamic response across different regions of the visual field 13 . Although these sizes grow with eccentricity, and across successive visual areas, they include additional factors, such as variability in receptive field position and non-neural hemodynamic effects, which may depend on both eccentricity and visual area. We chose to compare our results . The value of the scaling parameter (used to determine the pooling regions of the model) was selected to yield 75% correct performance in discriminating such synthetic images (see Fig. 4 ). The two images, when viewed with fixation at the center (red dot), should appear to be nearly identical to the original and to each other, despite gross distortions in the periphery (for example, a woman's face is scrambled and dissolves into the spray of the fountain). (e) Psychophysical ABX task. Human observers viewed a sequence of two synthetic stimuli ABX, each randomly selected from the set of all images having model responses matched to an original image, followed by a third image that was identical to one of the first two. Observers indicated which of the first two images matched the third. Figure 3 Metamer experiment results. Each graphs shows, for an individual observer, the proportion of correct responses in the ABX task as a function of the scaling parameter (ratio of receptive field diameter to eccentricity) of the model used to generate the stimuli. Data were averaged over stimuli drawn from four naturalistic images. Dark gray indicates the mid-ventral model (see Fig. 2 ), whereas light gray indicates the V1 model (see Supplementary Fig. 2 ). Shaded region indicates the 68% confidence interval obtained using bootstrapping. 1 1 9 8 VOLUME 14 | NUMBER 9 | SEPTEMBER 2011 nature neurOSCIenCe a r t I C l e S to single-unit electrophysiological measurements in non-human primates. Receptive field size estimates vary systematically, depending on the choice of stimuli and the method of estimation, so we combined estimates reported for ten different physiological datasets to obtain a distribution of scaling values for each visual area (see Supplementary Analysis). This analysis yielded values of 0.21 ± 0.07 for receptive fields in V1, 0.46 ± 0.05 for those of V2 and 0.84 ± 0.06 for those of visual area V4 (mean with 95% confidence intervals). Moreover, for studies that used comparable methods to estimate receptive fields in both V2 and V1, the average receptive field sizes in V2 were approximately twice the size of those in V1, for both macaques and humans 11, 13, 38 . As expected, the critical scaling value estimated from the V1 metamer experiment was well matched to the physiological estimates of receptive field scaling for V1 neurons. For the mid-ventral model, the critical scaling was roughly twice that of the V1 model, was well matched to receptive field sizes of V2 neurons and was substantially smaller than those of V4. This correspondence suggests that the metamerism of images synthesized using our mid-ventral model arises in area V2. If metamerism reflects a structural limitation of the visual system, governed by the eccentricity-dependent scaling of receptive field sizes, then the effects should be robust to experimental manipulations that alter observer performance without changing the spatial properties of the stimuli. To test this, we performed two variants of the mid-ventral metamer experiment, designed to alter performance through bottomup and top-down manipulations of the experimental task. First, we repeated the original experiment with doubled presentation times (400 ms instead of 200 ms). Fitting the observer model to data from four observers (Fig. 4a) , we found that the gain parameter (α 0 ) was generally larger, accounting for increases in performance, but that the critical scaling (s 0 ) was statistically indistinguishable from that estimated in the original experiment (P = 0.18, two-tailed paired t test). In a second control experiment, we manipulated endogenous attention. At the onset of each trial, a small arrow was presented at fixation, pointing toward the region in which the two subsequently presented stimuli differed most (see Online Methods). The fitted gain parameter was again generally larger, accounting for improvements in performance, but the critical scaling was statistically indistinguishable from that estimated in the original experiment (P = 0.30; two-tailed paired t test; Fig. 4b ). In both control experiments, the increase in gain varied across observers and depended on their overall performance in the original experiment (some observers already had near-maximal performance). The scaling for the two control experiments were similar to those of the original experiment, were closely matched to the scaling of receptive fields found in area V2 and were much greater than the scaling found in the V1 metamer experiment (P = 0.0064, extended presentation task; P = 0.0183, attention task; two-tailed paired t test; Fig. 5 ). Our model predicts severe perceptual deficits in peripheral vision, some of which are revealed in the well-studied phenomenon of visual crowding 18, 19 , which has been hypothesized to arise from pooling or statistical combination in the periphery [20] [21] [22] [23] [24] . Crowding is typically characterized by asking observers to recognize a peripheral target object flanked by two distractors at varying target-to-flanker spacings. The critical spacing at which performance reaches threshold increases proportional to eccentricity 18, 19 , with reported rates ranging from 0.3 to 0.6. Our estimates of critical scaling for the mid-ventral model are in this range, but the substantial variability (which arises from different choices of stimuli, task, number of targets and flankers, and threshold) renders this comparison equivocal. Moreover, a direct comparison of these values may not even be warranted, as it implicitly relies on an unknown relationship between the pooling of the model responses and the degradation of recognition performance. a r t I C l e S We performed an additional experiment to determine directly whether our mid-ventral model could predict recognition performance in a crowding task. The experimental design was inspired by a previous study linking statistical pooling in the periphery to crowding 24 . First, we measured observers' ability to recognize target letters presented peripherally (6 deg) between two flanking letters, varying the targetto-flanker spacing to obtain a psychometric function (Fig. 6a) . We then used the mid-ventral model to generate synthetic metamers for a subset of these peripherally presented letter stimuli and measured the ability of observers to recognize the letters in these metamer stimuli under foveal viewing. Recognition failure (or success) for a single metamer cannot alone indicate crowding (or lack thereof), but the average performance across an ensemble of metamer samples quantifies the limitations on recognizability imposed by the model. Average recognition performance for the metamers is well matched to that of their corresponding letter stimuli (Fig. 6a) for metamers synthesized with scaling parameter s = 0.5 (the average critical scaling estimated for our human observers). For metamers synthesized with scaling parameters of s = 0.4 or s = 0.6, performance was significantly higher or lower, respectively (P < 0.0001, two-tailed paired t test across observers and conditions). These results are consistent across all observers, at all spacings, and for two different eccentricities (Fig. 6b) . We constructed a model for visual pattern representation in the mid-level ventral stream that computes local correlations amongst V1 responses in eccentricity-dependent pooling regions. In addition, we developed a method for generating images with identical model responses and used these synthetic images to show that when the pooling region sizes of the model are set correctly, images with identical model responses are indistinguishable (metameric) to human observers, despite severe distortion of features in the periphery. We found that the critical pooling size required to produce metamericity is robust to bottom-up and top-down manipulations of discrimination performance; that critical pooling sizes are consistent with the eccentricity dependence of receptive field sizes of neurons in ventral visual area V2; and that the model can predict degradations of peripheral recognition known as crowding, as a function of both spacing and eccentricity. Perceptual deficits in peripheral vision have been recognized for centuries. Most early studies focused on the loss of acuity that results from eccentricity-dependent sampling and blurring in the earliest visual stages. Crowding is a more complex peripheral deficit 39 . In 1976, Jerome Lettvin gave a subjective account of this phenomenon, describing letters embedded in text as having "lost form without losing crispness, " and concluding that "the embedded [letter] only seems to have a 'statistical' existence. " 20 . This article seems to have drifted into obscurity, but these ideas have been formalized in recent reports that explain crowding in terms of excessive averaging or pooling of features [21] [22] [23] [24] . One study in particular hypothesized that crowding is a manifestation of the representation of peripheral visual content with local summary statistics 24 , and showed that human recognition performance for crowded letters was matched to that of foveally viewed images synthesized to match the statistics of the original stimulus (computed over a localized region containing both the letter and flankers). Our model provides an instantiation of these pooling hypotheses that operates over the entire visual field, which, in conjunction with the synthesis methodology, enabled several scientific advances. First, we validated the model with a metamer discrimination procedure, which provides a more direct test than comparisons to recognition performance in a crowding experiment. Second, the parameterization of eccentricity dependence allowed us to estimate the size of pooling regions and to associate the model with a distinct stage of ventral stream processing. Third, the full-field implementation allowed us to examine crowding in stimuli extending beyond a single pooling region and to account for the dependence of recognition on both eccentricity and spacing, the defining properties of crowding 18 . Finally, the fact that our model operates on arbitrary photographic images allows generalization of the laboratory phenomenon of crowding to complex scenes and everyday visual tasks. For example, crowding places limits on reading speed, as only a small number of letters around each fixation point are recognizable 40 . Model-synthesized metamers can be used to examine this 'uncrowded' window (Fig. 7a) . We envision that our model could be used to optimize fonts, letter spacing or line spacing for robustness to crowding effects, potentially improving reading performance. There is also some evidence linking dyslexia to crowding with larger-than-normal critical spacing 18, 41, 42 , and the model might serve as a useful tool for investigating this hypothesis. Model-synthesized images also show how camouflaged objects, which are already difficult to recognize foveally, blend into the background when viewed peripherally (Fig. 7b,c) . The interpretation of our experimental results relies on assumptions about the representation of, and access to, information in the brain. This is perhaps best understood by analogy to trichromacy 14 . Color metamers occur because information is lost by the cones and cannot be recovered in subsequent stages. However, color appearance judgments clearly do not imply direct conscious access to the responses of © 2011 Nature America, Inc. All rights reserved. © 2011 Nature America, Inc. All rights reserved. 1 2 0 0 VOLUME 14 | NUMBER 9 | SEPTEMBER 2011 nature neurOSCIenCe a r t I C l e S those cones. Analogously, our experiments imply that the information loss ascribed to areas V1 and V2 cannot be recovered or accessed by subsequent stages of processing (two stimuli that are V1 metamers, for example, should also be V2 metamers). However, this does not imply that observers directly access the information represented in V1 or V2. Indeed, if observers could access V1 responses, then any additional information loss incurred when those responses are combined and pooled in V2 would have no perceptual consequence and the stimuli generated by the mid-ventral model would not appear to be metameric. The loss of information in our model arises directly from its architecture, the set of statistics and the pooling regions over which they are computed, and this determines the set of metameric stimuli. Discriminability of non-metameric stimuli depends on the strength of the information preserved by the model, relative to noise. As seen in the presentation time and attention control experiments, manipulations of signal strength did not alter the metamericity of stimuli and therefore did not affect estimates of critical scaling. These results are also consistent with the crowding literature. Crowding effects are robust to presentation time 43 , and attention can increase performance in crowding tasks while yielding small or no changes in critical spacing 19, 44 . Certain kinds of exogenous cues, however, may reduce critical spacing 45 , and perceptual learning has been shown to reduce critical spacing through several days of intensive training 46 . If either manipulation were found to reduce critical scaling (as estimated from a metamer discrimination experiment), we would interpret this as arising from a reduction in receptive field sizes, which could be verified through electrophysiological measurements. From a physiological perspective, our model is deliberately simplistic. We expect that incorporating more realistic response properties (for example, spike generation, feedback circuitry) would not substantially alter the information represented in model populations, but would render the synthesis of stimuli computationally intractable. Despite the simplicity of the model, the metamer experiments do not uniquely constrain the response properties of individual model neurons. This may again be understood by analogy with the case of trichromacy: color-matching experiments constrain the linear subspace spanned by the three cone absorption spectra, but do not uniquely constrain the spectra of the individual cones 14 . Thus, identification of V2 as the area in which the model resides does not imply that responses of individual V2 neurons encode local correlations. Our results, however, do suggest new forms of stimuli that could be used to explore such responses in physiological experiments. In a single pooling region, the model provides a parametric representation of local texture features 17 . Stochastic stimuli containing these features are more complex than sine gratings or white noise, but better controlled (and more hypothesis driven) than natural scenes or objects, and are therefore well suited for characterizing responses of individual cells 47 . Finally, one might ask why the ventral stream discards such a substantial amount of information. Theories of object recognition posit that the growth of receptive field sizes in consecutive areas, as well as with eccentricity, confers invariance to geometric transformations, and cascaded models based on filtering, simple nonlinearities and successively broader spatial pooling have been used to explain such invariances measured in inferotemporal cortex [8] [9] [10] 48 . Our model closely resembles the early stages of these models, but a b c Figure 7 Effects of crowding on reading and searching. (a) Two metamers, matched to the model responses of a page of text from the first paragraph of Herman Melville's Moby Dick. Each metamer was synthesized using a different foveal location (the letter above each red dot). These locations are separated by the distance readers typically traverse between fixations 40 . In each metamer, the central word is largely preserved; farther in the periphery, the text is letter-like, but scrambled, as if taken from a non-Latin alphabet. Note that the boundary of readability in the first image roughly coincides with the location of the fixation in the second image. We emphasize that these are samples drawn from the set of images that are perceptually metameric; although they illustrate the kinds of distortions that result from the model, no single example represents what an observer sees in the periphery. (b) The notoriously hard-to-find Waldo (character with the red and white striped shirt) blends into the distracting background and is only recognizable when fixated. Cross-hairs surrounding each image indicate the location of the fovea used by the model during synthesis. (c) A soldier in Afghanistan wears patterned clothing to match the stoney texture of the environment and similarly blends into the background.
In the visual cortex, spatial pooling over receptive fields of increasing size, latency and complexity REF is akin to a lossy signal compression, where only the most important information is conveyed from one level to the next.
Metamers of the ventral stream
Nature Neuroscience
Nature Neuroscience
Psychology Medicine