A hierarchical definition of optical variability is proposed that links physical magnitudes to visual saliency and yields a more reductionist interpretation than previous approaches. This definition is shown to be grounded on the classical efficient coding hypothesis. Moreover, we propose that a major goal of contextual adaptation mechanisms is to ensure the invariance of the behavior that the contribution of an image point to optical variability elicits in the visual system. This hypothesis and the necessary assumptions are tested through the comparison with human fixations and state-of-the-art approaches to saliency in three open access eye-tracking datasets, including one devoted to images with faces, as well as in a novel experiment using hyperspectral representations of surface reflectance. The results on faces yield a significant reduction of the potential strength of semantic influences compared to previous works. The results on hyperspectral images support the assumptions to estimate optical variability. As well, the proposed approach explains quantitative results related to a visual illusion observed for images of corners, which does not involve eye movements. Keywords: optical variability, contextual adaptation, saliency, efficient coding, eye fixations, face saliency, hyperspectral Citation: Garcia-Diaz, A., Leborán, V., Fdez-Vidal, X. R., & Pardo, X. M. (2012). On the relationship between optical variability, visual saliency, and eye fixations: A computational approach. Journal of Vision, 12(6):17, 1-22, http://www. journalofvision.org/content/12/6/17, doi:10.1167/12.6.17. Biological vision establishes unrivaled benchmarks in terms of efficiency, robustness, and general performance in active visual tasks. These capabilities demand an active and dramatic selection of information that poses a main cause for visual attention. Evidence indicates that bottom-up processing (plenty of adaptation mechanisms) and data-driven saliency play a central role in the control of human visual attention and determine visual priority in cooperation with topdown relevance. The terms saliency as a data-driven property of image points, relevance as a semantic property, and priority as the combination of both are used for the sake of clarity, similar to Fecteau & Munoz, 2006 . Concerns on the understanding of the human visual system (HVS) as much as on the development of active vision systems have fostered an important and cross-disciplinary effort to improve the estimation of saliency and the efficiency of low level representations. As a result, the bio-inspired modeling and the applications of saliency have registered a steady increase of research activity. However, there is a lack of computational models that address the relationship between the contextual data-driven adaptation observed in early visual coding and the perception of saliency. Most existing models decompose the image through its projection on a predefined and fixed basis of low level features. They leave all the adaptive work well in a rigid process of normalization and weighted summation of the initial responses, or well directly in a subsequent local measure of dissimilarity or improbability of the ensemble of features. Approaches to this problem are very interesting for both the understanding of the HVS and for computer vision applications, as far as they may yield improved models of adaptive low level features and saliency. Journal of Vision (2012) 12(6) Most theorists agree that the early description derives from spatial groupings of a small set of simple primitives that are registered in parallel across the visual field. These primitives, or functional features, need not correspond to simple physical dimensions like wavelength or intensity. Models of the second group already point to a more reductionist approach and they ultimately claim to compute an efficient approximation of the inverse of the probability density of the low level content present in the image. However, both approaches lack specification of the physical sources involved, and more importantly, of the different ways in which they contribute to visual saliency. Remarkably, there are a number of evidences pointing to an invariant behavior of the HVS in the way it manages low level content, and particularly saliency. B. W. Tatler, Baddeley, & Gilchrist (2005) showed that while consistency between subjects decreases over time even without forcing a common starting location, there is no evidence for variation in the discrimination between the saliency at fixated and nonfixated locations. They used a number of specifically modeled low level features to account for saliency. Recent results by Foulsham & Underwood (2008) (1998), c) random selection with distance weighting independent of bottom-up and top-down processes as proposed by Melcher & Kowler (2001) , and d) strategic divergence, which as proposed by the authors means that top-down strategies chosen by observers are different, while the bottom-up frame of reference remains the same. This last possibility is the only one compatible with a decrease in the consistency between observers, even with free starting locations, and the constancy of low level content of fixations over time, both reported in the study. From comparison of eye fixations on natural images between patients with visual agnosia and healthy subjects, Mannan, Kennard, & Husain (2009) showed that consistency between observers in the very first fixations was equivalent for healthy and unhealthy subjects. However, for subsequent fixations, only unhealthy subjects (impaired to understand the image) maintained the consistency between fixation patterns. This result also points to a constant influence of saliency and an increasing and divergent influence of relevance in the spatial distribution of fixations in healthy subjects. All of this suggests invariance in the perception of visual saliency-strictly data-driven-that makes even more interesting the development of efficient computational approaches to yield an accurate estimation of the same. In previous works we have shown that the decorrelation of local scale features is sufficient to explain a variety of psychophysical results and to predict human fixations at state-of-the-art performance (Garcia-Diaz, Fdez-Vidal, Pardo, & Dosil, 2009) . As a generalization, in this paper we propose an estimation of optical variability that involves few magnitudes-intensity, spectral wavelengths, and spatial frequencies. This measure is shown to rely on a contextually adapted representation of the image arising from biologically plausible operations. Unlike most previous approaches to visual saliency that use a fixed basis of features to decompose the input image, the basis of components employed to compute optical variability is adapted for each specific scene. That is, our approach is rooted on a physical rather than only informational theoretic ground and links a parsimonious contextual adaptation of the low level representation to the computation of saliency. Therefore, a major contribution is the explicit proposal of the invariance of the HVS to cope with relative optical variability. A concrete implementation of such a measure is able to outperform many other state-of-the-art models of saliency in the prediction of human fixations in natural scenes, using two open access eye-tracking datasets. The impact of several approximations is assessed as well and a high robustness in the management of scales and spectral sensitivities is demonstrated. Besides, results on an additional open access dataset of images with faces suggest a lower influence of face relevance than recently reported using a classical model of saliency. Beyond a variety of psychophysical results also reproduced by different previous computational models, the proposed measure is able to quantitatively explain the linearity of perceived saliency versus corner angle. To our knowledge, this behavior is not correctly reproduced by any other model. Overall, the proposed approach exhibits an improved performance and robustness in major benchmarks and yields new insights in the perception of visual saliency, beyond the reach of existing models. The paper starts with a presentation of some necessary background information, followed by the definition of optical variability and the proposed hypothesis of invariance of saliency. Next, implementation details, datasets, and evaluation procedures used are described. Then results for each of the four selected experiments are shown. Finally, a discussion of the results and their implications is given.
The work in REF introduces the optical variability, and modeling the relationship between the magnitudes between optical flow and visual saliency.
On the relationship between optical variability, visual saliency, and eye fixations: A computational approach
Journal of vision
Journal of vision
Psychology Medicine