Visual clutter concerns designers of user interfaces and information visualizations. This should not surprise visual perception researchers because excess and/or disorganized display items can cause crowding, masking, decreased recognition performance due to occlusion, greater difficulty at both segmenting a scene and performing visual search, and so on. Given a reliable measure of the visual clutter in a display, designers could optimize display clutter. Furthermore, a measure of visual clutter could help generalize models like Guided Search (J. M. Wolfe, 1994) by providing a substitute for "set size" more easily computable on more complex and natural imagery. In this article, we present and test several measures of visual clutter, which operate on arbitrary images as input. The first is a new version of the Feature Congestion measure of visual clutter presented in R. Rosenholtz, Y. Li, S. Mansfield, and Z. Jin (2005). This Feature Congestion measure of visual clutter is based on the analogy that the more cluttered a display or scene is, the more difficult it would be to add a new item that would reliably draw attention. A second measure of visual clutter, Subband Entropy, is based on the notion that clutter is related to the visual information in the display. Finally, we test a third measure, Edge Density, used by M. L. Mack and A. Oliva (2004) as a measure of subjective visual complexity. We explore the use of these measures as stand-ins for set size in visual search models and demonstrate that they correlate well with search performance in complex imagery. This includes the search-in-clutter displays of J. M. Wolfe, A. Oliva, T. S. Horowitz, S. Butcher, and A. Bompas (2002) and Bravo and Farid (2004) , as well as new search experiments. An additional experiment suggests that color variability, accounted for by Feature Congestion but not the Edge Density measure or the Subband Entropy measure, does matter for visual clutter. Keywords: clutter, visual search, set size, Feature Congestion, Subband Entropy Citation: Rosenholtz, R., Li, Y., & Nakano, L. (2007). Measuring visual clutter. Journal of Vision, 7(2):17, 1-22, http://journalofvision.org/7/2/17/, doi:10.1167/7.2.17. Clutter is an important phenomenon in our lives and an important consideration in the design of user interfaces and information visualizations. It can interfere with searching for an important item, for example, a threat in a baggage X-ray, a document on our desktop, or a vehicle or pedestrian while driving. Clutter can interfere with quickly and veridically gathering visual information and making decisions. However, we lack a clear understanding of what clutter is; what features, attributes, and factors are relevant; why it presents a problem; and how to identify it. In practical applications, a computational measure of clutter could help either by allowing optimization of the level of clutter in displays over which we have control or by providing system alerts when clutter might impair task performance, for example, when road clutter might impair driving performance. In addition, a measure of visual clutter would also be useful for basic research in visual search. Most visual search research has used simple displays like that shown in Figure 1A . Researchers interested in search in more complex, naturalistic displays, however, need to be able to perform experiments on images more like that in Figure 1B and to generalize models of visual search to such images. This introduces numerous difficulties. Researchers have begun to address some of these difficulties, for example, by generalizing models of visual saliency to more complex imagery (Itti, Koch, & Niebur, 1998; and by exploring the influence of top-down information in more natural tasks and images (Torralba, Oliva, Castelhano, & Henderson, 2006) . One of the remaining difficulties in generalizing models to more complex and natural images concerns the notion of "set size," that is, the number of "items" in the display. Research on visual search has focused a great deal on reaction time (RT) versus set-size functions as a measure of search difficulty, and set size accounts for a significant proportion of the variance in simple search experiments. Furthermore, models such as Wolfe's (1994) Guided Search use the set size of the display to set a criterion for when to stop looking when the observer has not found the target. However, even in complex displays generated by the experimenter, the set size of the display is often unclear. In the map shown in Figure 2 , is a mountain range an item or multiple items? Does a single raindrop constitute an item? The outline of a state? In natural images, where the experimenter does not control the display, determining the number of items in the display becomes extremely difficult even given a reasonable definition of what constitutes an "item." We propose that visual search research requires, for complex imagery, a concept of visual clutter as a stand-in for the standard concept of set size. In the following section, we discuss what one might want from a measure of clutter. Then, we suggest a broad operational definition, followed by more specific candidate definitions, which will allow us to derive several measures of the level of clutter in a display. One such measure, the Feature Congestion measure of clutter, makes use of extensive modeling of what makes items in a display visually salient. Another is based on the notion that visual clutter is related to the amount of visual information in a display. Finally, we test Edge Density measure of clutter, suggested by Mack and Oliva (2004) , as a measure of subjective image complexity. In general, these clutter measures correlate well with the influence of a complex background on search performance both in previous search results from other researchers and in our own experiments. Visual search is a common subtask in many real-world visual tasks. A user must find buttons or other components of a user interface. An alert system must draw attention to a relevant part of a display so that the user can find it easily. Comprehending information visualizations also has a significant visual search component. By having an understanding of how clutter plays a role in visual search, we take a significant step toward understanding the role of clutter in many real-world visual tasks. Clutter as a stand-in for set size In substituting a notion of clutter for one of set size, we do not merely want to try to count the number of items in more complex displays. In the first place, this is currently unrealistic, given the state of computer vision algorithms. Furthermore, as mentioned in the previous section, items are ill-defined. What is an item in a natural scene such as that in Figure 1B : Is it a person? A shirt? A cup in a stack of cups? A branch on a tree? Furthermore, Bravo and Farid (2004) have argued that items are not even the unit of relevance, suggesting instead that the number of "parts" is more relevant to search performance. By analogy with the notion of set size, we should not expect a measure of clutter to predict, by itself, search performance. In standard RT versus set-size performance curves, as in Figure 3 , set size interacts with something like target-distractor discriminability to determine search difficulty. (Here, we may not mean target-distractor discriminability in exactly the sense of threshold in a single-item discriminability experiment because we may need to take into account greater search inefficiency in spatial-configuration or conjunction search situations, which may not entirely be captured by traditional discriminability. We use the term "discriminability" here more loosely.) In a simple view of visual search, targetdistractor discriminability determines which curve describes search performance. Set size determines essentially where a stimulus lies on a particular performance curve; that is, it determines the x-value. Together, the two (curve plus x-value) predict the performance, for example, RT. Similarly, we would expect clutter to interact with discriminability to predict search performance in complex imagery. Additional factors also influence search performance. Of particular importance for the issues of clutter and set size is the effect of top-down information on visual search. When a cue indicates that the target will appear at a subset of possible locations in a display, RTs are a function of the relevant (cued) rather than the nominal set size (Palmer, 1994; Palmer, Ames, & Lindsey, 1993) . Features of the target may also cue a subset of potential items as possible targets, again effectively reducing set size. Furthermore, researchers have shown that in complex natural scenes, expertise such as prior knowledge about regions likely to contain a target, such as a pedestrian, can limit eye movements during search to those regions (Torralba et al., 2006) . In testing our measure of visual clutter, we propose to initially minimize these top-down effects by focusing on search for categorical targets in situations that minimize prior knowledge about likely target locations. (If one were trying to determine whether set size is relevant for search performance, one would not first perform experiments in which relevant set size is unknown and differed from nominal set size.) Once we have confidence in a measure of visual clutter, such a measure can help us better evaluate experimental results involving more complex stimuli and search tasks, including those with significant top-down components. Our goals for this article are to derive, implement, and test several initial measures of visual clutter. These measures should be able to operate on arbitrary images, rather than requiring a list of the items in the display and their properties (e.g., "green tree at location (5, 2.6)"). The measures should behave sensibly on standard simple psychophysical displays. Furthermore, they should correlate well with performance in search experiments, at least when one (approximately) controls for target-distractor discriminability and when such experiments have a minimal top-down component to the visual search task. None of the candidate clutter measures explicitly deal with objects, but they will be a function of the number of objects in the display, as well as of their appearance and organization. Furthermore, the measures may be applied to any static display because they take an image as input and do not require a list of items in the display. Clutter is the state in which excess items, or their representation or organization, lead to a degradation of performance at some task. Excess and/or disorganized display items can cause crowding (Stuart & Burian, 1962) , masking (Legge & Foley, 1980) , decreased object recognition performance due to occlusion, and impaired visual search performance (see Wolfe, 1998 , for a review). More items can also stretch or exceed the limits of short-term memory (Miller, 1994) . In the case of short-term memory, the relevant factor seems not to be merely the number of objects, but their features (color, orientation, etc.); the Figure 3 . RT versus set-size (or clutter) curves. Set size (clutter) and target-distractor discriminability interact to predict performance. Target-distractor discriminability select which curve describes the RT data; for example, a difficult feature search might put performance on the green curve. Set size (clutter) selects location on this curve (circle or triangle), to predict RT. (2007) 7(2):17, 1-22 Rosenholtz, Li, & Nakano 3 capacity for certain simple features is higher than for more complex features (Alvarez & Cavanagh, 2004) . Sometimes, more items lead to performance benefits. This tends to occur when the items are low entropy, meaning that the appearance of one item is easily predicted from its neighbors. Under such conditions, it can be easier, for instance, to spot a trend in data when there are a larger number of points contributing to that trend or to notice a grouping of items with similar characteristics. Given a consistent trend or group of items, more items can aid detection of a deviationVan outlier with features or trend different from the group, or a boundary between two differing groups. Because clutter is a state when excess items lead to performance degradation, ideally, a measure of visual clutter should be modulated by low-entropy conditions in which additional items may actually lead to performance benefits. It should also capture the various phenomena in which additional or disorganized items degrade performance. However, although all of these phenomena have been studied extensively, many lack adequate models. There exist a number of models of visual search, and in the next section, we first suggest a measure of visual clutter based on such a model. When is a desk cluttered? In deriving a measure of visual clutter, consider a situation in which one wants to leave a note on a colleague's desk, with the hope that it will draw attention and the colleague will act upon it. If the colleague's desk is uncluttered, it seems easy to place a note such that one is confident that the colleague will notice it. If the colleague's desk is cluttered, one cannot be confident of drawing attention, and often, one leaves a note on an uncluttered chair instead of the desk. This analogy suggests that the level of visual clutter in a display or scene is related to the ease of adding an attention-drawing target to that display or scene. However, predicting, given a background image, the difficulty of adding an attention-grabbing target is not a usual task for a visual search model. Most such models are instead built to predict the ease of searching for a particular target among particular distractors and would need to be run iteratively to predict the difficulty of adding an item that would draw attention. Luckily, one of the authors (Rosenholtz, 1999 (Rosenholtz, , 2001a (Rosenholtz, , 2001b has, for some time now, been developing the Statistical Saliency Model. Rosenholtz' model tries to capture human performance at a functional rather than biological level, utilizing the notion that the visual system is designed to characterize various statistical aspects of the visual display. This statistical framework leads to easier intuitions for why a target in a display is or is not salient and can suggest features for a salient target or predict the ease of selecting features to create a salient target. Furthermore, this model has recently been implemented so it can run on arbitrary images . In the following subsection, we describe the Statistical Saliency Model for visual search and show how it can predict the ease with which one can add an attentiongrabbing target to a display. Then, we will develop our measure of visual clutter and describe its implementation. Rosenholtz begins with the premise that the visual system has an interest in detecting "unusual" items. She suggests that an item is unusual and, thus, salient if its features are outliers to the local distribution of features in the display. Following a long line of visual search researchers, these features are likely to include such things as contrast, color, orientation, and motion (see Wolfe, 1998 , for a review). Rosenholtz suggests a measure like a z-score for the degree to which a feature vector, T, is an outlier to the local distribution of feature vectors, represented by their mean, 2 D , and covariance, @ D . The saliency, $, is given by the following equation: where ðT j 2 D ÞVindicates a vector transpose. The higher the target saliency, the easier the predicted search. The saliency, $, can be thought of as a formalization of Duncan and Humphreys ' (1989) notion of the different roles of target-distractor versus distractor-distractor similarity in search performance. Rosenholtz's model predicts the results of a wide range of search experiments involving basic features such as those mentioned above (Rosenholtz, 1999 (Rosenholtz, , 2001a (Rosenholtz, , 2001b Rosenholtz, Nagy, & Bell, 2004) , including experiments that were previously thought to involve search asymmetries (Rosenholtz, 2001a) . More recently, this model has been implemented to run on arbitrary images and shown to be predictive of eye movement data . Consider the graphical interpretation in Figure 4 . The Statistical Saliency Model essentially represents the local distribution of features by a set of covariance ellipsoids in the appropriate feature space, as shown. The local covariance, @ D , specifies the size, aspect ratio, and orientation of the covariance ellipsoids. The innermost, 1A, ellipsoid indicates feature vectors 1 SD away from the mean feature vector, 2 D . The 2A ellipsoid indicates feature vectors that are 2 SD away from the mean, and so on. A target with a feature vector on the nA ellipsoid will have saliency $ = n. The farther out the target feature Journal of Vision (2007) 7(2):17, 1-22 Rosenholtz, Li, & Nakano 4 vector lies on these nested ellipsoids, the easier the predicted search. One can also use this model to choose attention-drawing features for an item in the display. If a designer wants to add an item to a given portion of the display, such that the saliency of that item is at least d, then any features outside of the local dA covariance ellipse will suffice. The Statistical Saliency Model can indicate the difficulty of adding a new, salient item to a local area of a display. Figure 5 demonstrates this. Figure 5A depicts a number of lines with identical orientation and variable luminance. Figure 5B shows a cartoon of the associated covariance ellipse, assuming an (orientation, luminance) feature space. The display in Figure 5A has high variance in the luminance direction and low variance in the orientation direction (its only variance is due to observation noise). Thus, the ellipsoid has low area, and it would be easy to draw attention to a target simply by giving it a significantly different orientation, as shown in Figure 5C . On the other hand, the display in Figure 5D has high variance in both orientation and luminance, as shown by the covariance ellipse in Figure 5E , which has a relatively high area. It would be difficult to draw attention to a target in this display using only orientation or luminance as features. The volume of the local covariance ellipsoid represented by @ D therefore gives a measure of the local clutter in a display, that is, of the difficulty of adding a new, salient item to a local area of a display. Locally measuring the ellipsoid size and pooling over the relevant display area gives a measure of clutter for the whole display. We call this the Feature Congestion measure of visual clutter. Displays with high clutter, according to this measure, are cluttered because feature space is already "congested" (filled by the covariance ellipsoid), so that there is little room for a new feature to draw attention. Too many colors, sizes, shapes, and/or motions are already clamoring for attention. Our discussion leads us to a surprisingly simple measure of clutterVThe clutter in a local part of a display is related to the local variability in certain key features. Implementation of the Feature Congestion clutter measure involves four stages: (1) compute local feature (co)variance at multiple scales and compute the volume of the local covariance ellipsoid, (2) combine clutter across scale, (3) combine clutter across feature types, and (4) pool over space to get a single measure of clutter for each input image. In the current implementation, we use color, orientation, and luminance contrast as features. Color, luminance contrast, and orientation have been used to model a number of perceptual phenomena, for example, pattern discriminability (Watson, 2000) and preattentive texture segmentation (Malik & Perona, 1990) . Color naturally seems important to our sense of clutter. Contrast-energy feature detectors are known to exist early in the visual system and can not only detect simple luminance contrast but also serve as a measure of size and shape (see Rosenholtz, 2000, for a review of evidence that such detectors may mediate preattentive processing of shape in the human visual system). Future implementations might include other features believed to be basic features in visual search and attention, for example, features suggested by the work of Treisman and Gelade (1980) . Below, we briefly describe our implementation. See Rosenholtz (2000) for more details on the feature covariance stage of processing because similar steps are used to segment an image into regions of different textureVa process thought to have much in common with visual search in the human brain. The Feature Congestion measure of visual clutter depends upon being able to estimate the perceptual distance between feature vectors. Therefore, we start by converting the input image into the perceptually based CIELab color space (C.I.E., 1978). We then process the image at multiple scales (currently, three) by creating a Gaussian pyramid by alternately smoothing and subsampling the image (Burt & Adelson, 1983) . Next, we find features at each scale. For luminance contrast, we compute a form of "contrast energy" by filtering the luminance band by a center-surround filter formed from the difference of two Gaussians and squaring the outputs. For  color, we extract a local mean color at each scale by pooling with a Gaussian filter. For orientation, we compute oriented opponent energy, a la Bergen and Landy (1991), which gives us a two-vector, (k cos(2E), k sin(2E)), at each image location and scale, where E is the local orientation and k is related to the extent to which there is a single strong orientation at the given scale and location. Then, for each feature, we compute the local (co) variance for each feature. This may be done efficiently, and in a biologically plausible way, through a combination of linear filtering (to average over a local area) and point-wise nonlinear operations (to compute the variance). From a covariance matrix, it is straightforward to compute the volume (area) of the covariance ellipsoid, our local measure of color (orientation) feature congestion, that is, clutter. The contrast feature congestion is simply the square root of the contrast variance. For each feature, we combine feature congestion across scale by taking the maximum at each pixel. We reason that a feature is locally congested if it is congested at any scale. Little has been done to examine the interaction of multiple scales in determining target saliency or the influence of clutter, and more basic research needs to be done to have a better understanding of how information combines across scale. At this point, we have three clutter maps for the image, representing the "color congestion," "texture congestion," and "orientation congestion." Next, we combine color, contrast, and orientation clutter at each point. We first take the cube root of color clutter (a volume) and the square root of orientation clutter (an area) to make these more comparable to contrast energy clutter (a scalar). Even so, the three clutter measures are not scaled equivalently. The true measure of how congested a feature space is is how much of feature space is taken up by the covariance ellipsoid relative to how much feature space is available. Therefore, it is appropriate to scale the clutter value in each feature dimension by essentially the range of possible clutter values for that feature. Currently, we approximate that range by normalizing by the standard deviation of clutter values for a given feature over a wide range of input images. We then combine the scaled color, contrast, and orientation clutter at each point by taking their sum. A final model of clutter will almost certainly involve a more complicated combination rule than that used here. The features might, for instance, be combined into a single large feature vector prior to computation of the covariance, as might be suggested by the search results and modelling of Eckstein, Thomas, Palmer, and Shimozaki (2000) . Some features might have priority over other features; for example, Callaghan (1989) has suggested that color dominates over geometric form in texture segregation. Much basic research needs to be done to adequately model feature interaction. In the absence of such research, our aim was to see how far we could go with a simple measure. Early attempts to allow a general linear combination of clutter across features did not greatly improve performance of this clutter measure. Combine across space Finally, we pool over space to get a single measure of clutter for each display by taking the average clutter value over the entire image. For a given number of objects in a scene, the scene will appear less cluttered the more "organized" it is. Organization may involve grouping similar objects together; aligning them; and making many of the objects a similar hue, luminance, size, and so forth. This principle is well known in the literature on decluttering one's home, and research has shown that perceptual organization of this sort affects search performance (Treisman, 1982) and other visual tasks. The degree of organization of a scene can be thought of in terms of the extent to which each part of the scene is predictable from the rest of the scene, or in terms of the amount of redundancy in the scene. The Feature Congestion measure of clutter captures this concept of "organization" to some extent implicitly; by looking at feature covariance, it essentially captures some measure of the grouping by similarity + proximity in the display. To the extent that an image contains redundancy, it can be represented, either in the brain or in a computer, with a more efficient code. If, for example, a region of the image forms a single homogeneous group, then that region can be encoded by noting its group characteristics and location instead of encoding each point within that region. Similarly, if a number of objects repeat regularly, they can be efficiently encoded by representing the object and its repetition pattern. Therefore, the less cluttered an image is, the more it is redundant and the more efficiently it can be encoded. We propose a second measure of clutter based on attempts to measure the efficiency with which the image can be encoded while maintaining perceptual image quality. For this purpose, one would ideally like an encoder that detects the sorts of redundancies exploited by the visual system. Both low-level visual processing and mid-level visual processing seem aimed at capturing redundancies in natural images. Early filtering by V1 receptive fields can be thought of as capturing low-level redundancies in the input. Researchers have suggested that a way to achieve efficient coding of natural scenes is to choose early filtering operations that maximize sparse representation and that this principle leads to receptive fields like that in V1 (Olshausen & Field, 1996) . Midlevel perceptual organization captures higher level redundancies in images by encoding groupings, symmetry, and so on. However, although one would ideally like to know how efficiently one can encode an image by making use of the same sorts of redundancies as the visual system, this is difficult to do given the current state of understanding and modeling of perceptual organization and mid-level processing. Therefore, for our purposes, we initially suggest using encoding efficiency of current, highly successful subband image coding methods such as JPEG 2000. Our Subband Entropy clutter measure is based on the notion that clutter is related to the number of bits required for subband (wavelet) image coding. A wavelet coder first decomposes the image into a set of subbands with different orientations and spatial frequencies, which is analogous to the decomposition that occurs early in human vision. Thus, it captures some of the same redundancy in images as early vision. We use steerable pyramids as the basis for our Subband Entropy measure (Simoncelli & Freeman, 1995) . A number of effective image coders such as JPEG and JPEG 2000 follow a subband transform of the image with entropy encoding. Similarly, in our Subband Entropy measure of clutter, we next compute the Shannon entropy within each subband. Shannon entropy is defined as Here, p is the probability distribution of coefficients in each subband and is estimated by binning (i.e., quantizing) the subband coefficients into bins indexed by i and computing a histogram. This Shannon entropy essentially captures the bits required to encode the subband, for a given level of fidelity, as specified by the coarseness of the bins (quantization). Higher fidelity, that is, finer bins, requires more bits to encode. In our computations, the number of bins is equal to the square root of the number of coefficients, meaning that bands with fewer coefficients also have, on average, fewer coefficients per bin. This implicitly says that it is more important to faithfully reproduce lower frequencies than high frequencies; at lower frequencies, a wavelet transform has fewer coefficients, and thus, this strategy leads to finer bins, more bits required, and more faithful encoding. The clutter measure is computed as a sum of these subband entropies. The algorithm is as follows: 1. Convert the RGB image into CIELab. 2. Decompose the luminance (L) and the chrominance (a, b) into wavelet subbands using a steerable pyramid. 3. Bin the wavelet coefficients within each subband and compute the Shannon entropy within each subband according to Equation 2. Journal of Vision (2007) 7(2):17, 1-22 Rosenholtz, Li, & Nakano 7 4. Sum the subband entropies for the luminance (L) and for the chrominance channels (a, b). 5. Compute a weighted sum of chrominance and luminance entropies. We used a weighting of 0.08 for each of the chrominance channels and 0.84 for the luminance channel. Image encoders typically use fewer bits for chrominance than luminance channels because chrominance need not be coded with as much fidelity, without compromising image quality. The Subband Entropy measure, however, is not very sensitive to changes in this weightingVa chrominance weighting of 0.22 to a luminance weighting of 0.56 gave nearly identical results on all examples in this article. There have been several past attempts to quantify something like the "clutter" in a display. Researchers in the field of information visualization have attempted to measure what they call "information density." In information visualization, one often has decisions to make about how much information to present to the user in a given display or part of a display. The hope has been that with a measure of the information density, designers would be able to choose the optimal level of information to present, trading off giving the user more information against making it more difficult for the user to quickly and veridically extract that information from the display. In this context, Woodruff, Landay, and Stonebraker (1998) experimented with several measures of information density: the number of visible objects and the number of vertices. Other researchers have suggested additional measures of information density or clutter. Dynamic Logic's (2001) study on the effect of clutter on the effectiveness of Web advertisements used the number elements on a Web page as a metric, where an element consisted of a word, graphic, or "interest area." Tufte (1983) suggested the number of entries in the source data matrixVthat is, the table of data used to generate the visualizationVper unit area. Nickerson (1994) enumerated a number of density measures, including the number of graphic tokens per unit area, the number of vectors needed to draw the visualization, and the length of program to generate the visualization. Frank and Timpf (1994) suggested the amount of "ink" per unit area as a metric for simple black and white maps. These metrics have a number of difficulties, and few have actually been implemented. Certainly, the amount of clutter has some dependency upon the number of objects, graphic tokens, or entries in the source data matrix in the display. The number of objects (or, conversely, the amount of blank space) has been shown to influence task performance at both a directory assistance search task (Springer, 1987) and at a number of tasks with a map (Phillips & Noyes, 1982) . However, column alignment helped in the directory assistance task, and Phillip and Noyes (1982) found that "like clutters like"; for example, additional lines on a map cause increased difficulty for tasks involving lines. Counting the number of objects does not take into account the appearance or organization of the objects. Merely counting the number of elements on a Web page did not prove to be a good measure of clutter in the sense of correlating with advertisement effectiveness (Dynamic Logic, 2001) , although human clutter judgments did correlate with advertisement effectiveness. Furthermore, as mentioned above, for complex visualizations, the number of objects can be ill-defined. A vertex may be similarly ill-definedVHow sharp a turn in a curve is required? The amount of ink is ill-defined for color displays. In addition to these difficulties in applying suggested clutter measures when we are given a list of drawing objects in a visualization, the situation becomes even bleaker if one wants to measure the clutter in an image, for example, in a natural scene. Oliva, Mack, Shrestha, and Peeper (2004) have attempted to determine what factors influence the human representation of "complexity," a concept clearly related to that of clutter. They had users hierarchically sort photographs according to their complexity and indicate at each hierarchical level the basis for the sort. Users indicated that complexity depended on the quantity and variety of objects, detail, and color, as well as on higher level, more global concepts like the symmetry, organization, and "openness" of the depicted space. Our Feature Congestion and Subband Entropy measures of visual clutter, as we will see, correlate with quantity and variety of objects, although we do not explicitly find objects. Our Feature Congestion measure explicitly incorporates a notion of the variety of color in an image, and the Subband Entropy measure also captures this, because more color variability means less predictability, and thus, more information = entropy = clutter. Both measures will implicitly deal with certain aspects of perceptual organization, such as grouping by a combination of proximity and similarity. In a later version of the Feature Congestion measure, we are interested in incorporating some of the higher level components of perceptual organization that Oliva et al. suggest. Mack and Oliva (2004) have implemented early versions of several measures of complexity: quantity of contours, degree of symmetry, global color variability, and degree of openness. (We argue that local color variability is more appropriate because it implicitly responds to local groupings of color. Local variability has been shown to affect search performance; e.g., Nothdurft, 1993.) They compared these measures to the mean subjective rankings of complexity on indoor scenes and found a correlation of r = .85. Edge densityVthe percentage of pixels that are edge pixelsValone led to a correlation with mean subjective rankings of r = .83. (Note, however, that the mean Spearman rank- order correlation between subjects was only r = .61, comparable with what we have found for subjective judgments of clutter in maps; Rosenholtz, Li, Mansfield, & Jin, 2005; r = .70. In that article, we found a correlation of r = .77 between median subjective judgments of clutter in maps and an earlier version of the Feature Congestion clutter measure. None of these differences in correlation coefficients is significant, p 9 .05.) This high correlation between subjective judgments of complexity in indoor scenes and such a simple measure as edge density suggests that this simple measure is worth examining further. In what follows, we also examine the performance of an Edge Density measure of visual clutter. To obtain the Edge Density measure for each image, we applied MATLAB's Canny edge detector to each image and measured the density of edge pixels. The Canny edge detector has several parameters: a low threshold, high threshold, and sigma. These parameters were set by hand to values that gave good results overall to the examples presented in this article. The low threshold and high threshold are used to find weak and strong edges, respectively, and the Canny edge detector keeps weak edges only if they are connected to strong edges. These thresholds were set to 0.11 and 0.27, respectively. The sigma parameter is the standard deviation of the Gaussian filter used in the computation of the gradient. It was set to the default, A = 1, comparable with the finest scale in the Feature Congestion and Subband Entropy measures. Our goal in this article is to design and test several measures of visual clutter: Feature Congestion, Subband Entropy (refer to http://hdl.handle.net/1721.1/37593 where the two clutters were implemented), and Edge Density. Because a good part of our interest in such measures is to have a stand-in for set size in models of visual search, we need to test whether these measures correlate well with performance in visual search tasks. In particular, we should hope and expect them to perform well in tasks involving minimal top-down information about the appearance and location of the target because such information could lead to the equivalent of a "relevant set size or clutter" as opposed to straight set size or clutter. The influence of topdown information on visual clutter could be included in a later implementation of any of these measures but is not included at present. In Experiment 1, we test the three measures against RT in a visual search task involving searching for a category target in map images and show that the measures perform well. Reaction times, however, are governed by a number of complicated factors. In Experiment 2, we instead use limited display times and find contrast thresholds necessary to correctly identify the target at a given level of accuracy. We show that the clutter measures also correlate well with this more traditional psychophysical measure of performance. Next, we demonstrate that the clutter measures give sensible answers on standard simple search displays. Then, we test these measures against previous results of search in clutter, predominantly limiting ourselves again to those previous results with minimal top-down components. Experiment 3 is aimed at distinguishing between these measures through more careful selection of displays and, in particular, at examining whether color variance is an important aspect of visual clutter. In our first experiment, we examine the influence of visual clutter, as determined by our various measures, on RT in a visual search experiment. In many visual search experiments, an observer searches for only one target per "scene," and only once. Although studied little in human vision, many real-world scenarios involve repeated search in the same scene for a number of different targets. We look for our keys, then our gloves, then our computer bag, and so on. When an observer searches only once in each scene, the target saliency is a key factor determining performance. In a simple scene, a salient target can save the observer as much as a few hundred milliseconds. On the other hand, in repeated-search scenarios, observers cumulatively spend a lot of time interacting with each scene. As a result, properties of the scene that enable or impair search performance become important. This is why some people spend so much time trying to organize and declutter their offices, homes, and other scenes over which they have control and within which they do many different tasks. A cluttered scene can cost us, over time, far more than a few hundred milliseconds. Due to our interest in more real-world search scenarios and in clutter, in particular, we use a repeated search task to study the extent to which the background scene within which the observer searches influences search performance. To focus on bottom-up influences such as clutter, as opposed to top-down influences such as priors on likely target location, the categorical targets can appear with equal probability in any of six isoeccentric locations. Rather than looking at performance at searching for a particular target in a particular location, which will tend to be strongly influenced by factors such as target saliency, we examine how, on average, across targets and locations, the background scene affects search performance. Observers searched for a categorical targetVa GaborVin a number of geographic maps. Although Gabors do not typically appear in geographic maps, this choice provided several advantages. First, we could, in principle, have had a more natural task in which observers search for targets already existing in the maps. But first, symbols already present in a map can hardly be expected to appear all at the same eccentricity, thus adding another source of variance to the data. Furthermore, searching for symbols already present in the maps would have been a more difficult experiment to set up because all the maps contain different symbols, and we would need to both determine which symbols appeared in which maps and inform the observer of a different target on each trial. We wanted observers to search for a variety of targets, not already present in the maps, over which we had a fair degree of control over their appearance, while keeping instructions to observers simple by having them search for a categorical target. We chose targets consisting of 16 grayscale Gabors. (Earlier pilot experiments had observers search for a bull's-eye target, which one might think of as a "you are here" symbol, and the results in terms of correlation with various measures of visual clutter were virtually identical to those presented here.) The wavelength of the carrier sine wave was always twice the standard deviation of the envelope. The Gabors varied in scale (envelope A = 0.55-or 0.83-), orientation (45-, 90-, 135-, or 180-), and phase (sine or cosine). Before the experiment, observers were shown a number of examples of Gabors, and each observer received a training block with feedback to ensure that he or she understood the task and nature of the targets. Each target appeared exactly once in each of the six general locations of the image. The exact location of the target was determined by the superposition of one of six locations in the image and a small random position jitter of up to 0.75-in both x and y directions. The prejitter target locations were isoeccentric at approximately 7.6-from the initial fixation. Nineteen colored map images served as the background against which the targets were superimposed. The targets were added as a semitransparent layer on top of the maps. The map images were found using a random search for maps on the Web to test the three candidate clutter measures on typical maps. The maps were chosen so that 8 were at the scale of showing an entire country, 3 were at the scale that showed several cities, 4 showed a large portion of a city, and 3 showed only a few streets. Again, this was done to get a sampling of typical existing maps. The maps were cropped to be uniform in size, to avoid effects in which search might take longer simply because there was more area to search, and spanned about 24-of visual angle when viewed during the experiment from 15 in. The maps were corrected to have a mean luminance of mid-level gray and to have approximately the same mean color (also a mid-level gray) for all. In principle, this was done to minimize a possible source of variance in which search in one map might be easier or harder merely because the map was darker or closer to the target (gray) color than another image. In practice, a pilot experiment suggests that this manipulation had virtually no effect on the fits of the clutter measures to the experimental data. The mean luminance of the Gabor target was always set to the local mean luminance of the map, again to minimally control for local contrast of the target, which might affect search performance. Figure 6 gives an example of a map with superimposed Gabor target. Each observer was seated in front of a computer display. Observers were instructed to search for a Gabor target against each background image and to respond as quickly and accurately as possible as to whether a target was present or absent. The image was displayed until subjects indicated their response by pressing "f" for target absent or "j" for target present. After each trial, visual feedback was given to indicate whether the response was correct. After the feedback, a fixation-cross appeared in the center of the display for 500 ms before the next trial began. A total of 3,648 trials (16 targets Â 19 images Â 6 locations Â target absent/present) were divided evenly among 12 blocks. We ran our experiments in MATLAB, using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) . Six subjects participated in Experiment 1. All observers had normal or corrected-to-normal vision and were compensated for their time.  We are interested in how the clutter of an image correlates with search performance within that image. As a result, in what follows (both for this experiment and later experiments), we average performance across target locations within an image and average across subjects. Error bars (which show standard error) on the plots give an indication of the variability across subjects and target locations. We report correlation coefficients between clutter and mean performance, essentially asking what percentage of the variance in mean performance is accounted for by a given clutter measure. All clutter measures were computed on the background images before adding the targets. Figure 7A shows mean log(RT) versus the Feature Congestion clutter measure. Figure 7B shows the results for the Subband Entropy measure. Figure 7C shows the results for the Edge Density measure. Each data point represents 1 of the 19 background images. We average RT from correct trials over target location, Gabor type, and subject to get a single measure of RT per image for both target-absent and target-present trials. We find a significant correlation between the average log(RT) and each of the clutter measures: Feature Congestion (r = .74, target present; r = .76, target absent; p G .001), Subband Entropy (r = .75, target present; r = .77, target absent; p G .001), and Edge Density (r = .83, target present and target absent; p G .001). All three measures of visual clutter do a good job of predicting the effect of the background imageVthat is, the effect of the display clutterVon search performance. None of these correlation coefficients differ significantly from each other (p 9 .05). In this experiment, we compare the outputs of the three clutter measures to a more traditional psychophysical measure of performance: contrast thresholds necessary for a given level of performance at a visual search task. In this, as in later experiments, unless otherwise specified, the methods and stimuli are as in Experiment 1. Observers searched for an arrow symbol in a number of geographic maps. The arrows were dark against a light rectangular background, so that their average color was mid-level gray. Each arrow pointed either to the left or to the right. Arrows were approximately 1.2-in length. Figure 8 shows an example of an arrow in a map. Twenty colored map images served as the background images against which the targets were superimposed. Fifteen of these maps were the same as in Experiment 1. Four of those maps were removed from this experiment because they already contained arrows, and 5 maps of approximately the same clutter were added in their place. The arrows were again added transparently to the background images; this meant that at 0 contrast, the arrows were invisible.
Visual complexity can also be approximated using algorithmic information theory and compression algorithms REF .
Measuring visual clutter
Journal of vision
Journal of vision
Psychology Medicine