Three levels of virtual environment (VE) metric are proposed, based on: (1) users' task performance (time taken, distance traveled, and number of errors made), (2) physical behavior (locomotion, looking around, and time and error classification), and (3) decision making (i.e., cognitive) rationale (think aloud, interview, and questionnaire). Examples of the use of these metrics are drawn from a detailed review of research into VE wayfinding. A case study from research into the fidelity that is required for efficient VE wayfinding is presented, showing the unsuitability in some circumstances of common metrics of task performance such as time and distance, and the benefits to be gained by making fine-grained analyses of users' behavior. Taken as a whole, the paper highlights the range of techniques that have been successfully used to evaluate wayfinding and explains in detail how some of these techniques may be applied. Navigation is one of the most common types of operation that people perform in virtual environments (VEs), and is a broad term that incorporates many different types of tasks, from wayfinding (the act of traveling between places), to planning routes and giving directions. Research into navigation has investigated all of these, in addition to component-level tasks that are involved in the process of navigation such as path integration, distance perception, estimates of direction and orientation specificity (e.g., Chance, Gaunet, Beall, & Loomis, 1998; Richardson, Montello, & Hegarty, 1999; Riecke, van Veen, & Bülthoff, 2002; Waller, 2000; Wilson, Foreman, & Tlauka, 1997) . This article focuses on wayfinding. Studies have investigated the effect on wayfinding of various aspects of VE systems design, for example, the content of the visual scene, movement interface, field of view (FOV) and wayfinding aids, but major limitations exist with the methods used for evaluation. In particular, commonly used metrics: (a) are sometimes insufficiently fine grained to detect the differences that exist between different configurations of system (e.g., Lessels & Ruddle, 2004) , (b) tend to treat extremes of behavior as a nuisance factor, even though it is these extremes that sometimes identify major problems in usability, and (c) are poor at providing a concrete explanation for any differences in wayfinding performance that are observed. *Correspondence to royr@comp.leeds.ac.uk To help overcome these limitations, and in common with frameworks for the evaluation of human-computer interaction (Sweeney, Maguire, & Shackel, 1993) , we propose the use of three distinct levels of metrics to evaluate wayfinding. At the bottom level are direct measures of task performance. At a secondary level are metrics that measure users' physical behavior while they navigated, and at the third level are metrics that provide an explanation for this behavior by trying to establish users' underlying cognitive rationale (or lack thereof!). The remainder of this article is divided into two major sections. The first reviews metrics in each of the three levels that have been used to evaluate wayfinding in a variety of user studies, and identifies certain deficiencies. The second major section then shows how some of these metrics have been applied to help evaluate wayfinding in a suite of the authors' own studies. The overall goal of the article is to make a substantial improvement in the techniques used to evaluate wayfinding, and to communicate these techniques to a wider audience. Over a period of time this should assist research into wayfinding and promote closer comparisons between the studies carried out by different researchers than was previously possible. The overall purpose of a set of metrics is to assimilate raw data gathered during an evaluation, and to allow key features of these data to be presented in a concise form that allows the meaning of the data to be readily comprehended. Metrics should provide valid data that supports the conclusions drawn in a written report or presentation, and be sufficiently reliable for the data to be recreated in future evaluations. As we progress from one level of metric to the next the data presented is of finer granularity, providing explanations for differences apparent in the levels above. At the bottom level (Level 1) are metrics that measure how well a user performs a task, which for wayfinding involves a user in finding a particular place. Physical behavior metrics (Level 2) provide information about what a user was doing during a given task, not just how long they took or how accurately they performed. Rationale metrics (Level 3) operate at the highest of the three levels, and can provide an explanation for why users exhibit given behaviors. Even though, at the end of the day, it is users' performance in wayfinding that counts (Durlach et al., 2000) , this performance is driven by the users' decision making processes and subsequent behavior. It is users' rationale that dictates their physical behavior and subtle changes in behavior can have a substantial effect on performance, as is well known within the general field of human-computer interaction (e.g., Gray & Boehm-Davis, 2000; Gray & Fu, 2001) . Rationale and behavior metrics facilitate detailed evaluation of wayfinding and can often identify important changes in usability caused by different designs of VE systems, even if those differences have yet to become apparent at the level of task performance. In fact, rationale and behavior metrics may be used in this way to identify the optimal components of each of several different designs, leading to an improved design that, overall, is superior. Another situation where cognitive metrics have successfully been used in VEs is to compare fidelity with the real world (Mania, Troscianko, Hawkes, & Chalmers, 2003) . The next sections review the metrics in the three levels. Details of the metrics used in each of the published studies that were included in the review are contained in Appendix A. The most common method of evaluating wayfinding is to directly measure task performance. The key characteristic of any given task performance metric is that it reports a single measure that provides an overview of users' performance during a task, and this can be used as a basis for the statistical analysis of the effect of different factors in a VE's design (e.g., interface or scene fidelity), or to compare users' performance against a previously established norm. Most evaluations of VE wayfinding use at least one direct measure of task performance, with the most common being the time taken to complete the task, distance traveled, or number of errors made (see Figure 1 later in 638 PRESENCE: VOLUME 15, NUMBER 6 this paper). Time and distance are well suited to situations where users may navigate freely, whereas counting errors or the number of correct turns that are made are used to measure the accuracy of users' wayfinding when they are not allowed to deviate from a given path. Care is needed to make sure that the performance metrics chosen are suited to the evaluation being performed, and in this three particular issues should be highlighted. First, consider two users performing a wayfinding task that involves searching for a set of locations. One of the users spends a considerable amount of time planning where to travel and trying to memorize where they have traveled, whereas the other user searches in a near-arbitrary manner (the word "random" is deliberately avoided, because wayfinding movements in VEs are rarely random; users typically travel forward much more often than they turn left or right; Ruddle, Howes, Payne, & Jones, 2000) . In terms of distance traveled the first user has a better wayfinding performance, but in terms of time it is the second user who may well have performed better. This issue arises not just in terms of individual differences, but is also caused by interfaces that impose different speed limits on the rate of travel, as has been shown for menus in a graphical user interface (O'Hara & Payne, 1998) and travel in VEs with a hyperlink interface that allowed users to jump between locations instead of walking (Ruddle et al., 2000) . The second issue also refers to a tradeoff between the motor and cognitive costs of wayfinding. When searching a spatial layout, users may choose to adopt certain strategies such as repeatedly traveling back to a given, known location (Elvins, Nadeau, Schuk, & Kirsh, 2001; Ruddle et al., 2000; Ruddle & Péruch, 2004) . Although this increases the distance traveled, compared with always traveling somewhere new, backtracking makes it easier to remember where one has searched, and so lowers the risk of leaving some sections of the environment completely untouched. Performance metrics sometimes need to take account of actions that were deliberately performed because, even if those actions could be considered to be inefficient at the motor level, they may reduce the workload involved in wayfinding and, therefore, actually boost efficiency from a cognitive standpoint. A similar tradeoff was observed in a computerized version of the game Tetris (Kirsch & Maglio, 1994) . Expert players rotated the pieces substantially more often than was necessary, and it was surmised that this occurred because pieces that entered the screen in certain orientations were first rotated to an orientation that made it easier to mentally solve the question of where a piece should be placed (epistemic action) and then rotated and moved to that place (pragmatic action). The third issue is particularly common in situations where users are assessed in terms of their performance when traveling to several different places. If overall performance is measured (e.g., total distance traveled) then very poor performance finding one of those places may mask the ease with which every other place was found. An example of a performance metric that was developed to overcome this issue is the perfect search metric used in the case study (see Section 3.2.1, Perfect Search Metric). This metric measures how close a user was to performing a wayfinding task in a perfect manner, so small mistakes such as narrowly missing a target do not result in a large penalty (e.g., traveling a large additional distance because the whole environment had to be searched again). These metrics measure what users were physically doing, but are much less widely used (22 of the 40 studies in Appendix A) than task performance metrics so there is less of a consensus about the most appropriate data to gather or the ways in which it should be presented. The overall pattern of users' movement encompasses a rich variety of behavior, including the path that is followed, periods of time spent in motion or at rest, head movements, and body orientation (Winkel & Sasanoff, 1970) . In the present paper, physical behavior metrics are divided into the following main categories: (i) physical actions (subdivided into locomotion, looking around, and general observation), (ii) classification of the time spent performing different types of action, and (iii) classification of any errors made (as opposed to the performance metric of simply counting the number Ruddle and Lessels 639 of errors made). Details of these are summarized in Table 1 and discussed in the next sections. Locomotion. The first aspect of users' physical actions that needs to be addressed is metrics to measure where users travel (locomotion) during their wayfinding. The most challenging problem lies in developing metrics that can either combine the movements of different users in a meaningful way or find patterns in the movements made by different users that may, at first glance, appear dissimilar. One approach is to count the number of times each part of an environment is visited or searched. If the environment is structured like a road network or a building, then it can be approximated as a graph (nodes and links) and a count can be made of the number of traversals of each link or visits to each node. A user's locomotion can then be expressed as either this number (Elvins et al., 2001) or the percentage of the environment that was visited more than a given number of times (Ruddle & Péruch, 2004) . This approach is also used to analyze hits on web pages (e.g., Tauscher & Greenberg, 1997) and is often presented as a histogram. If an environment has an open structure (e.g., virtual seascapes, campuses, and rooms) then it is necessary to divide it into zones, instead of nodes and links. Once this is done, the number of times a user's path enters each zone (Darken & Sibert, 1996; Ruddle & Jones, 2001) , or each zone falls within a user's field of view, may be counted. For both types of environmental structure, number of visits data may be superimposed on a plan view of the environment using color, grayscale, or line thickness to indicate the relative frequency of visits (Darken & Sibert, 1996; Elvins et al., 2001 ) and these techniques may be extended to provide time-dependent visualizations that take account of the speed of users' movements (Chittaro & Ieronutti, 2004) . A second approach is to break down users' locomotion into a sequence of events (movements in a straight line or along curves) interspersed by periods when the users pause (Thiel, 1970). Once this has been done it is possible to characterize the nature of users' locomotive movements in terms of the average or profile of distances traveled (or time taken) between pauses, the frequency with which movements are made in different directions (e.g., forward versus backward), the number or pattern of inputs made to the relevant interface devices (Gamberini, Cottone, Spagnolli, Varotto, & Mantovani, 2003; Sas, O'Hare, & Reilly, 2004; Strommen, 1994) , the frequency with which collisions are made with the fabric of the environment (Gamberini et al., 2003; Waller, Hunt, & Knapp, 1998) or use biomechanics techniques to derive motion profiles (Whitton et al., 2005) . In the case study discussed in this paper, users' locomotion was characterized by the profile of the distance traveled during each period of discrete movement (see Section 3.2.3, Movement Breakdown; see also Figure 5 later in this paper). A third approach is taken by metrics that provide information about wayfinding paths. Plan views have been used to great effect to show the actual path taken by a typical user (Gamberini et al., 2003) or illustrate the extreme difficulty some users have wayfinding in VEs (Ruddle & Jones, 2001 ). The paths taken by different users may be combined onto a composite map to highlight commonalities in the actual paths, with separate maps sometimes being created for users in different categories, for example, users initially turning left versus right on entering an environment (Winkel & Sasanoff, 1970) . Path information may also be captured using a comprehensive notation for describing users' physical actions during wayfinding (Theil, 1970) . Different channels of information (time, distance, rate, direction, and ascent) are presented in parallel, and these may be expressed in terms of movement to, along, or across districts, nodes, paths, edges, and landmarks, the five basic elements of cities proposed by Lynch (1960) . However, the authors are not aware of research that has adopted this notation. Finally, path information may be expressed by identifying the macro or micro heuristics that users have used. Macro heuristics characterize a user's movement through an environment as a whole. Heuristics that have been identified in VEs include moving directly to a place versus traveling via somewhere else (relevant when revisiting a place), and following the edge (perimeter) of major environmental features, searching from the center of an environment outwards, or following a lawnmower-type pattern (applicable to revisiting places or making an exhaustive search of an environment; Darken & Sibert, 1996; Ruddle, Payne, & Jones, 1999a; Ruddle & Péruch, 2004) . In all of these studies the heuristics were assessed subjectively, in some cases by two human raters who worked independently. However, it is recommended that a more objective approach is used whereby heuristics are described either in concrete terms or as a mathematical equation and tested for goodness of fit with users' actual paths. An example of this approach is shown in the case study, where searches were classified as perimeter-first or lawnmower (see Section 3.2.2 Macro-Level Search Heuristic; see also Figure  4 later in this paper), and another comes from research into the paths drawn by participants who were asked to solve the traveling salesman problem for six, 10, or 18 points that were displayed on a piece of paper (Hirtle & Gärling, 1992) . Four heuristics were analyzed (nearest neighbor; straight-line, in which a set of points will be taken in order along the line; zig-zag, or lawnmower; and cluster, where locations are partitioned into clusters, which are then connected using one of the other heuristics). Micro heuristics have generally been assessed within the context of an environment's graph (node and link) structure, with the most commonly identified heuristics being backtracking and loops (Elvins et al., 2001; Ruddle et al., 2000; Ruddle & Péruch, 2004; Ruddle, 2005) . The second aspect of users' physical actions is how users look around while they travel. For many years, researchers have investigated the effect that the freedom of movement provided by head-tracked rather than stationary displays has on performance in spatial tasks such as targeting radiotherapy treatment beams, target detection, and maintaining one's orientation (e.g., Bakker, Werkhoven, & Passenier, 1999; Chung, 1992; Pausch, Shackelford, & Proffitt, 1993) . These studies highlight Ruddle and Lessels 641 the benefits of head-tracking in terms of speed and accuracy, but few studies have actually reported behavioral data that relate to how people look around in VEs. Looking around during wayfinding takes place in three primary frames of reference: (i) view-referenced, (ii) body-referenced, and (iii) world-referenced. In a view reference frame, looking around relates to the movements a user makes to attend to different parts of a display or aspects of a scene that are shown on a display at any given moment in time. Users often have to make specific inputs (e.g., a key press) to view parts of a display such as a map (Darken & Banker, 1998) or a guidebook (Elvins et al., 2001 ), so it is straightforward for VE software to record the times for which each part was being displayed. Determining the aspects of a virtual scene that a user is attending to, and for how long, involves eye tracking. The use of eye tracking to investigate navigation within hypertext and the web is becoming commonplace and, for data analysis and presentation, the path of a user's eye fixation may be overlaid onto a view of the environment, annotated with time stamps and accompanied by a transcript of the user's physical actions (e.g., Card et al., 2001 ). An identical approach could be applied to VEs. Looking around within a body reference frame concerns the direction of a user's view in a VE relative to the orientation of their virtual body, whereas looking around in a world reference frame concerns changes in the global direction of the user's view. For a user who is "walking" through a VE, the body and world reference frames are identical for changes of direction in pitch and roll. Changes of heading (yaw) are also identical in the two reference frames if movement is made using the most commonly implemented walking metaphor, which is view-direction movement (also known as gazedirected movement; Bowman, Koller, & Hodges, 1997). However, changes of heading are different in body and world reference frames if the user's view and body directions are decoupled, although the basic methods used to quantify the changes remain the same. One of these methods involves annotating a plain view of the path taken by a user with either lines or view cones that show how the view heading changed during wayfinding (Zanbaka et al., 2004) . Animation of the view heading is useful for detailed analysis of users' actions and snapshots may be used to illustrate behavioral phenomena (see Section 3.2.5 Looking Around; see also Figure 8 later in this paper). A second method determines the amount of time for which each part of an environment was displayed, taking into account occlusions caused by walls and objects in a VE (Chittaro & Ieronutti, 2004) . A third method is to identify patterns of behavior that can be summarized verbally, for example, users systematically looking in every direction at junctions (Strommen, 1994) . A fourth method is born out of the fact that it is often desirable to be able to quantify the differences that occur between one design of VE system and another, although examples of this are rare. Useful data that may be reported include the mean rate of direction changes, which have showed that participants look around twice as much when navigating using a head-mounted display (HMD) than a desktop display (Ruddle, Payne, & Jones, 1999b) , the total angle users turn by and the number of separate movements in which this is achieved (Sas, O'Hare, & Reilly, 2004) , the cumulative angle of all direction changes made during a given task, and the profile of those changes (see Section 3.2.5, Looking Around; see also Figure 7 later in this paper). The profile and rate of change of pitch viewing angles have also been used to assess the causes of different levels of VE sickness in wayfinding and other tasks (Ruddle, 2004) . Observation. Most studies of VE wayfinding have only evaluated what users did within a VE, but occasionally studies have formally recorded and related users' on and off screen activities (Gamberini et al., 2003; Murray, Bowers, West, Pettifer, & Gibson, 2000) . The former adapted existing forms of notation to combine information about each user's physical real-world actions, verbally reported comments, movements in the VE, and events beyond the user's control that happened in the VE (e.g., a virtual fire starting) into a single transcription. These transcriptions were then analyzed in conjunction with other behavioral and performance data to provide quantitative and qualitative information about the VEs that were being evaluated. By contrast, Murray et al. made a qual-642 PRESENCE: VOLUME 15, NUMBER 6 itative assessment of similarities in their observations of different users and, drawing on reporting techniques used in areas such as conversation analysis (Sacks, 1992) , provided quotes from users that were exemplars of the observed behaviors. These metrics classify behavior in terms of the amount or proportion of time users spend performing different types of action. In wayfinding the primary candidates for classification are: (a) whether or not a user is locomoting (stationary versus traveling), and (b) whether or not a user is changing the direction in which the user is looking within a given frame of reference (static versus looking around). Some studies have only discriminated between the time users spent stationary versus traveling (Bowman, Johnson, & Hodges, 2001; Ruddle & Jones, 2001 ) but others include all four combinations of locomoting and changing direction (Elvins et al., 2001; Gamberini et al., 2003) . The latter is recommended, and easily implemented if a user's position and view direction are recorded at the same rate as the VE's frame rate and each time step coded using bitwise ORing. The metrics of looking around that were described in the previous section can then be calculated for the whole of a wayfinding task, or for periods when locomotion is either taking place or not (see Section 3.2.5 Looking Around; see also Figure  7 later in this paper). Sometimes finer grained classifications of time are performed to distinguish between the amount of time spent traveling in each direction allowed by an interface (e.g., forward versus backward; Gamberini et al., 2003) . Other studies have adopted different classifications for particular purposes with examples being the amount of time spent viewing environments from different altitudes (Witmer, Sadawski, & Finkelstein, 2002) , looking down at one's virtual feet to see the obstacles with which one has collided (Ruddle & Jones, 2001 ) and traveling along wide versus narrow pathways (Zhai, Kandogan, Smith, & Selker, 1999). Although the number of errors that users make during wayfinding is frequently used as a measure of performance (see above) it is rare that the errors are classified, even though this can highlight some of the root causes of difficulty users have navigating in a given VE. The type of error that is most commonly identified in wayfinding is a miss, which occurs when a user travels within sight of a given location without turning to look at it (Ruddle, Payne, & Jones, 1998; Ruddle & Péruch, 2004) . The most likely causes, of course, are the impoverished FOV that is provided by most VE systems and restrictions on the speed at which users can look around, the equivalent of glancing momentarily over one's shoulder in the real world usually being impossible to achieve. For VEs such as virtual buildings, which are structured like a graph, errors can be classified in terms of where along a path each error was made (e.g., the first versus a subsequent decision point; Ruddle et al., 1998) . VEs that have an open structure can make use of Delaunay triangulation or Voronoi regions (see Section 3.2.4, Error Classification; see also Figure 6 later in this paper). A small number of wayfinding studies (6 of the 40 studies in Appendix A) have attempted to gain direct insight into users' decision-making processes. The bestknown technique that provides data while wayfinding is actually taking place is for a user to think aloud. This has been successfully used in studies involving wayfinding/navigation in VEs (Gamberini et al., 2003; Grammenos, Filou, Papadakos, & Stephanidis, 2002; Murray et al., 2000) , the web (Card et al., 2001) , and the real world (Hayes-Roth & Hayes-Roth, 1979). Other techniques that provide information about users' decisionmaking are interviews and questionnaires (Elvins et al., 2001; Whitton et al., 2005; Zhai et al., 1999) , and these usually take place after wayfinding has finished. The think aloud method can provide detailed information during the process of wayfinding, albeit with the risk that the requirement to think aloud will impede a user's wayfinding performance. Interviews and questionnaires do not affect the process of wayfinding but rely more on the user having an accurate memory, although in an interview the user could be prompted by an observer using notes made during the wayfinding. All of Ruddle and Lessels 643 these rationale metrics rely to a certain extent on users being able to elucidate the reasons for the decisions that they make, but wider adoption of the metrics for the evaluation of VE wayfinding would be a positive development. It is well known that people frequently experience great difficulty wayfinding in VEs, and typically take much longer to learn the layout of virtual spaces than equivalent real world spaces (Richardson et al., 1999; Witmer, Bailey, Knerr, & Parsons, 1996) . The authors' current research is investigating what is required if wayfinding tasks are to be performed as easily in a VE as they are in the real world, focusing on the effects of visual fidelity versus FOV (Lessels & Ruddle, 2004) and visual versus movement fidelity (Lessels & Ruddle, 2005) . In the research, participants' task is to find targets that are placed in a small number of explicitly identified, possible locations in a 10 ϫ 10 m virtual room. Eight of the locations contain a target and the other eight are decoys (see Figures 1 and 2 ). Looking once in each location guarantees success. By using room-sized VEs, the effects of fidelity can be studied in the absence of the additional complexity caused by wayfinding in a large-scale space such as a building. Ultimately, however, it is hoped to extend the research findings to largescale VEs. The philosophy that guides the research is as follows. If it were possible to build a "virtual reality," which in all respects was indistinguishable from its real-world counterpart, then it follows that tasks would be as easy to perform in the virtual version of the environment as they were in the physical version. Given the task we are using in our experimental studies is straightforward to perform in the real world (Lessels & Ruddle, 2005) , it would also be straightforward in a virtual reality. Unfortunately, this is not possible given the limitations of current VE technology and so users often find the virtual version of the task difficult to perform. The behavioral experiments completed so far have all used desktop VEs. This section provides an overview  644 PRESENCE: VOLUME 15, NUMBER 6 of the method. Section 3.2 then explains the problems that were encountered when interpreting the data and how these problems were overcome by applying some of the performance and behavior metrics described in Section 2. For full details of the method and results of the experiments, readers are referred to Lessels & Ruddle (2004 , 2005 . The interfaces have allowed participants to travel through the VEs by holding down keys that allow movement to be made either: (a) only in a forward direction, or (b) forward/backward/sideways/diagonally. With all of the interfaces, participants' direction of view was the same as the orientation of their virtual body, and the same method was used to control changes of direction. The amount a participant looked up or down was directly mapped to the vertical position of the cursor on the screen (zero order control), and participants could turn left or right at a rate that increased with the cursor's horizontal offset from the center of the screen (first-order control). The maximum rate of turning was 135 deg/s and the view direction did not change at all if the cursor was within the middle 10% of the screen. A slip collision response algorithm was implemented that automatically guided participants around the cylindrical obstacles in the environment and along the walls. The VE application was written in Cϩϩ and OpenGL Performer and ran on an SGI Onyx 3400 with a frame rate of 60 Hz, giving an overall system latency of approximately 30 ms. Six different visual scenes have been utilized. All of the scenes used the same visual representation of the cylinders and target/decoy boxes, but differed substantially in terms of the texture maps used to represent the environment's walls and floor (see Figure 2) . Rendering used directional light sources and Gouraud shading. Participants either viewed one computer monitor (48 ϫ 39 deg; a normal FOV for a VE) or three monitors that were arranged in an arc (144 ϫ 39 deg; a wide FOV). A between participants design was used, with different participants taking part in each experiment. After a period spent practicing the interface, participants performed two practice trials that allowed them to become familiar with the search task, and then completed four test trials. Each trial began at the starting point in the boundary recess (see Figure 1) and participants searched until they had found and selected all eight targets. Participants were informed that the targets were always in the boxes that were on top of the cylinders, but that the position of the boxes and targets changed between trials. The primary problem faced in interpreting the experimental data was as follows. When participants performed the task in a real world version of the environment they completed 93% of the trials by checking each target/decoy box only once, which, in terms of the number of places checked, represents a perfect search. In each experimental condition used in the VEs, participants sometimes completed the task perfectly, but on other occasions experienced great difficulty and revisited a substantial proportion of the environment. Conventional task performance metrics such as the time taken or distance traveled proved to be insufficiently fine-grained to detect differences that did exist in the way that participants searched VEs that were constructed using different combinations of the visual scenes, FOVs, and interfaces outlined above. As a result, the data were analyzed using a selection of metrics, namely: 1. Task performance a. Proximity to conducting a perfect search (related to the number of errors) 2. Behavior a. Locomotion: Macro-level search heuristic b. Locomotion: Movement breakdown (start-stop frequency, and time classification) c. Error classification d. Looking around (quantity, profile, and impact on errors made) 3.2.1 Perfect Search Metric. Overall, participants completed 47% of the VE trials by checking each target/decoy box only once. Of the remaining trials, participants completed some in a near-perfect manner, for example, doublechecking just one target or decoy, Ruddle and Lessels 645 but on other occasions searched large parts of the VE several times. Sometimes this was a direct consequence of having previously traveled past a target without stopping to check it so, once the participant had completed the initial phase of their search, they had no idea of the location of the missing target because they had already traveled through the whole environment (see Figure  3a) . When this occurred, some participants searched the environment again in a systematic manner, but others changed their strategy and appeared choose the boxes to recheck on an arbitrary basis. On other occasions the general region that contained a target was left untraveled during the initial search (see Figure 3b) . To distinguish between the above cases, a performance metric was developed that measured how close each trial came to being completed in a perfect manner (checking each target/decoy only once). This metric was calculated by summing the proximity score for each target up until the point when the first target or decoy was revisited. Targets that had been found scored zero, as did targets that the participant had bumped into but not checked. For other targets the score was the closest distance to the target so far. An analysis of variance showed that participants came significantly closer (p Ͻ .05) to conducting a perfect search when a high fidelity visual scene was combined with a wide FOV than with other combinations of scene and FOV, even though there was no significant difference in terms of total distance traveled or time taken (see Lessels & Ruddle, 2004) . Group means for the perfect search ranged from 1.2 m (high fidelity/wide FOV) to 5.3 m for searches conducted in VEs, compared with 0.05 m for those conducted in the real world. In the real world study, 93% of trials used a heuristic that, by dividing the VE into quadrants and writing down the sequence in which they were entered, could unambiguously be classified as either perimeter-first or lawnmower (see Figure 4) . Perimeter-first searches visited the quadrants in the order 1-2-3-4-1 (clockwise) or 1-4-3-2-1 (anticlockwise). Lawnmower searches involved a sequence of passes that crossed the VE's centerline. The same two heuristics were used in 97% of the VE trials but, as has already been explained, only half of these were conducted in a perfect manner. Thus, although real world and VE behavior was similar at a macro level, that did not translate to a similarity in search performance. This identified the need to analyze participants' behavior at a finer level of detail, in terms of their movement, the errors made, and the extent to which they looked around. 646 PRESENCE: VOLUME 15, NUMBER 6 3.2.3 Movement Breakdown. Two aspects of movement were analyzed: the extent to which participants traveled in one continuous movement from one target/decoy to the next, and the percentage of time spent pausing in between targets and decoys. On average in the VEs participants paused once en route to every target/decoy in each trial, but never paused between targets and decoys when the task was performed in the real world. The effect of the pauses on the distance participants traveled during each period of continuous movement is highlighted in Figure 5 . Overall, these pauses accounted for approximately one third of the time participants took to perform the trials in the VEs and, although a significantly smaller proportion of time was spent pausing when a wide rather than a normal FOV was used (p Ͻ .05), indicating a shift toward a real world pattern of behavior, the magnitude of the difference was small, 30% versus 38% (see Lessels & Ruddle, 2004) . Taken together, these data highlight a clear difference between the smooth, continuous movements participants made when conducting the task in the real world, and the start-stop nature of their movements in a VE. The use of a view-direction walking metaphor for movement in the VE prevented participants from looking around while they traveled in a straight line and that is one factor that is likely to have caused them to pause. However, similar behavior was found in an earlier study that used an HMD and decoupled the view and body directions, which, in theory at least, made it straightforward for participants to glance to the side while they traveled (Ruddle & Jones, 2001 ). Therefore, it seems likely that the stop-start behavior arises from more fundamental limitations imposed by VE technology, with candidates being the abstract mechanism (key presses) that is used to control translationary movements even in most immersive VEs, and the impoverished FOV that prevents users from seeing much of their immediate surroundings at one moment in time and, therefore, impedes the planning of movements that are going to be made in the short-to medium-term future. The chunking together of path segments makes routes much easier to learn than if a long sequence of individual cues and wayfinding decisions that make up that route have to be memorized (Golledge, 1992) . The changing of movement from continuous (real world) to discrete segments (VE) is likely to have reduced the extent to which participants were able to chunk together sequences of targets and decoys to make it easier to remember the parts of the environment that had been searched. The perfect search metric highlighted the fact that participants sometimes passed near a target without checking it and on other occasions left substantial areas of a VE untraveled until the rest of the VE had been repeatedly searched. This led to the division of the VE into regions using Delaunay triangulation and the classification of three types of error (miss, local neglect, and global neglect; Lessels & Ruddle, 2004) to describe the targets that were not found during the initial stage of a participant's search. A miss was recorded if the participant had previously touched the cylinder on which the target's box was located. Local neglect was recorded if the participant had previously traveled through any of the Delaunay triangles connected to the target's cylinder. Global neglect was recorded for all other errors, indicating that the participant had not been in the target's immediate vicinity (see Figure 6 ). On average, 1.6 targets were not found during the initial search of each trial and, of these, misses accounted for 5%, local neglect 29%, and global neglect 66% (see Lessels & Ruddle, 2004 , 2005 . The Ruddle and Lessels 647 percentage of errors in each category was similar for each combination of visual fidelity, FOV, and movement interface, but categorization of the errors allowed the effect of FOV to be assessed in detail (see below). For each trial, the cumulative amount by which the view heading was changed was calculated for periods when a participant was: (i) standing in one place, or (ii) traveling through the VE. The majority (73%) of the change took place when participants had stopped traveling. Due to the fact that all of the VE interfaces used a view-direction walking metaphor for movement, changes of view heading while traveling took place whenever participants changed their direction of travel. Some changes of view direction occurred when participants were standing in one place looking around. Other participants stopped, chose a new direction in which to travel, and then recommenced movement. It was predicted that participants would look around less when they were provided with a wide FOV, because that allowed them to view a large amount of the environment by scanning the monitor displays rather than turning their virtual head, and this would be exhibited by a change in the profile of the angles by which participants changed their view heading when standing in one place. However, there was little difference in the profiles for wide and normal FOV (see Figure 7) . A finer-grained analysis of looking around was performed to relate changes in participants' view heading to the types of error that were identified above. For each trial where a miss or local neglect occurred, an animation of participants' FOV was overlaid on the path they followed and used to investigate where they were looking during the time when each error was made. In 94% of the misses that participants made with a normal FOV they looked directly at the target concerned when they were standing beside it (Lessels & Ruddle, 2004) . With a wide FOV, the target was displayed on the central monitor for 80% of the misses and on either the left or right monitor for the remaining 20%. However, it was a different story for the case of local neglect (see Figure 8 ). With a normal FOV, 98% of the targets that were neglected never appeared in participants' view when they were in the immediate vicinity of a target (i.e., within the Delaunay triangles that defined the local region). On the other hand, with a wide FOV 56% of these targets were visible on either the left or right monitor when participants were in a target's immediate vicinity, with the other 44% never being visible. Wayfinding is one of the most often performed and frequently studied tasks in VEs. However, although 648 PRESENCE: VOLUME 15, NUMBER 6 the vast majority of studies measure users' performance in at least one way, there is a tendency to rely on hypotheses and anecdotal observation to explain any differences that occur. This paper reviews metrics that may be used to measure various aspects of wayfinding behavior and the decision making process that accompanies it. By highlighting the techniques used by some researchers, this paper aims to stimulate the wider adoption of metrics of behavior and rationale, leading to greater insights into the whole process of wayfinding and allowing easier comparison between the results of the studies carried out by different research groups. To illustrate the practical use of some of the metrics, the authors' current research into the effects of visual fidelity, FOV, and movement fidelity on VE wayfinding was used as a case study. These metrics showed that participants used the same general heuristics (perimeter or lawnmower search pattern) to guide their wayfinding in both real world and VE versions of the task, but this did not translate into a similarity in terms of performance. Therefore, it was necessary to make a more detailed analysis of behavior. Breaking down participants' wayfinding into periods when they were either traveling or stationary highlighted the contrast between the continuous nature of participants' movements between targets and decoys in the real world, with the start-stop nature of their movement in VEs. This, and its impact on the ease with which participants could chunk together segments of their path through an environment, provide one explanation for the general difficulty that participants experienced performing the task in a VE. A major challenge for the developers of VE systems and technology is to provide interfaces that allow free-flowing movement during wayfinding. The role of FOV was highlighted by a fine-grained analysis of the errors that participants made in conjunction with the extent to which they looked around. Every target that was missed was visible within a participant's FOV when the person was standing beside it, so an important question is why did the person not check those targets? Did the participant think that these had already been checked, or did the person miss them because of attending to other aspects of the visual scene? Use of a think aloud methodology would help to provide an answer. Analysis of participants' looking around also showed that, with a normal FOV, almost every time a local target was neglected it was simply because the target was outside the FOV. One way of resolving this would be to substantially reduce the time cost of looking around and allowing participants to perform the real world equivalent of glances to one side while they traveled through a VE. This could be achieved by mapping the view direction directly to the orientation of a sensor (e.g., an Intersense InertiaCube2) instead of using a steering metaphor whereby participants rotated their view direction at a rate that depended on the offset of a cursor from the center of the display. This would increase the speed of precise direction changes, as would be expected with zero-order rather than first-order control. Even when a wide FOV was provided, participants frequently neglected targets that were visible on the sides of the combined display. This highlights the need for investigation into different methods of displaying a wide FOV, including distorting the geometric FOV relative to the Ruddle and Lessels 649 physical FOV, and the use of eye tracking to determine the parts of a display to which participants attended. Finally, a large proportion of participants' errors occurred because parts of the VEs were completely neglected during the initial period of wayfinding in each trial. Future use of a think aloud methodology may help explain why this neglect occurred.
Various metrics for exploring way-finding in an interactive 3D virtual environment, including player movement and orientation, involving field-of-view perception and memory have been studied REF .
Three Levels of Metric for Evaluating Wayfinding
PRESENCE: Teleoperators and Virtual Environments
PRESENCE: Teleoperators and Virtual Environments
Computer Science