In Augmented Reality, see-through HMDs superimpose virtual 3D objects on the real world. This technology has the potential to enhance a user's perception and interaction with the real world. However, many Augmented Reality applications will not be accepted until we can accurately register virtual objects with their real counterparts. In previous systems, such registration was achieved only from a limited range of viewpoints, when the user kept his head still. This paper offers improved registration in two areas. First, our system demonstrates accurate static registration across a wide variety of viewing angles and positions. An optoelectronic tracker provides the required range and accuracy. Three calibration steps determine the viewing parameters. Second, dynamic errors that occur when the user moves his head are reduced by predicting future head locations. Inertial sensors mounted on the HMD aid head-motion prediction. Accurate determination of prediction distances requires low-overhead operating systems and eliminating unpredictable sources of latency. On average, prediction with inertial sensors produces errors 2-3 times lower than prediction without inertial sensors and 5-10 times lower than using no prediction at all. Future steps that may further improve registration are outlined.  CR Categories and Subject Head-Mounted Displays (HMDs) and Virtual Environments have been a subject of great interest in the past few years. Less attention has been paid to the related field of Augmented Reality, despite its similar potential. The difference between Virtual Environments and Augmented Reality is in their treatment of the real world. Virtual Environments immerse a user inside a virtual world that completely replaces the real world outside. In contrast, Augmented Reality uses see-through HMDs that let the user see the real world around him. See-through HMDs augment the user's view of the real world by overlaying or compositing three-dimensional virtual objects with their real world counterparts. Ideally, it would seem to the user that the virtual and real objects coexisted. Since Augmented Reality supplements, rather than supplants, the real world, it opens up a different class of applications from those explored in Virtual Environments. Augmented Reality applications attempt to enhance the user's perception and interaction with the real world. Several researchers have begun building prototype applications to explore this potential. A group at Boeing uses a see-through HMD to guide a technician in building a wiring harness that forms part of an airplane's electrical system [6] [33] . Currently, technicians use large physical guide boards to construct such harnesses, and Boeing needs several warehouses to store all of these boards. Such space might be emptied for better use if this application proves successful. Other construction and repair jobs might be made easier if instructions were available, not in the form of manuals with text and 2D pictures, but as 3D drawings superimposed upon real objects, showing step-bystep the tasks to be performed. Feiner and his group demonstrated this in a laser printer maintenance application [11] . Feiner's group is also exploring displaying virtual documents in a sphere around the user, providing a much larger workspace than an ordinary workstation monitor. Medical applications might also benefit from Augmented Reality. A group at UNC scanned a fetus inside a womb with an ultrasonic sensor, then displayed a three-dimensional representation of that data in the same physical location as the fetus [4] . The goal is to provide a doctor with "X-ray vision," enabling him to gaze directly into the body. Conceptually, anything not detectable by human senses but detectable by machines might be transducted into something that we can sense and displayed inside a see-through HMD. Robinett speculates that ultimately Augmented Reality is about augmentation of human perception, about making the invisible visible (or hearable, feelable, etc.) [29] . While promising, Augmented Reality is barely at the demonstration phase today, and its full potential will not be realized until several technical challenges are overcome. One of the most basic is the registration problem. The real and virtual objects must be properly Permission to make digital or hard copies of part or all of this work or personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. points and view directions. We use a custom optoelectronic head tracker that provides sufficient range and accuracy. We also developed calibration techniques for determining the viewing parameters. The robust static registration is demonstrated by several still photographs taken from a single video sequence where the user walked 270 degrees around the registration object. Improved dynamic registration: To reduce dynamic errors caused by the end-to-end latency in the system, we use predictive tracking techniques that guess where the user's head will be in the future. We equipped the see-through HMD with inertial sensors to aid headmotion prediction. A method for autocalibrating the orientation of the sensors on the user's head, along with other parameters, was developed. The reduction in dynamic error for three different motion runs is listed in a table. The rest of the paper is organized as follows: first we give a brief overview of our system. Next, we describe and evaluate the static registration procedure. Then we do the same for dynamic registration. Each section describes problems encountered and points out limitations. The interested reader will find supplementary materials available in the CD-ROM version of this paper and many more details about our work in a technical report (the first author's dissertation) to be released later in 1994 by UNC Chapel Hill. The main implications of our work are: • Robust static registration within a few mm is possible, but it requires trackers with higher accuracy than Virtual Environment applications demand. • Inertial-aided predictors can greatly reduce dynamic registration errors. On average, inertial-based prediction is 5-10 times more accurate than doing no prediction and 2-3 times more accurate than a representative non-inertial predictor. • Augmented Reality systems that use predictive tracking require low-overhead operating systems and the elimination of unpredictable sources of latency. Our system uses an optical see-through HMD. Its position and orientation are measured by an optoelectronic tracking system, and the images are generated by the Pixel-Planes 5 graphics engine. Readings taken from inertial sensors mounted on the HMD are digitized by an A/D board in a 486 PC. Figure 10 shows the overall setup, and Figure 3 provides a system diagram. The optical see-through HMD [16] is shown in Figure 8 . Optical combiners placed in front of both eyes overlay images on top of the user's view of the real world. The displays are color LCD monitors containing 340x240 pixels each. The field-of-view in each eye is approximately 30 degrees. We chose an optical see-through approach because it does not delay the user's view of the real world (see Section 7). Tracking is provided by four optical sensors mounted on the back of the HMD, as seen in Figure 8 . These are aimed upwards at an array of infrared LEDs mounted in ceiling panels above the user's head. By sighting several LEDs, and given the known geometry of the sensors on the head and the known locations of the beacons in the ceiling, the system is able to compute the position and orientation of the user's head. The data collection and processing are performed by three single-board 68030 and i860-based computers installed in the VME chassis of a Sun4 host [36] . The inertial sensors consist of three Systron Donner QRS-11 angular rate gyroscopes and three Lucas NovaSensor NAS-CO26 linear accelerometers. The gyros measure angular rates within the range of ±300 degrees/second, and the accelerometers detect acceleration within ±2 g. A 12-bit A/D board (National Instruments AT-MIO-16D) in a 486 PC digitizes the signals. To minimize noise, we built special power regulation circuits, used shielded twisted-pair wire, differential-mode A/Ds, and analog prefilters. A Bit3 bus extender sends the digitized readings to the optoelectronic tracker boards in the Sun4. The virtual images are generated by Pixel-Planes 5 (Pxpl5), a highly parallel graphics engine consisting of i860-based Graphics Processors (GPs) to do geometry transformations and Renderer boards that rasterize primitives [14] . One of the GPs is used to run our prediction routine. The computed head positions and orientations from the optoelectronic tracker and the measured inertial signals are fed to this GP, which uses them to estimate future head aligned with respect to each other, or the illusion that the two coexist will be compromised. This is difficult to do because of the precision required. The human visual system is very good at detecting even small misregistrations, due to the resolution of the fovea and the sensitivity of the visual system to differences. A more tractable bound is provided by the relatively low resolution of the displays in modern HMDs. Errors of just a few pixels are noticeable. Some applications have strict requirements for accurate registration. For example, imagine a surgeon wearing a see-through HMD displaying virtual objects that identify where and where not to make incisions. Unless the registration errors are kept below a few millimeters, the surgeon is not likely to trust the equipment. Without good registration, Augmented Reality may never be accepted in serious applications. What causes registration errors? The main sources are: • Distortion in the HMD optics • Mechanical misalignments in the HMD • Errors in the head-tracking system • Incorrect viewing parameters (field of view, tracker-to-eye position and orientation, interpupillary distance) • End-to-end system latency The first four categories may be classified as static errors, because they cause registration errors even when the user keeps his head completely still. The 5th category, end-to-end latency, we call a dynamic error, because it has no effect until the user moves his head. No one has achieved perfect registration with a see-through HMD. The current demonstrated state-of-the-art, as reported in the text and pictures of [4][11][18] achieves static errors on the order of 0.5 inches for an object at arm's length away from the HMD, from a small number of viewpoints. Dynamic errors can be much larger. With an end-to-end system latency of 100 ms, and a moderate head rotation rate of 100 degrees/sec, the angular dynamic error will be 10 degrees. At arm's length, this results in registration errors of 5 inches. This paper describes a system we built that tackles three of the main sources of registration errors: the head tracker, the determination of viewing parameters, and dynamic errors caused by system latency. We demonstrate improved static and dynamic registration as follows: Improved static registration: Pictures and videos of existing Augmented Reality systems show registration from only a small number of viewpoints. The user is not allowed to translate or rotate the HMD very far from the initial viewpoint. There are two reasons for this limitation. First, most commercially available head-tracking systems do not provide sufficient accuracy and range to permit such movement without greatly increasing the static registration errors. Second, determining viewing parameters that work from just one viewpoint is much easier than determining parameters that work from many different viewpoints. We show that parameters yielding good registration from one viewpoint may result in static errors of a few inches at another viewpoint. Our system is capable of robust static registration: keeping a virtual and real object closely aligned across many widely spaced view-  Registration of real and virtual objects is not limited to see-through HMDs. Special effects artists seamlessly blend computer-generated images with live footage for films and advertisements. The difference is in time and control. With film, one can spend hours on each frame, adjusting by hand if necessary, and each shot is carefully preplanned. In Augmented Reality we have no such control: the user looks where he wants to look, and the computer must respond within tens of milliseconds (ms). Deering [10] demonstrated an impressive registration of a real and virtual ruler in a head-tracked stereo system. Registration is a somewhat easier task in head-tracked stereo vs. an HMD-based system because the images do not change nearly as much for the same amount of head translation or rotation [9] . An extensive literature of camera calibration techniques exists in the robotics and photogrammetry communities (see the references in [21] as a start). These techniques digitize one or more pictures of an object of fixed and sometimes unknown geometry, locate features on the object, then use mathematical optimizers to solve for the viewing parameters. However, it is not clear how to directly apply these techniques to an optical see-through HMD, where no camera exists. Asking a user to identify the locations of many different points simultaneously while keeping his head still was judged too difficult to be reliable. We have already mentioned several papers on Augmented Reality, but most focus on applications rather than on the details of calibration and registration. The sole exceptions are [6] [18]. We compare our results with theirs in Section 4.4. Methods used to calibrate helmet-mounted sights on helicopter gunships provided the initial inspiration for our approach. We reduce the problem to one real object, one set of virtual objects, and a desired registration linking the two. The real object is a wooden frame (Figure 1 ). The virtual objects are three mutually orthogonal extruded squares that form a coordinate system. The goal is to register the intersection of the three virtual bars with the front left corner of the frame, where the three bars run along the edges that touch the corner (Figures 2 & 14) . This task is a good registration test because it's easy to detect small position and orientation errors along the edges of the frame. Determining parameters that accomplish this task robustly is harder than one might first think. The naive approach, tried by several people in our laboratory, is the following: put a real object at a known or unknown position, wear the see-through HMD, then manually adjust the viewing parameters and the location of the virtual object until the registration "looks right." This rarely yields robust registration, because parameters and locations that work at one viewpoint may generate large registration errors at another. Figure 9 illustrates this. The picture on the left shows good registration at the initial viewpoint. But the same parameters yield a few inches of registration error when used at a different viewpoint, as seen in the picture on the right. locations. The prediction GP also converts the predicted head locations into view matrices that the rest of Pxpl5 uses to generate the graphic images the HMD-wearer sees. Since the normal Pxpl5 software is optimized for maximum throughput and not minimal latency, we use different rendering software written by Mark Olano and Jon Cohen that minimizes Pxpl5 latency [8] . Special care was taken to use fast communication paths and lowoverhead operating systems. Interprocessor communication is through shared memory, across Bit3 bus extenders, or through the 640 MByte/sec ring network within Pixel-Planes 5. UNIX is avoided except for initial setup and non-time-critical tasks, like reading buttons. This is discussed more in Section 5.3. The end-to-end system latency varies from 50-70 ms, with 15-30 ms coming from the tracker, ~12 ms from the predictor, and 16.67 ms from Pxpl5. The rest comes from communication paths and delays caused by the asynchronous nature of our system. Recording the images that the user sees inside the HMD to create the pictures in this paper was done by mounting a video camera in the right eye of a bust. The HMD was tied to this bust, then carried around. Since camera calibration techniques seemed too difficult to apply in this domain, we thought of ways to directly measure the viewing parameters, using simple tasks that rely on geometric constraints. If the tasks are sensitive enough and our tracker is accurate enough, then this simple approach might work. We now describe these procedures that systematically determine the viewing parameters. The steps in order are: • Measure the frame's location • Determine the apparent center of the virtual image • Measure the transformation between tracker space and eye space • Measure the field-of-view (FOV) We use only the right eye of our HMD, due to mechanical misalignments, color mismatches between the two display systems, and because a monocular display is sufficient to demonstrate registration. A digitization probe attached to a "hat" with four optical sensors returns the 3D position of the probe tip ( Figure 13 ). We measure eight points on the frame edges where the red and green bars will lie, fit a pair of orthogonal lines through those points, and those determine the axis going down the third edge. 2) Apparent center of virtual image: The center of our 640x512 NTSC frame buffer need not be the center of the virtual image seen by the right eye, requiring off-center projections to properly render the images [12] . Assuming that the frame buffer covers the entire area visible through the optics, we can measure this center by drawing a 2D, non-head-tracked crosshair in the frame buffer (Figure 4) . Four numbers specify this crosshair: the (X,Y) center coordinate, and the X and Y radii. The user determines the center by adjusting the X center and radius until the left and rightmost lines are equally spaced from the extreme visible edges of the display. This is tested by increasing the radius; both lines should disappear simultaneously or the center is incorrect. A similar procedure determines the Y center. Our measured center is (330, 255), which differs from the frame buffer center by about 10 pixels. 3) Eye->Tracker transformation: This is measured by the boresight operation, where a user wearing the HMD looks straight down the left top edge of the frame with his right eye (Figure 4) . A 0.25" diameter pipe sticking out along the edge (Figure 1 ) helps the user line up accurately. Simultaneously, he centers the virtual crosshair with the corner of the frame and aligns the horizontal and vertical crosshair lines with the edges of the frame (Figure 11 ). Then the Eye coordinate system has the same orientation as the Frame coordinate system, and the Z axes coincide. The boresight establishes the following relationship: Q wf = Q we where we define Q wf to be the quaternion that rotates points and vectors from Frame space to World space, and Q we rotates from Eye space to World space [30] . Then the desired Eye->Tracker orientation Q te is computed by: Q te = Q tw * Q we Q te = (Q wt ) -1 * Q wf where Q wt is what the head tracker returns, and Q wf is known from step 1. The Eye->Tracker position offsets are measured by the boresight and one additional task. The position of the corner of the frame in World space is known, due to step 1. The position of the tracker origin in World space is returned by the head tracker. Therefore, we can draw a vector in World space from the corner of the frame to the Figure 5 : Side view of FOV calibration results. To measure the remaining variation, we had three users repeat the boresight and FOV steps five times, moving their heads away in between each measurement. The average standard deviations in computed orientation, position, and FOV were 0.32 degrees, 4.8 mm (mostly along the Z offset), and 0.1 degrees respectively. While not fatal, this variation does mean users may have to try the procedures more than once to achieve desired results. The static registration demonstrated in Figures 15-22 holds only when the user stands still. When the user moves and rotates his head, the virtual objects appear to "swim around" the real objects, because of the system latency. The system requires time to measure the head's location, compute the corresponding images for that location, and display those in the HMD. This delay between measuring the head location and displaying the corresponding images means the images will be incorrect if the user moves his head during this delay. The virtual objects appear to lag behind their real counterparts, causing large dynamic registration errors. To reduce these dynamic errors, we predict head motion. Instead of using the reported head location to generate the graphic images, we predict where the head will be when the displays get updated. If the prediction is correct, the computed virtual images will match reality at the time they are viewed. How does our system differ from previous work? One difference is the application. We use prediction to aid registration of real and virtual objects in a see-through HMD and evaluate it in that context, something the other systems did not attempt. We also use gyros and accelerometers to aid both orientation and position prediction, which no previous system does. Our work also contributes the following: • Evaluation of how much inertial sensors help • Measurement and control of prediction distance • Autocalibration of inertial sensor parameters We run separate predictors for orientation and position. Each consists of two parts: 1) an estimator that computes our best guess of the current position, velocity, and acceleration, and 2) a predictor that takes those guesses and extrapolates the desired distance into the future. Every time a tracker position and orientation measurement arrives, along with the corresponding inertial measurements, the estimator updates its guesses. Whenever Pxpl5 is ready to compute a new scene, the predictor sends it an output corresponding to the time when that scene will appear. The estimator used is the Kalman filter [19] , which many previous works also use. Space does not permit a detailed description of the filter; please read [22] for that. The Kalman filter is a linear estimator that minimizes the expected mean-square error. For orientation, we use a nonlinear variant called the Extended Kalman Filter (EKF). The filter requires some variables to be estimated, occasional noisy measurements of those variables, and a model of how those variables change with time in the absence of new measurements. It is optimal only if the model is an accurate reflection of reality and if the uncertainty in both the model and the measurements is accurately represented by additive white noise. Even though tracker origin. Rotating this vector by (Q te ) -1 *Q tw transforms it to Eye space. Since Eye space and Frame space share the same orientation and their Z axes coincide, the X and Y values of the vector in Eye space are the X and Y Eye->Tracker offsets, in Eye space. To determine the Z offset, we need one more operation. Two nails are on top of the frame, one in front and one in the rear (Figures 1, 4, &  14) . While performing the boresight, the user must also position himself so the front nail covers the rear nail. A red LED mounted on the rear nail helps the user detect when this occurs. The known locations of these two nails identify a specific distance along the frame's Z axis where the user's eye must be. Subtracting that from the corner->tracker vector in Eye space yields the Z component of the Eye->Tracker offset. The user performs two boresights: one from a few feet away for greater orientation sensitivity, and one less than a foot away (matching the two nails) for greater position sensitivity. It suffices to measure FOV along the vertical Y direction in screen space, since scaling that by the frame buffer's aspect ratio yields the horizontal FOV. The crosshair's Y radius is set to 125 pixels so the top and bottom lines are easily visible. The user stands in front of the frame and lines up the top and bottom virtual crosshair lines with corresponding real lines drawn on the frame's front surface (Figures 12, 14) . This forces the Eye space X axis to be parallel to the Frame's X axis. From the information in steps 1, 2 and 3, we can compute the locations of the real lines in Eye space. By intersecting the lines with the X=0 plane in Eye space, we reduce the geometry to 2D, as shown in Figure 5 . We can always get right angles for y1 and y2 by using the line Y=0 as the basis. Then: ß1 = (-1.0) tan -1 (y1/z1) ß2 = tan -1 (y2/z2) z1 and z2 are positive, y2 is positive and y1 is negative as drawn. This still works if the user's eye is above or below both of the real lines: the signs of y1, y2, ß1 and ß2 change appropriately. Since the crosshair does not cover the entire frame buffer height (512 pixels), we must scale the result to compute the total FOV ß: ß = (ß1+ß2)(512/(2*125)) The parameters measured in steps 1-4 are sufficient to implement registration of the virtual axes. Step 1 tells us where to put the virtual axes. The other parameters tell us how to generate view matrices for the right eye, given reports from the head tracker. The only unusual aspect is the need for an off-center projection. These procedures, used with our optoelectronic tracker, generate parameters that work well from many different viewing angles and positions. To demonstrate this, we recorded a video sequence, using only one set of parameters, of a user walking around and looking at the corner of the frame from many different places. If space allows, excerpts from this will be on the CD-ROM. At several places during the run, the HMD was kept still. These viewpoints are identified by the numbered circles in Figure 14 , which correspond with the still images in Figures 2 and 15-22 . The red and green bars have a 5x5 mm 2 cross-section, while the blue bar is 7x7 mm 2 since it's harder to see at long distances. Note that the corner and edges usually stay within the width of the extruded rectangles at the static viewpoints, which puts the registration within ±4 mm for the red and green bars and ±5 mm for the blue bar. How do these results compare to the two previous works? Janin [18] takes a very different approach. He directly measures parameters with instruments and runs a camera calibration optimizer that requires the user to identify the location of ~20 object points from several different viewpoints. The best accuracy he achieves is ±12mm. In contrast, step 3 of our procedure is similar to Caudell's [6] registration platform, which computes the Eye->Tracker position and orientation offset by having the user line up two circles and two lines. He does not provide any information on how accurate his static registration is, and his registration procedure lacks equivalents to our steps 2 and 4. Registration accuracy depends on how successfully the user can complete the registration procedures. Users reported difficulty in keeping their heads still during the boresight and FOV operations, because of the weight of the HMD. To compensate, we use the most recent 60 tracker reports to compute each operation, averaging the Eye space orientation these assumptions are usually not met, the Kalman filter is still popular because it tends to perform well even with violated assumptions and its recursive formulation makes it efficient to compute. Building a Kalman filter is easy. The difficult parts are determining an appropriate model and finding good noise parameters. For the latter task, we collected several runs of tracker and inertial data while the HMD-wearer performed "typical" head motions. Then we ran Powell's method [27] to search for parameters that minimized prediction error at a fixed prediction distance. In practice, this heuristic is able to find a fairly wide range of parameters that meet this goal. We now outline the orientation estimator and predictor, later describing the translation case by how it differs from orientation. T where Q is a quaternion rotating points and vectors from Tracker space to World space, W is omega, the angular rate of rotation in head space, and X is the Nx1 state vector, where N=10. P is an NxN covariance matrix representing the uncertainty in X. The initial value of X holds the starting quaternion and has zeroes for omega and its derivative. P is initially a diagonal matrix with zeroes in all off-diagonal positions, 1 for the first four diagonal terms (quaternion), and 50 for the remaining six diagonal terms (omega and its derivative). The initial covariances are large so the filter will replace the initial X with new measurements as they arrive. X and P are maintained at the current time t. When each new measurement arrives, say at time t1, the filter performs two operations: 1) a time update that advances the state variables to time t1 based on the model, and 2) a measurement update that blends in the values measured at time t1. 1) Time update: A 4th-order Runge-Kutta ODE solver [27] integrates the derivatives of X and P from time t to t1. The derivatives are: (1) where E is an NxN matrix representing the noise in the model, and A is the NxN Jacobian matrix of the nonlinear function a() that returns the derivatives of X by computing: where for the derivative of Q, the multiplications are quaternion multiplications and W is written as a quaternion with zero w term [7] .  Q m = Normalize(Q), W m = W and the measurement update itself generates a new X and P given Z as follows: where K is an NxF matrix called the Kalman gain, H is the FxN Jacobian matrix of the nonlinear function h(), and R is the FxF covariance matrix representing the noise in the measurements. At the end of the measurement update, we explicitly renormalize the quaternion part of X. This isn't standard, but without it the quaternion terms quickly become unnormalized. Noise matrices E and R are determined during the offline optimization. Both are diagonal matrices with zeroes in all off-diagonal terms. The first six diagonal terms of E are set to 0.004452: a tiny amount of noise added to the measured quaternion and omega to help the stability of the EKF. The remaining three diagonal terms are set to 351.0. For R, the first four diagonal terms are 0.0001, representing Q m noise, and the remaining three diagonal terms are 0.005921, representing W m noise. Predictor: When the scene generator is ready to draw a new image, the predictor bases its extrapolation on the estimated values in X. The predictor is the closed-form solution of integrating the quaternion and omega under the assumption that the derivative of omega is constant over the integration interval t0 to t. We define a 4x4 matrix M(t) as satisfying: Q = (0.5)(Q)(W) = (M(t))(Q) where M(t) essentially rewrites the quaternion multiplication as a matrix multiplication. The solution of this is: where Q t0 is the original quaternion at time t0 and: The position estimation uses three separate linear Kalman filters, one each for X, Y and Z. Since they are identical in form, we look at the Y case only. This section lists the differences from the orientation case:  The inertial outputs must be rotated into the Tracker coordinate system, because the inertial sensor packs are tilted with respect to Tracker space. To perform these rotations, we must know the orientation of each pack on the HMD. While it is possible to mechanically build a mount that holds each pack at a specified orientation, it's easier to mount the packs at some rigid, but unknown, orientations, then measure them. Also, we would like to measure other sensor parameters, like the biases and scales. Autocalibration refers to mathematical methods that determine such constant parameters by applying geometrical constraints to collected data. One demonstration of this measured the locations of the beacons in the panels of our optoelectronic tracker [15] . Two such methods that we use with our inertial sensors are described in Appendix C on the CD-ROM. They are good at determining the orientation and biases, but not the scales. From the user's perspective, prediction changes dynamic registration from "swimming around the real object" to "staying close." Without prediction, registration errors are large enough to strain the illusion that the real and virtual coexist. With prediction, the real and virtual objects stay close enough that the user perceives them to be together. Although the prediction is not perfect, it demonstrably improves the dynamic registration. The predictor was run on three recorded motion datasets that are considered representative of this registration task. During each motion sequence, the user keeps the corner of the frame visible in his field-of-view. In the Walkaround dataset, the user walks slowly around the corner of the frame. The Rotation dataset has the user yawing, pitching, and circling his head while standing in place. The Swing dataset combines fast translation and rotation motion. We compared our inertial-based predictor on these three datasets against doing no prediction and against a predictor that does not use inertial sensors. Directly comparing our predictor against previous work is difficult because our system is unique. Instead, we wrote a Kalman-filter-based predictor that is representative of many previous works that do not use inertial sensors. We ran that on the datasets, keeping all other variables constant. Three error metrics evaluate the accuracy of the predicted outputs vs. the actual tracker measurements. Angular error is computed in degrees as follows: [qw]) Position error is the distance between the predicted and actual translations. Screen error combines orientation and translation errors by measuring the difference, in pixels, between the 2D projection of the real frame and the 2D point where the three virtual axes intersect on a hypothetical 512x512 screen. That is, it measures error in terms of what the user sees inside an HMD. The peak and average errors are summarized in Figure 6 . On average, our inertial-based predictor is 5-10 times more accurate than doing no prediction and 2-3 times more accurate than prediction without inertial sensors. Appendix B on the CD-ROM provides additional materials that demonstrate the results of prediction. Depending upon the space allocation on the CD-ROM, these may include error graphs, the motion datasets, and a short QuickTime video. Figure 7 shows how the average screen-based errors for the Rotation run change as the prediction distance is varied from 25-200 ms. Again, inertial sensors clearly help. But what this graph does not show is that at prediction distances of ~100 ms or more, the jitter in the predicted outputs often reaches objectionable levels. In practice, the only solution is to keep system latency at tolerable levels, below ~80 ms. Thus, prediction cannot compensate for arbitrary amounts of latency; to be effective, it must be combined with efforts that minimize system lag. See Appendix D on the CD-ROM. Because one cannot accurately predict without knowing how far to predict, our system requires accurate clocks and control over latency. Our tracker and graphics engine run asynchronously, requiring an estimation of the prediction distance at each iteration. Miscalculating the prediction distance by as little as 10 ms leads to visible registration errors. The clocks in the tracker boards and Pxpl5 are good to under a millisecond. Synchronization occurs through a message that takes less than 1 ms to make a round trip of all the processors. Since clocks that differ by one second every six hours change by 8.3 ms every three minutes, skew rate compensation and occasional resynchronizations are performed. Controlling latency means removing all unpredictable delays, so we have direct communication paths not shared with other users, and we run a lowoverhead operating system called VxWorks. Processes running on UNIX can suffer unbounded amounts of delay. Pauses of 60-200 ms are common occurrences on our Sun4 host. Therefore, we avoid UNIX for all time-critical tasks, directly injecting data from the tracker into Pxpl5 without going through the Sun4 host. While these steps reduce flexibility and make it harder to debug the system, they are needed to insure accurate prediction. Appendix E on the CD-ROM demonstrates the accuracy of our prediction distance estimation. Augmented Reality demands higher accuracy from head trackers than Virtual Environment applications do [3] . The main difficulty in duplicating our demonstration of static registration is in acquiring a head tracker that one can trust at long ranges. Wooden crates are easy to build, and the calibration techniques are straightforward and applicable to any tracking system or see-through HMD. But many commercially available trackers commonly used for Virtual Environments do not provide sufficient performance. For example, in our laboratory the widely used Polhemus magnetic trackers give distorted outputs at long distances because of the metal in the environment. A coworker experimenting with our optoelectronic tracker discovered a distortion that, when the tracker "hat" yaws 360 degrees about its origin, causes the reported positions to trace an ellipse about an inch wide. Since this distortion seems to be systematic, we were able to compensate for it. The fact that this distortion was undetectable in the Virtual Environment applications we run but was quite noticeable in Augmented Reality only serves to underscore the latter's need for accurate, long-range trackers. The need to accurately measure time and avoid unpredictable sources of latency has serious ramifications on the design of effective Augmented Reality systems. Tracker measurements must be timestamped, a feature not provided by most commercial trackers. Almost all interactive graphics applications in our laboratory use UNIX because of its convenient programming environment. We build applications to the desired complexity, then we extract as much speed as possible. Flight simulators and some other researchers [20] take the opposite approach: set a minimal standard for performance, then see how much complexity can be supported. Since accurate prediction requires guaranteed performance, future Augmented Reality applications may need to take this latter approach. Much work remains to further improve static registration. We only match one virtual object with one real object, where the real object is the calibration rig itself. Because our optoelectronic tracker loses accuracy when the sensors are not aimed at the ceiling beacons, we cannot move the HMD far away from the wooden frame, nor can we tilt the HMD far from horizontal. Our system is monocular; while our static registration procedure could be applied to both eyes, stereo displays involve additional issues like convergence that we have not addressed. We have not compensated for the optical distortion in the see-through HMD. Because our HMD has narrow field-of-view displays, this distortion is small and detectable only near the edges of the displays. We can eliminate this error by mapping the distortion, then predistorting the graphic images before displaying them [31] . More sophisticated prediction methods might further reduce dynamic registration errors. Adaptive methods that adjust to varying head motion deserve more exploration. Using a nonadaptive predictor is like trying to race a car at constant speed; slowing down on the curves and speeding up on the straight-aways will improve your time. Analyzing head motion for recognizable patterns or high-level characteristics may aid prediction. Other researchers have begun doing this [32] , and Fitts' Law has been shown to apply to head motion [2] [17]. Our work has not dealt with video see-through HMDs, where a video camera provides a view of the real world and the graphics are composited with the digitized images of the real world. With this class of see-through HMD, standard camera calibration techniques could determine the viewing parameters. And since the computer has digitized images of what the user sees, it may be possible to use image processing or computer vision techniques to detect features in these images and use them to aid registration. The disadvantage of this technology is that the video camera and digitization hardware impose inherent delays on the user's view of the real world. Therefore, even if the graphics are perfectly registered with the digitized images, a problem remains: the latency in the video stream will cause the user to perceive both the real and virtual objects to be delayed in time. While this may not be bothersome for small delays, it is a major problem in the related area of telepresence systems and may not be easy to overcome.
Azuma used inertial sensors to predict future head location combined with an optoelectrical tracker REF .
Improving static and dynamic registration in an optical see-through HMD
SIGGRAPH '94
Computer Science