# Augmented Reality

## Summary


With the coming of the social web, mobile web and Big Data, information has become plenty and easily accessible. As late as the mid-2010s, much of this information remained within the digital world of servers, computers and smartphones. Augmented Reality (AR) attempts to remedy this by bringing digital information into the real world. In essence, AR re-imagines human-computer interaction. 

Augmented Reality is a term that requires our presence and interaction in the real world. At the same time, our experience of the real world is augmented by adding virtual elements created by computers. These virtual elements are typically visuals and sounds. The idea is to give us extra information to do things more efficiently and enjoyably.

## Discussion

### What's the definition of Augmented Reality?

One possible definition of AR has three characteristics: it combines real and virtual; it's interactive in real time; and it's registered in 3D. This definition is regardless of the display hardware. It applies equally well to visuals that are optical see-throughs or rendered on monitors. A movie such as Jurassic Park is not AR since it's not interactive for viewers. 

Since this definition requires 3D integration into a real environment, 2D overlays on live video are also excluded from AR. Some writers consider this as AR as well, calling it *TV AR*. Augmenting live video with digital information for a sports game or a weather report would be considered TV AR. 

Others define AR in terms of the interfaces/hardware used. But conceptually, the definition is still includes merging the real and the virtual while retaining presence and interaction with the real world. 


### How is AR different from VR and MR?

The acronyms AR, VR and MR refer to Augmented Reality, Virtual Reality and Mixed Reality.

While AR experiences are rooted in the real world, VR experiences are rooted in the virtual world. In VR, the user is immersed in a synthetic world created by computer graphics. Such a virtual world may be inspired by the real world but it's also free to break the laws of physics that govern real-world interactions. 

AR and VR can also be differentiated by user presence. A person sitting in New York, visualizing the Eiffel Tower in Paris surrounds himself in VR, a world that represents the Tower and its surroundings. VR enables him to experience the Tower without going to Paris. A tourist who is already near the Tower, could use an AR app to get directions. When he is at the Tower, the AR app could display or read out relevant information while the user points his camera at the Tower. 

MR is an environment where users can interact with both real and virtual objects. 


### What are some possible applications of AR?

Early ideas for AR were mostly in the industrial space rather than the consumer space: medical, engineering, military, robotics, etc. Due to smartphone adoption, AR in the consumer segment is gaining interest with education, gaming and advertising showing good potential. 

In education, live 3D models can help medical students visualize veins in patients. Pokémon Go and Real Strike are example AR games. GPS-based AR apps can help walkers or drivers navigate. AR can add value in the treatment of Post-Traumatic Stress Disorder. In real estate, AR can help you visualize your living room for its interior decor. 

Advertisers can give more information about a product via an interactive AR app, such as the BMW mini ad of 2008. A shopper can try out a dress or a makeup in AR before the actual purchase. AR can be used to create public awareness such as the WWF-Coca-Cola Arctic Home Campaign of 2013. Tourist spots or operators can offer self-guided AR-enabled tours. Actors can prepare for their roles in front of AR mirrors. 


### What are the different types of AR?

AR can be categorized based on the technique used:

    + **Marker-based AR**: This involves a camera and image/pattern recognition. Markers in the real world are recognized. They are then overlaid with virtual objects or information. This is also called **Recognition-based AR**.
    + **Location-based AR**: The use of GPS plus sensor data (in smartphones) enables apps that are location-centric such as finding relevant shops nearby or showing directions. Information is overlaid based on present location. This is also called **Markerless AR**.
    + **Projection-based AR**: This works by projecting light onto real-world surfaces. Your empty desk can have a virtual keyboard. Your palm can be lit up with a phone dialler. User interactions are detected based on changes in the projected light. This type of AR includes 3D interactive holograms.
    + **Superimposition-based AR**: Using object recognition, real-world objects are replaced or augmented with digital equivalents. Virtual objects are thus superimposed on the real world.
    + **Outlining AR**: Using object recognition, relevant features are outlined virtually. This may be seen as a specialization of superimposition-based AR.It's possible to combine many of these within a single app. 


### I've heard of SLAM. What is it?

**Simultaneous Localization And Mapping (SLAM)** is a well-known problem in mobile robotics. A robot in an unknown environment has the twin tasks of mapping its surroundings while also determining its own location within the map. 

SLAM is being increasingly used in AR and is seen as a successor to marker-based AR. The good thing is SLAM doesn't need a marker. Multiple cameras with depth sensing technology are used in SLAM to create a map of the surroundings. Unlike marker-based AR, SLAM lacks context. GPS can be used to give context. For indoor navigation apps, SLAM maps (databases) can be used to create context. The idea is to obtain context with nothing more than visual input. 

Solutions to the SLAM problem can be combined with other techniques in AR to effectively combine real and virtual worlds. Apple's ARKit uses SLAM and so does Google's Project Tango. The latter's effort is also towards building context. 


### Isn't a QR Code the same thing as AR?

A Quick Response (QR) code is basically a 2D barcode that's machine readable. Once scanned, say with a smartphone, it can give more information about the item to which it's attached. A marker-based AR can do this and more. QR codes have to conform to a specific layout while in AR the marker can be any object or pattern. 

QR codes were invented in the 1990s when image recognition had to be simple and fast. In comparison, AR requires more processing but it's also more flexible because any object can be a marker. Moreover, contextual information can be shown at different parts of the same object. There's no need to reduce the object to a code like in QR code. Finally, using AR apps, tracking and analytics can be applied. This is rarely done with QR codes. 

With QR codes, information is typically displayed by redirecting the user to a video, URL, registration form, etc. With AR, information is overlaid on top of real-world objects. Immersion of 3D objects in the real world is possible in AR but not so with QR codes. This is a fundamental difference. 


### What are the components of an AR system?

An AR system can be said to contain the following components: 

    + **Tracking**: Via sensors and camera, the system tracks the user's viewpoint.
    + **Registration**: Virtual objects must be spatially registered or anchored in the real world.
    + **Visualization**: Based on current location and viewpoint, the visualization of virtual objects has to be adjusted.
    + **Spatial Model**: This consists of both the real world and the virtual world. Both must be registered in the same coordinate system.

### Could you mention the technical foundations of AR?

For AR to work, virtual objects must be placed accurately in the real world. We can identify the following essentials:

    + **Visual Keypoint Matching**: Also referred to as *Marker Detection*, this requires image processing, feature extraction and marker detection. The marker's surface is determined so that virtual objects can be placed on the surface.
    + **Spatial Mapping**: The idea is to map the real world to a virtual model. Depth sensing is involved. The virtual model can be used to detect surfaces (walls, floors, tabletops). When virtual objects are placed, occlusion becomes important.
    + **Sensing**: Viewer becomes the anchor of the virtual space and content. Viewpoints are adjusted based on inputs coming from sensors: GPS, accelerometer, gyroscope, etc. Since sensing accuracy may be limited, this can be combined with visual tracking.

### What sort of hardware is needed to realize AR?

AR typically requires some sort of a display; camera and other sensors to enable detection and interaction; computer processing to blend the real and the virtual. Smartphones have all of these, thus lowering the cost of adoption for end users. Today, AR can be delivered as smartphone apps and easily reach a worldwide audience. Wearables including head-mounted display (HMD) or eyeglasses such as Google Glass are AR-specific hardware. HMDs can be see-through or screen based.

To bridge real and virtual worlds, cameras and sensors are used. This information is processed to create a virtual model of the real world. For projection-based AR, miniature projectors are needed and these may be part of a headset wearable. 

A typical AR wearable would need sufficient processing power and memory, wireless connectivity and GPS. Sensors may include accelerometer, gyroscope and magnetometer to detect movements and thereby adjust the views of virtual objects. Some devices use mirrors to assist in aligning images to the viewer's eye. 

Explicit user control of AR could be via a touchpad or voice commands. 

## Milestones

1965

Ivan Sutherland, often called "Father of Computer Graphics", publishes an essay titled "The Ultimate Display". Here he describes ideas that are today part of AR/VR systems. 

1968

Ivan Sutherland creates an optical see-through head-mounted display (HMD). The system uses computer-generated graphics. Sutherland calls it the *The Sword of Damocles*.

1990

The term **Augmented Reality** is coined by Boeing researcher Tom Caudell. The idea was display schematics of wire bundle assembly on a see-through HMD to assist workers in an airplane factory. By 1994, the term is increasingly used in literature. 

1992

Louis Rosenberg develops an AR system called *Virtual Fixtures* at the U.S. Air Force Research Laboratory (AFRL). Because 3D graphics in the early 1990s were too slow for realistic experiences, Virtual Fixtures employed physical robots controlled by an exoskeleton worn by the user. 

1994

Researchers Milgram and Kishino attempt to create a taxonomy for **Mixed Reality (MR)**. They see AR as a subset of MR. AR consists of real environment augmented with virtual objects. They also define **Augmented Virtuality (AV)** as a virtual environment to which real objects are added. 

1999

Hirokazu Kato creates *ARToolkit*. It's open source. It overlays computer graphics on real video. In 2009, this is ported to Adobe Flash, thus bringing AR to the web. 

2003

Using ARToolkit, researchers at the Vienna University of Technology create an AR system with a Personal Digital Assistant (PDA). This system includes on-device real-time tracking that can at times be offloaded to the network via a wireless connection. 

2013

Volkswagen *MARTA (Mobile Augmented Reality Technical Assistance)* app provides technicians assistance on a repair process. 

May  
2014

Google releases **Google Glass** to the public after about a year of beta testing. It's a head-mounted display designed as eyeglasses. It features Bluetooth connectivity to the Internet via user's smartphone, voice control, touchpad, camera and display. 

2015

Microsoft announces **HoloLens** in January. A live demo of the same is done in April. 

Jul  
2016

Niantic releases a location-based smartphone AR app named *Pokémon Go*. It quickly becomes one of the most popular games, breaking multiple records (downloads and revenue). The game involves players visiting real-world locations to capture virtual creatures. 

Sep  
2017

Apple shows off capabilities of its **ARKit** along with its AR-capable iPhone 8 and iPhone X. Just two weeks earlier, Google launched its **ARCore**.