# Developing AR Apps

## Summary


Augmented Reality (AR) started with simple apps that offered either location-based information or visualization based on the scanning of 2D markers. As AR moves into the domain of sophisticated image processing, 3D rendering and interactions, there's a need for better AR authoring tools. 

In general, 3D tools are complex and difficult to master. Creating high quality 3D models also require particular expertise. If AR has to take off, authoring tools must offer simple and intuitive workflows. This is already happening. The idea is to enable domain experts, even if they lack 3D modelling expertise, to easily create AR experiences. It's expected that the AR ecosystem will eventually offer tools and SDKs for developers at all levels.

## Discussion

### What sort of data can I use for AR content?

AR content can include a variety of data input sources: CAD files and wireframes, 3D video, holographs, 360-degree images, 3D representations of real environments, and 3D sound. Increasingly, authoring tools will provide ways to integrate 3D design with AR. For example, 3D gaming engine Unity can import ARToolkit for tracking purpose. In 2015, CAD vendor PTC acquired AR authoring system Vuforia. 

Authoring systems are also capable of exporting AR content targeting a variety of devices: smartglasses, smartphones, tablets, head-mounted displays, etc. 


### What's a typical AR app development workflow?

For any project, it's customary to identify the problem, propose a solution or approach, list the requirements, study technical feasibility, select suitable tools and technologies, etc. Specifically for AR app development, there are some unique things to consider: 

    + **Data acquisition**: In medical field, for example, data can come from CT or MRI scans. In retail, data may come from printed catalogues. In engineering, data may come from sensors. To build accurate models, data may need to be cleaned and preprocessed.
    + **Modelling**: 3D models are generated based on the data. If models are already available, then they may be simply imported into AR authoring environments.
    + **Interface development**: Developer defines the scenes and their transitions. Models are imported into scenes. Within scenes, user interactions are enabled, such as clicking, zooming or rotating 3D models. The AR experience is then tested within the development environment and later on target devices.

### As an AR app developer, what tools should I adopt?

Because 3D content is an important aspect of AR, and many game engines already work with 3D content, game engines are commonly adopted by AR developers for authoring AR experiences. Top game engines include Unity and Unreal Engine. The development process can be simplified by using an AR SDK. Popular AR SDKs can be imported into game engines. For example, Vuforia can be used from with Unity. 

When selecting an SDK look for these capabilities: licensing, supported platforms (iOS, Android, Universal Windows Platform), target devices (phones, tablet, smartglasses, HoloLens, etc.), supported game engines, recognition (cloud or on-device), 3D tracking, geolocation, Simultaneous Localization and Mapping (SLAM)... 

AR SDKs are many: PTC Vuforia, Apple ARKit, Google ARCore, Wikitude, Kudan, MaxSt, EasyAR, and more. Some SDKs may be better at marker-based AR while others may be better at SLAM. Some SDKs such as Wikitude and Vuforia allow the use of other SDKs such as ARKit. 


### Do I need specific math skills to be an AR developer?

When creating basic AR experiences, the use of AR authoring tools is more important. Developers can get by with basic math skills. For example, in Unity, you can scale or rotate 3D models by simply changing some settings. With Google's Sceneform SDK and its runtime API, developers can avoid learning the whole 3D development stack. With Blippar SDK, object recognition is made easy by calling computer vision APIs. 

For more complex AR experiences, knowledge of linear algebra, vectors and matrices will help. To build or manipulate 3D models, 3D math, 3D design, rendering and UX design become important. When handling sensor data, you need to know about transforms and filters. These are the same skills that game developers possess. Thus, game developers could move into AR development roles. 

Increasingly, Machine Learning is being used to process sensor data including live video feeds. This would mean that knowledge of statistics and ML models is relevant. 


### Could you share some tips for creating compelling AR apps?

John Fan, CEO and Founder of Kopin, gives us five rules for building great AR: 

    + Humans first: User comfort must come before technology. To wear something, users must be convinced of its real value.
    + Physical world first: Too much virtual content can overwhelm users. Use overlays only to complement real-world experiences.
    + Maintain situational awareness: All senses must continue to be in touch with the real world. Physical reality cannot be replaced.
    + Voice is the new touch: Keyboards and touch screens are compromises. Voice should be used effectively for command and control.
    + Balance design with benefits: Don't overdesign. Remove unnecessary features. Design for specific AR benefits.


## Milestones

2014

Google announces **Project Tango**. It enables AR by its ability to map its surroundings using sensors, motion tracking camera, 3D depth sensing, depth perception and area learning. It's available in limited devices that meet its hardware requirements. With the coming of ARCore in 2017, it's discontinued in 2018. 

Jun  
2017

Apple announces its AR platform named **ARKit**. This uses *Visual Inertial Odometry (VIO)*. Using camera and motion sensors, it marks the environment with a bunch of points and tracks them as the user moves around. 

Aug  
2017

Google releases **ARCore**, an AR SDK for the Android platform. Unlike Google's Tango, ARCore can work on many Android phones without any specialized hardware. ARCore works with Java/OpenGL, Unity and Unreal. It's capable of motion tracking, environmental understanding and light estimation. While not as advanced as Tango, ARCore has potential to reach mass market. 

Oct  
2017

Unity 2017.2 is released with support for Vuforia, ARCore, ARKit and Windows Mixed Reality immersive headsets. 

May  
2018

ARCore 1.2 enables multiplayer AR across both Android and iOS. It's capable of wall detection. *Cloud Anchors* allows virtual objects to be placed in 3D space and synchronized to other devices via the cloud. **Sceneform SDK** is also released.