Develop your HelloAR app in Android studio using ARCore and Sceneform
Tags: arcore, android, sceneform, ardevelopment
Published: 2022-04-16 07:44:13

AR turning accessible with the increasing interest within the metaverse trend daily. It is changing the traditional way of interactions into more digital.

### What is Augmented Reality
Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory — Wikipedia

### What is ARCore
ARCore is a software development kit developed by Google that allows for augmented reality applications to be built.
ARCore uses three key technologies to integrate virtual content with the real world as seen through the camera of a smartphone.
- Six degrees of freedom allow the phone to understand and track its position relative to the world.
- Environmental understanding allows the phone to detect the size and location of flat horizontal surfaces like the ground or a coffee table.
- Light estimation allows the phone to estimate the environment’s current lighting conditions.

### Sceneform
Sceneform is the Google official AR SDK to enable developers to make AR apps while not having to be told OpenGL.
Sceneform comes with options such as:
- An automatic compatibility check for ARCore enabled phones.
- Checking for camera permissions.
- A plugin for manipulating 3D assets.
- A scene graph API to abstract all the complexities.

## Getting Started
You first need to enable ARCore in your project. This is simple as we will be using Android Studio and Sceneform SDK. There are two major operations Sceneform performs automatically:
Checking for availability of ARCore
Asking for camera permission
Create a new Android Studio project and select an empty activity.

![android project setup 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dpwim3p17b1r7zxhsh1z.png)
![android project setup 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/biwcj9n2ur6f3z7k8bqx.png)
![android project setup 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3p5kwb51wml4jr35iyp5.png)

### Adding Dependencies
Add the following dependency to your project level build.gradle file:

```
dependencies {
    ...
    classpath 'com.google.ar.sceneform:plugin:1.17.1'
}
```
Add the following dependency to your app level build.gradle file:

```
dependencies {
    ...
    implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1'
    implementation 'com.google.ar:core:1.27.0'
}
```
Add the following lines in your AndroidManifest.xml file:

```
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.INTERNET"/>
<uses-feature android:name="android.hardware.camera.ar" android:required="true"/>

<application
    ...>
    ...
    <meta-data
        android:name="com.google.ar.core"
        android:value="required" />
</application>
```
Then sync your project with Gradle files and wait for the build to finish (click on the sync icon). This will install the Sceneform SDK to the project and Sceneform plugin to AndroidStudio.

Now, we can begin with creating our first ARCore application.
In the first place, we have to add the Sceneform fragment to our design layout. This will be the container for rendering the 3d models and deals with the camera instantiation and other ar related functionalities.
Add the sceneform ar fragment into the activity_main.xml like this,

```
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical"
    tools:context=".MainActivity">

    <fragment
        android:id="@+id/ux_fragment"
        android:name="com.google.ar.sceneform.ux.ArFragment"
        android:layout_width="match_parent"
        android:layout_height="match_parent" />

</LinearLayout>
```

### Checking compatibility at runtime
Add the method below in your class for the compatibility check:

```
private boolean checkIsSupportedDeviceOrFinish(final Activity activity) {
        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
            Log.e(TAG, "Sceneform requires Android N or later");
            Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();
            activity.finish();
            return false;
        }
        String openGlVersionString =
                ((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))
                        .getDeviceConfigurationInfo()
                        .getGlEsVersion();
        if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {
            Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");
            Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
                    .show();
            activity.finish();
            return false;
        }
        return true;
    }
```
### Adding 3D models to our application
Now, we can download and import the 3D models into our application. You will get 3D models from the repositories like cgtrader, free3d and turboquid. Google also provided an excellent repository POLY to download 3D models for your application. You can download the models in .fbx, .obj or .gltf format. We will be doing it in the .fbx format.
In this demo, we will be augmenting a 3D cube on the user tap.
Open the project and expand the app folder in the project view. Create a folder named **sampledata** inside the app folder. Then you have to add the downloaded model file inside this **sampledata** folder. Here I have added a cube.fbx model. You can add your own model.

### Importing the model using the Sceneform plugin
Apply sceneform ar plugin in your app’s build.gradle file.
Add the following dependencies:

```
apply plugin: 'com.google.ar.sceneform.plugin'
```

Add the following lines at the end of the app’s build.gradle file:

```
sceneform.asset('sampledata/cube.obj',
        'default',
        'sampledata/cube.sfa',
        'src/main/res/raw/cube')
```

Finally, the build.gradle of the app looks like this:

```
apply plugin: 'com.android.application'
apply plugin: 'com.google.ar.sceneform.plugin'

android {
    compileSdkVersion 31
    defaultConfig {
        applicationId "com.example.arcorefirst"
        minSdkVersion 24
        targetSdkVersion 31
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner 'androidx.test.runner.AndroidJUnitRunner'
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    implementation 'androidx.appcompat:appcompat:1.3.1'
    implementation 'androidx.constraintlayout:constraintlayout:2.1.1'
    testImplementation 'junit:junit:4.13.2'
    androidTestImplementation 'androidx.test.ext:junit:1.1.3'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
    implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.17.1'
    implementation 'com.google.ar:core:1.27.0'
}

sceneform.asset('sampledata/cube.obj',
        'default',
        'sampledata/cube.sfa',
        'src/main/res/raw/cube')
```

### Integrating the Model
Add the following code to your java file:

```
@Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        if (!checkIsSupportedDeviceOrFinish(this))
            return;

        setContentView(R.layout.activity_main);

        arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);
        arFragment.setOnTapArPlaneListener(
                (HitResult hitresult, Plane plane, MotionEvent motionevent) -> {
                    if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING)
                        return;

                    Anchor anchor = hitresult.createAnchor();
                    placeObject(arFragment, anchor, R.raw.cube);
                }
        );
    }
```

### Adding the Model to the AR Scene
The AR fragment is a container to hold all 3d objects. So, we will add a model to the fragment whenever it is clicked by implementing `onTapListener` in our fragment.

```
private void placeObject(ArFragment arFragment, Anchor anchor, int uri) {
        ModelRenderable.builder()
            .setSource(arFragment.getContext(), uri)
            .build()
            .thenAccept(modelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable))
            .exceptionally(throwable -> {
                    Toast.makeText(arFragment.getContext(), "Error:" + throwable.getMessage(), Toast.LENGTH_LONG).show();
                    return null;
                }
            );
        }

        private void addNodeToScene(ArFragment arFragment, Anchor anchor, Renderable renderable) {
        AnchorNode anchorNode = new AnchorNode(anchor);
        TransformableNode node = new TransformableNode(arFragment.getTransformationSystem());
        node.setRenderable(renderable);
        node.setParent(anchorNode);
        arFragment.getArSceneView().getScene().addChild(anchorNode);
        node.select();
    }
```

**Let’s understand the code:**
Using the `hitResult`, get the tapped location value and create an anchor node on that position. The anchor node is used to place the object in the real world position but it can’t perform any operation over the model. So, we create a `TransformableNode` which can enable the user to apply the various transformation on the object like translate, rotate and scale.
**Let’s have a look at some terminologies here:**
**Scene**: It’s the place where our 3D world will be rendered.
**HitResult**: It is an imaginary ray of light coming from infinity and its first point of intersection with the real world is the point of the tap.
**Anchor**: A fixed location in the real world. Used to transform local coordinates into real-world coordinates.
**TransformableNode**: A node that can react to user interactions such as translation, rotation, scale and drag.
Finally, the MainActivity.java file looks like this:

```
package com.example.arcorefirst;

import android.app.Activity;
import android.app.ActivityManager;
import android.content.Context;
import android.net.Uri;
import android.os.Build;
import android.os.Bundle;
import androidx.appcompat.app.AppCompatActivity;
import android.util.Log;
import android.view.MotionEvent;
import android.widget.Toast;

import com.google.ar.core.Anchor;
import com.google.ar.core.HitResult;
import com.google.ar.core.Plane;
import com.google.ar.sceneform.AnchorNode;
import com.google.ar.sceneform.rendering.ModelRenderable;
import com.google.ar.sceneform.rendering.Renderable;
import com.google.ar.sceneform.ux.ArFragment;
import com.google.ar.sceneform.ux.TransformableNode;

public class MainActivity extends AppCompatActivity {
    private static final String TAG = MainActivity.class.getSimpleName();
    private static final double MIN_OPENGL_VERSION = 3.0;

    ArFragment arFragment;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        if (!checkIsSupportedDeviceOrFinish(this))
            return;

        setContentView(R.layout.activity_main);

        arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);
        arFragment.setOnTapArPlaneListener(
                (HitResult hitresult, Plane plane, MotionEvent motionevent) -> {
                    if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING)
                        return;

                    Anchor anchor = hitresult.createAnchor();
                    placeObject(arFragment, anchor, R.raw.cube);
                }
        );
    }

    private void placeObject(ArFragment arFragment, Anchor anchor, int uri) {
        ModelRenderable.builder()
            .setSource(arFragment.getContext(), uri)
            .build()
            .thenAccept(modelRenderable -> addNodeToScene(arFragment, anchor, modelRenderable))
            .exceptionally(throwable -> {
                    Toast.makeText(arFragment.getContext(), "Error:" + throwable.getMessage(), Toast.LENGTH_LONG).show();
                    return null;
                }
            );
    }

    private void addNodeToScene(ArFragment arFragment, Anchor anchor, Renderable renderable) {
        AnchorNode anchorNode = new AnchorNode(anchor);
        TransformableNode node = new TransformableNode(arFragment.getTransformationSystem());
        node.setRenderable(renderable);
        node.setParent(anchorNode);
        arFragment.getArSceneView().getScene().addChild(anchorNode);
        node.select();
    }

    private boolean checkIsSupportedDeviceOrFinish(final Activity activity) {
        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
            Log.e(TAG, "Sceneform requires Android N or later");
            Toast.makeText(activity, "Sceneform requires Android N or later", Toast.LENGTH_LONG).show();
            activity.finish();
            return false;
        }
        String openGlVersionString =
                ((ActivityManager) activity.getSystemService(Context.ACTIVITY_SERVICE))
                        .getDeviceConfigurationInfo()
                        .getGlEsVersion();
        if (Double.parseDouble(openGlVersionString) < MIN_OPENGL_VERSION) {
            Log.e(TAG, "Sceneform requires OpenGL ES 3.0 later");
            Toast.makeText(activity, "Sceneform requires OpenGL ES 3.0 or later", Toast.LENGTH_LONG)
                    .show();
            activity.finish();
            return false;
        }
        return true;
    }
}
```

**Let’s see what’s happening here:**

1. First, we need to get the ar fragment with its id from the layout file using `supportFragmentManager` .

2. Then we need to load the model into the scene. For this, we use the `ModelRenderable` class provided by the Sceneform SDK. We load the model by passing the name of the generated .sfb file as the argument of the `setSource()` method.

3. The model is being built on a background thread. After that, it renders in the scene using the main thread.

4. We receive the model inside the `thenAccept` method. An exception is thrown if there’s an error in building the model.

5. Now, we create an `anchorNode` and a `transformableNode`. Also, refer to the renderable object inside the `setRenderable()`. Finally, we set `anchorNode` as the parent of the `transformableNode` and add the `anchorNode` to the scene.

There you have it! Your own AR App in Android studio :)

![app demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5nc0bi7h4rbhljbqa9nn.gif)

Thanks for reading this article.
If you enjoyed this article, please click on the heart button ♥ and share to help others find it!
The full code of the project in this tutorial is available on

[codemaker2015/ar-object-augmentation-android-studio](https://github.com/codemaker2015/ar-object-augmentation-android-studio)

Originally posted on Medium - 
[Develop your HelloAR app in Android studio using ARCore and Sceneform](https://medium.com/@codemaker2016/develop-your-helloar-app-in-android-studio-using-arcore-and-sceneform-ae9e1b7a1b5b)