Navigating the AR Landscape in React Native with Kotlin
Tags: reactnative, kotlin, arcore, android
Published: 2023-12-15 16:26:31

## Introduction

In the [first part](https://dev.to/9bytes/building-ar-vr-experiences-with-react-native-part-1-bridging-the-gap-with-kotlin-1dii) of this series, we laid the groundwork by setting up _Kotlin_ within our _React Native_ environment, a crucial step in bringing augmented reality (AR) to our applications. This segment is designed for developers who are ready to take the next step in their AR journey, willing to learn how to integrate ARCore into their hybrid apps.

This is part two of a multi part article series about the integration of AR services via native modules into a _React Native_ app. You will learn how to integrate [ARCore](https://developers.google.com/ar/develop) and how to display a 3D-model in this article.

- Part one: [Setting up a native module via Kotlin](https://dev.to/9bytes/building-ar-vr-experiences-with-react-native-part-1-bridging-the-gap-with-kotlin-1dii).
- Part Three: [Setting up a native module via Objective-C](https://dev.to/9bytes/seamlessly-integrating-native-modules-in-ios-for-react-native-ar-apps-5alp)
- Part Four: [Integrating ARKit](https://dev.to/9bytes/managing-ios-augmented-reality-views-with-objective-c-in-react-native-apps-7ko)

The full code for this part is [here](https://github.com/Gh05d/RN3DWorldExplorer/tree/part-2).

### What is ARCore

> [ARCore](https://developers.google.com/ar/develop) is Google's platform for building augmented reality (AR) experiences. It enables phones to understand and interact with the real world. Using three key capabilities – motion tracking, environmental understanding, and light estimation – ARCore makes it possible for apps to merge virtual content with the real world in a seamless and realistic way.

### Add dependencies to the project

First we need to add the necessary dependencies to our project. Go into your _android/app/build.gradle_ and add the following implementations to your `dependencies` section:

```gradle
// AR
implementation 'com.google.ar:core:1.40.0'
implementation("io.github.sceneview:arsceneview:0.10.0")
```

It should look like this:

![app level build.gradle file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3xwm0fxx96l0ui59yl2i.png)

Then we have to bump our `minSdkVersion` to 24 in our _android/build.gradle_. Afterwards sync the project.

### Download a model

Now we need a 3D model to display. Download one from [here](https://ar-code.com/blog/ar-portals-3d-models-in-glb-and-usdz-formats) for example. Make sure to select the _GLB_ version!

### Add file to Android

To add a GLB model to your Android project in Android Studio, you should create an _assets_ folder and place your GLB file there. Here's how to do it:

1. Create the _assets_ Folder:

- Right-click on the _app_ folder in the Android Studio project view.
- Go to _New_ > _Folder_ > _Assets Folder_.
- Click _Finish_ in the dialog that appears to create the folder.

2. Add Your GLB File:

- Copy your GLB model file.
- Paste it into the _assets_ folder you just created.

### Create an Activity XML file

Now we need to create the layout file where our model will be displayed:

1. Locate the _res_ (Resources) Folder: In your project hierarchy in Android Studio, it is typically found under _app_ > _src_ > _main_.

2. Create the _layout_ Folder:

- Right-click on the _res_ folder.
- Choose _New_ > _Directory_.
- Name the directory _layout_.

3. Add Your XML File:

- Right-click on the newly created _layout_ folder.
- Choose _New_ > _Layout resource file_.
- Name the file _ar_activity.xml_ and click 'OK'.

Android Studio will open the file in Design mode, but we will need to change into Code mode. In order to do this, click on the Code button (should be in the upper right and looks like 5 vertical lines). Copy the following code into the file:

```xml
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    >

    <io.github.sceneview.ar.ArSceneView
        android:id="@+id/sceneView"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        tools:layout_editor_absoluteX="25dp"
        tools:layout_editor_absoluteY="16dp" />

    <ProgressBar
        android:id="@+id/progressBar"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerInParent="true"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="@+id/sceneView"
        app:layout_constraintTop_toTopOf="@+id/sceneView" />

    <Button
        android:id="@+id/closeButton"
        android:layout_width="48dp"
        android:layout_height="48dp"
        android:layout_alignParentTop="true"
        android:layout_alignParentEnd="true"
        android:layout_marginTop="30dp"
        android:layout_marginEnd="30dp"
        android:text="X"
        android:textColor="#fff"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        tools:layout_editor_absoluteX="365dp" />
</androidx.constraintlayout.widget.ConstraintLayout>
```

If there is a warning / error about a missing package `androidx.constraintlayout.widget.ConstraintLayout´ right click on it, select _Show Context Actions_ and select _Add dependency_. Android Studio will then add an import and sync the project.

The button at the end is so that we can close the screen, when we want to return to the non-ar part of our app. In order for this button to be displayed correctly, we need to

### Update the ARModule file

Now we can reference the created layout file in our module. Open the file _ARModule.kt_ and replace the `showAR` method with the following:
`
```
@ReactMethod
  fun showAR(fileName: String) {
      val assetPath = "file:///android_asset/$fileName"
      val intent = Intent(reactContext, ModelDisplayActivity::class.java)
        intent.putExtra("MODEL_PATH", assetPath)

        // If starting the activity from a non-activity context, set this flag
      intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)

      reactContext.startActivity(intent)
  }
```

We will pass the filename of our model to our native code, which will in turn load it from the assets folder and start a custom [Activity](https://developer.android.com/reference/android/app/Activity) to display the model. In simple terms, an activity represents a single screen with a user interface.

### Create the Activity file

Right click on the _com.rn3dworldexplorer_ folder and select _New_ > _Kotlin Class_. Name the file _ModelDisplayActivity.kt_. Copy the following code into it:

```kotlin
package com.rn3dworldexplorer

import android.os.Bundle
import android.util.Log
import android.view.View
import android.widget.Button
import android.widget.ProgressBar
import android.widget.Toast
import androidx.appcompat.app.AppCompatActivity
import com.google.ar.core.Config
import io.github.sceneview.ar.ArSceneView
import io.github.sceneview.ar.node.ArModelNode
import io.github.sceneview.ar.node.PlacementMode
import io.github.sceneview.math.Position
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch

class ModelDisplayActivity : AppCompatActivity() {
    private lateinit var progressBar: ProgressBar
    private lateinit var closeButton: Button
    private lateinit var sceneView: ArSceneView
    private lateinit var modelNode: ArModelNode

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.ar_activity)

        // Initialize views
        closeButton = findViewById(R.id.closeButton)
        progressBar = findViewById(R.id.progressBar)
        sceneView = findViewById<ArSceneView>(R.id.sceneView).apply {
            this.lightEstimationMode = Config.LightEstimationMode.DISABLED
        }
        sceneView.onArSessionFailed = { exception: Exception ->
            Log.e("ARModule", "${exception.message}")
            // If AR is not available, we add the model directly to the scene for a 3D only usage
            sceneView.addChild(modelNode)
        }

        // Set close button listener
        closeButton.setOnClickListener {
            finish()
        }

        // Load model if path is provided
        intent.getStringExtra("MODEL_PATH")?.let { modelPath ->
            loadModel(modelPath)

        } ?: showMessage("Model path is null")
    }

    private fun loadModel(glbFileLocation: String) {
        Log.d("ARModule", "loadModel path: $glbFileLocation")
        progressBar.visibility = View.VISIBLE

        CoroutineScope(Dispatchers.Main).launch {
            try {
                modelNode = ArModelNode(sceneView.engine, PlacementMode.INSTANT).apply {
                    loadModelGlbAsync(
                        glbFileLocation = glbFileLocation,
                        scaleToUnits = 1f,
                        centerOrigin = Position(0.0f)
                    )
                    {
                        sceneView.planeRenderer.isVisible = true
                        val materialInstance = it.materialInstances[0]
                    }
                }
                sceneView.addChild(modelNode)
                modelNode.anchor()
                sceneView.planeRenderer.isVisible = false
            } catch (e: Exception) {
                showMessage("Error loading model: ${e.message}")
                Log.e("ARModule", "Error occurred: ${e.message}")
            } finally {
                progressBar.visibility = View.GONE
            }
        }
    }

    private fun showMessage(message: String) {
        Toast.makeText(this, message, Toast.LENGTH_LONG).show()
    }
}
```

Here is a brief overview of the file:

#### Properties

- UI Elements: The class defines UI elements like a ProgressBar, a Button (closeButton), and an ArSceneView (for AR content display).
- Model Node: modelNode of type ArModelNode is used to represent the 3D model in AR.

#### onCreate Method

- UI Setup: In the onCreate method (called when the activity is starting), it sets up the user interface from a layout resource (R.layout.ar_activity).
- Initializing Views: Finds and initializes views like the close button, progress bar, and AR scene view.
- AR Session Failure Handling: Sets an error handling mechanism for the AR session. If AR is not available, it directly adds the model to the scene for non-AR 3D viewing.
- Close Button Listener: Implements an action for the close button - when clicked, it finishes (closes) the activity.
- Model Loading: Checks if a model path is provided via the intent; if so, it calls loadModel to load the model.

#### loadModel Method

- Functionality: This method loads the 3D model (GLB format) for display in the AR scene.
- Coroutine: Uses a coroutine for asynchronous execution to avoid blocking the UI thread.
- Model Node Setup: Configures the modelNode with the model path, scale, and origin. Once the model is loaded, it is anchored to a position in the AR scene.
- Visibility Settings: Manages the visibility of the plane renderer and progress bar based on the model loading status.
- Error Handling: Catches exceptions, logs errors, and shows error messages if model loading fails.

#### showMessage Method

Functionality: A utility method to show a toast message on the screen. It's used to provide feedback to the user (like successful model loading or error messages).

### Added the necessary permissions to the Androidmanifest.xml file

Replace the contents of your _AndroidManifest.xml_ with this one:

```xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools">

    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.CAMERA" /> <!-- You may need these if doing any screen recording from within the app -->
    <uses-feature
        android:name="android.hardware.vr.headtracking"
        android:required="true"
        android:version="1" />

    <uses-permission android:name="android.permission.NFC" />
    <uses-permission android:name="android.permission.WAKE_LOCK" /> <!-- Other camera related features -->
    <uses-feature android:name="android.hardware.camera" />
    <uses-feature
        android:name="android.hardware.camera.autofocus"
        android:required="false"
        tools:replace="required" />
    <uses-feature android:name="android.hardware.camera.ar" /> <!-- Specifying OpenGL version or requirements -->
    <uses-feature
        android:glEsVersion="0x00030000"
        android:required="false"
        tools:node="remove"
        tools:replace="required" /> <!-- Usage of accelerometer and gyroscope -->
    <uses-feature
        android:name="android.hardware.sensor.accelerometer"
        android:required="false"
        tools:replace="required" />
    <uses-feature
        android:name="android.hardware.sensor.gyroscope"
        android:required="false"
        tools:replace="required" />

    <application
      android:name=".MainApplication"
      android:label="@string/app_name"
      android:icon="@mipmap/ic_launcher"
      android:roundIcon="@mipmap/ic_launcher_round"
      android:allowBackup="false"
      android:theme="@style/AppTheme">
      <activity
        android:name=".MainActivity"
        android:label="@string/app_name"
        android:configChanges="keyboard|keyboardHidden|orientation|screenLayout|screenSize|smallestScreenSize|uiMode"
        android:launchMode="singleTask"
        android:windowSoftInputMode="adjustResize"
        android:exported="true">
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
      </activity>
        <activity android:name=".ModelDisplayActivity" />

        <meta-data
            android:name="com.google.ar.core"
            android:value="optional"
            tools:replace="android:value" />
    </application>
</manifest>
```

We request several device features needed for _AR_ like the gyroscope as well as register our custom activity via this line: `<activity android:name=".ModelDisplayActivity" />`. Not all of them may be used in the end, so it is your responsibility to choose carefully. This list just aims to be as comprehensive as possible out of convenience.

### Adjust our JavaScript React code

We are finally back in the cozy surroundings of your _JavaScript_ code. All that is left is to adjust our _ARModule.tsx_ and our _App.tsx_. For the _ARModule_ file, adjust your `showAR` function to this `showAR(path: string): Promise<void>;` to reflect the new type signature. After that, in the _App.tsx_ you can adjust the onPress function of the button to pass the name of your chosen model. For me it is like this:
`onPress={async () => await ARModule.showAR("AR-model.glb")}`.

Now just press the button and it should look similar to this:

![3D Sphere displayed on mobile phone](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lj60nxxp4ey45g1twukm.jpg)

## Conclusion
As we conclude this second part of our series, we have navigated through the integration of _ARCore_ in a _React Native_ environment via _Kotlin_. It can be pretty overwhelming if you haven't delved too deep into native code integration, but hopefully, it was manageable. From here on out, you can expand on the basic functionality and create your own features.

Moving forward, we will be doing the setup for native modules using _Objective C_ in _iOS_ in the next entry.