Getting Started with RealityKit: Models
Tags: tutorial, ios, realitykit, swift
Published: 2021-02-01 09:48:34

There are two main ways of adding models to your RealityKit scene, importing from a USD (usd, usda, usdc, or usdz) or a Reality file, or creating a mesh using the MeshResource generators provided with Xcode. There are a few shapes that can be generated via the MeshResource generators, other shape must be imported via a USDZ model or experience file from Reality Composer.

This post also includes examples of how to import USD files from remote URLs, and customise imported models.

- [Importing from USDZ](#importing-from-usdz)
  * [Loading with Entity Hierarchy](#loading-with-entity-hierarchy)
  * [Loading an AnchorEntity](#loading-an-anchorentity)
  * [Loading a BodyTrackedEntity](#loading-a-bodytrackedentity)
  * [Loading a Flattened Model](#loading-a-flattened-model)
  * [Loading a Remote USDZ File](#loading-a-remote-usdz-file)
- [Generating Shapes](#generating-shapes)
- [Customising Models](#customising-models)
  * [With Hierarchy](#with-hierarchy)
  * [Without Hierarchy](#without-hierarchy)


# Importing from USDZ <a name="importing-from-usdz"></a>
One of the more popular and exciting methods of importing a model to RealityKit would be importing it from a USD file. There are several ways to do this, and not knowing the subtle differences can lead you to miss parts of the imported models such as animations.

All the import methods either require a name and an optional bundle (omit the bundle to use the app's main bundle), or require a url and a unique resource name. To see how to use each see [this link](https://developer.apple.com/documentation/realitykit/entity/3244083-load) for passing a name and bundle, and [this link](https://developer.apple.com/documentation/realitykit/entity/3244082-load) for the url and resource name methods. The rest of this post will only discuss the name + bundle examples for simplicity.

## Loading with Entity Hierarchy <a name="loading-with-entity-hierarchy"></a>
To maintain information bundled in the USD file such as animations, the best import methods are `Entity.load` and `Entity.loadAsync`. The difference between these two is that `load` will block the thread it is called on until the file has been imported, while `loadAsync` will use a [`LoadRequest`](https://developer.apple.com/documentation/realitykit/loadrequest) (where the Output is an Entity) to get the model imported while allowing you to continue on the current thread. 
Another difference between the two, is that `load` can throw an error, while the error found in `loadAsync` can be caught in the `LoadRequest` slightly differently.

### Examples:

Simple loading of a USDZ file named `toy_robot_vintage.usdz` synchronously:
```swift
guard let entity = try? Entity.load(
  named: "toy_robot_vintage"
) else {
  // failed to load entity
  return
}
// do something with entity
```

Loading a USDZ file named `toy_robot_vintage.usdz` asynchronously:
```swift
_ = Entity.loadAsync(named: "toy_robot_vintage").sink(
  receiveCompletion: { loadCompletion in
    // Added this switch just as an example
    switch loadCompletion {
      case .failure(let loadErr):
        print(loadErr)
      case .finished:
        print("entity loaded without errors")
    }
  }, receiveValue: { entity in
    // do something with entity
  }
)
```

Loading multiple USDZ files asynchronously, where the parameter in receiveValue is of type `[Entity]`, and will only reach there once all models have been imported:
```swift
_ = Entity.loadAsync(named: "toy_robot_vintage")
  .append(Entity.loadAsync(named: "gramophone"))   
  .append(Entity.loadAsync(named: "fender_stratocaster"))
  .collect()
  .sink(receiveCompletion: { loadCompletion in
    // Add switch case for loadCompletion
  }, receiveValue: { entities in
    // Do something with returned entities array.
  }
)
```
> Note: You may notice that there is a small freeze  and when you add the entity to your scene, even when using loadAsync. The freeze is due to large meshes or images stored in the materials being loaded into memory, which must be executed on the main thread.


## Loading an AnchorEntity <a name="loading-an-anchorentity"></a>
In a very similar way to `load` and `loadAsync`, [`loadAnchor`](https://developer.apple.com/documentation/realitykit/entity/3244085-loadanchor) can load your USD file in as an `AnchorEntity`. This lets you place it directly in your scene without needing to make a separate anchor for it. Any animations in your USDZ file will also be available, as with `load` and `loadAsync`.
Example:
```swift
guard let entity = try? Entity.loadAnchor(
  named: "toy_robot_vintage"
) else {
  // failed to load entity
  return
}
entity.anchoring = AnchoringComponent(
  .plane(
    .horizontal, classification: .floor, minimumBounds: [1, 1]
  )
)
arview.scene.addAnchor(entity)
```

In this example, the AnchoringComponent is set to place the contents of the USDZ file to a horizontal plane categorised as the floor with a size of at least 1m x 1m. Only loadAnchor is shown here, but [`loadAnchorAsync`](https://developer.apple.com/documentation/realitykit/entity/3244087-loadanchorasync) works the same way as `loadAsync`, it will just return an AnchorEntity instead of an Entity.

## Load a BodyTrackedEntity <a name="loading-a-bodytrackedentity"></a>
When you want to import a model that is rigged up to be animated by a real person, you instead want to use [`loadBodyTracked`](https://developer.apple.com/documentation/realitykit/entity/3255555-loadbodytracked) or [`loadBodyTrackedAsync`](https://developer.apple.com/documentation/realitykit/entity/3255557-loadbodytrackedasync) to import a BodyTrackedEntity.
```swift
guard let entity = try? Entity.loadBodyTracked(
  named: "robot"
) else {
  // failed to load entity
  return
}
// do something with body tracked entity
```

For more information on tracking human bodies in AR, this session from WWDC 2019 is a great resource:

https://developer.apple.com/videos/play/wwdc2019/607

## Loading a Flattened Model <a name="loading-an-flattened-model"></a>
When you just want to import a Model from a USDZ file, removing entity hierarchy as well as animations, you can import using [`loadModel`](https://developer.apple.com/documentation/realitykit/entity/3244091-loadmodel) or [`loadModelAsync`](https://developer.apple.com/documentation/realitykit/entity/3244093-loadmodelasync). This will turn your 3d model of what would be various levels of nodes and children into a single Entity containing meshes and an array of all the materials.

In my opinion, this difference can be confusing to someone who is new to RealityKit and wanting to import their model that has an animation from a USDZ file; as seen by various questions on Stack Overflow asking what happened to their USDZ animations when importing a file using these methods. However, once aware of this difference, it is very useful to not have all the overhead of a large hierarchy of entities that you may not need.

## Loading a Remote USDZ File <a name="loading-a-remote-usdz-file"></a>
Currently there is no straightforward or builtin way to import USDZ files from remote URLs, but thankfully there are a few solutions, here's one from StackOverflow:

{% stackoverflow 59242131 %}

I've also altered the solution into a synchronous function with an example usage at the bottom in this gist:

{% gist https://gist.github.com/maxxfrazer/8aa6952f52236d23dea88d009414d337 %}

# Generating Shapes  <a name="generating-shapes"></a>
In its current state, the provided generators include Box (cuboid), Plane, Sphere or Text. Ironically, out of the three shapes in the RealityKit logo, only one of those (a sphere) can be produced without importing USDZ files.

![Objects that can be made with RealityKit](https://dev-to-uploads.s3.amazonaws.com/i/f2nq2e25j2wsspi29xn4.jpg)

All the mesh initialisers can be found on this page:

https://developer.apple.com/documentation/realitykit/meshresource

The easiest way to add one of these shapes to your RealityKit scene would be to use the ModelEntity initialiser:
```swift
let newModel = ModelEntity(mesh: .generateBox(size: 1))
```

![cube_without_material](https://dev-to-uploads.s3.amazonaws.com/i/3cqu2ew14lg165vclfx3.png)

Notice here, no material was specified in the ModelEntity initialiser, which means this broken shader sort of look is applied to the cube shape. This is the case for all the other MeshResource initialisers.

For in-depth information on how to create materials in RealityKit and a few tricks with them, see my previous post:

{% medium https://maxxfrazer.medium.com/getting-started-with-realitykit-materials-f97ab406ac61 %}

# Customising Models <a name="customising-models"></a>

Once you bring a USDZ file in your scene there may be some customisation you want to do to it, such as changing a material, or actually moving around one part of the model.

If you wanted to change the material of a model imported from a USDZ file, there are a couple of ways you can do so.

## With Hierarchy <a name="with-hierarchy"></a>
If you import your model with anything other than `loadModel` or `loadModelAsync` you can replace the material of your choosing with ease.
Take this file, `gramophone.usdz`:

![xcode_gram](https://dev-to-uploads.s3.amazonaws.com/i/thpl11y20teo4tkpt43b.png)

In the above image I've selected the record in the middle, and on the right we can see its name is `GramophoneRecord1`, and if we select the materials tab we can see that it only has one material named `pxrUsdPreviewSurface1SG`:

![xcode_gram2](https://dev-to-uploads.s3.amazonaws.com/i/fywgpgsgm57hgkakppj6.png)

> There is actually only one material in this whole USDZ file, however different parts of it are used on different meshes in the file.

Next all we need to do is find the Entity named `GramophoneRecord1` and replace its material with our own green one.

```swift
if let gramRecord1 = gramophoneModel.findEntity(
  named: "GramophoneRecord1"
) as! HasModel {
  gramRecord1.materials = SimpleMaterial(
    color: .green, isMetallic: true
  )
}
```

![gram_green](https://dev-to-uploads.s3.amazonaws.com/i/ywrshhe3fpwupjvww97w.jpg)

> Original on the left, altered material on the right. 

## Without Hierarchy <a name="without-hierarchy"></a>

If you imported your model using `loadModel` or `loadModelAsync`, then as discussed earlier there is no hierarchy of entities, just the one ModelEntity consisting of a ModelComponent with a single mesh and a collection of materials. Take this file, `fender_stratocaster.usdz`:![img](https://cdn-images-1.medium.com/max/800/1*Iu3vFEZRiLCzjHJ0mHbgOA.png)

The USDZ file itself consists of two meshes one for the stand and one for the guitar. Each of those meshes contain materials, the stand has one, and the guitar actually has 19, so once imported with `loadModel` the ModelEntity will have one mesh and a list of 20 materials.

Let’s replace the guitar body with a generated SimpleMaterial. It could take a bit of guess work to figure out which one in those 20 is that material, but by clicking on the part of the model that has the material you want to change Xcode highlights it in the side bar, see that it is the very first material in the guitar’s list of materials. It isn’t incredibly clear, but the first material when imported with `loadModel` is actually that of the guitar stand, leaving the guitar body material the second in the list, index 1.

After importing the model, simply running this I was able to change the guitar body to an electric blue:

```swift
modelAdjusted.model?.materials[1] = SimpleMaterial(
  color: .blue, isMetallic: true
)
```

![img](https://cdn-images-1.medium.com/max/1200/1*ooqvj5sIth4xVZG9Nf9Wwg.jpeg)
> Original on the left, altered material on the right

The version without hierarchy leads to a bit less code, but doesn't allow for embedded animations, and can lead to issues if our USDZ file ever changes such that the material order differs. Whereas the version with hierarchy lets us find a specific part of the model, which would be preferred with a more complex model.

I hope you found something useful in this post, give it a like if you did! 👏👏👏

Send me a message on [twitter](https://twitter.com/maxxfrazer) or leave a comment if there’s something I’ve missed out or if there’s something that isn’t working for you in RealityKit.