SDK - Importing Custom Avatars

### Getting the model ready for Unity

Avatars rigs must be compatible with Unity's Mecanim humanoid rigging system. Optional bones, such as toes, clavicles, eyes and jaw are not required by Mocap Fusion VR.

Ideally, avatars should be fairly similar in proportion to the individual who will be performing the motion capture. For example, an avatar with arms twice the length of its legs may perform oddly during motion captures.

**Apply All Transforms** in blender before exporting.

Export the avatar as an **.fbx** or a **.blend** file and place it, alongside the required textures, inside your Unity project **Assets** folder.

### Set up the humanoid rig

1. In Unity, select the model file in the **Project** panel
2. In the **inspector** panel, select the **Rig** tab
3. Change **Animation Type** to **Humanoid**
4. If errors appear, click **Configure** and ensure the bones are properly assigned
	1. The **Pose > Reset** and **Pose > Enforce T-Pose** options may be useful

### Build the Avatar

1. Drag the model file from the **Project** panel and drag it into the **Hierarchy** panel
2. Select the model in the **Hierarchy** panel
3. In the **Inspector** panel, click **Add Component** and add the **APS\_SDK** script (you may use the search bar)
4. Click **Build Avatar From FBX** (it may have a slightly different title)

The avatar will be built at this point.

### Transferring the Built Avatar to Mocap Fusion VR

The avatar files are exported into the unity project, so there is one last step you must do every time you build or rebuild an avatar.

1. Copy/Paste the avatar folder into the Mocap Fusion VR's **VR\_MocapAssets** folder, located at `%USERPROFILE%\AppData\LocalLow\Animation Prep Studios\LUXOR\VR_MocapAssets\`

The **.meta** files can be ignored.

## Optional Avatar Features

*Please note that you still have to **Re-build** and copy your avatar into the VR\_MocapAssets folder after changing it.*

### Set the View Position

Click **Add HMD Eye Target Helper** and place it where your headset's screen should be. Non-standard characters, such as any with large heads or abnormal eye placement may especially benefit from an added helper placed where standard human eyes would otherwise be.

Adding the HMD Eye Target Helper allows specifying the desired eye height and head location after T-Pose calibration. If no HMD Eye Target Helper was added, then T-Pose calibration will default to the center of the two eye bones. If no eye bones are present, then T-Pose calibration will default to the head bone, possibly making the avatar much taller than intended, resulting in unwanted mis-calibration.

[TODO- good screenshot of a big-head avatar with appropriate eye target placement]

### Add Materials

You may add materials and textures to your avatar in the scene view if they are not already set up. These materials will be seen in-app, but will not transfer to blender exports.

Custom shaders are supported.

### Add Visemes for Lipsync

You can animate a characters mouth during a capture by automatically moving its mouth based on the performers microphone input, if the **Lipsync** option is enabled. The avatar must be equipped with the appropriate blendshapes (also known as shapekeys), in order to morph the avatar's mouth to the various mouth-sounds (e.g. Ah, Oh, Ee)

1. In the **Armature Linker** script, ensure that the **Face Renderer** is set to your avatar's face
2. Add the **Visemes Linker** script to your avatar's root object (the same as the Armature Linker)
3. Click **Guess Links** or manually assign each blendshape

*You can save time by adding the prefix to your naming convention. For example, the prefix for visemes generated by the CATS blender plugin are `vrc.v_`*

### Add Face-Tracking Support

Face-tracking hardware is supported, but the avatar must be equipped with the appropriate blendshapes (also known as shapekeys), in order to morph the avatars face into smiles, frowns, etc.

1. In the **Armature Linker** script, ensure that the **Face Renderer** is set to your avatar's face
2. Add the **Facecap Linker** script to your avatar's root object (the same as the Armature Linker)
3. Click **Guess Links** or manually assign each blendshape

See arkit-face-blendshapes.com for a reference of how each blendshape should look like.

### Animation Builder

The **Animation Builder** allows you to create pre-made animations that can be cued in-app during a recording. These are in addition to whatever in-app capture occurs.

This can be used to spawn an object at an avatar's limb, play sound or particle effects, or make motions a performer isn't capable of. Skeletal animations will override the performers movements in-app, for the duration of the animation.

*Note: To make blendshape toggles, use instead the **Emotion Builder***

#### Set up custom animations on your avatar

1. Select the **animation** object (It is a child of your avatar that should have been added the SDK on the first build)
2. Click **Add New Animation**
3. Configure the new **Animation Builder** component
4. Repeat steps 2 and 3 as necessary
5. When all animations are added, click **Generate Controller** on the **Animations Builder Base**

#### Additional Animation Settings

**Follow Type** determines whether the character's location will be set by the animation. Set it to **Animation Follows Player** if you want the character to move normally while being animated. *This is irrelevant if Lower body weight is set to 0. [TODO- factcheck this]*

**Animation End** determines whether the character freezes in the last frame pose after the animation ends, such as for death animations. **Resume to player** instead returns control of the character back to the mocap performer.

**Blend Curve** represents the transition between mocap control of the character to the animation's control of the character. The curve can be edited to be more smooth or sharp to make the transition more smooth or sharp. Too sharp of a transition can lead to weird looking results, such as snapping from one position to the next faster than humanly possible.

**Weights** allow you to selectively animate certain sections of the body. This allows for motion capture to continue while playing an overlapping animation, for example for custom running animations.

**Optional Audio Clip** will play when the animation starts, or after **Start Delay** seconds. **Bypass Effects** determines whether scene sound effect filters will impact this audio or not.

**Optional Spawn GameObject** allows you to select a prefab to be spawned at the **Spawn Point**, at **Start Delay** seconds after the animation starts, and to be removed from the scene after **Lifetime Seconds.** This is useful for particle effects, physics objects, etc.

### Emotion Builder

The **Emotion Builder** allows you to setup blendshapes to be independently enabled and toggled in-app using hotkeys. This is in addition to face tracking and lipsync blendshapes. This is useful if you don't have a face tracker, or if you want additional expressions or toggleable props.

#### Set up emotions on your avatar

1. Ensure that the **Body** mesh is set in your **Armature Linker**
2. Select the **emotion** object (It is a child of your avatar that should have been added the SDK on the first build)
3. *(Optional) Set the Blink (L) Shape and Blink (R) Shape according to their blendshape indexes and then click **Generate Blink Clips***
	1. See the official tutorial on how to find the blink indexes
4. *(Optional) Move the **PortraitCamera** to your avatar's face to create preview emotions*
5. Add a new **Emotion Builder** component to the **emotion** object
6. Expand **Blendshape Weights** and set them to your intended settings.
7. *(Optional) Add a **sound effect** to be played when the emotion is enabled*
8. *(Optional) Enable **Ignore Blinking** if you would like automatic blinking to be disabled while this emotion is enabled*
9. Click **Save Changes**
10. Repeat steps 5-11 for each emotion *Note: There are unused options that don't seem to affect in-app behaviour, and can be ignored: **Keyboard Binding**, **On Timer***

## Tutorials

https://youtu.be/mlAgW8KlUtk

https://youtu.be/7pmvmmwTu2c

https://www.youtube.com/watch?v=oEwkhIr4ffw

https://www.youtube.com/watch?v=bDVLjqRBl88

## Troubleshooting

### The avatar disappears when I click Build Avatar from FBX

The APS SDK scales avatars to a standard size (less than 2 meters, or 2 unity units). If your avatar's fbx is relatively gigantic, it will appear to disappear when scaled down. Press **F** in the 3D window of unity to focus and zoom on the currently selected object.

### The avatar works, but animation exports fail

Ensure all the required bones are mapped, including all finger bones

### The avatar does not have a thumbnail in-app

Something went wrong with the avatar building process. Re-build the avatar in Unity and look for errors in the Unity console.

### I restarted Unity and the avatar is missing

Check in the **APS\_Prefabs** folder and drag that avatar prefab into any scene. You can save the scene or just use the prefab again. Each time you build the avatar, the prefab is saved.

### The textures are lower quality in-app than in blender

By default, Unity downscales texture resolution to 2048x2048. This can be changed in the Texture import settings. See the Unity manual on texture importing

### Actions cause the avatar to move far from its original position

In Unity, select the **.anim** file in the project panel and set **Based Upon (at start)** to **Center of Mass** under **Root Transform Position (XZ)**

**Build Avatar from FBX fails, leaving a (Clone) avatar and a broken Armature Builder**

Ensure that the FBX's rig import settings do not have any errors. You may need to configure the humanoid rig and use **Pose** -> **Enforce T-pose**