Help - Sidekick App

Main -> **Help - Sidekick App**

## Introduction

The LUXOR Sidekick IOS facecapture app which is free for beta testers is available on Apple Testflight. Anyone who would like to Testflight the Sidekick app please message me with an email that I can send an invite so you may download the app from Testflight.

Using iPhone's Truedepth sensor based facial motion capture app for the Sidekick was developed to stream ARKit's blendshape weights values (and now AR head tracker) as a high speed UDP binary stream and update the facial blendshapes of the avatar in Luxor in real-time and as exportable animations. When adding a FacecapLinker.cs component to custom avatars the APS\_SDK allows mapping to the avatar's face blendshapes. Luxor is compatible with all 52 ARKit convention blendshapes (also includes AR gaze tracking): https://youtu.be/5-CeXhZHb1E

It's also possible to utilize avatars that don't include ARKit convention blendshapes. For example if adding facecap to standard VRChat avatars it's still possible to partially map the FacecapLinker.cs even to just a few basic blendshapes: https://streamable.com/kmo3tg This also shows the new AR head tracking feature available in the latest Sidekick v1.2.1, for adding full AR localized headtracking to avatars (without needing to wear a head tracker)

The Sidekick IOS app uses ARKit and streams blendshapes values for adding realtime facecap on avatars in Luxor if the avatar had a FacecapLinker component added when building it using the APS\_SDK. This app currently provides facecap streaming and SLAM capabilities and many additional features are still in development!

Facecap Streamer: A very basic interface allows configuring a destination address (running APS game) for streaming (animoji) facecap data over WIFI to update a real-time 3D avatar's facial expressions (mapped blendshapes). This allows animators, storytellers, Vtuber, live streamers and many other creators to quickly and simultaneously record real-time full body motions capture plus natural facial expressions all synced to a player's own custom 3d characters.

SLAM scanning tool: Allows artists to create point-cloud data using Apple's TrueDepth depth camera and visualize point-clouds directly inside of the APS viewport. Artists may export their point-clouds as binary Stanford .ply polygon files. Point-clouds can be used for reference points when sculpting or modeling in external 3D editors.

## Steps for getting Sidekick app

### Installing

Now where do I get the Sidekick app and how do I set it up?

First download testflight on the IPhone, then visit this page and click the download link from the iphone: https://www.mocapfusion.com/face-capture

Be sure the IP entered in the Sidekick's IP Address field matches the IP set in Luxor's facecap settings (if you have multiple Ethernet/Wifi then the IP selected in Luxor's settings must be one that is on the same network and reachable by the IPhone):

### Overall setup steps

I need some help running it. First I launched Luxor v3\_1\_5 and then launched APS\_Sidekick app on my iphone X. The app has an IP address and port number. The IP address is different than the one on the Luxor VR menu so I changed the IP address on the Sideckick app and pressed the START button. I checked the yellow "Enable Facecap" box on the Luxor VR menu. My character isn't responding. Did I miss a step?

After launching Luxor, open the "Face Cap" tab from the main menu and there you should see two fields showing the receiver IP and Port - set the destination IP and port in the Sidekick app to match those values.

Avatars must have a FacecapLinker.cs script attached for facecap to work (when building them in the APS\_SDK). This maps facial blendshapes of the avatar to AR kit's standard face cap shapes name. I was able to add facecap mappings for many of the default avatars (supermodels, large\_lady, warrior, firefighters and trainers models should all have default mappings). Which avatar were you testing?

Many of Luxor's default avatars also have VisemeLinker mappings - which can add extra realism to facecap if you also enable the mic in APS while using facecap. The FacecapLinker should be compatible with any avatar so long as the avatar includes some basic facial blendshapes. The facial blendshapes mappings are very optional and you don't need to map them all (however more is typically better). And the FacecapLinker.cs script is attached to the avatar when building it in Unity, here's a video that shows adding the FacecapLinker component to an avatar: https://youtu.be/oEwkhIr4ffw?t=326 (edited)

Also here's Apples's list of blendshapes supported by ARKit (most are optional): https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation#overview

Here are the default mappings for one of the Reallusion CC3 avatars. Since all CC3 avatars are supported and have standard facial blendshaps you can easily press the "guess links" button to apply the default map. However note that not all mapping items are set, and CC3 does not include all of the same blendshapes used by ARKit - however the missing blendshapes could be created manually in Blender before importing to Unity or adding a FacecapLinker.cs:

### Connecting with APS

Ensure firewall rules allow for Luxor to receive inbound connection from the Sidekick. Windows needs to allow inbound UDP connection for Luxor.

The port 9000 should be set on both sides, then enable the Sidekick App checkbox:

### Adjusting blendshapes

Adjust your avatar blendshapes to match the expression ranges you want, Go to Desktop - BlendShapes.

When you refer to the EE, Oh and CH shapes those are Visemes and are part of the lipsync which requires the microphone. The microphone lipsync can be used simultaneously with facecap to add even more depth.

## About AR head tracking

AR head tracking is actually a full blown head tracker feature that uses the Iphone depth sensor instead of needing to wear any VR trackers. It sends the user's head rotation and translation to APS (as well as facecap). It's still a work in progress but I think is just as responsive as a vive tracker..

## About Head tracking feature in Sidekick app

01/10/2021
i tested out the app again for those buttons that werent working, and i made sure that i updated my phone, the aps sidekick app and luxor and yet when i press the turn buttons for example luxor sort of makes a dinging sound but doesnt turn the model or anything. all of the arrow/avatar control buttons seem to do this. I also tried out the head tracking feature, but the facecapture would work but not the head tracking.

The AR head tracking feature works when the iphone is used as a VR selfie-cam, and needs a tracker attached for localization. I will try to create a better tutorial how to setup a VR selfie-cam, but basically just attach a tracker to the back of the iphone with charger port facing up:

Then in Trackers Setup panel select "Camera" for that tracker from the dropdown:

"oh but isn't that the same as just having a head tracker?"

It requires a tracker to be used on the phone, but the user doesn't need to wear a head tracker.

"I'm confused so it's called a head tracker but in fact it's a camera tracker?"

It is a head tracker (and optionally works as a VR selfie cam too) for using facecap. The idea is that the phone uses the tracker, mainly for convenience while live streaming. It definitely would be advantageous if the head tracker were worn by the user. But this allows for setting the phone on a desk and the AR head tracker stays aligned with full body tracking.

This also works even if the user is holding the iPhone and walking around the playspace because a vive tracker is being used as reference and the algorithm remains synchronized in 3d so that AR headtracking along with full body tracking remain aligned no matter where the phone is placed or is moving.

It's a bit to explain but basically the projection point for AR head tracking must be localized in world space. I tried using the iPhone's AR world tracking but it was extremely high latency and low accuracy (also uses a different coordinate system and origin from steamVR) this caused offsets and shakiness in the final (composite space).

So to avoid tons of complexity and to achieve a tighter tolerance in the final computed position I used the already working VR selfie-cam feature as a localized projection point for determining the real-world position of the user's local AR head position as projected out into Unity's world space.

Each phone may have a slightly different alignment but most are typically fine. If the tracker is mounted to the phone's case facing in the same direction relative to the phone's Truedepth sensor as shown in this picture. I use an iphone XR and otterbox defender case and was able to secure the tracker to the case using the clip and held with a rubber band (double sided tape probably works too..):
https://www.amazon.com/OtterBox-Defender-Case-iPhone-Packaging/dp/B07GBHRJY9

Set the iphone tracker to "camera". Then load the offsets, the camera should snap into place. Then enable AR head tracking from the Sidekick app and the avatar's AR head tracking should then be aligned with full body tracking.