Draft - Tutorials - Building Custom Scenes for APS

Easiest way I can think would be to first create a scene/world in Blender (to scale!!) then save and export the entire scene as a .fbx to Unity, then simply import the .fbx and create a scene in Unity (using identical same model, origin and scaling) to create a mocap environment to be used as "reference mesh" for creating mocap around or as a guide for avoiding obstacles or to aid in the aiming of cameras for creating camera motion profiles, etc.. Then save the Unity scene and after importing the APS\_SDK use it to build the Unity scene to a luxor asset by clicking the Build Current Scene button located on the Scene Builder panel (be sure the Unity scene is open in the viewport!)

Once you have the luxor scene asset you could use it as "reference" for creating motion capture or interacting with the scene as reference meshes for determining where to stand, where to walk to while avoiding obstacles, where to point cameras, etc.. then after exporting the mocap data back into Blender (and using the same scene again) everything should line up properly ready for final rendering!

Check out this short I created entirely in luxor all together as a single scene and exported to Blender (and rendered in EEVEE). I created a drum set room in Blender and a "reference" drum set scene in Unity that I built as a scene asset for luxor. This allowed me to align my avatar to the seat prior to recording the mocap. I could see my avatar's hands and drumstick in VR to make it appear that I was hitting the drums. Also I recorded the multiple cameras all in the same environment together while seeing the avatars, the camera mocap was recorded just after creating the avatar mocap.Took roughly 15 minutes to record all of the elements. But no second takes and no post processing for on any keyframes: https://youtu.be/IpTYaUFLvTw

To create a scene/world for APS it's recommended to design the scene in Blender (to scale!!) then save and export the entire scene as a .fbx to Unity, then simply import the .fbx in Unity (keeping origin and scale) to be used as "reference mesh" for creating mocap around or as a guide for avoiding obstacles or to aid in the aiming of cameras for creating camera motion profiles, or just go all out and add beautiful materials, animations, particles etc. to the environment for use in live productions/streaming.

Then save the Unity scene and give it a name. Finally if you've already imported the APS\_SDK into Unity you can create a new scene asset for APS by clicking the Build Current Scene button located in the Scene Builder menu (be sure the Unity scene is opened and visible in the viewport before clicking the button!). After that you should see a new .scene file that you can copy to the VR\_MocapAssets folder.

Most Unity scenes can be built as a mocap asset using the Scene Builder from the APS\_SDK menu. Create a new scene in Unity and in the Scene Builder click "build current scene". A new assetbundle should be created in the scenes output folder which can be copied to the VR\_MocapAssets folder.

Also the SteamVR library is optional - however it's recommended to add SteamVR into the project when building scenes to allow adding the "LuxorPlayer" object. the luxor player can be used for checking the scene in the Editor (by pressing play) in VR to ensure it will look correct in luxor. Also the luxor player works as a spawn point for the mocap player. And the luxor player's VR camera has a default post processing profile applied which can be adjusted for adding bloom and vignette to the mocap (or live stream) scene.

## Notes about interaction

Would interactive things we program into our scenes work after exporting to Luxor/MF? Like objects a person can pick up or moving lights or the like? If so, can we make more complicated things like button presses which trigger game logic like a wheel of fortune for example?

Possibly some of the elements from the Unity SteamVR plugin interaction system could be added to custom scenes. I haven't tested this much but luxor would not remove any of those components from scene objects, and maybe they could be used to add intractable objects into custom scenes. Check out the example scenes found in SteamVR plugin's interaction system example scenes folder.

Currently there are a few Sandbox scripts already available in the APS\_SDK that can be used for adding light switches, or for triggering animations and explosions. However i would really like to focus on improving the Sandbox and add more features/examples at some point! Here are a few examples found in the **Studio** area that show some of the current Sandbox features: https://youtu.be/CFcPStN-ZxA

## Notes about lightning for scenes in Unity

Has anyone tried making custom scenes yet? how do we use baked lights in it?
Baked lightmaps can be added to scenes, check out the Treadmill scene,(Treadmill Room scene - credit: Muhtesemozcinar https://blendswap.com/blend/12781 )where I baked the scene using the Enlighten lightmapper. Also reflection and light probes should be supported.

"ok cool. is there a tutorial you would recommend? I'm not as great at reverse engineering as id like to be. also btw i just tested the camera control with the ps4 controller and it works wonderfully! thanks for implementing it."

I created a video showing some steps for backing lightmaps and adding lightprobes to a scene:

The video was created as a tutorial for a Unity plugin I was developing but is no longer being maintained, but I think the tutorial still has some useful info..

## About scene models that contain animations

Can that scene contain models that play animations?

Sure, just about anything that is supported in a Unity scene should work in Luxor including sounds, animations, particle systems, skyboxes, animated shaders like water and post processing effects. here's a scene I created that shows adding animations and synchronized sound: https://drive.google.com/file/d/1eDI7GXy0ygZdaBSdi4AFKGorkCFnP-CT

## Avatar spawn location on the scene

How do I specify the spawn location for a scene? Is there a prefab or sorts or some form of script?

If you are using the scenebuilder in the APS\_SDK before clicking "build current scene" you can click "Add VR Player" to as a player object to the scene which will act as a spawn point.

Also the SteamVR library is optional - however it's recommended to add SteamVR into the project when building scenes to allow adding the "LuxorPlayer" object. the luxor player can be used for checking the scene in the Editor (by pressing play) in VR to ensure it will look correct in luxor. Also the luxor player works as a spawn point for the mocap player. And the luxor player's VR camera has a default post processing profile applied which can be adjusted for adding bloom and vignette to the mocap (or live stream) scene.

## About other animations in the scene

"I also want to be able to import animations with that scene, Ideally I want to use the camera to film the action within the scene(then export the camera data)."

It should be possible to include native Unity animations with the Unity scenes and use those as guides for recording camera mocap. All animations in the scene should already sync to the start of the mocap recording in luxor, this allows adding sequential animations such as this Falcon heavy launch scene example created with sequential animation and sound effects. I included motion captured avatars and then added a VR camcorder and filmed the launch (to record the the camera's zoom/fov and motion), I did not render this video in Blender, but was able to export the camera motion capture: https://youtu.be/XkzzgFZiAuU

## More notes on scenes

Add the scene.blend to a new Unity project, place the .blend into the Unity scene, be carful the model is not moved from it's default location, save the Unity scene as ConcertHall, then from the APS\_SDK select Scenes>Scene Builder and click the build scene button, once it has completed you can copy the ConcertHall.scene file to VR\_MocapAssets folder, then also copy the original scene .blend and name it as ConcertHall.blend and place it next to the ConcertHall.scene. Then in Luxor load the scene, record mocap and export, the output folder should then contain a copy of the Blend scene and the sceneloader should import all models and the scene model.

## Is there a video on how to export scenes in luxor?

"Is there a video on how to export scenes in luxor? I'm not that experienced when it comes to scene making in unity so i thought to parent the scene together and add the aps sdk component to the parent object, set it to scene and export, but nothing happens when i press the button" I'm guessing that's not the proper way to do it"

Are you using the scenes in Unity only as reference (eg. doorways, table tops or other static objects)? If so then the Unity scene can be simplified to just a few primitives (cubes and quads) which are only needed (during mocap) to indicate position of objects that will exist in the final render scene. And the render scene is the actual .blend file that is imported when running by the SceneLoader. However it is sometimes nice to 'decorate' the Unity scene so that it appears has resemblance (during mocap) to the final render scene - but is not necessary!

Alternatively you could place your scene blend file(naming not important) directly into the folder next to the mocap data (next to the SceneLoader.blend), then open your pre-created scene, create a new txt window, and copy/paste the sceneloader.py script into the scene and press run to import to avatars into the pre-created scene. I think using this method is convenient when wanting to preserves the entire original scene (aka. merely adding the avatars to it) and will keep all settings like world textures and skyboxes and any previous mocap or animations untouched. Here's the sceneloader.py script from SceneLoader.blend

(provided link in google drive, but script no longer available)

If you are creating high quality scenes for use in both Unity (aka. real-time scene) and Blender (final rendering scene) and want both to match exactly then you may create the final scene in Blender, then copy the .blend to your asset builder Unity project, create a new Unity scene, Add the .blend scene to the new Unity scene (ensure no to offset the scene model in the unity scene so that the model position in Unity agrees with the model position in blender), finally, select the SceneBuilder from the APS\_SDK menu and press Build Scene to create the .scene asset for LUXOR.

When copying the .scene asset to the VR\_MocapAssets folder be sure to include the original .blend file next to the .scene file and that they share the same name. For example here are some .scenes files that have accompanying .blend files, and when exporting from LUXOR the .blend will be copied to the output folder and renamed to scene.blend:

## SDK notes

There's some documentation about building scenes located at the legacy CustomSceneBuilder on github. It's mostly still relevant however the only major change when using the newer SceneAssetBuilder from the latest All-in-One APS\_SDK is that users no longer need to set the Asset Label manually and may skip that step: https://github.com/guiglass/CustomSceneBuilder -- I created a simple scene in Unity and built it to LUXOR. Here's a screenshot of the output:

## Notes about post processing

01/06/2021
"Can we apply post processing effects to our scene? I added some in mine but it didnt seem to translate over into Luxor even though it's added as a component to the camera of my scene."

Sure you may include post-processing effects with scenes. You would need to include the post processing stack V2 and you'd need to add the SteamVR unity plugin to your project, then after adding the "player" prefab somewhere in the scene (which will also act as the spawn point) find the VRCamera object and add a Post-Process Layer and Post-porocess Volume script to the camera and create a new profile.

Check out the CustomSceneBuilder (legacy) project on github. The example scene includes a steamVR player rig with basic post processing setups on the VRCamera object (and should still be compatible with the latest version of Luxor): https://github.com/guiglass/CustomSceneBuilder

By adding the SteamVR plugin to a scene provides many additional feature and allows using many of the included SteamVR scrips and prefabs when creating scenes for Luxor. For example, you can instantly test the scene in VR without needing to build or export it to Luxor. Also you can add teleport points and teleport areas. You can also change the spawn point where Luxor will place the avatar by default when the scene is loaded.

And you may add post processing effects to the VRCamera...There's probably a few other features that I am not aware of but could be included in the scene. Basically any of the scripts or features that are included in the SteamVR plugin (and the interaction system) can be included in a custom scene and most likely would be available in the scene after being built and loaded in Luxor.

The same principals also apply when using the latest All-in-One APS\_SDK and/or the most recent SteamVR Unity plugin. Similarly I believe it would be possible to also include scripts in the scene or on objects from any of the supported addons such as Unity's Standard Assets, Unity physics, Unity animators, and possibly even DynamicBones Though I am not certain how versatile this approach would be in all scenarios and by including certain scripts could cause unforeseen bugs.

However I believe it would be possible to add particles and wind using Unity physics components, or add vehicles and followtargets using Unity standard assets, or for animating lights and translating or rotating objects/text/textures using Unity animators and may even be possible (untested) to add dynamic bones to objects in the scene such as foliage, cloth, curtains, flags.

"So is the steam vr camera also that of the desktop view for Luxor? The desktop view camera is the one I’d expect to need the post processing on since I’m not currently using Luxor in vr yet, just the trackers for now. I don’t have version 2 but v1 is built into unity 2018."

The steam vr camera added to custom scenes serves as a template, it's actually destroyed when Luxor loads the scene, however the postprocessing profile is copied to the main HMD camera and all camcorders/streaming cameras also will inherit the profile.

the APS\_SDK has be updated to v3.2.7 and now includes a "Add Player" button when building scenes and adds a Player prefab to the current scene which includes post processing. Before importing the APS\_SDK.unitypacking it is important to have first installed **Post Processing (v2)**!! which can be located from the Unity Packages manager. Also important is to install SteamVR plugin from the Unity Asset Store. Once those two packages are included you may add the APS\_SDK into the project and create a new scene and press "Add VR Player" from the SceneAssetBuilder to add the Player prefab object to the current scene. Once the Player prefab has been added you should see post processing effect in the game view. If not, then make sure to have deleted all other cameras in the scene. After customizing the scene/postprocessing you can press "Build Current Scene" and add the outputted .scene to *VR\_MocapAssets* folder for Luxor. Then in Luxor after loading the scene you should see the player/avatar is now placed at the new spatial offsets and the post processing profiles have been applied to all cameras.