The ETC CMU EDU Magic Leap Wiki

## Getting Started

### Prerequisites

To get started with Magic Leap, you need to install Lumin SDK and Lumin Build support for Unity. To do this, follow these steps:

1. Go to Magic Leap Creator Portal, sign up your personal Magic Leap account following the instruction.
2. Download Magic Leap Package Manager from column **Download**. Download for Windows or Mac according to the OS you use.
3. Open the Magic Leap Package Manager and install the latest version of the **Magic Leap Unity Package**, which should be listed under *Unity Packages* section. You should also get the latest version of **Lumin SDK** in this way. If not, install **Lumin SDK** under *Common Packages*. This setup tutorial works for **Lumin SDK** version **0.22.0.**
4. Double check the **change list** at the bottom, make sure there are **Magic Leap Unity Package** and **Lumin SDK** in the list, then click **Apply.**
5. Open Unity Hub, make sure your unity version is at least **2019.2.0f1**. Click **Installs** and then click the 3-dot menu for the unity version you use. Click **Add Modules** and install **Lumin OS (Magic Leap) Build Support**. If you could not find **Add Modules** option, try downloading Unity Installer corresponding to the version you use from Unity website, and then run the installer to add the module.

### Basic Setup

1. Create new Unity 3D Project
2. Go to Edit->Preferences->External Tools, set **Lumin SDK** path. The default path will be `C:/Users/<CURRENT_USER>/MagicLeap/mlsdk/<VERSION>`.
3. Go to File->Build Settings, select **Lumin** and then click **Switch Platform**
4. Go to Window->Package Manager, click **Advanced** and select **Show preview packages**, choose Yes for the warning message. Find **XR Management** and click install
5. Go to Edit->Project Settings->XR Plugin Management, select **Lumin**, click **+** then download **Magic Leap Loader**. Then click **+** again and select **Magic Leap Loader** under **Available**. Do the same for **PC, Mac & Linux Standalone**.
6. Go to Edit->Project Settings->Player, select **Lumin** and go to **Other Settings**. Under **Rendering**, change **Color Space** from *Gamma* to *Linear*. Under **Identification**, change **Bundle Identifier** to a unique name, like `edu.cmu.etc.bvw19.team1`. Under **Configuration**, change **Api Compatibility Level** from *.Net Standard 2.0* to *.Net 4.x*.
7. Go to Edit->Project Settings->Player, select **Lumin** and go to **Splash Image**. Under **Splash Screen**, uncheck **Show Splash Screen**.
8. Go to Edit->Project Settings->Player, select **PC, Mac & Linux Standalone** and go to **Other Setting**. Under **Rendering**, uncheck *Auto Graphics API for Windows*, then click **+** to add *OpenGLCore*. Drag *OpenGLCore* and put it on top of *Direct3D11*. You will need to restart Editor to take effect.
9. Import the **Magic Leap Unity Package** into your project. To locate the package, open Magic Leap Package Manager and explore to Unity Packages-> Magic Leap Unity Package and you will see the folder option. Double click the .unitypackage file to import. If Unity needs to update API on importing, click **I Made a Backup, Go Ahead!** to continue. Now you should see [MagicLeap] folder in your Assets window.
10. If you receive compilation error messages like **MLSpatial Audio is not found** after importing the package, go to Edit->Project Settings->Player->Lumin OS->XR Settings and check "Virtual Reality Supported" to try enabling legacy XR. After a while Unity will tell you multiple XR SDK providers have been found and will disable legacy VR then (the checkbox will not be checked as well), and the compilation error should be resolved.
11. Now you are all set! Be ready to enjoy developing with this charming device!

Note: The Official Guide provides more details on how to setup Magic Leap for Unity projects.

### Scene Setup

1. Remove **Main Camera** from Scene.
2. Drag **Main Camera** from `MagicLeap/Core/Prefabs` into the scene.
3. According to the types of function you want to use, go to Edit->Project Settings->MagicLeap->Manifest Settings, check the function you will use. Remember make sure **Minimum API Level** should at least be 4 and **LowLatencyLightwear** should always be checked. For more details, check Privileges page from official documents.

### Preview Game with ML Remote Simulator

1. **Make sure build setting has changed to Lumin Platform and the Unity renderer is set to OpenGL (with <OpenGL> on your title bar)**
2. Go to Magic Leap->ML Remote, click **Import Support Libraries**
3. Go to Magic Leap->ML Remote, click **Launch MLRemote**
4. Click **Start Simulator** and after **all signal lights turn green**, you can hit **play** button in Unity Editor and see the scene in ML Remote.
5. Follow the instructions here to navigate and control in the simulated scene.
6. To stop the game, click stop button in Unity Editor. **You have to stop the game first before stopping the simulator, or Unity will become suspended and may crash.**
7. To stop the simulator, click the 3-line menu->Stop Magic Leap Remote Server.
8. **Do not close MLRemote unless you have stopped the Remote Server, or the server will be still running in the background.**
9. **Do not close MLRemote unless you have stopped the Remote Server, or the server will be still running in the background.**
10. **Do not close MLRemote unless you have stopped the Remote Server, or the server will be still running in the background.**

**Note: You must launch MLRemote from Unity editor to correctly connect to the simulator when running the game.**

### Preview Game on Device

1. **Make sure build setting has changed to Lumin Platform and the Unity renderer is set to OpenGL (with <OpenGL> on your title bar)**
2. Go to Magic Leap->ML Remote, click **Import Support Libraries**
3. Go to Magic Leap->ML Remote, click **Launch MLRemote**
4. Make sure your Magic Leap device is connected to the computer. Check Connect to PC for instructions.
5. Click **Start Device** and after **the signal light turns green**, you can hit **play** button in Unity Editor and see the scene in your device.
6. You can debug your game as you normally do in Unity.
7. To stop the game, click stop button in Unity Editor. **You have to stop the game first before stopping device connection, or Unity will become suspended and may crash.**
8. To stop the device connection, click the 3-dash menu->Stop Magic Leap Remote Server.
9. **Do not close MLRemote unless you have stopped the Remote Server, or the server will be still running in the background.**
10. **Do not close MLRemote unless you have stopped the Remote Server, or the server will be still running in the background.**
11. **Do not close MLRemote unless you have stopped the Remote Server, or the server will be still running in the background.**

**Note: You must launch MLRemote from Unity editor to correctly connect to the simulator when running the game.**

### Build and Run a Game

1. **Make sure build setting has changed to Lumin Platform.**
2. Make sure your Magic Leap device is connected to the computer. Check Connect to PC for instructions.
3. Go to Magic Leap Creator Portal, under **Publish**->**Certificates**, create your certificate. It will automatically download the `privatekey.zip` which contains a `.privkey` file. **Be sure to keep the file safe.** In certificate page, find the certificate you just create and download it. The certificate should be a file with extension name `.cert`. **Important: Keep** `.privkey` **file and** `.cert` **file in the same folder**. Otherwise you can't build successfully.
4. In Unity Editor, go to Edit->Project Settings->Player, select **Lumin**, and go to Publish Setting. Set the path of ML Certificate. **Note that the path is an absolute path, so you need to change it when you change to another computer.**
5. Click Build and Run and a signed `.mpk` package will be generated under your build folder. If you have your Magic Leap device connected, Unity will try installing this `.mpk` package and run it in Magic Leap. You need to accept the certificate you signed before running the application. You can also use `mldb install` command to install the `.mpk` package manually following this guide.

Note: If you have your game installed in your Magic Leap device and want to update your game, try `mldb install -u` command instead of `mldb install`.

### Remove Game from Device

If you switch between development and non-development builds, uninstall the app when you're done testing each build.

1. On the Control, press the Home button for 3 seconds.
2. Select your app in the app Launcher, and then press and hold with the touchpad.
3. Select the trash can and press the trigger.
4. You can also remove it with mldb using `mldb uninstall <BUNDLE_IDENTIFIER>`command.

## Advanced Usage

### Magic Leap Device Bridge (MLDB)

Magic Leap Device Bridge (MLDB) is a command line tool provided with the Lumin SDK that allows developers to communicate with the Magic Leap One device from a PC over a USB cable or WiFi connection (see Connect to PC). MLDB provides direct access to Magic Leap device, and developers can install, run, debug and manage applications directly using this command line tool.

If you know ADB (Android Debug Bridge), you can take MLDB as the "ADB" for Magic Leap devices.

To use MLDB, you need to enable the MLDB access on your Magic Leap device:

1. Navigate to Settings > Device > Creator, and enable Creator Mode.
2. Select **Enable MLDB Access** to allow MLDB access.

To run commands from MLDB, open Magic Leap Package Manager, locate **Lumin SDK**, and click **Shell** to open a command line environment configured for the Lumin SDK.

A basic MLDB command is to check device connection to your computer:

Check this guide to explore the potentials of this powerful tool.

### Resetting the device via a computer

**WARNING: This method will delete all of the user information from the device!**

If you manage to lock yourself out of the Magic Leap or cause the device to no longer render anything despite being on and functional, don't fret! Getting the device back up and running from initialization is straightforward and quick.

1. Connect the Magic Leap to one of your computer's back USB 3.0 ports using a USB-C to USB-A adapter (you *must* connect the device to a back USB port, as the front ones may not support the device)
2. If you haven't already, install the Magic Leap Package Manager
3. Open the Lumin SDK shell for the MLDB tool.
4. Then enter `mldb reboot bootloader`. If your device is locked, then it will take 60 seconds for it to reboot. The bootloader is used for fastboot commands, which we need for doing a hard reset.
5. Enter `fastboot erase userdata`. This will erase all of the user information from the device, including any improper configuration information and any private access locks.
6. Lastly, enter `fastboot reboot` . This will reboot the device into it's typical startup state.

That's all you need to do! You can now go through the initialization process to sign into the device and set up the locking information on it again.

### Connect to PC

1. This tutorial only works for Magic Leap with OS 0.97 or higher.
2. Open **Magic Leap Package Manager**. Under **Common Packages**, find **Device Stream** and install it.
3. Launch Device Stream.
4. Use USB-A 3.0 to USB-C cable to plug your Magic Leap device to the computer.
5. Turn on Magic Leap device, accept USB connection in Magic Leap and accept the connection request of **Device Stream**, then you should see the device identifier on **Device Stream** with green signal light and streaming display.
6. You can also connect Magic Leap using Wi-Fi. Make sure your device has Internet connection. Go to **Device Stream** and click the **reverse triangle button** next to device identifier and then click the **gear menu icon**. Click **Use Network**, and you need to accept the connection request again in Magic Leap. Then you should see the ip address of the device on **Device Stream** and see the streaming display. Note that you don't need to use the USB cable after the device is connected via Wi-Fi, but you need to use the cable to reconnect your device after device restart.
7. To check the connection status. Open **Magic Leap Package Manager** and explore to **Lumin SDK**, click **Shell** and use `mldb devices` command to check.

### Streaming to Device

#### To PC

The Magic allows for streaming of headset on **Device Stream** on OS 0.97.

1. Make sure you have **Device Stream** installed. If not, open **Magic Leap Package Manager**. Under **Common Packages**, find **Device Stream** and install it.
2. Connect your Magic Leap device to PC. For detail instruction, check Connect To PC.
3. You will see the stream display in the display window of **Device Stream**.
4. Don't use the recording function inside Device Stream now, no audio will be recorded in the current beta version. To record the video, check Recording Video for more information.

#### To Android or iOS

The Magic Leap allows for streaming of headset visuals to a mobile device as on OS 0.95.

1. Download the Magic Leap companion app from the iOS or Android store to mobile device
2. Make sure Bluetooth is enabled on mobile device
3. Open the Magic Leap Companion App and move to the Magic Leap Headset
4. On the Headset, enter the settings Device Settings menu.
5. Under "Creator", make sure that Enable Device Streaming is checked affirmative
6. Under Connection Settings, enable Bluetooth and pair to your mobile device
7. Under Controllers, scroll down to Scan for Companion App
8. Connect and pair to mobile device there as well.
9. You will be asked to pair/accept connections on your mobile device as well as on the Magic Leap. Accept them.
10. On the Companion App, make sure that "waiting to connect" now displays "connected'
11. On the sidebar menu, scroll and select to Device Streaming (beta).
12. Magic Leap will pop up a notification to allow device streaming. Allow the device to stream.

**Your device should now show a stream of the Magic Leap's view.**

* Note: Device Streaming and Device Recording/Screenshot can not be used at the same time (as of 0.95)
* To record the stream, use the mobile device's built in screen recording software
	+ In iOS you must start a screen recording first before connecting to Device Streaming. This is because leaving the Companion App while streaming breaks the stream.

### Recording Video

Please download the recording script from here and follow the instructions. The script assumes that you have Lumin SDK v0.22 installed in the default location (`C:/Users/xxx/MagicLeap`). This script utilizes `mldb capture video` command to record the video.

You could also record video using **Device Stream**. **Don't use the recording function of Device Stream (audio will be lost), instead, use OBS and treat Device Stream as normal video output to record.**

## Programming Information

### Using Controller

https://creator.magicleap.com/learn/guides/control-in-unity

### Using Hand

https://creator.magicleap.com/learn/guides/gestures-in-unity

https://creator.magicleap.com/learn/guides/hand-tracking-key-points-unity

https://creator.magicleap.com/learn/guides/sdk-hand-meshing-in-unity

### Using Reconstructed World Mesh

https://creator.magicleap.com/learn/guides/meshing-in-unity

https://creator.magicleap.com/learn/guides/plane-extraction-in-unity

https://creator.magicleap.com/learn/guides/raycast-in-unity

### Placing Your Scene

The Magic Leap Unity Plugin is shipped with a component [Placement] that can help place your scene or any objects aligned with world mesh. You can check the scene [Placement] in the plugin's examples to learn how to do so.

### Unity API Reference

You can find the Unity API Reference here, or you can install it on your local PC following these steps:

1. Open Magic Leap Package Manager.
2. Open Unity API Documentation tab.
3. On the right side panel, download the latest version.
4. On the left panel in the bottom, "Apply".
5. The right hand panel now shows a button that opens the reference when you click on it.

### Single Pass Instanced Rendering

Magic Leap uses **single pass instanced rendering** by default. It may introduce some problems to customized shaders and only the left eye will render the correct result if the shaders lack the support of this mode. You need to manually modify the shader and related scripts to make sure the camera poses are correct for each eye and to support instanced rendering. Please check Unity's reference for more information.

If you have no idea how to work correctly with single pass instanced rendering, the easier way is to fall back into muti pass rendering with some performance lost. You can turn it on through [Edit -> Project Settings -> XR Plugin Management -> Magic Leap Settings] and enable [Force Multipass] option.

## Art (2D & 3D) Information

The Official Art & Anmiation Guide provides best practices for artists.

## Sound Information

The Unity Magic Leap plugin is shipped with Magic Soundfield Audio (MSA) plugin that supports spatial audio in Magic Leap. Check this guide to learn how to set up MSA in Unity.

The Official Sound Design Guide provides best practices for sound design of Magic Leap Apps.

## Design Guidelines & Tips

The Official Design Guide is a good starting point to learn how to design a game or an immersive experience for Magic Leap.

* The near clip plane distance of Magic Leap One is 37cm (14.57 inches). Hand interaction within this distance should be avoided.
* Magic Leap One has a horizontal FOV of 40 degrees, a vertical FOV of 30 degrees, and a diagonal FOV of 50 degrees. The FOV range is a bit narrow. Considering using Vignette effect and guiding arrows when you are designing a 360 degree scene.

## Example Projects

We provides a template project configured with Magic Leap Unity Package 0.22.0.

**To set up for this template project, follow the steps below:**

1. Make sure you have finished all the steps in the Prerequisites section first.
2. Extract the zip file and rename the extracted `MagicLeapTemplate` folder to your team name (e.g. `Team01`).
3. Open **Unity Hub**, click **Add** and select this folder. You will find this project in the **Projects** list. Click this project to launch Unity editor.
4. Go to Edit->Preferences->External Tools, set **Lumin SDK** path. The default path will be `C:/Users/<CURRENT_USER>/MagicLeap/mlsdk/<VERSION>`.
5. Go to File->Build Settings, select **Lumin** and then click **Switch Platform**.
6. Go to Edit->Project Settings->Player, select **Lumin** and go to **Splash Image**. Under **Splash Screen**, uncheck **Show Splash Screen**.
7. Go to Edit->Project Settings->Player. Change **Product Name** to your team name (e.g. `Team01`). Then select **Lumin** settings tab below and go to **Other Settings**. Under **Identification**, change **Bundle Identifier** to a unique name, like `com.bvw19.team1`. This is the unique name for the mpk package that you will build.
8. If you are using MLRemote for debug, don't forget Menu->MagicLeap->ML Remote->Import Support Libraries.
9. If you are using Perforce, configure your Perforce settings in Edit->Project Settings->Editor.
10. Now you are all set! Example scenes are located in `Assets/BVW/SampleScenes`. Be ready to enjoy developing with this charming device!

## Previous ETC Projects

1. https://www.etc.cmu.edu/projects/artales/

## BVW

Currently, there is no BVW-specific information to know about Magic Leap.

```
#This has been a 1:1 copy of The ETC CMU EDU Magic Leap Wiki
#You can find more Wikimidia Pages on cool XR stuff there.

```