SteamVR Overlay with Unity: Follow Device
Tags: unity3d, steamvr, openvr, vr
Published: 2024-06-16 19:05:18

## Follow the HMD

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ql6lml9c94kxid987g5b.gif)

*Overlay following the HMD*

### Remove the position code

First, remove the absolute position code we added in the previous part.

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);
        
    SetOverlaySize(overlayHandle, 0.5f);

-   var position = new Vector3(0, 2, 3);
-   var rotation = Quaternion.Euler(0, 0, 45);
-   SetOverlayTransformAbsolute(overlayHandle, position, rotation);
        
    ShowOverlay(overlayHandle);

    ...
```

### Device Index

In SteamVR, connected devices are identified with Device Index that automatically allocates from the system. (read the wiki for details)
For HMD, it is defined as [OpenVR.k_unTrackedDeviceIndex_Hmd](https://valvesoftware.github.io/steamvr_unity_plugin/api/Valve.VR.OpenVR.html#Valve_VR_OpenVR_k_unTrackedDeviceIndex_Hmd
) and is always 0.

### Prepare position and rotation

Let’s display the overlay at 2 m ahead (Z-axis) of the HMD. As in the previous part, let `position` and `rotation` variables.

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

+   var position = new Vector3(0, 0, 2);
+   var rotation = Quaternion.Euler(0, 0, 0);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);
    
    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
}
```

### Set relative position based on the HMD

Use [SetOverlayTransformTrackedDeviceRelative()](https://valvesoftware.github.io/steamvr_unity_plugin/api/Valve.VR.CVROverlay.html#Valve_VR_CVROverlay_SetOverlayTransformTrackedDeviceRelative_System_UInt64_System_UInt32_Valve_VR_HmdMatrix34_t__) to set the relative position based on the HMD. (read the [wiki](https://github.com/ValveSoftware/openvr/wiki/IVROverlay::SetOverlayTransformTrackedDeviceRelative) for details)

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
    var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
    var matrix = rigidTransform.ToHmdMatrix34();
+   var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, ref matrix);
+   if (error != EVROverlayError.None)
+   {
+       throw new Exception("Failed to set overlay position: " + error);
+   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
} 
```

Pass the HMD device index (`OpenVR.k_unTrackedDeviceIndex_Hmd`) and the transformation matrix.
Run the program, and check the overlay is shown 2 m ahead of the HMD.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ql6lml9c94kxid987g5b.gif)

### Organize code

Move the relative position code into `SetOverlayTransformRelative()`.

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
-   var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
-   var matrix = rigidTransform.ToHmdMatrix34();
-   var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, ref matrix);
-   if (error != EVROverlayError.None)
-   {
-       throw new Exception("Failed to set overlay position: " + error);
-   }
+   SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
} 

...

+ // Pass deviceIndex as argument.
+ private void SetOverlayTransformRelative(ulong handle, uint deviceIndex, Vector3 position, Quaternion rotation)
+ {
+     var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
+     var matrix = rigidTransform.ToHmdMatrix34();
+     var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(handle, deviceIndex, ref matrix);
+     if (error != EVROverlayError.None)
+     {
+         throw new Exception("Failed to set overlay position: " + error);
+     }
+ }
```

## Follow the controller

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/08cvlc1tipb7w8ibg931.gif)

*Overlay following the controller*

Use controller device index instead of HMD to make the overlay follow a controller.

### Get the controller device index

Get the left controller’s device index with [GetTrackedDeviceIndexForControllerRole()](https://valvesoftware.github.io/steamvr_unity_plugin/api/Valve.VR.CVRSystem.html#Valve_VR_CVRSystem_GetTrackedDeviceIndexForControllerRole_Valve_VR_ETrackedControllerRole_) of `OpenVR.System`.

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
+   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
}
```

The argument is:
Left controller: `EtrackedControllerRole.LeftHand`
Right controller: `EtrackedControllerRole.RightHand`

If it fails to get the device index like the controller is disconnected, `GetTrackedDeviceIndexForControllerRole()` returns `k_unTrackedDeviceIndexInvalid`.

### Follow the controller

We have got the left controller index, then make the overlay follow the controller. Pass the controller index to `SetOverlayTransformRelative()` that we previously created for the HMD.

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);
+   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+   {
+       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);

    ShowOverlay(overlayHandle);
}
```

Make sure the left controller is connected to the SteamVR window, then run the program.

The overlay should follow the controller instead of the HMD.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/08cvlc1tipb7w8ibg931.gif)

## Adjust overlay position

To make a watch application, we will adjust the overlay position on the left wrist. Make position parameters editable at runtime on the Unity editor.

### Add member variables

Add size, position, and rotation variables as class members. Use Range() attribute to show sliders on the inspector.

```diff
public class WatchOverlay : MonoBehaviour
{
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

+   [Range(0, 0.5f)] public float size = 0.5f;
+   [Range(-0.2f, 0.2f)] public float x;
+   [Range(-0.2f, 0.2f)] public float y;
+   [Range(-0.2f, 0.2f)] public float z;
+   [Range(0, 360)] public int rotationX;
+   [Range(0, 360)] public int rotationY;
+   [Range(0, 360)] public int rotationZ;

    ...
```

Sliders are shown on the inspector.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0rwcmeuudxeu6qfsohll.png)

### Replace variables in the code

Replace the existing size and position variables in the code with the added member variables.

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

-   var position = new Vector3(0, 0, 2);
-   var rotation = Quaternion.Euler(0, 0, 0);
+   var position = new Vector3(x, y, z);
+   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
    {
        SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
    }

    SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

-   SetOverlaySize(overlayHandle, 0.5f);
+   SetOverlaySize(overlayHandle, size);
    ShowOverlay(overlayHandle);
}
```

### Update size and position in Update()

Make the size and position editable at runtime by adding code to the `Update()`. Note that this code will be deleted later. It is temporary for determining new positions and rotation.

```diff
private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(x, y, z);
    var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
    {
        SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
    }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, size);
    ShowOverlay(overlayHandle);
}

+ private void Update()
+ {
+     SetOverlaySize(overlayHandle, size);
+ 
+     var position = new Vector3(x, y, z);
+     var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
+     var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
+     if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+     {
+         SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+     }
+ }

...
```

Run the program. Ensure the inspector slider changes the overlay size and position at runtime.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/79hvoidyb6dcwpmkg0ih.png)

### Adjust overlay position

Move sliders to adjust the overlay position to the left wrist. I recommend changing sliders from the desktop window that is opened in the SteamVR dashboard.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lqjbfksjjm0hy3tjqwtx.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rltzoo88vrrp6rkbvf4f.gif)
*Control the Unity editor from the SteamVR dashboard.*

Another way is looking at the HMD view on the desktop and adjusting parameters.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tv8nbl7qw0hn8nqqa4bb.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/htyzqso4fvv3pulq9w8m.gif)
*It’s helpful if you don’t want to put on the HMD.*

Here are sample parameters.

```
size = 0.08
x = -0.044
y = 0.015
z = -0.131
rotationX = 154
rotationY = 262
rotationZ = 0
```

When adjusting is done, right click on the WatchOverlay component name, and select Copy Component.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j3janr5p3zvi3729050q.png)

Stop the program, right click the WatchOverlay component again, and paste the copied values with Paste Component Value.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/28akbb7hcyfwqkil2q47.png)

Run the program. Check the overlay is on the left wrist.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z4hkf0bknwwnbx2vfngq.png)

### Remove temporary code

Remove the code from the `Update()`.

```diff
private void Update()
{
-   SetOverlaySize(overlayHandle, size);
-   
-   var position = new Vector3(x, y, z);
-   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
-   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
-   {
-       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
-   }
}
```

## When the controller is not connected

Currently, the controller must be connected at launch because we get the controller device index at the `Start()`.
Move the getting device index code from `Start()` to Update() to support the cases where the controller connects or disconnects in the middle.

```diff
private void Start()
{
    InitOpenVR();
    OverlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

-   var position = new Vector3(x, y, z);
-   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
-   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
-   {
-       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
-   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, size);

    ShowOverlay(overlayHandle);
}

+ private void Update()
+ {
+     var position = new Vector3(x, y, z);
+     var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
+     var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
+     if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+     {
+         SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+     }
+ }
```

Now, it works even if the controller is connected in the middle.

## Final code

```cs
using UnityEngine;
using Valve.VR;
using System;

public class WatchOverlay : MonoBehaviour
{
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

    [Range(0, 0.5f)] public float size;
    [Range(-0.2f, 0.2f)] public float x;
    [Range(-0.2f, 0.2f)] public float y;
    [Range(-0.2f, 0.2f)] public float z;
    [Range(0, 360)] public int rotationX;
    [Range(0, 360)] public int rotationY;
    [Range(0, 360)] public int rotationZ;
    
    private void Start()
    {
        InitOpenVR();
        overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

        var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
        SetOverlayFromFile(overlayHandle, filePath);

        SetOverlaySize(overlayHandle, size);
        ShowOverlay(overlayHandle);
    }

    private void Update()
    {
        var position = new Vector3(x, y, z);
        var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
        var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
        if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
        {
            SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
        }
    }
    
    private void OnApplicationQuit()
    {
        DestroyOverlay(overlayHandle);
    }

    private void OnDestroy()
    {
        ShutdownOpenVR();
    }

    private void InitOpenVR()
    {
        if (OpenVR.System != null) return;

        var error = EVRInitError.None;
        OpenVR.Init(ref error, EVRApplicationType.VRApplication_Overlay);
        if (error != EVRInitError.None)
        {
            throw new Exception("Failed to initialize OpenVR: " + error);
        }
    }

    private void ShutdownOpenVR()
    {
        if (OpenVR.System != null)
        {
            OpenVR.Shutdown();
        }
    }

    private ulong CreateOverlay(string key, string name)
    {
        var handle = OpenVR.k_ulOverlayHandleInvalid;
        var error = OpenVR.Overlay.CreateOverlay(key, name, ref handle);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to create overlay: " + error);
        }

        return handle;
    }

    private void DestroyOverlay(ulong handle)
    {
        if (handle != OpenVR.k_ulOverlayHandleInvalid)
        {
            var error = OpenVR.Overlay.DestroyOverlay(handle);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to dispose overlay: " + error);
            }
        }
    }

    private void SetOverlayFromFile(ulong handle, string path)
    {
        var error = OpenVR.Overlay.SetOverlayFromFile(handle, path);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to draw image file: " + error);
        }
    }

    private void ShowOverlay(ulong handle)
    {
        var error = OpenVR.Overlay.ShowOverlay(handle);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to show overlay: " + error);
        }
    }

    private void SetOverlaySize(ulong handle, float size)
    {
        var error = OpenVR.Overlay.SetOverlayWidthInMeters(handle, size);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay size: " + error);
        }
    }

    private void SetOverlayTransformAbsolute(ulong handle, Vector3 position, Quaternion rotation)
    {
        var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
        var matrix = rigidTransform.ToHmdMatrix34();
        var error = OpenVR.Overlay.SetOverlayTransformAbsolute(handle, ETrackingUniverseOrigin.TrackingUniverseStanding, ref matrix);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay position: " + error);
        }
    }

    private void SetOverlayTransformRelative(ulong handle, uint deviceIndex, Vector3 position, Quaternion rotation)
    {
        var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
        var matrix = rigidTransform.ToHmdMatrix34();
        var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(handle, deviceIndex, ref matrix);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay position: " + error);
        }
    }
}
```

Here, We attach the overlay on the left wrist. Let’s draw Unity camera output to the overlay instead of the static image file in the next part.