【visionOS/ARKit】Simplest sample code for hand tracking
Tags: visionos, applevisionpro, arkit, visionpro
Published: 2024-05-24 02:50:57

The basis of ARKit's hand tracking API in visionOS is to obtain data such as coordinates and rotation of each joint of the hand.

![Joints Example SessionVideo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yvy7aj6vy5wsavii6y8t.jpg)

![Joints List SessionVideo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fiehqggye0cn79wo9zwc.jpg)

Here is a sample code of ARKit's hand tracking API in visionOS that works with only 41 lines of code.


## Basic Knowledge
First, watch this session video. It gives a good overview of ARKit and hand tracking in visionOS.

[Meet ARKit for spatial computing - WWDC23 - Videos - Apple Developer](https://developer.apple.com/wwdc23/10082)

Hand tracking is explained from 15:05.


## Two APIs
There are two APIs to get joint data. Both can be accessed from the HandTrackingProvider instance.

- anchorUpdates: receives the latest values via AsyncSequence.
- latestAnchors: contains the latest values.

![Observing hand anchor data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/16y6zrukqclrauv4r43u.png)

In this case, I implemented it with anchorUpdates.


## Overview of the Sample Code
- Starts hand tracking when the app launches.
- Places simple sphere objects at all joints of both hands.
- Updates each object’s position to match the latest joint position.
- Ensures the objects are not hidden by the hand.

![Example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lbaoi1ul79cog5a8plcn.jpg)


## Full Source Code
```
import SwiftUI
import RealityKit
import ARKit

@main
struct MyApp: App {
    private let session = ARKitSession()
    private let provider = HandTrackingProvider()
    private let rootEntity = Entity()
    
    var body: some SwiftUI.Scene {
        ImmersiveSpace {
            RealityView { content in
                content.add(rootEntity)
                for chirality in [HandAnchor.Chirality.left, .right] {
                    for jointName in HandSkeleton.JointName.allCases {
                        let jointEntity = ModelEntity(mesh: .generateSphere(radius: 0.006),
                                                      materials: [SimpleMaterial()])
                        jointEntity.name = "\(jointName)\(chirality)"
                        rootEntity.addChild(jointEntity)
                    }
                }
            }
            .task { try! await session.run([provider]) }
            .task {
                for await update in provider.anchorUpdates {
                    let handAnchor = update.anchor
                    for jointName in HandSkeleton.JointName.allCases {
                        guard let joint = handAnchor.handSkeleton?.joint(jointName),
                              let jointEntity = rootEntity.findEntity(named: "\(jointName)\(handAnchor.chirality)") else {
                            continue
                        }
                        jointEntity.setTransformMatrix(handAnchor.originFromAnchorTransform * joint.anchorFromJointTransform,
                                                       relativeTo: nil)
                    }
                }
            }
        }
        .upperLimbVisibility(.hidden)
    }
}
```

Copy and paste this code to use it.


## Additional Steps
- Set any text for “NSHandsTrackingUsageDescription” in Info.plist.
- Set “Preferred Default Scene Session Role” to “Immersive Space” in Info.plist.

![Info.plist Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d4ml8zfbj8w71v39do20.png)


## Comments
_The explanations in the session video and the basic knowledge of SwiftUI and RealityKit are omitted._

### Managing Each Entity
```
jointEntity.name = "\(jointName)\(chirality)"
```

```
let jointEntity = rootEntity.findEntity(named: "\(jointName)\(handAnchor.chirality)")
```

Each entity is managed by its name using HandSkeleton.JointName for joint names and HandAnchor.Chirality for left or right hand.

### Setting Access Permissions
To request access permissions, you need to set any text for “NSHandsTrackingUsageDescription” in Info.plist.

This key does not appear in the pull-down menu, so enter it directly.

![Xcode Info.plist pull-down-menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/afo03qzji96ucjo33xhh.png)


### Launch the App in Full Space
```
@main
struct MyApp: App {
	...
    var body: some SwiftUI.Scene {
        ImmersiveSpace {
        	...
        }
        ...
	}
}
```

When you create a new visionOS app project in Xcode, it generates code to launch in a window. For simplicity, I made it launch in full space.

Set “Preferred Default Scene Session Role” to “Immersive Space” in Info.plist. If you do not make this setting, the app will crash right after launch.


## Note: A Physical Device is Required
You need a physical device to test the ARKit Hand Tracking API. It does not work at all in the simulator.


* * *


## Next Step
- Check the current authorization status: _session.queryAuthorization(for:)_
- Explicitly request authorization: _session.requestAuthorization(for:)_
- Check the state of anchors: _AnchorUpdate.Event_
- Check if each anchor or joint is being tracked: _TrackableAnchor.isTracked_
- Observe the session state: _ARKitSession.Events_
- Check if the current runtime environment supports it: _HandTrackingProvider.isSupported_


* * *


## Links
[Meet ARKit for spatial computing - WWDC23 - Videos - Apple Developer](https://developer.apple.com/wwdc23/10082)

[ARKit in visionOS | Apple Developer Documentation](https://developer.apple.com/documentation/arkit/arkit_in_visionos)

[upperLimbVisibility(_:) | Apple Developer Documentation](https://developer.apple.com/documentation/swiftui/scene/upperlimbvisibility(_:))

* * *

[HandsRuler on the App Store](https://apps.apple.com/app/id6475769879)

[FlipByBlink/HandsRuler: Measure app by hand tracking for Apple Vision Pro](https://github.com/FlipByBlink/HandsRuler)