What's New in Apple AR/VR 2023
Tags: realitykit, visionpro, visionos, wwdc23
Published: 2023-06-07 02:32:27

> AR/VR/XR Highlights from Apple's WWDC Developer Conference 2023

It's been six years since ARKit's debut at WWDC 2017, and we've seen big milestones since then; but none come close to what was announced at WWDC 2023 this week.

From a full AR/VR simulator, to Apple Vision Pro, to full skeletal hand tracking, you'll find the AR/VR highlights from the main keynote listed below; as well as other things hiding around in Apple's beta API References. It won't go into too much technical details on how to implement some of these new features and new frameworks, you'll have to [follow me on twitter](https://twitter.com/maxxfrazer).

Although many of the APIs are not yet public (expected later in June 2023), we'll cover an outline of the main new features coming our way soon.

## ARKit

Starting with the core of AR at Apple, there only seems to be one feature available for now, and only on the newly introduced OS.

### Skeletal Hand Tracking

As well as tracking surfaces, images and bodies, ARKit will now have skeletal hand tracking, starting with visionOS. This seems to be enabled due to a new device that has multiple cameras to better percieve depth.

![A hand with a globe floating above](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o4w47nsvoigr3i7boali.gif)


## RealityKit

Not known to be the fastest horse in terms of rolling out features, especially when compared to its older sister [SceneKit](https://developer.apple.com/scenekit/). But this year there were some much needed advancements, which were of course necessary for a certain piece of hardware.

### RealityKit x SwiftUI

With the introduction of a whole new SwiftUI view, RealityView, RealityKit can now fully take advantage of SwiftUI layouts, and vice versa.

You can also add SwiftUI views into RealityKit in the form of attachments.

![Rendering of an island dipslayed in AR, with a couple of SwiftUI views added as attachments](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zn4mvp1efsd0n57e0qmn.gif)

This now allows us to add content with sharp, readable text, especially those with complex layouts and animations.

### MaterialX

RealityKit is adopting the open standard MaterialX for surface and geometry shaders.

MaterialX is supported across a broad variety of tools. Meaning people already familiar with those tools can now write already shaders in RealityKit, as well as those already using RealityKit can now have more developer tools and countless shader examples at their disposal.

## visionOS

Apple's exciting new operating system, visionOS brings many of the AR/VR concepts that RealityKit developers may be familiar with together in a stunning new headset dubbed "Vision Pro". While many of the 

### Fundamentals

There are a few key objects that you can create within Vision Pro's OS, visionOS:

- Windows, planes in space.
- Volumes, which can hold 3d objects, such as a game board.
- Full Space, which are fully immersive environments, and can not necessarily be manipulated by the user.

![Screenshot of the platforms state of the union presentation showing the three fundamentals of visionOS](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yajk7u0k9r36g74uqmpe.jpg)

### Apps

Existing iOS/iPadOS apps are supported as a scalable 2d window shown in 3d space on visionOS. This window keeps their original look and feel.

![Existing iPad application, brought into a 3d scene with visionOS](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5er7j44ofvr88sthb93w.gif)

If you simply add the build target of visionOS to your application, it immediately gets those nice visionOS materials, a fully resizable window, and displays the eye focus of the user, as you find in all visionOS applications.

![Updated iPad application, with visionOS materials](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y7aqyk2y16nmbnixa9n7.gif)

With visionOS SwiftUI or UIKit can be used display content, RealityKit for 3d content, and ARKit to understand the user's space. The apps designed specifically for visionOS can create a collection of windows, volumes, and spaces.

### Simulator

My favourite part of the developer tooling at WWDC 2023. This simulator lets you move and look around the scene, and interact with the app by simulating the system gestures. There are several different builtin scenes, with both day and night coditions to help better understand how users will interact with your apps.

![visionOS simulator, showing an application in the middle, camera is moving towards it and rotating slightly to show depth](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o545s5wfwg6g3tg4or8x.gif)

### Mac Virtual Display

Simply by looking at your Mac through a Vision Pro, your Mac's screen will pop out and instead a high fidelity 4k monitor will appear in its place. This enables a whole new end-to-end development experience, where you can code, test, and debug all in the same place.

![Visualisation of a mac window on the right containing Xcode, and the built visionOS application on the left. macbook with blank screen below both.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wui76t3tp17zlc7ciew8.jpg)

Bluetooth keyboards and mice can be used for data input across visionOS, so of course you won't have to type on a virtual keyboard.

### Reality Composer Pro

Using Reality Composer Pro, you can add and prepare 3d content for visionOS apps.

## What's next?

You may have noticed some sections are a little bit bare, I'll be expanding upon the content over the next week or so as more videos and software are released.

To stay tuned, drop a follow on [dev.to](https://dev.to/maxxfrazer) as well as [twitter](https://twitter.com/maxxfrazer). I'm always posting about the latest advancements in AR/VR, and specifically those in the Swift ecosystem.