Vision Pro's Next Leap: visionOS 2 Unlocks 3D Object Tracking for Enhanced AR

Apple officially announced visionOS 2 at its Worldwide Developers Conference (WWDC24) on June 10, 2024. Among the significant enhancements is an entirely new capability: 3D object tracking. This feature allows the Vision Pro to identify and interact with specific physical objects in a user's environment, moving beyond the 2D image tracking available since the device's launch.

UploadVR reports that this advanced tracking, alongside improvements to hand tracking and overall scene understanding, was not highlighted in the main WWDC24 keynote. Instead, details emerged from developer sessions made available online by Apple engineers. According to UploadVR, the Vision Pro previously could only track 'specified 2D images, such as markers and game boards,' but visionOS 2 extends this to three-dimensional physical items.

The outlet clarifies the process for developers, explaining that they must supply 3D models (USDZ format) of the objects to be tracked. These models are then processed through Apple's Create ML tool during app compilation on a Mac. This training generates a neural network for each object, which is then embedded within the application itself. They note this means users cannot track arbitrary objects of their own, and adding new objects requires an app update.

UploadVR suggests this functionality holds considerable promise, particularly for 'business and education use cases, or even for exhibitions like museums.' The article describes how visionOS provides a bounding box with accurate scale and orientation for each recognized object, allowing developers to anchor virtual content directly to them or augment them with digital effects. The publication expresses keen interest in 'what visionOS developers end up building' with this new capability, which it characterizes as enabling 'true augmentation of specific objects in your space, not just the environment.'

Our take: This 3D object tracking is a critical step forward for the Vision Pro’s augmented reality capabilities. While the current implementation requires pre-specified objects, limiting spontaneous interaction, it lays the groundwork for more sophisticated AR experiences in industrial, educational, and even retail sectors. We've been tracking the slow but steady convergence of digital and physical realities, and this update confirms Apple's commitment to enabling deeper, more interactive spatial computing. Future iterations will likely address the limitation of user-defined object tracking, but for now, this provides a robust framework for dedicated AR applications.