This chapter is from the book
This chapter is from the book
This chapter is from the book
Scene Reconstruction
Hand-tracking can enable experiences where interactions with the environment seem very natural. However, the problem is that the environment itself still doesn’t seem very natural. Plane detection provides the ability to place virtual objects on real-world surfaces like seats and tables, but it doesn’t consider things like pillows on couches and the fact that literally no living human has ever kept a table surface completely clean for more than 47 seconds. As a result, virtual objects added to the planes could exist inside real-world objects that happened to be in the same location on the plane. Let’s face it, plane detection is cool, but it just doesn’t give us a very “exact” representation of all the different surfaces that virtual objects may encounter.
Scene reconstruction takes plane detection to another level. Think of scene reconstruction as plane detection on steroids. Rather than just looking for flat surfaces, a SceneReconstructionProvider (https://developer.apple.com/documentation/arkit/scenereconstructionprovider) considers all the incoming data from the Vision Pro to recreate the geometry of all the surroundings where the user is located. It’s like taking a giant sheet and covering everything with it, tucking the sheet into all the spaces around all the different objects.
This data is provided by multiple MeshAnchors, each with a mesh (shape) that’s constantly tracked in the environment. By adding these meshes to your content, you effectively “reconstruct” the real world within a virtual space.
With the right meshes in place, you can have objects interact with the miscellaneous “stuff” you place around yourself. Objects can roll off pillows and under tables and even fall in places that make them difficult to retrieve—making virtual life just as annoying as the real thing.
ARKit MeshAnchors
Yes, a MeshAnchor works in a very similar way to the hand anchors and plane anchors, so you’re gonna be experiencing more déjà vu. Let’s quickly cover the properties you might need when you process mesh anchor updates:
.originFromAnchorTransform: The location and orientation of the detected shape in world space.
.geometry: A collection of the different shapes that make up a mesh anchor.
.geometry.classifications: A classification of each face of the geometry that makes up a mesh. Because a mesh may span multiple objects, one must look at all the different geometry classifications to see everything that has been detected. Review https://developer.apple.com/documentation/arkit/meshanchor/meshclassification if you’re interested in what objects can be reported by a MeshAnchor.
You can learn more about MeshAnchors at https://developer.apple.com/documentation/arkit/meshanchor, but probably the most important thing to understand is that it takes work to turn a MeshAnchor into something useful. With planes, for example, you need to create a plane ModelEntity and add it to your content. MeshAnchors come to use with geometry information but not in a form you can use.
Generating Collision Shapes
To use a MeshAnchor, you need to turn it into something that can be used in your Reality View content. To do this, you take advantage of a ShapeResource class method that turns a MeshAnchor into a shape that can be used as a collision component.
You might be wondering, “What good does that do? Are you saying it doesn’t give a shape I can use to style and present a virtual object?” Yes, that’s exactly what I’m saying. You can create an entity and assign a collision component based on the anchor, and then add it to the content. This will have the effect of creating an invisible object that matches the shape and placement of real-world objects, but it only serves the purpose of allowing objects to collide with it realistically.
To generate a collision shape from a mesh anchor, you first generate the shape:
let <shape mesh> = try! await ShapeResource.generateStaticMesh(
from: <mesh anchor>)
Then, you can create a new ModelEntity, set its collision component to the generated mesh, and add a physicsBody for good measure:
let <model entity> = ModelEntity()
<model entity>.collision = CollisionComponent(shapes: [<shape mesh>],
isStatic: true)
meshEntity.physicsBody = PhysicsBodyComponent(mode: .static)
Like all the other data providers, this process must be repeated over and over as the headset detects or stops detecting new surfaces, so you need another new class for the implementation (which you make momentarily). But, before you do that, there’s “one more thing” I need to discuss because it will truly bring your projects to life.
Occlusion
Apple has built a heck of a device, but the Apple Vision Pro’s development tools are still in their early stages. Some tasks that have worked great on the iOS/iPadOS platforms can be painful on the Vision Pro. One of these is occlusion, or the process of hiding one object behind another. Your hands, for example, occlude virtual objects, which is necessary for interactions. Virtual objects hide other virtual objects that are behind them. What’s missing is for real-world objects to occlude virtual objects.
You may have noticed over the past several exercises that if you place a virtual object somewhere in the environment then walk behind a wall or put a physical object in front of it, you can still see the virtual object. It’s like having virtual X-ray vision but can also be quite jarring and bring you out of an experience really quickly.
Occlusion Material
Apple provides a special material, called an occlusion material, that can be applied to virtual objects. The object becomes invisible to the viewer but still blocks virtual objects behind it:
let material = OcclusionMaterial()
You should be able to take this occlusion material, apply it to the model entities you create during scene reconstruction, and gain the effect of real objects blocking the virtual.
But it’s not going to work. The collision shape you add to a model entity isn’t a visible surface. You can’t apply a material or see a model that only has a collision shape. I suspect Apple will remedy this in the future, but for now, occlusion is not simple.
Or is it?
Occlusion Meshes
As it turns out, the occlusion mesh problem has been solved reasonably well by a GitHub user named XRealityZone. Within their GitHub repository, they maintain a visionOS project called what-vision-os-can-do. This has some useful code snippets that you can use in your creations and is a combination of community contributions and code that Apple has published in its examples.
You can access the repository here:
https://github.com/XRealityZone/what-vision-os-can-do/tree/main
Within the project is a method that translates a MeshAnchor into a MeshResource, which is exactly what you need to do. You make use of a modified version of this code when you build a scene reconstruction class next. You create entity models with collision shapes and model meshes that can use any material or shader you want—including the occlusion material.
I’m sure Apple will eventually make the process easier, but if you use the SceneReconstructor class you’re about to code, you’ll have that functionality now.