- Hand-Tracking
- Hands-On: Creating a Hand Tracker Class
- Scene Reconstruction
- Hands-On: Creating a Scene Reconstructor Class
- Hands-On: Reconstruction
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Hands-On: Creating a Scene Reconstructor Class
Here you are, once again, about to build a class that uses an ARKit session to collect data. This is yet again the same code pattern used for plane and hand-tracking. It’s also the last time you’re going to have to hear me say that. Once you’ve finished the class, you’re going to jump into a third exercise that puts it and the hand-tracking class to good use.
In this project, you create another new class, SceneReconstructor, that employs a SceneReconstructionProvider to generate MeshAnchors. Each MeshAnchor is used to position a ModelEntity that is built using the geometry in the anchor. It has both collision shapes and a surface with applied material. You track all of them in an EntityMap collection.
In ImmersiveView.swift, you add these model entities to the RealityView. Users will see a version of their surroundings covered in any material you choose, as shown in FIGURE 8.3.
)
FIGURE 8.3 You are now living in the Matrix.
Setting Up the Project
Create a new Mixed Immersive project in Xcode named Room Virtualizer and then follow these steps:
[Optional] Update the ContentView.swift file to include an introduction and the <App Name>App.swift file to size the content appropriately.
Remove the extra code from the ImmersiveView.swift file. Edit the RealityView so that it is empty.
The project uses world-sensing capabilities; the project’s Info.plist file (Info within the Project Navigator) needs to be updated with the key NSWorldSensingUsageDescription, along with a string prompt to ask for permission.
Adding the SceneReconstructor Class
Add a new Swift file named SceneReconstructor to your project. Save the file to the same location as the other Room Virtualizer Swift files. Leave the other settings at their defaults.
Open the SceneReconstructor.swift file in the Xcode editor then enter the code in LISTING 8.2.
LISTING 8.2 Tracking Shapes Detected by the Vision Pro
import ARKit
import RealityKit
import Foundation
@MainActor class SceneReconstructor: ObservableObject {
private let session = ARKitSession()
private let sceneData = SceneReconstructionProvider()
private var entityMap: [UUID: Entity] = [:]
@Published var parentEntity = Entity()
func startReconstruction() async {
try! await session.run([sceneData])
if SceneReconstructionProvider.isSupported {
for await update in sceneData.anchorUpdates {
switch update.event {
case .added, .updated:
let shape = try! await
ShapeResource.generateStaticMesh(from:
update.anchor)
updateMesh(update.anchor, shape: shape)
case .removed:
removeMesh(update.anchor)
}
}
}
}
func updateMesh(_ anchor: MeshAnchor, shape: ShapeResource) {
if entityMap[anchor.id] == nil {
let entity = Entity()
let meshEntity = ModelEntity(mesh:
anchorToMeshResource(anchor))
let material = SimpleMaterial(color: .red, isMetallic: true)
meshEntity.collision = CollisionComponent(shapes:
[shape], isStatic: true)
meshEntity.components.set(InputTargetComponent())
meshEntity.model?.materials = [material]
meshEntity.physicsBody =
PhysicsBodyComponent(mode: .static)
entity.addChild(meshEntity)
entityMap[anchor.id] = entity
parentEntity.addChild(entity)
} else {
let entity = entityMap[anchor.id]!
let meshEntity = entity.children[0] as! ModelEntity
meshEntity.collision?.shapes = [shape]
meshEntity.model?.mesh = anchorToMeshResource(anchor)
}
entityMap[anchor.id]?.transform = Transform(matrix:
anchor.originFromAnchorTransform)
}
func removeMesh(_ anchor: MeshAnchor) {
entityMap[anchor.id]?.removeFromParent()
entityMap.removeValue(forKey: anchor.id)
}
func anchorToMeshResource(_ anchor: MeshAnchor) -> MeshResource {
var desc = MeshDescriptor()
let posValues = anchor.geometry.vertices.asSIMD3(ofType:
Float.self)
desc.positions = .init(posValues)
let normalValues = anchor.geometry.normals.asSIMD3(ofType:
Float.self)
desc.normals = .init(normalValues)
do {
desc.primitives = .polygons(
(0..<anchor.geometry.faces.count).map { _ in UInt8(3) },
(0..<anchor.geometry.faces.count * 3).map {
anchor.geometry.faces.buffer.contents()
.advanced(by: $0 *
anchor.geometry.faces.bytesPerIndex)
.assumingMemoryBound(to: UInt32.self).pointee
}
)
}
let meshResource = try! MeshResource.generate(from: [desc])
return(meshResource)
}
}
extension GeometrySource {
func asArray<T>(ofType: T.Type) -> [T] {
assert(MemoryLayout<T>.stride == stride,
"Invalid stride \(MemoryLayout<T>.stride); expected \(stride)")
return (0..<self.count).map {
buffer.contents().advanced(by: offset + stride *
Int($0)).assumingMemoryBound(to: T.self).pointee
}
}
func asSIMD3<T>(ofType: T.Type) -> [SIMD3<T>] {
return asArray(ofType: (T, T, T).self).map
{ .init($0.0, $0.1, $0.2) }
}
}
The logic should be obvious by now: An ARKit session is created along with an instance of SceneReconstructionProvider (sceneData). Supporting data structures parentEntity and entityMap hold all the mesh model entities and a mapping between anchor IDs and model entities, respectively.
The startReconstruction function first verifies you have permission to monitor the environment (SceneReconstructionProvider.isSupported). Assuming there are no issues, it waits for an incoming MeshAnchor and calls updateMesh or removeMesh depending on whether an anchor has been updated/added or removed. For new and updated meshes, a shape is created; this is the collision shape you can easily generate from the anchor.
When updateMesh is called, the shape and the anchor are provided as arguments. The function checks entityMap to see if the anchor has been seen before. If it hasn’t, a new entity is created—our version of an AnchorEntity. A ModelEntity named meshEntity is defined with the generated collision shapes, a metallic red color, a physics body, and an input target component.
The meshEntity is then added to entity, which, in turn, is added to the published parentEntity.
If an anchor has been seen before and needs an update, the code fetches the entity from entityMap, grabs the meshEntity from that, and changes its collision shapes to the updated shape as well as updating the visible model mesh with anchorToMeshResource.
When a MeshAnchor is no longer being tracked, the removeMesh function removes the entity (and the ModelEntity it contains) as well as any entityMap references to it.
The remainder of the code (anchorToMeshResource, asArray, and asSIMD3 functions) is provided as-is with minor modifications from the community code at https://github.com/XRealityZone/what-vision-os-can-do/blob/ed7adb8c281d68aaf2cdc472986127fc11f44cca/WhatVisionOSCanDo/ShowCase/WorldScening/WorldSceningTrackingModel.swift#L70.
RealityView { content in
content.add(sceneReconstructor.parentEntity)
}
Finish up by adding a task that starts scene reconstruction immediately after the RealityView block. Apple indicates that any scene reconstruction tasks should be started with low priority, which you can indicate with the priority argument:
task(priority: .low) {
await sceneReconstructor.startReconstruction()
}
You can now run the application on your Apple Vision Pro and watch as your familiar surroundings are turned into a metallic red nightmare.
Congratulations! You’ve built hand-tracking and scene reconstructions classes that can be used in future applications. Let’s wrap up by building an application that uses these classes to build a fully interactive physics playground that blends virtual and reality seamlessly.