Reconstructing Reality

This chapter is from the book

Apple Vision Pro for Creators: A Beginner's Guide to Building Immersive Experiences

Learn More Buy

This chapter is from the book

This chapter is from the book 

Apple Vision Pro for Creators: A Beginner's Guide to Building Immersive Experiences

Learn More Buy

Hands-On: Reconstruction

One of the nice things about creating reusable code is that once it is built, it can just be used without thinking about it again. By reusing the HandTracker and SceneReconstructor classes in this project, you can focus solely on the functionality you want to provide without getting into the nitty-gritty of data providers and ARKit sessions and all that fun. You just get to build and play.

This exercise is designed to be a playground for you, the developer. You can try different indirect and direct object interactions, mess with gravity, and just practice with all the capabilities you’ve been learning throughout the book.

In this project, Reconstruction, you use tap gestures to drop random objects from your fingertips. Then you can (carefully) use your hands to scoop up the objects and move them, flick them around, or use an indirect gesture to pick them up and position them throughout the environment. Using scene reconstruction, the application considers the shapes in your space in its physics simulation and virtual objects react to physical objects as you’d expect.

The finished project will likely result in a significant mess around your room, as shown in FIGURE 8.4. Thankfully, cleaning up is just a matter of closing the application.

FIGURE 8.4 Place and interact with random objects scattered around your room.

Setting Up the Project

Create a new Mixed Immersive project named Reconstruction.

If desired, update the ContentView.swift file to include an introduction and the <App Name>App.swift file to size the content appropriately.

Remove the extra code from the ImmersiveView.swift file. Be sure the RealityView code block is empty.

This project needs both world-sensing and hand-tracking; update the project’s Info.plist file to include keys and string values for NSWorldSensingUsageDescription and NSHandsTrackingUsageDescription.

Adding the HandTracker and SceneReconstructor Classes

Now add the HandTracker.swift and SceneReconstructor.swift files you created in the previous two projects to the Reconstruction project. The easiest way to do this is to choose File, Add Files to “Reconstruction” from the Xcode menu. When prompted, as shown in FIGURE 8.5, drill down into the Hand Skeleton project and select the HandTracker.swift file.

FIGURE 8.5 Importing the class files to the Reconstruction project

Leave the other settings with their defaults (Copy Items, Create Groups, and Add to Targets are all selected) and then click Add. The file now appears in your Xcode Project navigator. Repeat these steps for SceneReconstructor.swift file found within the Room Virtualizer project.

Before going any further (and before I forget), open the newly added SceneReconstructor.swift file and comment out the material definition for the red metallic surfaces by adding two forward slashes to the line:

// let material = SimpleMaterial(color: .red, isMetallic: true)

Add a new line that initializes material to the occlusion material:

let material = OcclusionMaterial()

This is typically the material you would want in the SceneReconstructor class. The red metallic material was only used to help visualize what the reconstruction was doing behind the scenes.

Generating Random Objects

In this project, I’m finally breaking free from the shackles of the lowly sphere and adding in codes, cylinders, and cubes (oh my!). The generateRandomSphere function you’ve used repeatedly is evolving to handle the additional shapes. Let’s get this crucial functionality out of the way now. Edit ImmersiveView.swift to include the new generateRandomObject function (and the old getRandomColor function) in LISTING 8.3. These should be placed inside the ImmersiveView struct, near the very bottom.

LISTING 8.3 Randomizing Objects and Materials

func generateRandomObject() -> ModelEntity {
    var object: ModelEntity
    let randomChoice = Int.random(in: 0...3)
    switch randomChoice {
    case 0:
        object = ModelEntity(mesh: .generateSphere(radius:
                            Float.random(in: 0.005...0.025)))
    case 1:
        object = ModelEntity(mesh: .generateCone(height:
                            Float.random(in: 0.01...0.09),
                            radius: Float.random(in: 0.02...0.03)))
    case 2:
        object = ModelEntity(mesh: .generateCylinder(height:
                            Float.random(in: 0.01...0.09),
                            radius: Float.random(in: 0.02...0.03)))
    default:
        object = ModelEntity(mesh: .generateBox(size:
                            Float.random(in: 0.01...0.05),
                            cornerRadius: Float.random(in: 0.0...0.009)))
    }

    let material : SimpleMaterial = SimpleMaterial(color:
                       getRandomColor(),
                       roughness: MaterialScalarParameter(
                       floatLiteral: Float.random(in: 0.0...1.0)),
                       isMetallic: Bool.random())
    object.model?.materials = [material]
    object.generateCollisionShapes(recursive: true)
    object.components.set(GroundingShadowComponent(castsShadow: true))
    object.physicsBody = PhysicsBodyComponent(
        massProperties: PhysicsMassProperties(mass: 2.0),
        material: .generate(friction: 1.0, restitution: 0.1),
        mode: .dynamic)
    object.physicsBody?.angularDamping = 0.1
    object.physicsBody?.linearDamping = 0.1
    return object
}

func getRandomColor() -> UIColor {
    let red = CGFloat.random(in: 0...1)
    let green = CGFloat.random(in: 0...1)
    let blue = CGFloat.random(in: 0...1)
    let color = UIColor(red: red, green: green, blue: blue, alpha: 1.0)
    return color
}

There are three primary additions to the generateRandomObject function versus the sphere-centric version you’ve been using.

First, you define a generic ModelEntity named object. To decide what kind of object it will be, a random integer between 0 and 3 is calculated and stored in randomChoice. A switch statement handles generating models from each of the possibilities of randomChoice:

0: Sphere

1: Cone

2: Cylinder

3 (or other): Box

The parameters (radius, height, and so on) of each shape are also randomized so that the appearance changes for each model entity created. The new randomized model is stored in object.

The second change is that you use a new component with the object. For the first time, you cast shadows with the GroundShadowComponent:

object.components.set(GroundingShadowComponent(castsShadow: true))

The final change is to define slightly more physics than you have in the past:

object.physicsBody = PhysicsBodyComponent(
    massProperties: PhysicsMassProperties(mass: 2.0),
    material: .generate(friction: 1.0, restitution: 0.1),
    mode: .dynamic)
object.physicsBody?.angularDamping = 0.1
object.physicsBody?.linearDamping = 0.1

Within the PhysicsBodyComponent, I specify a mass of 2.0 kilogram. I found this value helpful for keeping the objects from bouncing everywhere at the slightest touch. Friction is set high (1.0), and restitution (bounciness) is low at 0.1. The physics mode is dynamic, meaning the objects can fully receive and transmit energy through collisions.

You also alter two additional physics body properties: angularDamping and linearDamping, which are values between 0 and infinity that define how quickly an object slows down when it is spinning or moving, respectively. You can play with all these values to see their effects. I used what I found to offer a pleasing experience after much trial and error.

The rest of the generateRandomObject (and getRandomColor) code is the same that you’ve already seen and used many times before.

Initializing the Data Providers

With the supporting functions under control, it’s time to initialize and start the two data providers via the HandTracker and SceneReconstructor classes. At the top of the ImmersiveView struct, add these lines:

@ObservedObject var sceneReconstructor = SceneReconstructor()
@ObservedObject var handTracker = HandTracker()

Use normal (hand-tracking) and low-priority (scene reconstruction) tasks to start each of the detectors running. Add these lines directly following the RealityView code block:

.task() {
    await handTracker.startHandTracking()
}
.task(priority: .low) {
    await sceneReconstructor.startReconstruction()
}

Now, all you need to do is make the application do something interesting. You have two data detectors up and running, so let’s make use of them.

Defining the Hand Objects

One of my goals with this project was to try to enable the user to use their hands to interact with the objects added to the Reality View using just the physics simulation. This isn’t (currently) a particularly easy thing to do because your hands can’t feel objects if you try to pick them up. Squeeze too hard and the object “squirts” out of your fingers. For this reason, I’ve decided to add a plane to the palms of my hands so that I can “scoop” objects into a hand or pick them up and drop them into a hand. In addition to the plane, adding spheres for the joints aids in the interactivity (and provides the ability to flick objects around or pull them toward you).

Importing ARKit

Because you need to access the finger joints by name, you need ARKit imported into the ImmersiveView.swift file. Add the required import line following the other import statements:

import ARKit

Creating Objects and Materials

Within the RealityView code, define the material to use for the finger joints as well as a fingerObject model entity that can be copied and used at each joint. This is virtually identical to what you did in the Hand Tracker project but with some additional physics properties and a clear material:

let material = UnlitMaterial(color: .clear)

let fingerObject = ModelEntity(
    mesh: .generateSphere(radius: 0.005),
    materials: [material]
)
fingerObject.physicsBody = PhysicsBodyComponent(
    massProperties: .default,
    material: .generate(friction: 1.0, restitution: 0.0),
    mode: .kinematic)
fingerObject.generateCollisionShapes(recursive: true)

This setup gives you a high-friction sphere you can use with your finger joints. The spheres are clear, so you can’t see them, but they’ll be able to interact with other objects. Note that the physicsBody mode is set to .kinematic, which means the object is being controlled by the user.

Next, define a palmObject that is used to cover the palm. It’s a plane and uses the same clear material and physics properties as the finger joints. Add this code following the fingerObject definition:

let palmObject = ModelEntity(mesh: .generatePlane(width: 0.09, depth: 0.09),
materials: [material])
palmObject.physicsBody = PhysicsBodyComponent(
        massProperties: .default,
        material: .generate(friction: 1.0, restitution: 0.0),
        mode: .kinematic)
palmObject.generateCollisionShapes(recursive: true)

You now have a finger and a palm object that are configured and can be used for your finger joints and palms.

Adding the Palm Entities

The location of the palm is based on the wrist joint, but it’s going to be offset slightly from the wrist so that it roughly covers the average person’s palm. Define rightPalmObject and leftPalmObject as clones of the PalmObject and then adjust their positions like this:

let rightPalmObject = palmObject.clone(recursive: true)
let leftPalmObject = palmObject.clone(recursive: true)

leftPalmObject.position.x += 0.07
leftPalmObject.position.y += 0.02
rightPalmObject.position.x -= 0.07
rightPalmObject.position.y -= 0.02

These positions, like so many other things, were a matter of trial and error. You can set the color of the material to something other than clear and see for yourself where they sit. You may want to adjust them further for your needs.

Next, add the left and right palm objects to the wrist entity contained in the handTracker.leftHandParts and handTracker.rightHandParts.

handTracker.leftHandParts[.wrist]!.addChild(leftPalmObject)
handTracker.rightHandParts[.wrist]!.addChild(rightPalmObject)

Finally, add left and right wrist entities to the RealityView content:

content.add(handTracker.rightHandParts[.wrist]!)
content.add(handTracker.leftHandParts[.wrist]!)

Adding the Finger Joint Entities

The finger joints are handled with a loop, just as you did with the Hand Tracker project. Iterate through the joint names, accessing each entity in rightHandParts and leftHandParts. For each entity, the code adds a child containing a clone of the fingerObject ModelEntity:

for joint in HandSkeleton.JointName.allCases {
    handTracker.rightHandParts[joint]!.addChild(
                              fingerObject.clone(recursive: true))
    handTracker.leftHandParts[joint]!.addChild(
                              fingerObject.clone(recursive: true))
    content.add(handTracker.rightHandParts[joint]!)
    content.add(handTracker.leftHandParts[joint]!)
}

Each entity in each hand is then added to the RealityView content.

Managing the User-Added Objects

Each object (sphere, cylinder, box, sphere) a user creates will be added to a parent entity named worldObjects. Define this variable at the top of the ImmersiveView struct:

private var worldObjects = Entity()

After the content additions you’ve already made, set worldObjects to be an input target for indirect gestures. This is used in conjunction with a drag gesture to move objects around. Finally, add worldObjects to the content:

worldObjects.components.set(InputTargetComponent(
                                       allowedInputTypes: [.indirect]))
content.add(worldObjects)

As objects are added to worldObjects, they subsequently appear within the RealityView.

Adding the Scene Reconstruction Shapes

The other objects you need to include in the content are possibly the most important: the scene reconstruction model entities. Without these, user-added objects have nowhere to land, so they will fall… and fall…. and fall.

Add the sceneReconstructor.parentEntity to the RealityView code as well:

content.add(sceneReconstructor.parentEntity)

The code is in place to store user-added models, finger joints and palms, and the surfaces that make up the environment. The remainder of the project is setting up the gestures that turn the environment into a playground of shiny trinkets.

Creating Random Objects with the Tap Gesture

When a user wants to add an object to the environment, they perform a tap (pinch) gesture with either of their hands. The object is created and appears to fall from their hand position. In general, objects fall from the hand that performs the gesture—or at least the hand that is being looked at when the gesture is detected.

Add a SpatialTapGesture after the closing brace in RealityView, as in LISTING 8.4.

LISTING 8.4 Detect and React to Tap Gestures

.gesture (
    SpatialTapGesture(count: 1)
        .targetedToAnyEntity()
        .onEnded { event in
            var releaseLocation = Transform()
            let tapLocation3D = Point3D(event.convert(event.location3D,
                                       from: .local, to: .scene))

            let distanceToRight = tapLocation3D.distance(to:
                        Point3D(handTracker.
                        rightHandParts[.indexFingerTip]!.position))
            let distanceToLeft = tapLocation3D.distance(to:
                        Point3D(handTracker.
                        leftHandParts[.indexFingerTip]!.position))

            if distanceToLeft<distanceToRight {
                releaseLocation =
                   handTracker.leftHandParts[.indexFingerTip]!.transform
            } else {
                releaseLocation =
                   handTracker.rightHandParts[.indexFingerTip]!.transform
            }

            let object = generateRandomObject()
            object.transform = releaseLocation
            object.position.y = object.position.y - 0.05

            worldObjects.addChild(object)
        }
)

The gesture block starts by declaring that a SpatialTapGesture with a count of 1 is the trigger. The gesture is then targeted to any entity with the .targetedToAnyEntity() modifier.

When the tap gesture ends (.onEnded), the calculations begin.

First, a release location (releaseLocation) for the random object is defined as an empty transformation matrix. Keep in mind that this is a transformation matrix, so it also carries orientation (rotation) information in addition to the location.

In this gesture, I make use of several instances of Point3D, a data structure containing x, y, and z coordinates in 3D space. Point3D also offers a useful distance function that calculates the distance to another Point3D.

The first use is in tapLocation3D, a Point3D data structure derived from the location where the spatial tap event took place, converted into world coordinates. Values distanceToRight and distanceToLeft are subsequently assigned using the Point3D distance function to find the distance between the tapLocation3D and the tip of the index finger on both the right and left hands.

If distancetoLeft value is larger than distanceToRight, you set the releaseLocation to be the same as the transform matrix of the left index finger entity. If not, you set it to the transform matrix of the right index finger.

Lastly, an object is generated from the generateRandomObject function, and its transform matrix is set to releaseLocation. For good measure, the object is lowered by adjusting its y position. This ensures that the object appears below the user’s physical hand.

Finally, the object is added as a child to worldObjects, at which point it appears in the environment and falls to the surface below it.

The project is now in a testable state and can be launched on the Vision Pro. You should be able to add objects, interact with them, and move them around with your hands.

As I mentioned earlier, however, trying to pick up objects with your fingers can lead to frustration. You add one more gesture: an indirect drag gesture that will make it easier to grab and move any object anywhere in the environment.

Dragging Objects

The last major piece of functionality needed in the application is the ability to look at individual objects, and then drag them to other locations (including dropping them in a user’s hands.) To do this, you use a second gesture—DragGesture—targeted to the worldObjects entity that contains anything a user adds to the environment.

Dragging objects that are moving or under the effect of gravity can have some strange side effects, so part of the code needs to “turn off” gravity for the duration of the drag.

Add the second gesture code block in LISTING 8.5 directly after or before the SpatialTapGesture.

LISTING 8.5 Reposition Objects with a Drag Gesture

.gesture(
    DragGesture()
        .targetedToEntity(worldObjects)
        .onChanged { event in
            let object = event.entity as! ModelEntity
            object.physicsBody?.isAffectedByGravity = false
            object.physicsBody?.angularDamping = 1.0
            object.physicsBody?.linearDamping = 1.0
            object.position = event.convert(
                   event.location3D, from: .local, to: .scene)
        }
        .onEnded { event in
            let object = event.entity as! ModelEntity
            object.physicsBody?.isAffectedByGravity = true
            object.physicsBody?.angularDamping = 0.1
            object.physicsBody?.linearDamping = 0.1
        }
)

In this gesture, you make use of both the .onChanged and .onEnded events. In .onChanged, you assign object to the entity referenced by the event (event.entity). You typecast the entity to ModelEntity because you know that the objects added are model entities, and you need to access specific features of model entities, namely the physics body.

Next, these lines “turn off” gravity and stop any spin or other motion on the object:

object.physicsBody?.isAffectedByGravity = false
object.physicsBody?.angularDamping = 1.0
object.physicsBody?.linearDamping = 1.0

If the changes to the physics body are not included, the object moves in unexpected ways while it is being dragged.

During the drag, the object’s position is updated in to match the event’s location3D attribute but converted to world coordinates.

When the drag gesture ends (.onEnded), you once again assign object to the entity targeted by the drag and reset its physics properties to their defaults. This means that gravity once again takes effect, and the object falls onto the nearest surface.

For an interesting effect, you can try leaving gravity disabled. Objects can then be positioned in the air and just hang in empty space. It’s cool, but do you really need any new ways to make a cluttered mess of your homes and office?

Cleaning Up

One last block and you’re done! After ImmersiveView is dismissed, you need to remove the entities you’ve added outside of the initial RealityView setup.

Add the code in LISTING 8.6 as yet another modifier to the RealityView, similar to what you’ve done in other projects:

LISTING 8.6 Remove Entities from the RealityView

.onDisappear {
    worldObjects.children.removeAll()
    for joint in HandSkeleton.JointName.allCases {
        handTracker.rightHandParts[joint]!.children.removeAll()
        handTracker.leftHandParts[joint]!.children.removeAll()
    }
    sceneReconstructor.parentEntity.children.removeAll()
}

This removes all worldObjects, all finger joints, and the surfaces added by the scene reconstruction, leaving a blank canvas for when the immersive view is opened again.

Run the application on your Apple Vision Pro and try scooping, throwing, and making a mess with the randomly generated objects. Cleaning up after throwing a tantrum just got much easier!