Starting with Apple Arcade, Light Brick Studio’s LEGO Builder’s Journey, an immersive multiplatform puzzle game, has consistently built momentum. Here’s what the team learned when they embraced the new era of spatial computing and brought the title to Apple Vision Pro.

How does a studio that’s new to mixed reality build for the immersiveness of Apple Vision Pro while staying true to the gameplay experience on other platforms?

Since LEGO® bricks are real physical products, Light Brick Studio was cognizant of the challenges of creating an experience where players interact with gameplay elements in the room. For the game’s Apple Vision Pro release, time was also a constraint. While the team had executed other platform ports with 2D interfaces within a month or two, the three-month timeline was cutting it close for this new entry into spatial computing.

Navigating a new dimension

“Once you know the platform’s boundaries for the game and its gameplay design, you can pretty much create any kind of experience you want,” explains Mikkel Fredborg, the technical lead at Light Brick Studios.

For the team, bringing a game to visionOS was “was mostly a technical challenge of getting it to run smoothly and then taking advantage of the full experience.” It meant implementing multiple technical reworks, including a full redesign of the user interface (UI) to place once-flat UI elements in the room. The team also had to update their rendering tools and methods to ensure the visual quality and immersiveness of the final product. Balancing both the Unity and RealityKit systems in parallel added further hurdles to performance optimization.

“We’ve done a lot of different ports, and this one was the most complex because Apple Vision Pro is very different from the other platforms,” says Karsten Lund, the managing director at Light Brick Studio. “The expectation was to see how far we could push the experience, and it worked out really well.”

The results

• Built the title’s 80 levels and 60,353 LEGO bricks for Apple Vision Pro in less than 100 days
• Saved weeks of development delay with support from the Unity PolySpatial team
• Kept all gameplay areas intact with only specific performance optimizations but no changes to the overall flow of the game
• Implemented a smooth transition with an open mixed reality style that grows more immersive as the player progresses

Nintendo Switch is a registered trademark of Nintendo.

Reworking the UI and gameplay design

LEGO Builder’s Journey has a minimal UI, containing at most 20 buttons for actions like loading saved games and restarting a level, so reworking it was a lower lift than it would be for many games. They did the UI redesign in Unity, taking GameObjects and attaching colliders to them to get the input to work properly while mimicking the design language of visionOS.

The team also had to take body movement flexibility into account. Except for the UI, everything in the game is built from LEGO bricks in a tool called LEGO Digital Designer. The team has always tried to simulate the 3D space in a 2D surface, but with Apple Vision Pro, they had to account for the fact that a player could move around anywhere. To accommodate a full 360-degree experience, the team reworked existing models designed to be viewed only from specific angles to ensure there weren’t any holes in the model geometry. To reduce draw calls, multiple meshes were also combined into single GameObjects.

Fredborg says, “We were able to take into account that a player can walk all the way around and then do a full 360 degree optimization of the model. We got rid of all the geometry that was inside the bricks when we imported them into the game, so we removed all the interior details, as you never see them. We were able to take the automated tools that we have and adjust them to work with the full simulation experience.”

Managing placement in Bounded mode

Early on, the team decided to design in Bounded mode, where they had to operate within a defined volume. If they attempted to work outside of that volume, elements were clipped and went unseen. They decided to embrace the limitations of Bounded mode, where there is no information about the player’s head, eye, or hand position/orientation unless an interaction is performed. Once the player performs a pinch gesture, the game knows where the hand is and which object is being interacted with. When the pinch ends, the information also ends. In that sense, it’s similar to a touchscreen, where you only know what the player is doing while they touch the screen.

“We didn’t know what people were looking at, which was a big change, especially since our gameplay necessitates having players look at a brick to select it,” says Fredborg. “Once we worked around all of these boundaries, it worked really well and it sits well within the system. It ended up being really nice to operate in this way since everything gets led in the same sort of user experience direction.”

Giving MaterialX shaders a hand

In the earliest versions of LEGO Builder’s Journey, the team used a custom version of Unity’s Lightweight Render Pipeline (LWRP) and then moved to the High Definition Render Pipeline (HDRP) for Xbox and PlayStation ports. For Apple Vision Pro, they had to first convert everything over to be based in the Universal Render Pipeline (URP) and Shader Graph, then transfer over to Apple’s RealityKit.

To do so, they translated shaders authored in Shader Graph into MaterialX shaders they could load into RealityKit. But the team ran into obstacles, finding that the results weren’t well optimized and needed more work than anticipated.

They had planned for hand optimization but found that they needed to automate the process for scale. They would do manual patches to the shaders, find duplicated code, then replace that section of the code to ensure that it would only be executed once. “It was a challenge to get things running smoothly,” admits Fredborg, “but I think the PolySpatial team is doing a great job in moving forward to get things optimized.”

Finding the best light

Due to the game’s short turnaround time they needed to find solutions on the fly. “We had to find solutions to push up the game’s realism and still support Realtime Lights and shadows,” says Fredborg. “That was one of the bigger changes.”

At first, they took the image-based lighting from the Apple Vision Pro, with the goal of adding it correctly on the materials they wanted to include, like plastic and different variations. They had to address tricky-to-solve-for elements like fingerprints and scratches on the bricks.

Since the game was initially developed for early iPhones, it has very simple lighting. The team used a volume texture – a 3D texture with ambient occlusion baked into it – so they could sample from anywhere in the scene. They would sample points and analyze how much ambient occlusion was happening. Since 3D textures are not supported on Apple Vision Pro, they had to convert the volume texture to 2D textures and do manual sampling from those. This is now an automated process in PolySpatial, but it was not available when the Light Brick Studio team started.

The team started by using plastic brick shading and this volume texture sampling system. Then they incorporated Realtime Lights, and, as the volume texture enveloped the scene, they needed to modify the outside lighting. When they started development, it wasn’t possible to overwrite the image-based lighting, so they had to sample a different cube map and then fade the light out from the image-based lighting from RealityKit.

“What we were doing wasn’t really possible, but we found a workaround by turning down the smoothness of the brick and upping different values before we sent it over to the RealityKit lighting,” explains Fredborg. “Developers will have an easier time since overwriting the image-based lighting is now possible in PolySpatial and RealityKit.”

Creating a soft shadow

The Light Brick Studio team also had to find a solution for shadows. While shadows can be added to certain elements in the Apple Vision Pro, it’s not standard. To do it, the team rendered the shadow in Unity as a height map or depth map, sent it over to RealityKit, and resampled it there. They rendered the scene as a shadow map on the Metal backend in Unity, then passed it over to the RealityKit rendering side.

“What you can push through rendering is limited if you still need to hit 90 frames per second. We did a very small shadow map, 64 x 64 pixels, and then we combined the mipmap levels into the RGBA channels of a single texture, which can then be sampled with a single sample in RealityKit to produce something that looks like a soft shadow,” explains Fredborg. “So when the brick is very close to the surface, the shadows are quite sharp, but when it’s up, the shadow’s softer. We did that with one texture sample.”

For the Realtime Lights from Unity, they used the PolySpatial Lighting Node in Shader Graph, which expanded into multiple Shader Graph nodes once it was converted to RealityKit.

“For RealityKit, the only way to alter shaders is through a Shader Graph-like interface,” says Fredborg. On the Unity side, the team found that it was possible to create a custom code function in Shader Graph, and, so long as this followed certain specific rules, it can be converted over to a MaterialX node graph afterwards. “This really helped us a lot,” he continues, “otherwise it was going to be a spaghetti monster inside a spaghetti monster in Shader Graphs.”

Optimizing at 90 fps

As the team was pushing visuals to a new dimension, they experienced some hurdles to keeping the platform running smoothly. Development ran as a two-part system, passing everything from Unity over to RealityKit, which has different rendering techniques. This meant that Unity handled the game logic, while RealityKit handled rendering the visuals, and optimization was needed to run smoothly at 90 frames per second to be smooth. One of the main challenges was identifying whether a bottleneck came from the Unity side or RealityKit side, and it would take the team multiple tries to find the solution.

“In the beginning, the PolySpatial package would reflect everything we were doing in Unity over to RealityKit, so every GameObject or transform would be there, and at that time we weren’t able to turn off specific layers from being transferred or reflected. It mirrored everything from the UI rendering, canvas rendering and animator particle system,” explains Fredborg. “Now, there’s a filter where you can set it to not track specific layers, but at that time, we had to figure out a way to clean everything out that we weren’t going to see on the RealityKit side.”

The team needed to optimize GameObject hierarchies. On other platforms, bricks are three or more linked GameObjects, but for visionOS, they set a maximum of two GameObjects per brick to limit the back and forth between Unity and RealityKit. This worked really well and was a big win for the team.

Enhancing stop motion animation was also a highlight. The game contains animated waterfalls that were exchanging meshes on bricks. They consisted of hundreds of different GameObjects, which were quite slow. “We ended up addressing it in a staggered way instead of doing every frame for every GameObject,” says Fredborg. “We’d wait and do it every few frames between the different waterfall streams. It helped a lot.”

Finding the best solutions and partner

Throughout their journey building for the Apple Vision Pro, the Light Brick Studio team had to stay nimble, solve problems quickly, and iterate to get things right.

“I think you have to wear the glasses to understand it,” says Fredborg. “We recommend doing multiple builds and using the device as much as possible to focus on touch and movement. You have the simulator that you can run on the Mac to get a sense of how things are working, but the input is different. It’s a much more physical platform.”

In the end, the team was really happy with how the game turned out, in part thanks to working with a strong support partner. “The PolySpatial team was very responsive both on Slack and in the discussion forums,” he continues. “When we were blocked, they would get back to us with solutions quickly, which was really helpful. They were really supportive and wanted to make the best version of PolySpatial possible.”

Build for Apple Vision Pro today with Unity 2022 LTS

Talk to our team to learn how we can help you leverage Unity’s powerful tools and workflows to build compelling spatial experiences.