How conversational AI in real-time 3D industrial applications can drive faster, better-informed decisions across teams

Nov 20, 2026|10 Min

Shanti Zachariah - Unity Technologies

Senior content marketing manager

Matti Gruener, researcher, the MIT Media Lab

This Unity U/Day Houston presentation is by Matti Gruener, a researcher from the MIT Media Lab.

Drawing on his extensive background in visual effects, augmented reality, and current research at the MIT Media Lab, Matti Gruener examines AI in real-time 3D through two lenses: AI agents that assist in building applications, and embedding AI directly into runtime applications as interactive, conversational partners.

On the application development side, utilizing tools like Unity CLI enables developers to delegate complex build and visualization tasks directly to AI agents, rather than opening the Unity Editor continuously.

Embedding conversational AI in real-time 3D creates intuitive systems that act as assistants to the end-users. By integrating large, multi-dimensional datasets, such as decades of lunar research, synthetic urban movement, or geological mining models – examples from Matti Gruener’s work – users can easily query, filter, and modify 3D representations through natural language in real time.

Key takeaways

  • Matti Gruener gives an overview of how Unity CLI has significantly sped up his Unity workflows, enabling him to automate multiple tasks through AI agents and verify outputs using simple HTML reporting.
  • He steps through three key example projects from the MIT Media Lab:
    • A lunar AR experience together with NASA
    • A building and urban design visualization
    • A simulation of a lithium mine
  • Each of these three applications, together with embedded conversational AI, empowers the user to ask for filtering, modeling, and rendering of visual responses through plain-language prompts. An example is a user in the lithium mine visualization who requests that the AI generate possible pit geometries, road planning, or economic forecasts.
  • A goal with his research is to develop specialized architectures for AI systems that need to operate in regulated industries, with the goal of zero fabricated data, so users can trust the output and take high-precision, data-driven decisions.
  • Conversational AI assistants can help bridge knowledge gaps across cross-functional teams, allowing diverse stakeholders, technical experts, and non-technical members to interpret complex spatial information with the goal of driving more efficient and accurate decisions.
Conversational AI in real-time 3D

Full transcript

Speaker: Matti Gruener, researcher, The MIT Media Lab

Notes:

  • This is a lightly edited transcript of a recorded presentation.
  • Iron Man and JARVIS are used for illustrative purposes only. Not affiliated with or endorsed by Marvel/Disney.

Introduction

[0:00] My name is Matti. I currently do research at the MIT Media Lab. Before going back to school and going back to MIT, I spent about 15 years in varying industries, about 10 in visual effects, where I worked on one of the Harry Potter movies, The Mandalorian, Game of Thrones; then I left visual effects at some point, and was then in fashion technology and eventually in augmented reality and artificial intelligence. So in the context of Unity applications, I want to look at AI from two different perspectives.

[0:58] The (first perspective) is the one on the left where AI helps us build these applications. So we have these agentic workflows that build applications and executables for us. And then the other one, on the right, is embedding AI into the application itself. We'll focus on the right, but I'll talk about the left just for a minute, about these agentic workflows to build applications for a second.

Building applications with Unity CLI and agentic workflows

[1:30] Okay. So I admit that I've been all in on AI for a little while. MIT is crazy busy. I have to delegate as much as I can to my agents to get through the storm that is MIT on certain days. And I have another confession to make, which is that I barely opened the Unity Editor over the last couple months. So there was a time where I would spend eight hours a day in Unity and those times seem to be gone. I don't know if that was the intended effect when Unity released Unity CLI. But with the release of CLI, just quite naturally, I just stopped opening the Editor.

Unity CLI has fundamentally changed how I approach building applications now.

[2:28] So right now, I instruct my agents to build things through Unity CLI, and then I instruct them, too, to build these HTML pages to provide visual proof that the task that I assigned was actually done. So in this case, I don't focus much on the visuals. I focus more on was the right data picked up, was it put in the right spot, was it treated the right way? And in this case, did the agent understand to produce massing around these node graphs? But those are details. It's more about the workflow and that I'm only looking at HTML pages these days as output to verify that the work has been done.

[3:03] And this has sped up my workflows quite a bit. And I found that, as we said this earlier today, we've gotten more busy since we've gotten all these tools. So they have not made life less busy necessarily, but somehow people like us just end up doing more stuff. So in my case, I often run, I don't know, 10 things at the same time, and this sort of workflow has helped me keep track of that.

[3:32] Having these HTML pages that have context about what I was doing at the time helps me review afterwards if the work has actually been done. So this was more of a procedural example, but I also used this to do creative work. In this case, I have data that I ingested into one of my Unity applications and I'm trying to figure out how I want to visualize that information.

[3:59] I instructed my agents to give me some options for how I might do that. And now I just scroll through and look at what I like about it. There's different color schemes, it's about scale, about whether I use perspective or orthographic cameras. And I can just go through and pick the things that I like. And what I would usually do afterwards when I've done this is to also build an interactive application that exposes all of these parameters that I might want to play with as UI on top of the application.

[4:27] So that makes it very fast for me to dial in the look, for example, and then take a screenshot of the UI, pass that back to my agent, and then tell the agent this is my default look from now on. So that has sped up my work tremendously.

From Iron Man to embedded AI

But I want to switch gears for a second and go back to what I was talking about before: embedding AI into Unity applications. And to set the stage for this, I want to take you back 15 years.

[4:59] I told you I was working in film for a long time and 15 years ago I was working on this sequence. So this is a very short clip from Iron Man 3. And it's a suit assembly test where Tony Stark is trying to figure out this new system for assembling suits on his body. Great sequence. That's why I'm showing it. But I'm also showing it because Tony has this AI agent that he's working with.

[5:31] I think Iron Man is probably the prime example of an engineer working with an AI-assisted system that can visualize information all around him to achieve these great engineering feats. And if you remember, JARVIS, (the AI assistant from the franchise) is able to bring in all this information. It goes away and runs asynchronous tasks, simulations, analyses, and brings that back into this magical XR environment. But Tony can see all that information.

[6:00] It (JARVIS) helps with answering questions and in this case makes a lot of snarky comments about Tony's safety practices. And let's go to this. So this is what this looks like, right? So in practice, there's information all around this robot in this case. And back then when we worked on this film 15 years ago, this was all science fiction. And you might be guessing where I'm going with this.

[6:28] It doesn't feel like that's science fiction anymore. A lot of this stuff is quite feasible today.

A lunar data demo from NASA and reducing AI hallucinations

[6:34] I want to show you a project that I did recently with a fellow researcher at MIT. What you'll see is me talking to a conversational agent in an augmented reality headset. And the agent brings the information about the moon. We ingested four decades of lunar data into the system so we can ask questions about it and get it visualized directly on the lunar surface.

[6:59] I'll play that. The following is audio from the demo that Matti shows the audience.

(Matti in the video): Using the neutron data that we have for the moon, can we explore where there might be water?

(The AI embedded assistant in the application that’s being shown in the video): I am now showing the neutron data, which ranges from 531.52 to 615.83 counts per 32 seconds. Lower neutron counts can indicate the presence of hydrogen, and thus potentially water ice, on the moon's surface. I've applied a thermal gradient to it.

[7:31] So this is a lot less snarky than JARVIS would be in the franchise, but still quite helpful. And you could see how this potentially helps mission planners or educators interact with lunar data.

[8:01] What you just saw was part of a publication that I was able to present at NASA’s Ames Research Center a couple of weeks ago. And what I think is really important to mention is that the architecture that we use in this system is specifically designed to reduce hallucinations.

[8:31] As we enter this world where we have AI agents in these very powerful 3D runtimes, I still want to be able to believe my eyes. Seeing is believing. And I think specifically with systems like this, it can be confusing or misinform people. And we are doing our best to keep this as safe as possible. In our evaluation of this particular work, the agent never showed any fabricated information and barely made any mistakes. And generally the response times were really good. So we were quite happy with this system.

[8:59] I'll show you a few more clips. I switched off audio here to be able to talk over it, but it's the same kind of setup. You just talk to it and it responds. In this case, I'm asking to see flight paths for certain spacecraft that have orbited the moon. I'm not just asking for what I want to see, but also how I want to see it. I'm asking for particular colors, which is a quick way to just get on the same page with the agent and understand what is being visualized.

[9:26] Again, you can see how this could be helpful if you want to maybe plan a mission to the moon, or if you just want to educate people. Here's another one. So this is an example of me asking for a modification of the visualization. In this case, the agent had already visualized all the elevation levels on the moon with this gradient. But I'm interested in particular elevation levels at four kilometers. So I'm asking for a modification to the visualization. And all of this just happens right in front of me as I keep talking to the system.

Urban data and synthetic populations

[10:01] The research group that I'm in is called City Science. We'll talk about an urban project now. This is a synthetic population that moves around in this building that was never built. The population is visualized using these dots that are moving around. Using the system where I can conversationally ask for information, I can ask for a random agent to be visualized on how they move around in the building.

[10:27] In this case, I just ask for a random one. But you could also ask for all the trips that students might take, students under 25, and trips that end at a bar. And that might help you understand why the elevators are always packed at 10:00 PM. So this sort of conversational way of analyzing data helps us to then make these buildings better and make the inhabitants happier and healthier in these buildings because we can analyze what's going wrong more easily. In our research group, we do a lot with synthetic populations and we learn from simulating how they interact with their environments.

[11:02] Embedding conversational agents, like what I'm demonstrating here, has made it much faster for us to then interact with these large data sets because you could simulate, I don't know, 100,000 of these agents. Making sense of that is not that easy.

Here's another example that shows the conversational nature of these interactions quite nicely. So I'm digging deeper and deeper into this visualization. I can just (give) instructions where I want to look at the visualization in

[11:33] different ways. But I can also filter and I can really dig in, isolate certain spaces that the agents move in. All of that is done in a conversational way.

I guess the question is, why should you care about all of this? Some of the conversations that I've had already today kind of indicated that a lot of you have large amounts of spatial information that you might already visualize in Unity. And I think this might be a great way to make sense of that sort of information.

[12:01] The question is, what could you do with an agent that curates information for you as you need it within these powerful 3D runtimes? Could you make decisions faster? Could you make better decisions? And maybe most importantly, could you get on the same page with other people faster by putting this in an XR environment and looking at it with more people, (while having) an agent curate the environment and the data that you want to look at?

[12:30] Everything that I've shown you so far was relatively simple in terms of the questions that were asked. I'm asking for a visualization and then the visualization comes back.

Mining use case and multistep reasoning

We can ramp this up a little as well. This is a mining use case. I'm not a mining or oil and gas person at all. So again, apologies if I use the wrong terminology. But this is a synthetic block model for a lithium deposit in the ground. And the idea here is to pick a point on the surface and to ask, what would happen if we start digging at this particular spot?

[12:58] The agent then goes away in an asynchronous manner, analyzes the data, figures out where I want to dig, and also runs a simulation of what the pit might look like that would emerge if we started digging in this spot. So it has a model in the background that can create these shell geometries. And you could put this on XR glasses, of course.

[13:25] Then five of us could look at it, you know, around the table. This is a different example where the agent calculates the changing shape of the open pit. It makes assumptions about an economic model and changing lithium prices to estimate what these changing pit shapes would be like. So again, it goes away, takes this task, and then comes back with a visual response that helps me understand what the result would be in a very visual way. And you could imagine plugging these models, plugging additional models into these sorts of systems.

[14:02] You could have a model that predicts how dust would move based on wind speeds and wind patterns, or the agent could suggest the location of a tailings site and the consequences on the environmental side, also for the bottom line of people developing in the site. And the agent could show you multiple options and then work with you to hit your targets, whatever those might be, whether those are environmental or economic.

[14:33] A particular research interest of mine is working on these multistep reasoning processes in these systems. An example would be, look at this, you're in this spot, this is where you want to mine. You already understand what the shell would look like, because maybe you are an engineer who understands this sort of stuff. But think about what that would do for road placement and how that would affect your bottom line in the end.

[14:59] So these sorts of multistep reasoning processes where the agent takes you from one place to another and shows you things is a particular interest of mine and I think it's very important. I hope that these examples help illustrate how powerful conversational agents in these environments could be. Because you might have expertise in a particular area and you don't need an agent to tell you something that you already know. But sometimes you work with these agents and there are things that you don't know.

[15:32] Maybe you are quite familiar with the financial modeling, but you're not familiar with the engineering task or vice versa. And as you make engineering decisions, you might want to understand what the downstream effects are on an economic side. If you think back to Tony Stark, he clearly has a lot of expertise in many areas, unrealistic levels of expertise in many areas. But JARVIS is still there to fill the gaps. So JARVIS runs these background tasks, brings in information and fills in all the blind spots that Tony might have. Often these are safety concerns that he's not fully aware of or just ignores.

Closing: Design goals and community engagement

I believe we will see these conversational agents much more in these environments, in quite powerful real-time 3D environments. And ideally we'd have all of these three characteristics in these systems:

[16:29] We want them to be powerful and expressive, and especially in regulated industries, we also want them to be safe and auditable. And then for users, whether they're internal or external users, we want these systems to be fast and intuitive. And as with most stuff, if you build these systems in a naive way, you might get two out of three or maybe just one out of the three. So I think there's a little bit more research that is needed to keep this specifically safe and auditable.

[17:01] But I think if we get this right, it'll be quite amazing. So imagine you can come to a system, ask all the questions about your data, get visual responses, have a system that helps you filter out and focus on data and explains the data to you. I feel quite encouraged to build these sorts of systems because I think they'll be incredibly powerful. You could think about onboarding a new colleague to a construction project that already exists.

[17:29] Finding innovative solutions alongside an AI partner or getting different stakeholders together around the same table and making decisions together and getting everybody on the same page. And on that note, what we've now also done is embed these sorts of systems into community engagement platforms. So in our research group, we have a lot of expertise when it comes to engaging the public so that certain projects can move forward. But there's a lot of resistance currently when it comes to certain construction projects.

[18:00] And we're hoping that with this particular type of system, we can get people to come together and drive consensus faster. Because a lot of the people that come to these sorts of meetings can't read a CAD drawing. But if you can explain this to them and have a personalized system that explains the situation to them and how it affects their personal situation in life, we hope that that gets people on the same page faster. So with that, as you can imagine, we could only scratch the surface.