AI in Virtual Reality Gaming

I just posted an epic, 17-episode Voices of VR podcast series on the intersection of XR & AI featuring lots of immersive artists & XR devs who are looking at how generative AI & conversational interfaces via Large Language Models & GPT are revolutionizing production pipelines & experiential design with intelligent NPCs and AI agents. Things are evolving quickly, and this series should help ground some of the promises and perils of AI. Total run time is 13 hours & 50 minutes that contains a survey over the last 4 months from NewImages, Laval Virtual, AWE, & Tribeca Immersive. (Rough transcripts are included) 1253: XR & AI Series Kickoff with Evo H. on a Promptcraft Guide to Generative Media 1254: Using AI to Upskill Creative Sovereignty with XR Artist Violeta Ayala 1255: Using GPT Chatbots to Bootstrap Social VR Spaces with “Quantum Bar” Demo with Christina Christina "XaosPrincess" Kinne & Guillermo Valle Perez 1256: Using GPT for Conversational Interface for Escape Room VR Game "The Unclaimed Masterpiece" with Alizee Calet & Maël Sellier 1257: Talk on Preliminary Thoughts on AI: History, Ethics, and Conceptual Frames with Kent Bye 1258: Using XR & AI to Reclaim and Preserve Indigenous Languages with Michael Running Wolf 1259: AWE Panel on the Intersection of AI and the Metaverse with Amy LaMeyer, Tony Parisi, Alvin Wang Graylin, & Kent Bye 1260: Using ChatGPT for XR Education and Persistent Virtual Assistant via AR Headsets with Javier Fadul 1261: Using ChatGPT for Rapid Prototyping of Tilt Five AR Applications with CEO Jeri Ellsworth 1262: Using in AI in AR Filters & ChatGPT for Business Planning with Educator Don Allen Stevenson III 1263: MeetWol AI Agent with Niantic, Overbeast AR App, & Speculative Architecture Essays with Keiichi Matsuda 1264: Inworld.ai for Dynamic NPC Characters with Knowledge, Memory, & Robust Narrative Controls with Matthew Kim 1265: Integrating Generative AI into Live Theatre Performance in WebXR with OnBoardXR with Brendan Bradley 1266: Converting Dance into Multi-Channel Generative AI Performance at 30FPS with "Kinectic Diffusion" with Brandon Powers 1267: Frontiers of XR & AI Integrations with ONX Studio Technical Directors & Sensorium Co-Founders with Matthew Niederhauser & John Fitzgerald 1268: Survey of Open Metaverse Technologies & AI Workflows by Adrian Biedrzycki 1269: Three XR & AI Projects: "Sex, Desire, and Data Show," "Chomsky vs Chomsky," and "Future Rites" with Sandra Rodriguez Links to all episodes here: https://lnkd.in/gvdDhCdG ICYMI: bonus episodes already published: 1242: Poetic Generative AI Film “In Search of Time” Uses Style Transfer To Explore Childhood Memories with Pierre Zandrowicz & Matt Tierney 1233: Intersection of AI and Spatial Computing for XR Accessibility with Mozilla’s Liv Erickson Thanks & help spread the word! Consider supporting my work on Patreon! Enjoy the series!

Fascinating: AI just learned how to build 3D worlds. From a single image 😳 Meet Marble, the new beta from World Labs (co-founded by Fei-Fei Li, the creator of ImageNet). It takes a photo or text prompt and turns it into a fully explorable 3D world. The demo you see below is a hobbit-style home you can walk through - curved doors, gardens, tunnels, and light shafts - all generated in minutes 🤯 It’s built on Large World Models (LWMs). Think of them as spatial cousins of LLMs, but instead of predicting words, they predict geometry, depth, and continuity. That’s why Marble’s worlds don’t “morph” or break. You can keep walking. Forever. The scene stays coherent. Creating 3D worlds used to take teams, months, and specialized software. Now it’s just one prompt. One click. One world. ↳ Artists → prototype scenes in minutes ↳ Game studios → cut production time from years to weeks ↳ Educators → turn photos into explorable lessons ↳ Architects → walk clients through designs before a single brick is laid Under the hood, it uses Gaussian Splatting - a new way to represent reality as 3D point clouds. Lightweight, photorealistic, persistent. And yes, you can export it to Unity, Unreal, or VR. NVIDIA’s Jensen Huang said it best: “AI will render most of the world’s graphics.” Marble is that prophecy coming to life. From gaming to film to robotics, we’re entering an era where AI doesn’t just describe the world - it builds it 🌍 Welcome to the age of spatial intelligence.

AI-generated Minecraft opens new possibilities for real-time video game creation and customization. AI companies Decart and Etched have developed a version of Minecraft entirely generated by artificial intelligence. This technology uses next-frame prediction to create a playable game environment without traditional coding, potentially revolutionizing game development and virtual world interactions. 🎮 AI generates Minecraft-like games using next-frame prediction: The system creates each frame of the game in real-time based on previous frames and player inputs, mimicking Minecraft's physics and gameplay without pre-programmed rules. 🧠 Extensive training on gameplay footage: The AI model learned game mechanics and physics by analyzing millions of hours of Minecraft gameplay, enabling it to replicate complex interactions and environmental behaviors. 🌈 Voice command potential for instant environment changes: Future iterations aim to allow players to verbally request changes to the game world, such as adding new structures or altering landscapes, which the AI would generate in real time. 💻 Current hardware limitations affect gameplay: The computational demands of real-time AI generation restrict smooth gameplay on consumer hardware, but custom AI chips are being developed to address this challenge. 🚀 Broader applications beyond gaming: Researchers envision using similar technology to create AI-powered virtual entities like doctors or tutors, capable of real-time interaction and adaptation in various fields. #AIGaming #MinecraftAI #NextFramePrediction #RealTimeGeneration #GameDevelopment #AIInnovation #VirtualWorlds #FutureOfGaming #AITechnology #GameAI

The Brain of XR is AI—And It’s About to Get Nuts. ReCamMaster just pulled off something cray cray; AI-powered dynamic camera angles in video footage. If you’re in XR, you should pay attention. Traditional VR setups? Clunky. Expensive camera rigs, multi-lens setups, insane post-production. Apple’s investment in Blackmagic cameras is gorgeous but $$$$. ReCamMaster? It recalculates camera angles in real time with AI. No extra rigs and pain needed. And this is just the beginning. Imagine not just watching a scene but being IN it—not in the usual “I’m standing in a 360-degree video like an upgraded View-Master” way. For VR: think John Wick cam but in VR. You’re dodging bullets, ducking under tables, AI shifting the camera to match your perspective. Today AR filters are locked to static angles. AR filters are limited to 2D screens but for AR spatial? Not just a Snapchat filter—an entire AR environment that adjusts in real time. For Virtual Production, Hollywood spends millions on LED walls, tracking tech, motion capture rigs. No $50K suits, no studio-sized volume—just AI interpolating motion in real-time. We’re stepping into GDC week, and AI is making XR more accessible than ever. Big studios might have the resources, but indie devs now have the tools. The power dynamic? Shifting. What used to cost millions? Now, it’s possible on a laptop. As you can see from the footage, it still hallucinates like it sipped the wrong kind of tea, but it’s a start. And if AI’s pace tells us anything, it won’t be long before it's near perfect. AI isn’t just assisting—it’s changing how we create. #AI #GDC #XR #VR #AR #VirtualProduction #GameDev #FutureTech #ReCamMaster #IndieDev #FilmTech #snapchat

Now, Google DeepMind has taken a massive leap in the direction of reel time by introducing Genie 2—a model capable of generating an endless variety of interactive 3D worlds. Technically, Genie 2 is an autoregressive latent diffusion model trained on a large-scale video dataset. It can create playable worlds from a single image prompt, dynamically responding to user actions via keyboard and mouse. Announcement Article: https://lnkd.in/gNYaFxfU System Capabilities: 🔸 Creates sequential worlds lasting up to one minute 🔸 Remembers and accurately recreates parts of the world that temporarily go out of view 🔸 Simulates complex physics, including gravity, smoke, water, and lighting 🔸 Animates characters and their interactions with the environment 🔸 Generates NPCs with advanced behaviors 🔸 Supports multiple perspectives: from first-person to isometric views It will be interesting to see how much impact this approach will have on prototyping ability. When designers can quickly turn concept art into fully interactive environments, what doors it will open in the game development. While the technology is still in its early stages, we need to start thinking how it will affect existing game engines in the next 5-10 years. #AI #GameDevelopment #DeepMind #Genie2 #3DWorlds #ArtificialIntelligence #GamingInnovation

For the past few years, AI's center of gravity has been language. The next fundamental shift is already underway, and it’s not about generating better text or images, but generating entire, interactive realities. This is the concept of "world models," and it represents a move from machine 𝘱𝘦𝘳𝘤𝘦𝘱𝘵𝘪𝘰𝘯 to machine 𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥𝘪𝘯𝘨. Instead of just classifying pixels, a world model develops a compressed, internal simulation of an environment. The true unlock is the ability for an AI to be trained 𝘦𝘯𝘵𝘪𝘳𝘦𝘭𝘺 𝘸𝘪𝘵𝘩𝘪𝘯 𝘪𝘵𝘴 𝘰𝘸𝘯 𝘥𝘳𝘦𝘢𝘮 and then transfer that knowledge to the real world. This is no longer just an academic curiosity. An ecosystem is emerging to productize this idea, moving us from static content to dynamic spaces, with major players and startups pushing the frontier.  • 𝗙𝗿𝗼𝗺 𝟮𝗗 𝗜𝗺𝗮𝗴𝗲 → 𝟯𝗗 𝗥𝗲𝗮𝗹𝗶𝘁𝘆: Startups like World Labs are collapsing the barrier to 3D creation. Their new RTFM (Real-Time Frame Model) can now generate 3D-consistent, interactive worlds in real-time, directly from a handful of 2D images.  • 𝗙𝗿𝗼𝗺 𝗣𝗮𝘀𝘀𝗶𝘃𝗲 𝗩𝗶𝗱𝗲𝗼 → 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝘃𝗲 𝗦𝗶𝗺𝘂𝗹𝗮𝘁𝗶𝗼𝗻: Companies like Odyssey are pioneering "interactive video models," turning passive viewing into active simulation for training robots or creating dynamic games.  • 𝗙𝗿𝗼𝗺 𝗦𝘁𝗮𝘁𝗶𝗰 𝗦𝗰𝗲𝗻𝗲𝘀 → 𝗣𝗹𝗮𝘆𝗮𝗯𝗹𝗲 𝗪𝗼𝗿𝗹𝗱𝘀: Google'𝘀 𝗚𝗲𝗻𝗶𝗲 takes this a step further, generating action-controllable, playable worlds from a single image - creating a virtual sandbox for an AI agent to learn or a user to explore. The core shift is this: today's AI learns from static datasets, finding correlations in things that have already happened. World models allow an agent to learn from cause and effect by running millions of experiments in its own simulated future. This leap from data-driven prediction to simulation-driven understanding will be as significant as the move from handcrafted software to deep learning. It will unlock new paradigms in robotics, drug discovery, and game design, creating a world where the primary mode of interaction with AI is not conversation, but exploration.