World Models & Gaussian Splats: A Papers Survey

Two research threads have been converging for the last few years, and the seam between them is where I spend most of my thinking. The first is world models: systems that learn an internal simulation of an environment — good enough to predict what happens next, and lately good enough to be the environment. The second is 3D Gaussian splatting: a scene representation that made photoreal, real-time 3D practical on ordinary hardware. One thread learns worlds; the other represents them. Prompt-to-world systems like World Labs’ Marble — and my own toy version of that pipeline — live at the intersection.

This is a curated reading list, not an exhaustive one. Each entry gets a line or two on why it matters. Papers link to arXiv; industry systems link to their announcements.

The world model idea

World Models — Ha & Schmidhuber, 2018. The paper that named the field. A VAE compresses pixels to a latent, an RNN learns the dynamics, and a tiny controller learns to act — famously, the agent can be trained entirely inside its own dream. Almost everything below is a scaled-up descendant of this three-part sketch.

Learning Latent Dynamics for Planning from Pixels (PlaNet) — Hafner et al., 2018. Made latent dynamics models actually useful for control, planning in latent space without ever decoding back to pixels.

The Dreamer line — Hafner et al.: Dreamer (2019), DreamerV2 (2020), DreamerV3 (2023). Reinforcement learning by imagining trajectories inside the learned model. V3 is the landmark: one set of hyperparameters across 150+ tasks, and the first agent to mine diamonds in Minecraft from scratch — a result that eventually landed in Nature.

A Path Towards Autonomous Machine Intelligence — LeCun, 2022. The position paper behind JEPA: predict in representation space, not pixel space, because most pixel detail is unpredictable and irrelevant. The clearest statement of the “world models are the road to real machine intelligence” thesis — V-JEPA 2 (2025) is the strongest empirical version of the argument, and LeCun has since left Meta to pursue it full-time at his own lab.

Video models as world simulators

Around 2024 the field noticed that a good enough video generator is a world model — and then made them interactive.

Video generation models as world simulators (Sora) — OpenAI, 2024. The technical report that reframed video generation as world simulation. Emergent 3D consistency and object permanence at scale, with no 3D representation anywhere in the system.

The Genie line — Google DeepMind: Genie (2024), Genie 2 (2024), Genie 3 (2025). From unlabeled video to playable worlds — Genie learned latent actions with no action labels at all. Genie 3 generates photoreal interactive worlds from a text prompt at 24 fps with minutes-long persistence. The strongest evidence yet that frame-by-frame generation can carry a coherent world.

Diffusion Models Are Real-Time Game Engines (GameNGen) — Valevski et al., 2024. DOOM, playable at 20 fps, with no game engine — every frame hallucinated by a diffusion model conditioned on actions. A proof point that stunned people outside the field, alongside DIAMOND (diffusion world models for Atari and CS:GO) and Oasis (Decart’s playable Minecraft, served to the public at scale).

Cosmos World Foundation Model Platform for Physical AI — NVIDIA, 2025. World models as infrastructure: pretrained video world models, tokenizers, and curation pipelines aimed at robotics and autonomous vehicles — generate the long tail of training experience instead of collecting it.

Runway GWM-1 — Runway, 2026. The video-generation company’s bet on “interactive video” as the world model substrate, extending the Genie/GameNGen lineage to general scenes.

The open question in this whole section: frames are not geometry. Video world models still struggle with long-horizon spatial consistency — walk far enough and the world reshuffles behind you. Which is exactly the itch the next thread scratches.

The representation: NeRF to Gaussian splats

NeRF: Representing Scenes as Neural Radiance Fields — Mildenhall et al., 2020. The big bang of neural scene representation. A tiny MLP plus volume rendering, photoreal novel views, and a thousand follow-ups. Instant-NGP (Müller et al., 2022) then collapsed training from days to seconds with hash-grid encodings.

3D Gaussian Splatting for Real-Time Radiance Field Rendering — Kerbl et al., SIGGRAPH 2023. The paper this whole section is named after. Replace the neural field with millions of explicit anisotropic Gaussians, rasterize instead of ray-march, and you get NeRF-quality scenes at 100+ fps. Explicit, editable, fast — this is why splats, not NeRFs, became the substrate for generated worlds.

Refinements worth knowing: Mip-Splatting fixes the aliasing that made splats shimmer across zoom levels; 2D Gaussian Splatting flattens Gaussians to surfels for far better geometry; gsplat is the open-source library (used by nerfstudio, and by my own NeRF service) that made all of this a pip install.

Feed-forward 3D — the optimization loop disappears: DUSt3R (Wang et al., 2023) regresses dense 3D from image pairs with no calibration, no SfM; pixelSplat and MVSplat predict Gaussians directly from two or a few images in a single forward pass; VGGT (2025) is a single transformer that outputs cameras, depth, and 3D points for dozens of views in seconds. This line matters because it turns “reconstruct a scene” into an amortized network call — the speed world generation needs.

Where they meet: generative 3D worlds

Score distillation and its escape — DreamGaussian (2023) brought text-to-3D from hours to minutes by optimizing Gaussians against a 2D diffusion prior; TRELLIS (Microsoft, 2024) skipped the per-asset optimization entirely with a native 3D latent diffusion model that decodes to splats, meshes, or radiance fields.

Outpaint-and-lift worlds — the lineage my own pipeline sits in: Text2Room (2023) established the loop — generate an image, estimate depth, lift to 3D, move the camera, inpaint, repeat. LucidDreamer swapped the mesh for Gaussians; WonderJourney (2023) made the loop wander through coherent scene sequences, and WonderWorld (2024) got interactive scene generation under 10 seconds. HunyuanWorld 1.0 (Tencent, 2025) scaled the panorama-first version: 360° pano as the world proxy, layered depth, exportable splats.

Camera-conditioned diffusion as the reconstructor — CAT3D (Google, 2024): a multi-view diffusion model hallucinates the missing views, a NeRF/splat fits what it dreamed. One image — or zero, just a prompt — becomes a full scene in minutes. This “generate views, then reconstruct” recipe quietly powers a lot of what shipped since.

World Labs — Marble — 2025. Fei-Fei Li’s team’s answer to the frames-vs-geometry question: generate the world as a spatial object (splats you can export and edit), not as a video stream. Marble launched in late 2025 as the first commercial prompt-to-3D-world product, and their RTFM work explores the real-time frame-model side of the same house. The bet — persistent geometry beats regenerated frames for worlds you keep — is the one I find most convincing, and it’s the architecture my toy version follows: prompt → 360° panorama → monocular depth (Depth Anything V2) → unproject → Gaussian splat, viewable in the browser.

Where this is heading

Three things I’m watching:

• Geometry vs. frames. Genie 3 and GWM-1 say generate frames and let 3D consistency emerge; Marble and the splat lineage say generate geometry and let rendering be trivial. My money is on hybrids — frame models for dynamics and interaction, explicit splats for persistence and editability.
• Feed-forward everything. VGGT-style transformers keep replacing optimization loops. The endgame is a single forward pass from prompt (or a few photos) to a navigable, persistent scene.
• Worlds as training data. Cosmos makes the case that the biggest customer for generated worlds isn’t people — it’s robots that need a million years of experience nobody can collect.

For going deeper: the 3DGS applications survey (TPAMI 2026) tracks the splatting side comprehensively, and Radiance Fields is the best running news source for the field.