L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory
World models stall because memory grows with the stream. L3G-World grows memory with the scene instead: 38.6x fewer primitives and real-time generation at 9 FPS.
The claim
Scene-sized memory keeps world generation real-time over long exploration.
The problem
Interactive worlds must stay consistent over long sessions.
Temporal context and 3D memory both grow with the stream.
Re-encoding RGB frames for memory blocks real-time generation.
How it works
L3G memory holds compact 3D Gaussians of the scene.
A memory updater spawns Gaussians in new regions.
It refines old ones, so memory grows with the scene.
Each Gaussian stores a VAE latent feature.
The video transformer reads memory directly in latent space.
Figures
What changes
Memory rendering runs 359x faster than the best prior baseline.
It uses up to 38.6x fewer 3D primitives.
The system generates worlds in real time at 9 FPS.
Limits
Each figure compares with the best prior baseline per metric.
Memory still grows as the explored scene grows.
Source · arXiv