Skip to main content
Back to timeline
arXivSource publication:

PlayGen2 splits a playable world into 3D state and rendering: corridors persist after you turn, with 5.2%–6.8% contact error over ten minutes

Synopsis

PlayGen2 splits playable-world generation in two: a learned world-state model keeps a persistent volumetric occupancy grid and identity-indexed entity states, while a separate neural renderer turns that state into first-person frames. Earlier models that used generated frames as the sole recurrent representation had to preserve geometry, track entities and synthesize appearance from one view-dependent visual signal. On a Doom testbed the model learns geometry-aware player motion and player–NPC interaction, reaching 5.2%–6.8% cumulative contact error and 82.1%–85.6% action accuracy over ten-minute exploration.

Source-provided article image: PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State
arXiv

Interpretation

Playable-world generation no longer treats generated frames as the only world memory; an explicit 3D state persists and is updated through interaction. Earlier models using generated frames as the sole recurrent world representation had to preserve geometry, track entities, predict interactions and synthesize appearance from one view-dependent visual signal, recovering out-of-view information only from visual memory. After training on Freedoom2 MAP01, cumulative contact error over ten simulated minutes of exploration is 5.2%–6.8% and action accuracy is 82.1%–85.6%, with both metrics relatively stable between five and ten minutes.

Within a fixed-size, player-centered 3D region, the world-state model jointly learns to advance entities under actions, complete unknown geometry and discover newly encountered entities. State updates are conditioned on player actions, entity histories and geometric context, and results are written back to persistent world memory, so generated regions remain on revisits and entity records outside the active volume are retained. Training uses roughly 500 hours and about 500 GB of automatically collected Doom gameplay; the world-state and rendering models have 41.2 M and 29.1 M parameters and train for about three days on four NVIDIA A100 GPUs.

The renderer synthesizes frames only from depth and semantic projections of the world state, and generated images never feed back into state prediction. Appearance synthesis and state evolution are trained separately; the renderer takes projected scene conditions and the last four RGB frames, with RGB history used only for visual continuity. On the training map the two systems have comparable visual quality at one minute; by five minutes PlayGen degrades substantially while PlayGen2 scores better on all five metrics under global initialization, with LPIPS staying within 0.41–0.44 across the one-, five- and ten-minute stages.

Perspective

The demonstrated scope is the Doom testbed: training uses only Freedoom2 MAP01, unseen-map evaluation uses seven Scythe levels, and interaction and efficiency are measured on a MacBook Pro with an M4 Pro chip. The intended readers are researchers and engineers interested in how interactive world models connect to neural rendering.

On unseen maps the reference-based metrics reflect both structural discrepancies and appearance differences, because depth and semantic conditions do not specify native Scythe textures, so those numbers should be read alongside qualitative results and their variation over time. Multiplayer composition and programmable gameplay are demonstrated without updating weights, and training data still comes only from single-player gameplay.