On the cover: a film set, the left half built by carpenters and the right half generated. Decorative, and also the whole argument.
The VLA post ended on an accounting problem. Language models ate the internet. Robots have a few thousand hours of somebody wiggling a joystick. The gap between those two piles is about five orders of magnitude, and nobody closes it by hiring more people to drive forklifts.
So you stop collecting the world and start generating it instead.
Which leaves us here: a model with billions of parameters, drawing a street that has to still be there when you turn back around.
Welcome to the era of learned simulators. This is the story of how the film set stopped being built by carpenters and started being drawn by a neural network, one frame at a time.
Picture the set properly, because the rest of this post runs on it. Old Hollywood built New York inside a studio in Burbank. A row of building fronts, painted, propped up from behind with scaffolding, and nothing behind them at all. Walk through a door and you are in a car park. It works because the camera only ever looks one way, and the street only has to hold up for one shot.
Every simulator is that street. The questions are who builds it and how long it holds up.
The street the carpenters built
You might think the answer is to build the street properly. That is what a classical simulator does. Somebody models the road, somebody models the traffic light, somebody writes the script where the cyclist swerves at 14 metres (the 14 came from a ticket, not from physics). Every object in the scene is there because a person put it there.
It works. It does not scale, because a person has to think of the thing first.
Waymo’s own wish list makes the point: a tornado, a flooded street, an elephant standing in the road. Nobody is going to model an elephant on the off-chance. And the tail is where the driving problem lives now, since the easy 99.9% of miles was solved years ago and what is left is made of things nobody expected.
There is a second bill. What carpenters build looks like what carpenters build: the light too clean, the road too even, the pedestrians walking like puppets. Train a perception model on that and it gets a shock the first time it sees a real road.
So what if you had the street generated by something that had watched an enormous amount of real footage?
What a world model actually is
The definition is older and plainer than the noise around it suggests, so let us get it down first.
A world model is a state predictor. Tell it the state of the world now and the action you are about to take, and it tells you the state you end up in:
\[s_{t+1} = f(s_t, a_t)\]where $s_t$ is the state at time $t$, $a_t$ is your action, and $f$ is the model. Jürgen Schmidhuber wrote this down in 1990 as a recurrent network trained to predict its own next input from the current input and the action, so an agent could think ahead without touching the world. Ha and Schmidhuber rebuilt it in 2018 as three parts: a vision model that compresses the frame, a memory model that predicts what comes next, and a small controller. They trained the controller inside the model’s own dream and moved it back to the real environment, which is the trick everybody in this post is still running.
Two words carry the weight. State is whatever you decide the world is made of, and every camp in this post picks something different. Action is what makes it a world model at all. Take the action slot away and you have a video generator, which is a film: gorgeous, and it does not care what you do. The JEPA post spent a table on this, because a model conditioned on position rather than action is doing latent inpainting and calling it prediction.
The generic recipe
Every model in this post is the same three pieces, and it is worth having them straight before any brand names turn up.
A tokeniser takes one frame, a million-odd pixels, and squashes it down to a few thousand numbers, the way a ViT chops an image into patches. This one squashes along time too, so a run of frames becomes a single small block.
A dynamics model is a transformer that reads the blocks so far and predicts the next one.
An action is a slot in the input where you put what the agent just did. That slot is the only thing separating a world model from a video generator.
In words, what the model learns is this:
\[\text{next description of the world} = f(\text{every description so far},\ \text{what you just did})\]specifically,
\[\hat{z}_{t+1} = f_\theta(z_{t-k:t},\, a_t), \qquad x_{t+1} = \mathcal{D}(\hat{z}_{t+1})\]where $z_t$ is the squashed description of frame $t$, $a_t$ is the action, $k$ is how far back the context window reaches, $f_\theta$ is the dynamics model, and $\mathcal{D}$ is the decoder that turns a description back into a picture. Then you feed the frame you just drew back in as the frame you look at, and go round again.
Figure 1: The whole family, in four boxes. The action slot at the bottom is the difference between a simulator and a very expensive screensaver. Source: Author
Notice what is missing. Nothing in those boxes holds a 3D model of the street, and nobody has fitted a NeRF or a cloud of gaussians to it. The street stays where you left it only because the model can still see the earlier frames.
Genie: the set that draws itself
Genie (Bruce et al., DeepMind, 2024) is where the recipe got its shape: 11B parameters, trained on unlabelled internet video of 2D platformer games.
The clever bit is the middle. Nobody uploads their keystrokes along with their Let’s Play, so the model has to work out for itself what button must have been pressed between one frame and the next. That is the latent action model, and its codebook holds eight actions. Somebody sat down, considered every platformer ever made, and concluded the genre is eight buttons wide. They were not wrong.
Genie 3 (DeepMind, August 2025) is the one everybody has seen. 720p at 24 frames a second, generated in real time from a text prompt, and consistent for a few minutes where Genie 2 managed 10 to 20 seconds. It takes promptable world events, which is the director shouting “now make it rain” halfway through the take. DeepMind dropped their SIMA agent into a generated warehouse and told it to walk to the red forklift, and it walked to the red forklift (a low bar, until you remember the forklift did not exist ten seconds earlier).
NOTE: there is no Genie 3 paper, just a blog post and some very good videos. Everything above is what DeepMind chose to say, not something anybody outside the building has checked.
Why the street forgets
Genie 3 has no map of the street. It has a context window, and the bakery stays where you left it for exactly as long as the frames containing the bakery are still inside it.
So how long is that? Get a calculator out, because this is the number the whole illusion rests on.
Take 720p, so 1280 by 720 pixels, at 24 frames a second. Squash it with a tokeniser at NVIDIA’s published Cosmos rates: 8x smaller in each direction on screen, 8x fewer frames. Across the picture you get $1280/8 = 160$ by $720/8 = 90$, so 14,400 numbers per squashed frame. Along time, 24 frames a second becomes 3. Multiply:
\[160 \times 90 \times 3 = 43{,}200 \ \text{tokens per second of video}\]One minute of street is 2.6 million tokens. The whole of War and Peace is about 750,000, so a minute of generated pavement costs three Russian novels, and the model has to read all of it again for every frame it draws.
Figure 2: How fast video eats a context window. A million-token context buys you 23 seconds of 720p at the gentler setting and 93 at the harder one. Which is roughly the horizon everybody reports, and I do not think that is a coincidence. Source: Author
Squash harder, 16x instead of 8x, and it drops to 10,800 tokens a second, so a million-token context holds a minute and a half. That is the trade everyone is making, and the detail you threw away is detail the model can no longer put back. This is the KV cache problem in a hard hat.
Run off the end of the window and nothing crashes. The model just makes something up out of whatever is still on screen.
Figure 3: What running out of context looks like from the inside. Nothing is broken. The model is drawing a perfectly reasonable street, which is the problem, because it is not the street you were standing in. Source: Author
Cosmos: the set you can download
Cosmos (NVIDIA, January 2025) took the same recipe and industrialised it. Diffusion models at 7B and 14B, autoregressive ones at 4B and 12B, trained on 20 million hours of video on a cluster of 10,000 H100s over three months. The weights are out under an open licence.
Twenty million hours is a shade over 2,280 years of footage. To watch it end to end you would have to have started in the third century BC and never once gone to bed.
The tokeniser is the part I would steal. Cosmos squashes 8x or 16x on screen and 4x or 8x along time, up to 2048x in total, and NVIDIA measured it running 12x faster than the tokenisers it replaced while looking better. Nobody has ever been promoted for a tokeniser, which is a shame, because every number in the last section came out of a tokeniser decision.
NVIDIA’s name for the thing is the one that stuck: a world foundation model, pretrained on everything and fine-tuned by you into a simulator for whatever your own robot has to survive.
The driving set, and who it is really for
Driving is where the money is, so driving is where this got serious fastest. And who is the customer for a generated street? For the first few years everybody assumed it was the perception team, hungry for training frames.
Wayve’s line runs GAIA-1 (9B, 2023) to GAIA-2 (March 2025). GAIA-2 swaps in latent diffusion, draws all the cameras at once so they agree with each other, and lets you specify the scene properly: ego speed and steering curvature, 3D boxes for every other car, weather, time of day, lane counts, speed limits, crossings. The conditioning list reads like an order form, which is exactly the point of it.
Then GAIA-3 (December 2025) doubles the parameters to 15B, doubles the tokeniser and trains on ten times the data. More interesting than any of that, it changes jobs. GAIA-3 is pitched at evaluation rather than data generation, and the number Wayve reports is a fivefold drop in the rejection rate of synthetic tests, meaning five times fewer generated scenarios that their own engineers threw out as not worth grading a driver on.
Waymo World Model (February 2026) takes Genie 3 and adapts it to driving. It adds the sensor nobody else generates: lidar alongside camera, so the whole stack sees one coherent world instead of a pretty picture.
XPENG X-World (April 2026) is the least glamorous and possibly the most telling, because it is already inside a production loop doing closed-loop simulation and online reinforcement learning. It ships a caching trick that skips the parts of a scene that did not change, worth a 2.7x speedup on denoising.
Figure 4: Three years of it. The interesting move is not left to right, it is that the driving row keeps arriving at evaluation while the games row is still talking about generation. Source: Author
| Model | What you steer it with | What it is for | Verdict |
|---|---|---|---|
| Genie 3 | Text prompt plus navigation actions | Agents, research demos | The most general, the least documented |
| Cosmos | Text, video, robot actions | A base to fine-tune | The one you can actually download |
| GAIA-2 | Ego dynamics, agent boxes, weather, road semantics | Rare-scenario training data | Control you can specify in a ticket |
| GAIA-3 | Same, at 15B | Grading the driver | Evaluation is the real product |
| Waymo World Model | Prompts, driving inputs, scene layout | Camera and lidar scenarios | Only one generating the whole sensor suite |
| Dreamer 4 | Mouse and keyboard | Training a policy inside it | The only one with a policy to show for it |
Does anybody actually learn to drive on a film set?
Every frame in this post is worth nothing unless something learns from it, so here is the result that says something does.
Dreamer 4 (Hafner et al., 2025) trains an agent by reinforcement learning inside its own world model, which they call imagination training. The agent is the first to mine diamonds in Minecraft from offline data alone (no environment interaction means exactly that: the policy never once played the actual game), a task more than 20,000 mouse and keyboard actions deep, from raw pixels. It beats OpenAI’s VPT offline agent on 100x less data, and the world model runs at 21 frames a second on one H100.
A policy that never touched the game rehearsed entirely inside a model of it, then went out and mined a diamond in the real thing. That is the result the whole field is chasing.
Now the ablation, and it is not a kind one.
Physics-IQ (Motamed et al., 2025) is 396 real videos that test whether generative video models have the faintest idea about fluids, optics, solid mechanics, magnetism and heat. The best model scored 29.5 out of 100. Worse than the score is what sits behind it: how well a model does on physics has nothing to do with how real its videos look. The prettiest model is not the one that gets the collision right, and there is no version of the leaderboard where it is.
Which is the set doing what a set does. It was built to survive a camera pointed at it. Nothing in that job description asks whether the wall holds when you lean on it.
So who is right?
Four camps are building this, and they disagree about one thing: what you are allowed to throw away.
Pixels. Cosmos, Genie 3, the GAIA line. The state is the picture. The bet is that pixels are general enough to cover a warehouse and a motorway and a kitchen, and that a human can watch the output and argue with it.
Descriptions. LeCun’s camp, and now his company: he left Meta in November 2025, and AMI Labs raised a $1.03B seed at a $3.5B valuation in March 2026 to build models that predict a description of the world and never draw it. LeWorldModel is the small version of the argument, about 15M parameters on one GPU and planning up to 48x faster than world models built on video backbones. The bet is that most of a pixel is noise, which is the JEPA case.
Actual 3D. Fei-Fei Li’s World Labs shipped Marble in November 2025: one image in, a persistent 3D scene out, built from gaussian splats and downloadable. Walk a different path through it and you get the same scene every time, because the scene is stored rather than redrawn. That is the drift in Figure 3 solved by having no context window at all, and the price is a room rather than a world with things happening in it.
Only what changes the decision. Silver’s camp at DeepMind. MuZero learns a model that predicts reward, value and the next move, and never predicts the picture at all. Its film set does not have walls, or a street, or a shopfront. It knows the score, and it beat everybody at Go.
| Camp | What it calls the state | Who | What you get | What you give up |
|---|---|---|---|---|
| Pixels | The picture | NVIDIA, DeepMind, Wayve | Anything you can film, watchable by a human | Cost, and physics you cannot check |
| Descriptions | An abstract vector | LeCun, AMI Labs, JEPA | Cheap planning, no wasted detail | You cannot look at what it believes |
| Actual 3D | Geometry on disk | World Labs | Consistency for free, forever | A scene, not a story |
| Decision-relevant | Reward and value | Silver, MuZero | Superhuman play, tiny model | Only works where you know the reward |
So who is right? Nobody can tell you, because the four camps are not measured on the same thing and none has beaten another on a task all four agreed to run. What you can do is pick by what you need to keep. A human signing off a scenario needs pixels. A robot planning in a warehouse does not, and pays for them anyway. Anything you walk through twice wants stored geometry. And if you know the reward and the rules, MuZero has been doing this since 2019 without any of the vocabulary.
Where things are going
WorldLens (CVPR 2026): somebody had to grade the grader. On a film set, continuity is the person who checks the coffee cup is still half full in the next shot, and WorldLens is the first benchmark doing that job for driving models. It scores them on five things at once: generation, reconstruction, action-following, downstream task and human preference. The authors collected 26,808 human ratings with written reasons attached, then distilled an automatic scorer off them.
Real time on one GPU: Dreamer 4’s 21 fps and XPENG’s 2.7x cache are the same trend arriving from opposite ends. A simulator you can only run overnight in a batch job is a dataset with extra steps.
The continuity report
WorldLens is blunt about the state of the set. None of the models wins on all five scores. The ones with rich texture get the geometry wrong, the ones with stable geometry get the behaviour wrong, and the best of them score 2 to 3 out of 10 on human realism. Three years into the driving effort, the strongest generated footage is about a quarter of the way to convincing a person.
Most of the impressive numbers in this post came from the companies selling the models. Wayve’s fivefold improvement, XPENG’s deployment, Waymo’s elephant: announced in press releases and measured on private fleets, against baselines nobody outside can look at. I have quoted them because they are the only numbers there are, which is not the same as trusting them.
Then there is the trap you get free with closed-loop training. A policy trained inside a learned simulator is marked by a model with soft spots, and reinforcement learning is a machine for finding soft spots. If the simulated pedestrians are a little too timid, the policy learns a merge that is a little too aggressive and gets rewarded for it, and you find out on a real road. That is an old sim-to-real lesson rather than a fresh result, and none of these papers has retired it.
And the most capable model here has no paper. You cannot reproduce a blog post or review one, so the whole public discussion of Genie 3 happens at the level of what the videos looked like.
Conclusion
A film set was always a trick for building the smallest amount of world that survives one shot from one angle. What changed is who builds it. The carpenters have been replaced by a model that watched two thousand years of footage and can put up any street you name in real time, elephant included, and it holds for about ninety seconds, which is roughly how long a context window can hold the pavement you are standing on.
The series thesis survives with one more turn on it. Image patches are tokens, words are tokens, robot actions are tokens, and now the next ten seconds of the world is a token stream with a steering wheel bolted to the side.
What nobody has hired is the rest of the film crew. The set builders are astonishing. The continuity person, the one with the clipboard and the half-full coffee cup, does not exist yet, and until somebody writes one, the honest description of every model here is the same: a very good street to look at, and one you should not lean on.
And now you know. Fin.