Nvidia's PixelUMM watches and makes video straight from raw pixels
Nvidia and the University of Waterloo built one model that both watches video and makes it, working straight on raw pixels. PixelUMM generated every clip here from a sentence. Most video models first squeeze each frame through a separate compressor, and read video through yet another vision model. This one uses neither.
A video becomes blocks four frames deep and 16 pixels across
Here's how it reads a video. It cuts the clip into blocks, four frames deep and sixteen by sixteen pixels across. Each block becomes one token, the way a word does, and sits in a row beside the words of the question. A still image is cut into sixteen-pixel squares the same way.
One Transformer, two experts: one writes text, one paints pixels
Those tokens go into one Transformer, built on Alibaba's Qwen3 language model, with eight billion parameters. Inside are two experts. One takes text and clean pixels, and writes answers. The other takes noisy pixels, and turns them into a new video. They share one attention layer, so each sees everything the other does.
Next to vision models its size, it counts best and trails on hard questions
CountBench 94.30 against Qwen3-VL 8B's 89.80; MVBench 70.53 against 69.00; MMMU 41.67 against 69.60. The page says training data differ across models, so these can't show which design is better.
PixelUMM project page, understanding benchmarks (LMMS-Eval), released checkpoint. Wei et al., arXiv 2609.38597.
How well does it watch? On counting things in a picture, it scores 94, the best in its table. On questions about video, it edges out Alibaba's Qwen3 VL, a model its size. But on college-level questions it scores 42, against 70. The team says itself that the models were trained on different data, so none of this shows which design wins.
Smooth skies show a faint 16-pixel grid, and the released model has it
Skipping the compressor has a visible cost. PixelUMM's output layer is deliberately simple, and in smooth areas it leaves faint sixteen-pixel squares. Here's a sky, magnified. And the wall behind a pianist. Heavier output layers cut the grid, but the released model, and every video on the page, still uses the simple one.
When wolf pups overlap, they merge, split and change in number
It also has the usual video-model failures, and the page shows them. Watch these wolf pups. As they run past each other, their bodies merge and split, so the number of pups keeps changing. Hands get the wrong number of fingers too, and objects can bend when they shouldn't.
The weights and code are out
Posted to arXiv on 29 September 2026 by Nvidia and the University of Waterloo. Weights on Hugging Face since 1 October, code on GitHub.
The paper went up on September 29th. The weights followed on Hugging Face on October 1st, with the code on GitHub. Anyone with the hardware can try it. The paper and the project page are linked below.
