A 4B open model beats GPT-5 at finding when things happen in video

2 months ago

A four billion parameter open model just beat GPT-5 at finding when something happens in a video. It is called VideoChat3. MCG-NJU shipped the weights under Apache two point oh.

Ask

Ask about this presentation

Answers are generated from this presentation.

Chapters

  1. 0:00A 4B open model beats GPT-5 at finding when things happen in video
  2. 0:12It points at the right span more often than GPT-5 does
  3. 0:34Eleven wins, and the big ones are all the same kind
  4. 0:55It compresses the video before the language model sees it
  5. 1:41The encoder tax is paid up front
  6. 1:55Weights and all three training sets, Apache-2.0
  7. 2:11Ask a video model when, not just what
Show transcript

A 4B open model beats GPT-5 at finding when things happen in video

From MCG-NJU, Shanghai AI Lab and NTU · 4B, Apache-2.0.
VideoChat3-4B MCG-NJU · Shanghai AI Lab · NTU Apache-2.0 · ungated video + text in, text out 4B · 8.96 GB bf16 256K context

A four billion parameter open model just beat GPT-5 at finding when something happens in a video. It is called VideoChat3. MCG-NJU shipped the weights under Apache two point oh.

It points at the right span more often than GPT-5 does

VideoChat3-4B best open baseline GPT-5
56.1
53.6
40.5
CharadesTimeLens · mIoUbest open: Eagle2.5-8B
54.6
53.7
42.9
ActivityNetTimeLens · mIoUbest open: Eagle2.5-8B
67.0
64.0
52.1
QVHighlightsTimeLens · mIoUbest open: Eagle2.5-8B
40.2
32.7
20.0
VUE-TR V2mIoUbest open: Molmo2-4B
VUE-TR V2: 40.2 vs 20.0 · and 39.7 for Gemini 3 Pro
Gemini 2.5 Pro still takes two of the three TimeLens columns — ActivityNet 58.1 and QVHighlights 70.4, against 54.6 and 67.0
each model at the frame budget it supports · paper Table 10 / §A.1 · arXiv 2607.14935

That task is temporal grounding. You describe a moment. The model points at the span where it happens. You have scrubbed a timeline hunting for one shot. On Charades, VideoChat3 scores fifty six point one. GPT-5 scores forty point five. On VUE-TR it roughly doubles GPT-5. Gemini two point five Pro still takes two of the three TimeLens tests.

Eleven wins, and the big ones are all the same kind

comprehension & reasoning · 5 wins, 8 losses
TOMATO−1.9
MotionBench+0.1
TVBench−9.8
TempCompass+1.2
Video-MME−2.3
LVBench+0.5
VideoEval-Pro MCQ+0.3
VideoEval-Pro Open−1.8
Video-MMMU−0.2
MMVU−2.5
Minerva−3.4
Video-MME-v2 Non-Lin+0.1
Video-MME-v2 Avg−0.9
temporal grounding · 6 wins, 0 losses
Charades+2.5
ActivityNet+0.9
QVHighlights+3.0
VUE-TR V1+1.9
VUE-TR V2+7.5
MomentSeeker+6.0
Every win bigger than 1.3 points is a grounding column
Biggest losses: TVBench −9.8 · Minerva −3.4 · MMVU −2.5 · Video-MME −2.3
margin vs the best of 11 open baselines, per column · paper Table 2 · arXiv 2607.14935

The full table runs nineteen columns. VideoChat3 wins eleven and loses eight. All six grounding columns are wins. Six of the eight losses are comprehension. Ask it what happened. GPT-5 pulls away. On Video-MME it is eighty three point three against seventy point one. The big wins all sit in one block. The reason sits upstream of the language model.

It compresses the video before the language model sees it

everyone else
Sample a few frames, shrink them
“fundamentally removes substantial video details at the input stage”
arXiv 2607.14935 §2.2
I3D-ViT · step one
0
1
2
3
Chunks of four frames, joint attention across space and time
config.json · temporal_merge_size 4
learned temporal position embeddings
I3D-ViT · step two
↓
×4 temporal · ×2×2 pixel shuffle = 16×
inside the encoder
LLM cost grows with the square of the sequence · encoder cost grows with the frames
config.json · merge_kernel_size [2, 2]
The receipt — swap I3D-ViT back for the plain image tokenizer it started from
Charades 30.7 → 45.0
ActivityNet 27.1 → 43.7
QVHighlights 33.9 → 58.4
Image Avg. 65.30 → 66.10
“we attribute this performance gain entirely to the frame sampling advantage” · §5.4 · I3D-ViT is initialized from MoonViT and gets more pre-training and ~4× the frames, so this is not a controlled swap

Video costs tokens. The language model has little room. So everyone samples a few frames and shrinks them. You throw the frames away before the model ever looks. VideoChat3 compresses inside the vision encoder instead. It groups the video into chunks of four frames. Those frames attend to each other across space and time. Then it pools along time. That is sixteen fold compression. It all happens inside the encoder. The language model's cost grows with the square of the sequence. The encoder's cost grows with the frame count. Move the work early, and the language model reads half the visual tokens. Swap that encoder for the plain image one it started from, and Charades grounding falls from forty five to thirty one. The image scores barely move. Four frames now cost what one used to, so the model watches about four times as many. The authors credit exactly that. You cannot time a moment you never saw.

The encoder tax is paid up front

50 s 20 s 10 s 5 s 2 s 1 s 256 512 1024 2048 input frames total latency, log scale at 256 frames it is the slower one: 1.652 vs 1.363 s crossover at 512 frames Qwen3-VL 44.449 s VideoChat3 20.412 s
At 2048 frames: 62% fewer FLOPs · 26 GB less GPU memory · half the visual tokens at every frame count
H200, HF pipeline, FlashAttention-2, same ViT patches per frame · paper Table 4 · arXiv 2607.14935

You pay for that heavier encoder. At two hundred fifty six input frames, VideoChat3 is slower than Qwen3-VL. The crossover lands at five hundred twelve frames. At two thousand forty eight frames, VideoChat3 finishes in twenty seconds. Qwen3-VL takes forty four.

Weights and all three training sets, Apache-2.0

huggingface.co/MCG-NJU/VideoChat3-4B arxiv.org/abs/2607.14935
Apache-2.0 ungated 8.96 GB bf16 4B parameters
the training data, all three sets
Academic2M
2.27M instances
Apache-2.0
LV116K
116.2K instances
Apache-2.0
OL617K
617,183 instances
Apache-2.0
Training code not released yet — still an unchecked box in the repo's own to-do list
Hugging Face model + dataset APIs and the GitHub tree API, pulled 2026-07-19 · the repo carries no license file

The weights are on Hugging Face under Apache two point oh, ungated. All three training sets shipped too, also Apache two point oh. The training code has not. It is still an unchecked box on their own to-do list. The data was the expensive part. That part is already there.

Ask a video model when, not just what

the wins tell you where the compression happens
Every figure here is the authors' own run. No third-party re-evaluation exists yet, and the training data is being pulled about twenty two times as often as the model itself.
huggingface.co/MCG-NJU/VideoChat3-4B

Ask a video model when, not just what. There is now a four billion parameter open answer. Every number here comes from the authors' own run. Nobody outside the lab has re-run it. And people are pulling the training data more than twenty times as often as the model.

Up next

2:58

Apodex 1.1 mini · the same file, run alone and run as a team

@ai-ml-today · 1 month ago

3:09

Ornith-1.5 · an open model trained on problems it wrote for itself

@ai-ml-today · 1 month ago

3:08

MOSS-VL · an open 11B video model trained to decide, after every frame, whether to speak or stay silent

@ai-ml-today · 1 month ago

3:29

Qwen3.8-27B · a 27B open model anyone can download averages 84.3 across real desktop jobs on Qwen’s own run, and loses the widest row on its own card by nine points

@ai-ml-today · 1 month ago

3:15

MiLMMT-46-v1.0 · Xiaomi finished training an open translator on sentences that had no translation attached, and the scores it now leads on are produced by the two models that graded it

@ai-ml-today · 1 month ago

3:31

BDH-CQ — a 150M-parameter system that learns a puzzle rule from the worked examples in front of it and does its thinking as numbers, never as words

@ai-ml-today · 2 months ago

3:29

Muse Glimmer 30B, Meta’s open agent model, and the second tiny model that guesses sixteen words ahead so it runs fast on one consumer graphics card

@ai-ml-today · 2 months ago

3:05

JoyAI-Video-Edit, JD’s open 16B model that edits a video while the video is still playing, at about 30 frames a second on one Nvidia B200

@ai-ml-today · 2 months ago

3:26

Inkling-Small, Thinking Machines’ 12B-active / 276B open mixture-of-experts model that out-reasons its own 3.5× bigger flagship

@ai-ml-today · 2 months ago

3:42

Maple-Preview, DeepGrove’s 20-billion-parameter ternary reasoning model that runs on a Mac mini

@ai-ml-today · 2 months ago

3:16

Hunyuan3D-Buffalo, Tencent’s one model that builds, understands, and edits 3D from words

@ai-ml-today · 2 months ago

3:28

MiniMax H3, an open model that makes a video and its soundtrack in one pass

@ai-ml-today · 2 months ago