llama.cpp v0.9.0 processes prompts nearly 3× faster on Gemma 4 E2B

llama.cpp v0.9.0 processes Gemma 4 E2B prompts almost three times faster. On an M4 Pro that is 4,600 tokens a second instead of 1,600, so long prompts finish in a third of the time.

Slide text

llama.cpp

November 2026

Gemma 4 E2B Q4_0 · M4 Pro, Metal · prompt tokens per second

Before

1,614

v0.9.0

4,603

llama.cpp v0.9.0 release notes · Core changes · #31356

01 Faster prompts

First, speed.

Slide text

llama.cpp

November 2026

01

Faster prompts

llama.cpp v0.9.0 release notes

01 Faster prompts

The upper layers of Gemma 4 reuse earlier layers' cache, so they now skip the prompt tokens. E4B gets about 60 percent faster, 26B gains nothing, and the speedup needs the unified KV cache.

Slide text

llama.cpp

November 2026

Gemma 4’s upper layers now skip the prompt

• E4B about 1.6× faster prompts
• 26B no gain, every layer fills the cache
• Generation same speed as before

llama.cpp v0.9.0 release notes · Core changes · #31356

01 Faster prompts

The unified KV cache used to slow down when several requests shared it. Now, four requests of 100,000 tokens each on Gemma 4 26B process almost three times faster on an RTX Pro 6000.

Slide text

llama.cpp

November 2026

Parallel long prompts run about 3× faster on CUDA

Gemma 4 26B A4B Q8_0 · 4 requests of 100K tokens · RTX Pro 6000 · prompt tokens per second

Before

2,982

v0.9.0

8,626

llama.cpp v0.9.0 release notes · Highlights · #29510

01 Faster prompts

Mixture-of-experts models process prompts up to 17 percent faster on CUDA. MXFP4 models generate about a sixth faster on Apple's Metal, and Intel GPUs get flash attention with a quantized KV cache.

Slide text

llama.cpp

November 2026

Faster MoE on CUDA, faster MXFP4 on Metal

• CUDA MoE prompts up to 17% faster on DGX Spark
• Metal MXFP4 generation 17% faster on M3 Ultra
• SYCL flash attention with a quantized KV cache

llama.cpp v0.9.0 release notes · ggml changes · #31188 #31075 #31125

01 Faster prompts

Next, speech.

Slide text

llama.cpp

November 2026

02

Speech in and out

llama.cpp v0.9.0 release notes

02 Speech in and out

NVIDIA's Nemotron 3.5 Transcribe now runs in llama.cpp. It transcribes recordings in 40 languages and labels each speaker. Word timestamps and streaming aren't supported yet, and llama-server needs reasoning turned off.

Slide text

llama.cpp

November 2026

Nemotron 3.5 Transcribe writes down who said what

40 languages Speaker labels Offline

Not yet word timestamps, streaming

llama.cpp v0.9.0 release notes · Multi-modality changes · #31471

02 Speech in and out

Chatterbox is a new text-to-speech model in llama.cpp. Pocket TTS now speaks in ready-made voices instead of cloning one from a recording, and llama-server lists a model's voices and accepts one by name.

Slide text

llama.cpp

November 2026

Chatterbox joins, and voices come built in

• Chatterbox a new text-to-speech model
• Pocket TTS ready-made voices, no cloning
• llama-server lists the voices, takes one by name

llama.cpp v0.9.0 release notes · Multi-modality changes · #30993 #31429

02 Speech in and out

New models you can run.

Slide text

llama.cpp

November 2026

03

New models

llama.cpp v0.9.0 release notes

03 New models

Mistral Large 4 now runs locally. So does Clover, a diffusion model from DiffuSpace, though only in ordinary token-by-token mode for now. Limite 1B Violetto, a small math model, arrives too.

Slide text

llama.cpp

November 2026

• Mistral Large 4
• Clover-35B-A3B-Preview a diffusion model, one token at a time
• Limite 1B Violetto a small math model

llama.cpp v0.9.0 release notes · New models · #31376 #31306 #29433

03 New models

Metis-1.6 is a new family trained from scratch. Praxis and Logos mix Mamba-2 and attention layers and run the whole stack twice. Praxis generates about 250 tokens a second on an RTX 5070 Ti.

Slide text

llama.cpp

November 2026

• Praxis about 2.9B parameters
• Logos about 10.2B parameters
• Mamba-2 and attention with a mixture of experts

llama.cpp v0.9.0 release notes · New models · #31134

03 New models

Then a new number format, and more machines.

Slide text

llama.cpp

November 2026

04

New formats, more machines

llama.cpp v0.9.0 release notes

04 New formats, more machines

llama.cpp now stores weights as FP8 E4M3, the 8-bit float format most used for inference. FP8 on CUDA and Vulkan, and an FP8 KV cache, are still open pull requests, so GPU users wait.

Slide text

llama.cpp

November 2026

FP8 E4M3 is a new weight type

The 8-bit float format most used for inference

Still in open pull requests CUDA, Vulkan, FP8 KV cache

llama.cpp v0.9.0 release notes · Highlights · #27512

04 New formats, more machines

Tensor parallelism now spans several machines, over NVIDIA's NCCL library. Splitting Qwen 3.8 27B across two DGX Spark machines lifts generation from about 13 to 24 tokens a second.

Slide text

llama.cpp

November 2026

Two DGX Sparks run one model almost twice as fast

Qwen3.8 27B Q4_K_M · tensor parallel over NCCL · generated tokens per second

One node

13.0

Two nodes

23.5

llama.cpp v0.9.0 release notes · Changelog · #28967

04 New formats, more machines

And the web interface.

Slide text

llama.cpp

November 2026

05

The web UI

llama.cpp v0.9.0 release notes

05 The web UI

In the web interface, tool settings get a checkbox per MCP group. A streaming reply stays at the bottom until you scroll up, and a tool call that never returns shows as interrupted.

Slide text

llama.cpp

November 2026

• MCP tools a checkbox per group
• Long chats stay at the bottom until you scroll up
• Tool calls marked interrupted when they never return

llama.cpp v0.9.0 release notes · UI changes · #31303 #31311 #31314

05 The web UI

GitHub's Octoverse report features llama.cpp for the third year in a row. This year it names the project one of the top open source projects on GitHub in 2026.

Slide text

llama.cpp

November 2026

A top open source project, a third year running

llama.cpp v0.9.0 release notes · Highlights · GitHub Octoverse 2026

llama.cpp v0.9.0

If you run the small Gemma 4 models on a laptop, I'd update today, since the speedup costs you nothing. If you build on the library, the release notes list every API change.

Slide text

llama.cpp

November 2026

Building on the library? The llama.cpp v0.9.0 release notes list every API change

github.com/ggml-org/llama.cpp · releases/tag/v0.9.0

llama.cpp v0.9.0