Q2_0 on WebGPU
llama.cppWebGPU now runs Q2_0 matrix multiplication.
Ternary Bonsai 1.7B and Bonsai 2 2B stop falling back to the CPU for matrix multiplication.
This change teaches the WebGPU backend to run Q2_0 weights. It adds Q2_0 to the get rows shader and to matrix multiplication with F32 activations. The models it is for are Ternary Bonsai 1.7B and Bonsai 2 2B. Until now their matrix multiplications fell back to the CPU, and with this change they stay on the GPU.
Where the work ran
llama.cppOn master, every Q2_0 matrix multiply ran on the CPU.
Here is what that meant on master. The scheduler asks each backend whether it supports an operation, and WebGPU said no to Q2_0. So the embedding lookup and every matrix multiply over those weights ran on the CPU. With this change the matrix multiplies move to WebGPU. The input embedding lookup still runs on the CPU for both models.
The Q2_0 block
llama.cppA Q2_0 block packs 64 weights into 18 bytes.
The format is compact. Each block holds 64 weights in 18 bytes, and the first two bytes are a half precision scale, d. The other 16 bytes are read as four 32-bit words, and each word carries 16 two-bit codes. A code q decodes to q minus one, times d. So zero, one and two give minus d, zero and plus d, the three values a ternary model needs.
Host side: four lines
llama.cppFour lines of C++ route Q2_0 to the new shaders.
static bool ggml_webgpu_supported_qtype(ggml_type type) { case GGML_TYPE_Q1_0: case GGML_TYPE_Q2_0:static bool ggml_backend_webgpu_device_supports_op(…) { case GGML_TYPE_Q2_0:
case GGML_TYPE_Q2_0:} else if (key.src_type == GGML_TYPE_Q2_0 || key.src_type == GGML_TYPE_NVFP4) { defines.push_back("BLOCK_SIZE=64u");
On the host side the change is small. In ggml-webgpu.cpp, Q2_0 joins the supported quant types, and supports op now says yes to it. In the shader library, Q2_0 joins the same switch as Q1_0 and Q4_0, and gets a block size of 64, the define NVFP4 already uses.
Three shaders
llama.cppThe same decode lands in three WGSL shaders.
Writes 64 F32 values per block.
Unpacks 16 codes per thread into shared memory as F16.
Four threads per block, 16 weights each.
The decode itself appears in three shaders. The get rows shader copies rows out of a Q2_0 tensor and writes 64 floats per block. The tiled matrix multiply unpacks 16 codes per thread into shared memory, as half precision. And the matrix-vector kernel splits each block across four threads, 16 weights apiece.
The vector kernel
llama.cppEach thread reads one 32‑bit word: 16 weights.
#define THREADS_PER_BLOCK 4 #define ELEMS_PER_THREAD (BLOCK_SIZE/THREADS_PER_BLOCK) // … let d = f32(load_f16_at_src0(block_byte_base)); let q_packed = load_u32_at_src0(block_byte_base + 2u + 4u * thread_within_block); for (var col = 0u; col < NUM_COLS; col++) { var row_sum = 0.0; for (var i = 0u; i < ELEMS_PER_THREAD; i++) { let q = (q_packed >> (2u * i)) & 3u; row_sum += (f32(q) - 1.0) * x_block[col][i]; } acc[col][row] += d * row_sum; }
Here is that kernel. Four threads share each 64 weight block, so each thread owns 16 weights. A thread loads the block's scale, then exactly one 32-bit word, which holds its 16 codes. Each code is shifted out two bits at a time, decoded as q minus one, and multiplied by the activation. The scale is applied once, after the sum.
Benchmark
llama.cppPrompt processing jumps from 35 to about 2,000 tokens a second.
pp512
tg128
On an RTX 3060 with native Dawn WebGPU, prompt processing for Ternary Bonsai 1.7B goes from 35 tokens a second to about 1,900. Bonsai 2 2B lands just over 2,000. Token generation goes from 14 to about 95 tokens a second on the smaller model, and from about 13 to about 72 on Bonsai 2.
Checked against the CPU
llama.cppQ2_0 results match the CPU backend.
Correctness is checked against the CPU backend. Q2_0 passes test backend ops there, and so do Q1_0, Q2_K and Q4_0. And the local WebGPU CI run passed.
What stays unsupported
llama.cppQ2_0 MUL_MAT_ID and F16 activations are still unsupported.
The change stops short in a few places. The input embedding lookup still runs on the CPU in both Bonsai models. Q2_0 isn't supported in the expert matrix multiply that mixture of experts models use. And activations have to be F32, because F16 activations aren't supported for Q2_0.
Q2_0 on WebGPU, merged
llama.cppQ2_0 matrix multiplies run on WebGPU now.
Dense Q2_0 models with F32 activations. The embedding lookup stays on the CPU.
So the scope is dense Q2_0 models with F32 activations, and the embedding lookup stays on the CPU. Within that, Ternary Bonsai 1.7B and Bonsai 2 2B now multiply on the GPU, and prompt processing went from 35 to about 2,000 tokens a second. It's merged, so if you run Q2_0 models on WebGPU, pull master and run llama bench on your own card.
