WebGPU now runs Q2_0 matrix multiplication.

This change teaches the WebGPU backend to run Q2_0 weights. It adds Q2_0 to the get rows shader and to matrix multiplication with F32 activations. The models it is for are Ternary Bonsai 1.7B and Bonsai 2 2B. Until now their matrix multiplications fell back to the CPU, and with this change they stay on the GPU.

Here is what that meant on master. The scheduler asks each backend whether it supports an operation, and WebGPU said no to Q2_0. So the embedding lookup and every matrix multiply over those weights ran on the CPU. With this change the matrix multiplies move to WebGPU. The input embedding lookup still runs on the CPU for both models.

The format is compact. Each block holds 64 weights in 18 bytes, and the first two bytes are a half precision scale, d. The other 16 bytes are read as four 32-bit words, and each word carries 16 two-bit codes. A code q decodes to q minus one, times d. So zero, one and two give minus d, zero and plus d, the three values a ternary model needs.

On the host side the change is small. In ggml-webgpu.cpp, Q2_0 joins the supported quant types, and supports op now says yes to it. In the shader library, Q2_0 joins the same switch as Q1_0 and Q4_0, and gets a block size of 64, the define NVFP4 already uses.

static bool ggml_webgpu_supported_qtype(ggml_type type) {
        case GGML_TYPE_Q1_0:
        case GGML_TYPE_Q2_0:static bool ggml_backend_webgpu_device_supports_op(…) {
                            case GGML_TYPE_Q2_0:

    case GGML_TYPE_Q2_0:} else if (key.src_type == GGML_TYPE_Q2_0 || key.src_type == GGML_TYPE_NVFP4) {    defines.push_back("BLOCK_SIZE=64u");

The decode itself appears in three shaders. The get rows shader copies rows out of a Q2_0 tensor and writes 64 floats per block. The tiled matrix multiply unpacks 16 codes per thread into shared memory, as half precision. And the matrix-vector kernel splits each block across four threads, 16 weights apiece.

Here is that kernel. Four threads share each 64 weight block, so each thread owns 16 weights. A thread loads the block's scale, then exactly one 32-bit word, which holds its 16 codes. Each code is shifted out two bits at a time, decoded as q minus one, and multiplied by the activation. The scale is applied once, after the sum.

#define THREADS_PER_BLOCK 4
#define ELEMS_PER_THREAD (BLOCK_SIZE/THREADS_PER_BLOCK)
// …
let d = f32(load_f16_at_src0(block_byte_base));
let q_packed = load_u32_at_src0(block_byte_base + 2u + 4u * thread_within_block);
for (var col = 0u; col < NUM_COLS; col++) {
    var row_sum = 0.0;
    for (var i = 0u; i < ELEMS_PER_THREAD; i++) {
        let q = (q_packed >> (2u * i)) & 3u;
        row_sum += (f32(q) - 1.0) * x_block[col][i];
    }
    acc[col][row] += d * row_sum;
}

On an RTX 3060 with native Dawn WebGPU, prompt processing for Ternary Bonsai 1.7B goes from 35 tokens a second to about 1,900. Bonsai 2 2B lands just over 2,000. Token generation goes from 14 to about 95 tokens a second on the smaller model, and from about 13 to about 72 on Bonsai 2.

Correctness is checked against the CPU backend. Q2_0 passes test backend ops there, and so do Q1_0, Q2_K and Q4_0. And the local WebGPU CI run passed.

The change stops short in a few places. The input embedding lookup still runs on the CPU in both Bonsai models. Q2_0 isn't supported in the expert matrix multiply that mixture of experts models use. And activations have to be F32, because F16 activations aren't supported for Q2_0.

So the scope is dense Q2_0 models with F32 activations, and the embedding lookup stays on the CPU. Within that, Ternary Bonsai 1.7B and Bonsai 2 2B now multiply on the GPU, and prompt processing went from 35 to about 2,000 tokens a second. It's merged, so if you run Q2_0 models on WebGPU, pull master and run llama bench on your own card.