Grouped Value Attention matches GQA on a 47% smaller KV cache

A chat model keeps a store of numbers for every word it has already read. That store is the KV cache, and it fills GPU memory in a long chat. For each word the cache holds a key, which the model matches against, and a value, the content it pulls out. Grouped Value Attention keeps the values and drops the keys. When the model needs a key, it rebuilds that key from the value. The paper's Figure 1 draws both designs. On the left, grouped-query attention stores keys and values. On the right, Grouped Value Attention stores values only.

Each new word asks a question, called the query. The model scores that query against every stored key, then pulls in the values whose keys match best. A model asks many of those questions at once, one per attention head. Grouped-query attention, or GQA, lets several heads share one stored key and one stored value. On models of about 350 million parameters, the paper counts a Grouped Value Attention cache 47 percent smaller than GQA's. On five reasoning quizzes, Grouped Value Attention averages 44.35 percent accuracy. GQA averages 44.36 percent.

In Grouped Value Attention, each head owns a small grid of learned numbers, called a map. Multiply a stored value by the map, and out comes that head's key. Scoring then multiplies the query by that rebuilt key. The map stays fixed once training ends. So the model can multiply the query by the map once, and score the query straight against the stored value. Equation 7 in the paper shows both routes give the same score, and the authors call the identity exact. The cache holds values only. Each head also gets its own key. In GQA, every head in a group shares one key.

The team tried a simpler cut first. They used each stored value as its own key, with no map. That version also halves the cache. Figure 3 plots its training error, how often the model guesses the next token wrong. Tokens are the word pieces a model reads. The error stays above GQA for the whole stretch the figure plots, the first 4.4 billion tokens of a 30 billion-token run. The gap narrows by the end of the plot. The map gives each key room to differ from its value.

Word order breaks the shortcut. Models mark word order by rotating the query and the key by an angle. That angle depends on how far apart the two words sit. This rotation is called RoPE, and it lands between the query and the map. The angle changes for every pair of words, so the map can no longer move onto the query once. The fix comes from MLA, DeepSeek-V2's attention design, another way to shrink the cache. Each head splits in two. The content part rebuilds its key from the value, without rotation, so the fold still works. A small position part carries the rotation. Its key is shared by every head and stored, 16 or 24 numbers per token. That slice is why the saving is 47 or 45 percent. Values alone would save exactly half.

The rebuilt keys also started at the wrong size. With the usual starting values, the keys came out far smaller than the queries. Attention then spread almost evenly across every word, and early training spent its steps recovering. The team now starts each map so key size matches query size. Figure 4 is the paper's sketch of that effect, on ten made-up positions. Each panel has its own colour scale. The busy middle panel is the flat one, because its colour bar barely changes.

Every model here has about 350 million parameters. Each trained on 30 billion tokens of FineWeb-Edu, a public set of educational web pages. The team trained each design three times from different random starting points and averaged the scores. The quizzes run with no worked examples first. GQA averages 44.36 percent. Grouped Value Attention with the 16-number position slice averages 44.35 percent, and with the 24-number slice, 44.29 percent. MLA averages 43.88 percent. The top row in the table, at 44.41 percent, uses ordinary rotation, so that version cannot use the fold.

The 45 to 47 percent saving counts stored numbers. The paper reports no decoding speed, latency, peak memory or batch size. The authors say they have built custom kernels, the low-level GPU code that would turn a smaller cache into faster serving. They are testing those kernels now, with a release planned. Every comparison uses one model size, one dataset and three runs. The paper does not state the context length and runs no long-context test. The arXiv listing's summary gives 44.18 percent for the headline version. Both versions of the paper say 44.35 percent. The model the team released is Lumma, a 0.6 billion-parameter base model. Lumma uses the first cut from Figure 3, where each value doubles as its key.

Skim it. Read section 3 for the fold and the shared position key, if the KV cache limits the models you run or design. Hold off on implementing it. The evidence is one model size of about 350 million parameters, with zero timings, and the kernels are unreleased. The authors say the throughput numbers come with that kernel release. The paper is Grouped Value Attention, by Vishesh Tripathi, Abhay Kumar and Ramsha Khan at FrontiersMind. It is an arXiv preprint, number 2609.13285.