Two dials past their usual range let Kimi Delta Attention turn its memory
The memory inside Kimi Delta Attention can fade and it can erase, but it can't turn. This paper makes it turn with two mirrors. Kimi Delta Attention, or KDA, is the fast kind of attention layer inside Kimi K3, Moonshot AI's language model. A language model is the kind of AI that writes a chatbot's replies. Kimi K3 uses KDA in sixty-nine of its ninety-three layers, the stages each word passes through in turn. Instead of rereading every earlier word, KDA keeps one running memory and updates it after every word. Turning is what that memory needs to keep track of order, like where a shuffled card ended up. The authors' version, Complex KDA, lets two of KDA's dials go past their usual range, and a single layer can then turn its memory. On language models with 1.3 billion parameters, the adjustable numbers a model learns, the scores come out the same as standard KDA.
A forget gate at minus one flips the slot’s sign, and that flip is a mirror
wiped
kept
flipped
Every word, KDA updates its memory in two moves. The first move is the forget gate, a dial on each slot of the memory. At one, the slot keeps its value. At zero, the slot is wiped. Standard KDA keeps every dial between zero and one. The authors let the dial reach minus one. At minus one, the slot keeps its value with the sign flipped. Picture one slice of the memory as an arrow. Flip one side of the arrow and you get its mirror image. So a gate at minus one is a mirror.
A write at double strength is a second mirror, and two mirrors make a turn
keeps
erases
mirrors
The second move is the write, called the delta rule. Before KDA writes a new fact, it subtracts what the memory already says about the current word. The write strength sets how hard it writes. At one, the write wipes the old answer for that word. Standard KDA stops the write strength at one. The authors let it reach two. At two, the write overshoots by the same distance again, and the arrow lands on its mirror image. The current word decides where that mirror stands. So one step can now hold two mirrors, the gate's mirror and the write's mirror. Reflect an arrow in one mirror, then in a second mirror tilted against the first, and the arrow ends up turned, by twice the angle between the two mirrors. A memory that can turn works like a clock hand. Each word moves the hand by some angle, and after enough moves the hand comes back to where it started. That is how a single layer keeps count around a cycle.
Complex KDA needs both dials to turn, and it turns with one write per word
Standard KDA holds neither mirror, because both of its dials stop at one. Kimi K3, for one, keeps its gate above zero. Stretch only one dial and you get one mirror, and a single mirror can only flip. The paper proves that the turn appears only when both dials are stretched. An earlier design called DeltaProduct already gets the turn by doing two writes for every word. Complex KDA gets the same turn with one write per word. The authors also prove the fit is exact. Every step of this shape that keeps the arrow's length the same is one of these two-mirror steps.
Only the layer with both dials stretched keeps track of long shuffles
The first test is a shuffle puzzle. A single-layer model reads a list of shuffles of three or four items and says where everything ends up. Training lists run up to thirty-two shuffles, and test lists up to five hundred and twelve shuffles. Scores run from zero for guessing to one for perfect, best of three training runs. Standard KDA scores near zero, even on short lists. With one dial stretched, the three-item score falls to about zero point two on long lists, and the four-item puzzle fails even on short lists. With both dials stretched, Complex KDA stays near perfect on three items to two hundred and fifty-six shuffles. On four items, it stays near perfect to a hundred and twenty-eight shuffles and ends near zero point six. DeltaProduct matches it on three items and leads on four items until about four hundred and fifty shuffles.
Complex KDA keeps a sound wave clean at nearly twice its training length
In the second test, a single-layer model hears the first half bar of a synthetic repeating groove, a short looped sound wave, then keeps the groove going on its own. Training uses sequences up to a hundred and thirty-six steps long. At two hundred and sixty-four steps, Complex KDA's continuation still has thirty-eight point one decibels of signal over noise, where higher means cleaner. A standard Transformer, the design behind most chatbots, falls to two point eight decibels. A GRU, an older kind of step-by-step network, stays more accurate than all of them.
At 1.3 billion parameters Complex KDA ties standard KDA, and training still uses the turns
The last test trains language models. The authors train Complex KDA and standard KDA at 1.3 billion parameters, on a hundred billion tokens of educational web text. Tokens are the word pieces a model reads. Across nine common-sense reasoning tests, Complex KDA averages fifty-four point oh six percent and standard KDA fifty-four point oh nine. That is a tie. On an Nvidia H100 chip, Complex KDA runs at about ninety-six to ninety-seven percent of KDA's speed. During training, the model starts to use the new range, and turns show up in every layer, most often in the first two.
One layer can hold a puzzle it never learns, and it turns in one plane per step
The first limit is learning. A harder puzzle uses the sixty even shuffles of five items, the ones you can make with an even number of swaps. That puzzle fits inside one Complex KDA layer on paper, and the layer failed to learn it from scratch. Second, each write touches one direction of the memory, so each step can turn in one plane at most. Third, Complex KDA has no speed edge over DeltaProduct, which already had this power at competitive speed. Fourth, the scale is small. The puzzles use one layer and the best of three runs, the language models stop at 1.3 billion parameters, and every result so far is the authors' own. The authors say the language model uses the turns in every layer, and what those turns do there is still unclear.
Read it for the two-mirror argument in Section 3
Read it. Section three shows exactly why a negative gate and a double-strength write make a turn. Hold off on implementing it. At 1.3 billion parameters, Complex KDA ties standard KDA on language, and DeltaProduct already offers the same turn at similar speed. The code and four trained models are public. The paper is a preprint on arXiv, the open archive for research papers, posted before peer review.
