Needle 2 · a model that turns what you say into an exact command
Needle 2
A watch has almost no memory to spare. Whatever turns your words into a command has to fit in what is left. Needle 2 does that job. The file the project trained is just over ninety megabytes. The file that ships to the device is thirteen point seven megabytes, and that is the whole model.
It answers only in the functions you declared
It answers only in the functions you declared.
You write an ordinary function in your own code. Then you say what you want. Dim the bedroom lights to twenty percent, and lock the front door. What comes back is the exact instruction your program takes: which action to run, and the values to run it with. Ask for a poem instead and the answer comes back empty, because writing poems is not one of your functions.
Memory is the hard limit on a watch or a phone
Memory is the hard limit on a watch or a phone.
On a watch, memory is the number that cannot move. A model eats memory as a conversation runs. The model keeps running notes on everything said so far. Needle 2 runs a whole session in about twenty-eight megabytes. And every answer Needle 2 sends back carries the memory that answer used, right there in the reply.
The memory budget is fixed first, and it decides the most this can hold
The memory budget is fixed first, and it decides the most this can hold.
Models are normally described by how much conversation they can hold. Needle 2 works backwards. The source picks the memory first: eleven and a half megabytes, written down as a constant. Then the code works out what one more chunk of text costs to remember. A chunk is a short word or a piece of one. That costs about sixteen thousand seven hundred bytes. Divide the budget by that cost, and the answer is a hard ceiling on how far back this model may ever see. The project writes that ceiling into the file that ships.
The model cannot forget what it is allowed to do
The model cannot forget what it is allowed to do.
The model that shipped keeps a window of two hundred and fifty-six chunks of text, well under that ceiling. A long conversation would push the description of what this model may do off the back of a window that short. So the project keeps that description exempt, and marks it on the model’s own diagram. The description stays in memory however far behind it falls.
The rule comes from your code, not from the model
The rule comes from your code, not from the model.
Needle 2 turns your own function into a rule. That rule decides, letter by letter, which characters are allowed next. If your function accepts only heat, cool or auto, there is no path through which any other word can be produced. The project’s own documentation says the model cannot emit anything else. With one function declared, it says, the shape of the answer is guaranteed rather than requested. The reasoning the model writes alongside is left free, so that part stays readable.
It trades five points of accuracy for a model a fraction of the size
It trades five points of accuracy for a model a fraction of the size.
On the project’s own chart, scored on a public set of everyday phone commands that Google put out, Needle 2 lands at sixty-three point seven percent. That is level with a model six times its size, and five points of accuracy off the highest score there. Five points, for a model five to sixty-seven times smaller that runs with the network off.
Builds for sixteen kinds of device, and a 13.7 MB model file
Builds for sixteen kinds of device, and a 13.7 MB model file.
The project publishes builds for sixteen kinds of device: phones, watches and TVs. Needle 2 is free and open, and it installs from the Python package index. After the first run, a machine with no connection at all can use Needle 2. And that thirteen point seven megabytes is the file that goes on the device, not the size of the download.
A model sized by what the device can spare
A model sized by what the device can spare.
Needle 2 is described by the memory it is allowed to spend, and how much conversation that buys is what falls out. The project’s own chart leaves open how much smaller this can get before the answers stop being good enough. Cactus dash compute slash needle, on GitHub. New repo tomorrow.
