turbovec · a search index for long lists of numbers, packed small
turbovec
Type beach into your photos app and it finds beach pictures. Search like that works by turning every photo into a long list of numbers that stands for what is in it. turbovec is a search index. It stores those lists, searches them, and keeps each one in a fraction of the space.
Five lines of Python, no training call
You tell turbovec how long your lists are and how hard to squeeze them. You add the lists. You search them. You write the index to a file. That is the whole surface. There is no step where you hand over a sample and wait.
Normally the rounding table is fitted to a sample of your own data
Shrinking a list means rounding: every number gets pushed to the nearest value on a short list, and you store which one you picked. That short list is the rounding table. To build one you normally take a sample of your own lists and see where your numbers tend to sit. So the table fits that sample. If the lists you add later sit somewhere else, the table you fitted no longer fits them.
The rounding table is solved from two numbers, and neither is yours
Here is the clever part. The function that builds turbovec's rounding table takes two things: how many bits you spend on each number, two, three or four, and how long the lists are. From those two numbers, the function solves for the best places to round to. There is nowhere in that list of inputs to put one of your lists, and the note above the function says the answer depends on those two numbers and nothing else.
Every list is scrambled the same fixed way before anything is rounded
You cannot work out the best rounding points for data in general. So turbovec makes the data stop being general. Every list gets its length taken off and stored as one number, so what is left is pure direction. Then every direction is pushed through the same shuffle. After that shuffle, the numbers in a list spread out in a shape you can work out in advance, whatever the list started as. Once you know the shape, where to round to is arithmetic, and the answer is the same for everybody. The small numbers get packed tight, and the length goes back on. The idea of scrambling first comes from Google Research.
The scramble is a constant in the source, so every machine encodes the same bytes
The scramble needs random shuffles and sign flips. Here is where it gets good: the randomness is thirty-two numbers typed straight into the source file. Change one of them and every index ever written under this format becomes unreadable. One round is a shuffle, a sign flip on every number, then a mixing step inside each block, and the whole round runs twice. Before this version, turbovec made that randomness while it ran, so the encoded bytes depended on the machine and its processor core count. An index written on one computer could decode differently on another. The new scramble also dropped a forty-two megabyte library.
Everything runs on your own machine
It is MIT licensed, and version one point oh is five days old. Install it from the Python package index or the Rust one, and it runs entirely on your own machine, with no service in the middle. One step is optional and it does read your data: calibration takes about a thousand of your lists, fits two numbers for each position in the list, and redoes every list you already stored.
The rounding table depends on your settings, never on your data
The rounding table depends on your settings, never on your data.
Compression normally means finding the patterns in your own data. turbovec makes your data stop being particular first, and then one answer serves everybody. Ryan Codrai slash turbovec, on GitHub. New repo tomorrow.
