LatticeDB · one file holding your data and everything built to search it
LatticeDB
You have searched your own email for a message you could see, and it wasn’t there. This is LatticeDB, and one question runs three kinds of search. It finds paragraphs that mean what you asked for, even without your words in them. It finds the paragraphs that do contain your words. Then the question walks from a paragraph to its document, and on to the author. All three are in one file.
There is no server to run and nothing to configure
You open LatticeDB the way you open a file. Every part of LatticeDB reads and writes through one patch of memory, and underneath that, one file. Everything you store ends up in that one file.
A search index is a second structure built from your data
Here is your data: your things, and the links between them. To search it quickly, LatticeDB builds two more structures out of it. The meaning index is layers of links between your items: a few long jumps at the top, every item at the bottom. The word index is a dictionary from each word to the items containing that word. Change your data, and both indexes change too.
The search index is made of the same blocks as the data
Here is the part worth knowing. This whole file is nothing but blocks, four kilobytes each. The meaning index keeps one record for each item, and that record is twenty-one bytes. Eight bytes name the item. One byte says how high it sits in those layers. And the last twelve bytes are two addresses: which block holds its numbers, and which block holds its links to its neighbours. The links are in blocks too. The project calls a block a page, and its own comment says the connections for many items get packed into shared pages. Every block carries a tag saying what it holds. The tags for the two search indexes sit in the same list as the tags for your data.
One numbered list says where every structure starts
The very first block of the file holds one numbered list. Your things are number zero. Your links are number one. Five, six and seven are the word index, and eight is the meaning index. The list is sixteen slots long, and each slot holds the block where one structure starts. Every one of those blocks also carries the same kind of checksum, so one routine can tell whether any block in the file is intact.
Open a file and start working
All of that has a name. LatticeDB is an embedded graph database: your things and their links, in a file your own program opens. One line installs the command line tool, and three languages have libraries: Python, TypeScript and Go. LatticeDB is open source under the MIT licence, and it depends on nothing else. Version zero point thirteen has builds for Linux and macOS, on both Intel and ARM. It is built for one program on one machine owning the file.
There is no second copy to keep in step
Searching means building a second structure out of your data. Here that second structure is made of the same blocks as the data, in the same file. That is jeff hajewski slash latticedb, on GitHub. New repo tomorrow.
