Persistence Implementation

The goal of a persistence project would be

• Automatic CRUD operation of designated objects
• Ability to define performant indexes
• Ability to define text searches

Cache

This could be implemented with a combination of a memory limit and priority list (able to set priority to something like last accessed). But that requires something to watch access and estimate memory use.

Less clean, but easier is a weak map with uintptr.

Storage

I don't think sql does me much good. It handles indexing, but that's not a really hard problem. Some engines sort of handle NLP search, but the results are clumsy and inconsistent.

BoltDB is nice. There are some very interesting combinations of using many files and many sub-buckets.

I could also do an abstract key/value store so that I could use any of the no-sql servers.

mmap

A memory mapped file would really be cool. Could be one file per object, one file per object Type, some sort of file key (like User ID). The really tricky part here is that I would like to support slices and maps but I don't know how to map that to a memory mapped file.

• https://golang.org/pkg/syscall/#Mmap

Slices are not too bad because the mmap is backed by a slice. But it means append wouldn't work. Maps wouldn't work. I'd need to build a new set of data structures on top of an mmap.

Indexing

I like the idea of just having data structures to do indexing. The nodes of a given structure just become another object. The roots need special values.

Structs

I think structs will end up like this

 type Task struct{
   TaskKey
   *FooIndex
   *FooText
   *FooData
 }
 type TaskKey []byte
 type TaskIndex struct{
   Done bool
   Label LabelKey
   Priority byte
 }
 type TaskText struct{
   Title string
   Notes string
 }
 type TaskData struct{
   Points byte
 }

Which is a little cumbersome, but not terrible. We may not need to break out the text indexing fields.

Rolling

When using a memory mapped file, it seems that I can only map one section of the file at a time. So rather than one large file, I think it makes sense to have a few smaller files. A rolling cache can have a set number of files mapped at any time. It can also have more files open in a read only mode. Active objects are kept in the memory mapped files. When a file is full, it goes into read only mode and it's active objects are moved to a mapped file.

This creates quite a bit of complexity. And it relies on statistical patterns. And will require regular clean ups.

Live objects will be tracked in an index, mapping them to the file they are stored in. The index will also keep track of how many live objects are stored in a file. If a file has 0 live objects, it can be deleted. A cleanup can also look for files with the fewest objects and move them to an active file.