Persistence

This document is meant to build an abstract model of a persistence engine. I am specifically trying to avoid implementation details.

Also See

• Immortal Object Server
• Gothic Vault]

Persistence is complex. Goals are

• Data should move between memory and disk transparently
• Types can be migrated to add, remove or rename fields, change the type of a field or perform arbitrary changes via a migrate function.

Model

• The Store is collection of all records
• Every record has an ID unique to the store
• Every record has a Type that describes it's fields
• Index are an ordered subset of fields and allow O(m log n) look up of sets where m is the members of the set and n is type membership
• A record can have many indexes
• A record can belong to many categories
• A category can belong to many categories
• Models can reference each other
• Natural(ish) language searches can be performed efficiently

Resources

• https://blog.kowalczyk.info/software/volante/database.html
• https://en.wikipedia.org/wiki/Comparison_of_object_database_management_systems
• https://en.wikipedia.org/wiki/Object_database

Persistence Implementation

External Indexing

One option to achieve indexing is to separate it from the core system. A data structure can fulfill the necessary interfaces for persistence and then that can act as the index. There just needs to be a listener to cause re-indexing.

Reflection

Recent experiments with reflection have shown that it is not as slow as I expected. I'm also working with using reflection to build an unsafe serialization strategy, though the results so far are mixed.

Migration Strategy

If I give every field an index then I can return a slice of unused fields that can be used for migrations. Say a field changes type, it would also change index. When an unmigrated record is pulled, unknown fields would be detected.