Persistence
This document is meant to build an abstract model of a persistence engine. I am specifically trying to avoid implementation details.
Also See
Persistence is complex. Goals are
- Data should move between memory and disk transparently
- Types can be migrated to add, remove or rename fields, change the type of a field or perform arbitrary changes via a migrate function.
Model
- The Store is collection of all records
- Every record has an ID unique to the store
- Every record has a Type that describes it's fields
- Index are an ordered subset of fields and allow O(m log n) look up of sets where m is the members of the set and n is type membership
- A record can have many indexes
- A record can belong to many categories
- A category can belong to many categories
- Models can reference each other
- Natural(ish) language searches can be performed efficiently
Resources
- https://blog.kowalczyk.info/software/volante/database.html
- https://en.wikipedia.org/wiki/Comparison_of_object_database_management_systems
- https://en.wikipedia.org/wiki/Object_database
External Indexing
One option to achieve indexing is to separate it from the core system. A data structure can fulfill the necessary interfaces for persistence and then that can act as the index. There just needs to be a listener to cause re-indexing.
Reflection
Recent experiments with reflection have shown that it is not as slow as I expected. I'm also working with using reflection to build an unsafe serialization strategy, though the results so far are mixed.
Migration Strategy
If I give every field an index then I can return a slice of unused fields that can be used for migrations. Say a field changes type, it would also change index. When an unmigrated record is pulled, unknown fields would be detected.