Persistence: Difference between revisions
Adamcolton (talk | contribs) No edit summary |
Adamcolton (talk | contribs) No edit summary |
||
| (8 intermediate revisions by the same user not shown) | |||
| Line 8: | Line 8: | ||
* Data should move between memory and disk transparently | * Data should move between memory and disk transparently | ||
* Types can be migrated to add, remove or rename fields, change the type of a field or perform arbitrary changes via a migrate function. | * Types can be migrated to add, remove or rename fields, change the type of a field or perform arbitrary changes via a migrate function. | ||
* Can translate and decorate a user search into a query | |||
Model | Model | ||
| Line 14: | Line 15: | ||
* Every record has a Type that describes it's fields | * Every record has a Type that describes it's fields | ||
* Index are an ordered subset of fields and allow O(m log n) look up of sets where m is the members of the set and n is type membership | * Index are an ordered subset of fields and allow O(m log n) look up of sets where m is the members of the set and n is type membership | ||
* A record can have many indexes | |||
* A record can belong to many categories | * A record can belong to many categories | ||
* A category can belong to many categories | |||
* Models can reference each other | |||
* Natural(ish) language searches can be performed efficiently | |||
Resources | Resources | ||
| Line 20: | Line 25: | ||
* https://en.wikipedia.org/wiki/Comparison_of_object_database_management_systems | * https://en.wikipedia.org/wiki/Comparison_of_object_database_management_systems | ||
* https://en.wikipedia.org/wiki/Object_database | * https://en.wikipedia.org/wiki/Object_database | ||
[[Persistence Implementation]] | |||
== External Indexing == | |||
One option to achieve indexing is to separate it from the core system. A data structure can fulfill the necessary interfaces for persistence and then that can act as the index. There just needs to be a listener to cause re-indexing. | |||
A basic index takes a query and returns a slice of IDs. | |||
A storage index can take an ID and return the data representing the section of the record indexed. This means that portion of the record can be stored in the index, not in the primary record. | |||
== Reflection == | |||
Recent experiments with reflection have shown that it is not as slow as I expected. I'm also working with using reflection to build an unsafe serialization strategy, though the results so far are mixed. | |||
== Migration Strategy == | |||
If I give every field an index then I can return a slice of unused fields that can be used for migrations. Say a field changes type, it would also change index. When an unmigrated record is pulled, unknown fields would be detected. | |||
[[Category:Code]] | |||
[[Category:Projects]] | |||
Latest revision as of 17:19, 19 March 2019
This document is meant to build an abstract model of a persistence engine. I am specifically trying to avoid implementation details.
Also See
Persistence is complex. Goals are
- Data should move between memory and disk transparently
- Types can be migrated to add, remove or rename fields, change the type of a field or perform arbitrary changes via a migrate function.
- Can translate and decorate a user search into a query
Model
- The Store is collection of all records
- Every record has an ID unique to the store
- Every record has a Type that describes it's fields
- Index are an ordered subset of fields and allow O(m log n) look up of sets where m is the members of the set and n is type membership
- A record can have many indexes
- A record can belong to many categories
- A category can belong to many categories
- Models can reference each other
- Natural(ish) language searches can be performed efficiently
Resources
- https://blog.kowalczyk.info/software/volante/database.html
- https://en.wikipedia.org/wiki/Comparison_of_object_database_management_systems
- https://en.wikipedia.org/wiki/Object_database
External Indexing
One option to achieve indexing is to separate it from the core system. A data structure can fulfill the necessary interfaces for persistence and then that can act as the index. There just needs to be a listener to cause re-indexing.
A basic index takes a query and returns a slice of IDs.
A storage index can take an ID and return the data representing the section of the record indexed. This means that portion of the record can be stored in the index, not in the primary record.
Reflection
Recent experiments with reflection have shown that it is not as slow as I expected. I'm also working with using reflection to build an unsafe serialization strategy, though the results so far are mixed.
Migration Strategy
If I give every field an index then I can return a slice of unused fields that can be used for migrations. Say a field changes type, it would also change index. When an unmigrated record is pulled, unknown fields would be detected.