Persistence Implementation - Revision history

Adamcolton at 20:13, 1 February 2019

2019-02-01T20:13:57Z

← Older revision		Revision as of 20:13, 1 February 2019
Line 21:		Line 21:

	Slices are not too bad because the mmap is backed by a slice. But it means append wouldn't work. Maps wouldn't work. I'd need to build a new set of data structures on top of an mmap.		Slices are not too bad because the mmap is backed by a slice. But it means append wouldn't work. Maps wouldn't work. I'd need to build a new set of data structures on top of an mmap.

			== Indexing ==
			I like the idea of just having data structures to do indexing. The nodes of a given structure just become another object. The roots need special values.

	== Structs ==		== Structs ==

2019-02-01T20:08:49Z

← Older revision		Revision as of 20:08, 1 February 2019
Line 1:		Line 1:
			The goal of a persistence project would be
			* Automatic CRUD operation of designated objects
			* Ability to define performant indexes
			* Ability to define text searches

	== Cache ==		== Cache ==
	This could be implemented with a combination of a memory limit and priority list (able to set priority to something like last accessed). But that requires something to watch access and estimate memory use.		This could be implemented with a combination of a memory limit and priority list (able to set priority to something like last accessed). But that requires something to watch access and estimate memory use.

2019-02-01T20:06:10Z

Rolling

← Older revision		Revision as of 20:06, 1 February 2019
Line 46:		Line 46:
	This creates quite a bit of complexity. And it relies on statistical patterns. And will require regular clean ups.		This creates quite a bit of complexity. And it relies on statistical patterns. And will require regular clean ups.

	Live objects will be tracked in an index, mapping them to the file they are stored in. The index will also keep track of how many live objects are stored in a file. If a file has 0 live objects, it can be deleted.		Live objects will be tracked in an index, mapping them to the file they are stored in. The index will also keep track of how many live objects are stored in a file. If a file has 0 live objects, it can be deleted. A cleanup can also look for files with the fewest objects and move them to an active file.

2019-02-01T20:04:03Z

← Older revision		Revision as of 20:04, 1 February 2019
Line 40:		Line 40:

	Which is a little cumbersome, but not terrible. We may not need to break out the text indexing fields.		Which is a little cumbersome, but not terrible. We may not need to break out the text indexing fields.

			== Rolling ==
			When using a memory mapped file, it seems that I can only map one section of the file at a time. So rather than one large file, I think it makes sense to have a few smaller files. A rolling cache can have a set number of files mapped at any time. It can also have more files open in a read only mode. Active objects are kept in the memory mapped files. When a file is full, it goes into read only mode and it's active objects are moved to a mapped file.

			This creates quite a bit of complexity. And it relies on statistical patterns. And will require regular clean ups.

			Live objects will be tracked in an index, mapping them to the file they are stored in. The index will also keep track of how many live objects are stored in a file. If a file has 0 live objects, it can be deleted.

2018-08-01T15:52:42Z

Structs

← Older revision		Revision as of 15:52, 1 August 2018
Line 39:		Line 39:
	}		}

	Which is a little cumbersome, but not terrible.		Which is a little cumbersome, but not terrible. We may not need to break out the text indexing fields.

2018-08-01T15:51:39Z

Structs

← Older revision		Revision as of 15:51, 1 August 2018
Line 38:		Line 38:
	Points byte		Points byte
	}		}

			Which is a little cumbersome, but not terrible.

2018-08-01T15:50:47Z

← Older revision		Revision as of 15:50, 1 August 2018
Line 16:		Line 16:

	Slices are not too bad because the mmap is backed by a slice. But it means append wouldn't work. Maps wouldn't work. I'd need to build a new set of data structures on top of an mmap.		Slices are not too bad because the mmap is backed by a slice. But it means append wouldn't work. Maps wouldn't work. I'd need to build a new set of data structures on top of an mmap.

			== Structs ==
			I think structs will end up like this
			type Task struct{
			TaskKey
			*FooIndex
			*FooText
			*FooData
			}
			type TaskKey []byte
			type TaskIndex struct{
			Done bool
			Label LabelKey
			Priority byte
			}
			type TaskText struct{
			Title string
			Notes string
			}
			type TaskData struct{
			Points byte
			}

2018-08-01T15:45:17Z

← Older revision		Revision as of 15:45, 1 August 2018
Line 8:		Line 8:

	BoltDB is nice. There are some very interesting combinations of using many files and many sub-buckets.		BoltDB is nice. There are some very interesting combinations of using many files and many sub-buckets.

			I could also do an abstract key/value store so that I could use any of the no-sql servers.

	=== mmap ===		=== mmap ===

2018-08-01T15:15:47Z

← Older revision		Revision as of 15:15, 1 August 2018
Line 9:		Line 9:
	BoltDB is nice. There are some very interesting combinations of using many files and many sub-buckets.		BoltDB is nice. There are some very interesting combinations of using many files and many sub-buckets.

			=== mmap ===
	A memory mapped file would really be cool. Could be one file per object, one file per object Type, some sort of file key (like User ID). The really tricky part here is that I would like to support slices and maps but I don't know how to map that to a memory mapped file.		A memory mapped file would really be cool. Could be one file per object, one file per object Type, some sort of file key (like User ID). The really tricky part here is that I would like to support slices and maps but I don't know how to map that to a memory mapped file.
	* https://golang.org/pkg/syscall/#Mmap		* https://golang.org/pkg/syscall/#Mmap

			Slices are not too bad because the mmap is backed by a slice. But it means append wouldn't work. Maps wouldn't work. I'd need to build a new set of data structures on top of an mmap.

2018-08-01T14:58:12Z

← Older revision		Revision as of 14:58, 1 August 2018
Line 10:		Line 10:

	A memory mapped file would really be cool. Could be one file per object, one file per object Type, some sort of file key (like User ID). The really tricky part here is that I would like to support slices and maps but I don't know how to map that to a memory mapped file.		A memory mapped file would really be cool. Could be one file per object, one file per object Type, some sort of file key (like User ID). The really tricky part here is that I would like to support slices and maps but I don't know how to map that to a memory mapped file.
			* https://golang.org/pkg/syscall/#Mmap