Data - Revision history

Adamcolton: /* Fundamental Structures */

2020-04-29T19:52:52Z

Fundamental Structures

← Older revision		Revision as of 19:52, 29 April 2020
Line 39:		Line 39:
	; Tagged Union		; Tagged Union
	: One of a set of types with a tag indicating the type.		: One of a set of types with a tag indicating the type.

			I was considering an enum, but the List may be sufficient.

	== Mono Container ==		== Mono Container ==

Adamcolton at 19:51, 29 April 2020

2020-04-29T19:51:42Z

← Older revision		Revision as of 19:51, 29 April 2020
Line 53:		Line 53:
	** Lookup : The combination of ContainerType, KeyType and KeyValue provides a lookup - not sure how to make this work.		** Lookup : The combination of ContainerType, KeyType and KeyValue provides a lookup - not sure how to make this work.
	** Type : All the data has the same type.		** Type : All the data has the same type.

			The language is completely regular. No parser is needed. I'm not sure if it should be stack based, register based or both. But the size must be defined in the script.

			The language also is not Turing complete. No loops and no conditionals.

	== Shared Context ==		== Shared Context ==

Adamcolton: /* IDs */

2020-04-29T19:48:19Z

IDs

← Older revision		Revision as of 19:48, 29 April 2020
Line 58:		Line 58:

	== IDs ==		== IDs ==
	~~There~~ are ~~two fundamental ID types - one is intended~~ to be ~~a unique identifier either~~ from ~~an increment source or a guid. The other is a meaningful tag, like setting the key column in sql to a string~~.		IDs are either set, which allows for references to be built up or are derived from hashes.

	== Quick Lookup and Compact Data ==		== Quick Lookup and Compact Data ==

Adamcolton: /* Fundamental Structures */

2020-04-29T19:46:29Z

Fundamental Structures

← Older revision		Revision as of 19:46, 29 April 2020
Line 22:		Line 22:
	: A type and bytes where the type indicates how to interpret the bytes.		: A type and bytes where the type indicates how to interpret the bytes.

	; ~~Array~~		; Anonymous Type
			: The type is part of the value.

			; List
	: An ordered list, all of the same type. Variable length.		: An ordered list, all of the same type. Variable length.

	~~; Tagged Union~~
	~~: One of a set of types with a tag indicating the type.~~

	; Tuple		; Tuple
	: An ordered list where the tuple type defines the length and the type of each element. Last variadic?		: An ordered list where the tuple type defines the length and the type of each element. Last variadic?

			; Dict
			: An unordered list where each element has key and value.

	; Struct		; Struct
	: A set of fields. This also necessitates a field as type. A field is a tag (field name) and a type.		: A set of fields. This also necessitates a field as type. A field is a tag (field name) and a type.

	; ~~Associative Array~~		; Tagged Union
	: ~~An unordered list where each element has key and value.~~		: One of a set of types with a tag indicating the type.

	~~; Anonymous Type~~
	~~: The type is part~~ of the ~~value~~.

	== Mono Container ==		== Mono Container ==

Adamcolton: /* Fundamental Structures */

2020-04-29T19:44:37Z

Fundamental Structures

← Older revision		Revision as of 19:44, 29 April 2020
Line 19:		Line 19:
	== Fundamental Structures ==		== Fundamental Structures ==

	~~===~~ Value ~~===~~		; Value
	A type and bytes where the type indicates how to interpret the bytes.		: A type and bytes where the type indicates how to interpret the bytes.

	~~===~~ Array ~~===~~		; Array
	An ordered list, all of the same type.		: An ordered list, all of the same type. Variable length.

	~~===~~ Tagged Union ~~===~~		; Tagged Union
	One of a set of types with a tag indicating the type.		: One of a set of types with a tag indicating the type.

	~~===~~ Tuple ~~===~~		; Tuple
	An ordered list where the tuple type defines the length and the type of each element.		: An ordered list where the tuple type defines the length and the type of each element. Last variadic?

	~~===~~ Struct ~~===~~		; Struct
	~~An unordered list where each element has~~ a tag and a type.		: A set of fields. This also necessitates a field as type. A field is a tag (field name) and a type.

	~~===~~ Associative Array ~~===~~		; Associative Array
	An unordered list where each element has key and value.		: An unordered list where each element has key and value.

			; Anonymous Type
			: The type is part of the value.

	== Mono Container ==		== Mono Container ==

Adamcolton at 15:04, 28 April 2020

2020-04-28T15:04:07Z

← Older revision		Revision as of 15:04, 28 April 2020
Line 70:		Line 70:

	This further makes an argument to place the variable length field headers at the top, in reverse order. So the very first piece of data is the record length.		This further makes an argument to place the variable length field headers at the top, in reverse order. So the very first piece of data is the record length.

			== Namespace ==
			A structured namespace can be defined. A secure model would be something like
			* Resource (url)
			* Public Key
			* Hash
			* Hash bits

			Then the ID would be the defined bits of the hash. This would allow for fewer bits to be used to identify static resources.

Adamcolton at 14:04, 25 September 2019

2019-09-25T14:04:21Z

← Older revision		Revision as of 14:04, 25 September 2019
Line 56:		Line 56:
	== IDs ==		== IDs ==
	There are two fundamental ID types - one is intended to be a unique identifier either from an increment source or a guid. The other is a meaningful tag, like setting the key column in sql to a string.		There are two fundamental ID types - one is intended to be a unique identifier either from an increment source or a guid. The other is a meaningful tag, like setting the key column in sql to a string.

			== Quick Lookup and Compact Data ==
			When data is going to be fully deserialized, compaction tends to be advantageous. But for data at rest this causes lookups to be slow. If a struct has 5 variable length fields and we want the fifth, we have to pass through the first four.

			To make a struct in place that allows for fast lookups, we need a constant header size. Lengths need to be a constant size. But even defining lengths will still require reading all four, so instead we encode offsets. For example if we have
			Person:
			ID: int64;
			Age: byte;
			RoleID: byte;
			First: string;
			Last: string;
			Then the first 3 fields are fixed length. We have to place an upper limit on the length of string. So lets say we allocate 4 bytes for this. Then we now First starts at byte (8+1+1+4+4) 18. The 4 byte value in the First header field will actually be the relative offset of the end of first. And Last will be the relative offset of the end of Last as well as the end of the record.

			This further makes an argument to place the variable length field headers at the top, in reverse order. So the very first piece of data is the record length.

Adamcolton at 12:21, 25 September 2019

2019-09-25T12:21:56Z

← Older revision		Revision as of 12:21, 25 September 2019
Line 53:		Line 53:
	== Shared Context ==		== Shared Context ==
	An important feature is the ability to read data from a shared context. But this requires collision avoidance so the keys in a shared context are large. We don't want these large values repeated in the data, so there is an operation to read from the shared context into a register.		An important feature is the ability to read data from a shared context. But this requires collision avoidance so the keys in a shared context are large. We don't want these large values repeated in the data, so there is an operation to read from the shared context into a register.

			== IDs ==
			There are two fundamental ID types - one is intended to be a unique identifier either from an increment source or a guid. The other is a meaningful tag, like setting the key column in sql to a string.

Adamcolton at 15:41, 19 September 2019

2019-09-19T15:41:33Z

← Older revision		Revision as of 15:41, 19 September 2019
Line 50:		Line 50:
	** Lookup : The combination of ContainerType, KeyType and KeyValue provides a lookup - not sure how to make this work.		** Lookup : The combination of ContainerType, KeyType and KeyValue provides a lookup - not sure how to make this work.
	** Type : All the data has the same type.		** Type : All the data has the same type.

			== Shared Context ==
			An important feature is the ability to read data from a shared context. But this requires collision avoidance so the keys in a shared context are large. We don't want these large values repeated in the data, so there is an operation to read from the shared context into a register.

Adamcolton at 15:39, 19 September 2019

2019-09-19T15:39:40Z

← Older revision		Revision as of 15:39, 19 September 2019
Line 36:		Line 36:
	=== Associative Array ===		=== Associative Array ===
	An unordered list where each element has key and value.		An unordered list where each element has key and value.

			== Mono Container ==
			I think I can represent all these types with a single container.

			* Key Type
			** Ordered: The key is an int taken from the position, an ordered list
			** Type: The keys all have the same type.
			* Key Value
			** Direct : The data will define the key value directly
			** Indirect : The data will define a location to get the key value.
			* Value Type
			** None : The data will define the value type
			** Lookup : The combination of ContainerType, KeyType and KeyValue provides a lookup - not sure how to make this work.
			** Type : All the data has the same type.