Data

Data; it's structure, formatting and storage.

Large Uint

A uint is a byte slice.

 (data uint) LoadUint(l uint) (this uint, rest uint) {
   if l==0{
     b = data[0]
     if b < 129 {
       return b, data[1:]
     }
     if b < 252 {
       return data[1:b-250], data[b-250:]
     }
     return LoadUint(b-251)
   }
   return data[1:l+1], data[l+1:]
 }

Fundamental Structures

Value

A type and bytes where the type indicates how to interpret the bytes.

Array

An ordered list, all of the same type. Variable length.

Tagged Union

One of a set of types with a tag indicating the type.

Tuple

An ordered list where the tuple type defines the length and the type of each element. Last variadic?

Struct

A set of fields. This also necessitates a field as type. A field is a tag (field name) and a type.

Associative Array

An unordered list where each element has key and value.

Anonymous Type

The type is part of the value.

Mono Container

I think I can represent all these types with a single container.

• Key Type
  • Ordered: The key is an int taken from the position, an ordered list
  • Type: The keys all have the same type.
• Key Value
  • Direct : The data will define the key value directly
  • Indirect : The data will define a location to get the key value.
• Value Type
  • None : The data will define the value type
  • Lookup : The combination of ContainerType, KeyType and KeyValue provides a lookup - not sure how to make this work.
  • Type : All the data has the same type.

Shared Context

An important feature is the ability to read data from a shared context. But this requires collision avoidance so the keys in a shared context are large. We don't want these large values repeated in the data, so there is an operation to read from the shared context into a register.

IDs

There are two fundamental ID types - one is intended to be a unique identifier either from an increment source or a guid. The other is a meaningful tag, like setting the key column in sql to a string.

Quick Lookup and Compact Data

When data is going to be fully deserialized, compaction tends to be advantageous. But for data at rest this causes lookups to be slow. If a struct has 5 variable length fields and we want the fifth, we have to pass through the first four.

To make a struct in place that allows for fast lookups, we need a constant header size. Lengths need to be a constant size. But even defining lengths will still require reading all four, so instead we encode offsets. For example if we have

 Person:
   ID: int64;
   Age: byte;
   RoleID: byte;
   First: string;
   Last: string;

Then the first 3 fields are fixed length. We have to place an upper limit on the length of string. So lets say we allocate 4 bytes for this. Then we now First starts at byte (8+1+1+4+4) 18. The 4 byte value in the First header field will actually be the relative offset of the end of first. And Last will be the relative offset of the end of Last as well as the end of the record.

This further makes an argument to place the variable length field headers at the top, in reverse order. So the very first piece of data is the record length.

Namespace

A structured namespace can be defined. A secure model would be something like

• Resource (url)
• Public Key
• Hash
• Hash bits

Then the ID would be the defined bits of the hash. This would allow for fewer bits to be used to identify static resources.