Framework

The open source community lacks a vm/framework like .Net or the JVM. I think it would be interesting to develop one with no primary language (C# for .Net or Java for the JVM).

Memory Management

This is (in my opinion) the most important function of the VM - it needs to be very efficient. I believe I like tri-color mark and sweep best, but I need to learn more.

I would like the GC to perform well out of the box (ootb, but also provide hooks for optimization. The GC-ootb would be non-deterministic, but have hooks to make it deterministic, allowing the program to trigger when it executes. The program should also be able to provide hints to the GC indicating that memory is either still in use or ready to be collected. For instance, this allows the programmer to label semantic garbage - in code, you can delete a root object and you lose that root reference. If there are other references to it, it will not be deleted, but it can no longer be accessed from its root reference (to do so would be semantically incorrect). It would also be interesting to have a triggerable stop-the-world giving the developer access to the low overhead that stop-the-world provides, but freeing them from the unpredictable interruptions.

Another aspect that could be handled automatically, but also provide hooks is heap management. By default, an appropriate heap size is maintained using - some algorithm - but, users could choose a different algorithm, or, it they know they're about to use a lot of data, preemptively request that the heap size be increased to what they will need.

Note: weighted reference counting looks interesting.

Concurrency

This is where everything is headed. It's important that at the core everything is threaded. I really like the cheap Go threads - I'm interested in learning how they did that. It should be easy to make things threadsafe, shared memory should be discouraged.

Types

There needs to be a type system, but I'm hesitant to get too attached to classical objects. I like the idea of structs with functions and functions as first-class-citizens.\

JIT

Important, but nothing special here. Native code is generated as (or before) it is needed.