NPM - New Programming Model

NPM defines a programming model for new languages and "application processors", that emphasises safety, security and modularity.

Public Chat
Similar projects worth following
Yeah it's another of those projects that steals another TLA (Three Letters Acronym) from another projects but the landscape is already so crowded. I needed a quick name that is easy to remember, I'll rename it later.

This is a medium-high-level discussion about general computer program organisation. For lower-level PCU design tricks, dos and don'ts, please check it out: #PDP - Processor Design Principles :-)

The way we are used to program computers is a long evolution, and selective pressure has favoured bad habits, among others.

Safety and security are not guaranteed, ASLR and NoExecute flags don't prevent  the constant stream of dumb and preventable vulnerabilities (out of range access, buffer overflows, stack smashing, various leaks, Return-Oriented-Programming and so many exploits). I'm not speaking about cache and buffer coherency/timing issues here, which are detailed architectural issues.

Of course, no program is flawless and "there will always be bugs" but so many of them depend on the platform allowing them. This project will not revolutionise software engineering but tries to define techniques, structures and features that prevent many of the above-mentioned exploitable side-effects of the current generation of monolithic kernels (which are themselves architectured around existing CPU's programming models).

This project also tries to enable a new generation of capability-based microkernels, which influence this work. I tried to generalise their structure and keep the most fundamental features for inclusion as processor features.

1. POSIX is a curse
2. Read as Zero
3. The cellular allegory
4. Introduction to the new model

  • Introduction to the new model

    Yann Guidon / YGDES05/30/2020 at 18:59 0 comments

    In the previous log (3. The cellular allegory), we find that there is some degree of similarity between eukaryote cells and the model I'm describing in this project.

    Differences are :

    • Binary coding is used, instead of quaternary (i'm nitpicking, I know)
    • Programs communicate with direct calls, register values and transient memory blocks, instead of mRNA
    • fine-grained rights are enforced by an extra sub-level, under control of a specific promgram that enforces protections, as well as housekeeping for the thread ID and other essential features (like allowing code to load in the executable area)
    • Programs are inherently parallel and can be executed by any number of threads simultaneously (whereas DNA transcription of one gene can only be physically performed once at a time, as in the following GIF
      Obviously a RNA or DNA strand can only be "sensed" by a single molecule at a time).

    I will now try to describe the elements of the programming model :

    • The rights
    • The programs
    • The threads
    • The data memory

    The rights

    are properties and/or credentials that enable or inhibit access to a critical resource, such as

    • the input-output ports, or communications outside de execution context
    • the paging mechanism
    • the program memory (loading and/or reading the code of programs)
    • the properties of the programs
    • etc.

    There is one rule here, inspired by other OSes : it is only possible to drop/lose rights ! Otherwise, any program could get access to resources it shouldn't, by mistake and/or malevolently. So the whole system is designed in a "top-down" fashion where a first/initial program starts with all the possible rights, dispatches them to other sub-programs, with each of them having only the minimum required rights to perform their job.

    Surrogate programs can serve as gatekeepers : they perform the I/O taks for example while filtering data and enforcing protocols. They have their own filters for who can use which provided service. This allows dynamic, fine-grained access to necessary features, and even cascading "server programs" while keeping the system "flat" (no "privileged program" because no program has all the rights).


  • The cellular allegory

    Yann Guidon / YGDES05/30/2020 at 05:06 0 comments

    Here is one way to explain the "vision" of the programming model I try to define.

    Initially I wanted to use robots in a factory, but a more natural example abounds on Earth : cells. More specifically : Eukaryotes. They have a nucleus that contains, among others, chromosomes, each made of genes, each written with codons of three nucleotides.

    The analogy would then be :

    • Each nucleotide equivalent to 2 binary bits, so it makes a quaternary "computer"
    • Each codon has 3 quaternary values, or 64 codes, vaguely equivalent to a processor instruction
    • Codons are assembled in sequences to make genes, or "computer routines"
    • The genes are packed in chromosomes, like a computer's program
    • The cell contains several chromosomes, or program, that can "work" in parallel...
    • Some extra functions are provided by Mitochondria, such as peripheral management or energy processing...
    • Communication is provided by "messenger RNA" (simplified)


    Our "new computer" can perform parallel execution but requires of course synchronisation and semaphores. The number of execution threads is not specified and can vary wildly, as it is hypothetical so far.

    Our computer has "threads" : this is an active instance of a program. Any number of instances could be running at any time... It depends on the implementation. The program starts from one "chromosome" and can then ask services to other programs/chromosomes. The callee can refuse or accept, depending on its own policy. This is performed with the "IPC/IPE/IPR" mechanism.

    Communication is essential and "zero-copy" is required. How is our mRNA implemented ? Small chunks of data would fit in the registers, larger blocks require memory. The paging system ensures that a block of data has only one owner and the sender "yields" a data block ownership to the callee. To prevent dangling and zombie blocks, in case the callee has an issue, the callee must "accept" the new block (otherwise the block is garbage-collected). This block could then be yielded again to another program, and passed along a string of "genes" to perform any required processing.

    Paging ensures that any access outside of the message traps. It's not foolproof because data will rarely fill a whole page... Smaller and bigger pages are required (with F-CPU we talked about 512, 4096, 32768 and 262144 bytes)

    This also means that there must be a sort of fast and simple page allocator to yield pages of data at high rates, and also provide some garbage collection and TLB cleanup.

    There is a shared space and a private area for all the threads. The private area implements the stack(s) and resides in the negative addresses. The positive addresses are shared though only one thread can "own" a block at once. So there are fewer chances of stack disruption.

    As described before, the "yield" operation marks a block as candidate to belonging to the callee. The "accept" operation can then validate and acknowledge the reception and eventually remap the block into its private space.

    Processor support is required for the best performance but these features could be implemented on existing systems through emulation. It would be slow (yes the Hurd was slow too) but will help refine the API.

    Similarly : no mention of a "kernel" can be found : any "program" is structurally identical. All of them provide a function of some sort and the only difference is the access rights they provide. Each "thread" either inherit these rights from the thread that creates it, or can selectively "drop" some of those rights. This is not even a "microkernel" approach if there is no kernel !

    However in the first implementations, no hardware provides those features and a nanokernel is necessary to perform these while we develop the software stack.

  • Read as Zero

    Yann Guidon / YGDES05/30/2020 at 03:36 7 comments

    I'm not sure why it's not widespread yet, or why it's not implemented or even spoken about but...

    Imagine you free some memory, and the kernel reclaims it for other purposes. It can never be sure the physical data in RAM can be read by another thread to exfiltrate precious information. So the kernel spends some of its precious time clearing pages after pages, just in case, writing 0s all over the place to clean up after the patrons.

    It's something the hardware could do, by marking a page as "read as zero" or "trap on read dirty" in the TLB. Writes would not trap and you can read your own  freshly written data. In fact it's as if you allocated a new cache line without reading the cache...

    The cache knows about the dirty bits, and a coarser "dirty map" could be stored to help with cache lines. That's 128 bits, one for each cache line if lines are 256 bits wide.

    It's still preliminary but I'm sure people have worked on this already... Because scrubbing data was a thing for a long time.

  • POSIX is a curse

    Yann Guidon / YGDES05/30/2020 at 03:14 0 comments

    I'm sorry, dear Mr Stallman, but POSIX is defective by design and even though it was an amazing project 30+ years ago, it is now biting us E V E R Y - S I N G L E - D A Y. What was the very best in the mini- and micro-computers era is now totally unadapted and no hardware tweak or supervisory tool could ever solve the fundamental problems. And all the new language (even the "safe" ones) perpetuate the misconceptions that you could make an unsafe system safe by using a safe language. You should already know that the weakest link breaks first, right ?

    POSIX was never meant to be safe by design. It is inherently tied to the standard libraries, which are inherently designed for and by the C language, which is well known to not be safe...

    • You can change the application programming language : you keep the standard libraries.
    • OK let's change the standard libraries : nobody will want to program for your platform, AND the core OS will remain tied to the standard interfaces...
    • Let's make a novel interface and kernel then ? Oh, others have tried and... they are stuck by performance issues because their HW platform is designed to run POSIX fast.

    This co-evolution creates a macabre dance and there is no solution. The whole thing is rotten to the core but held in place because everybody uses it !

    Of course the "new programming model" requires an adapted platform to run smoothly. However this is not a hard requirement and the system can be emulated at some level or another, to help bootstrap the transition.

    Let's simply start with the most common source of vulnerabilities : buffer overflows. These should NEVER happen. Yet they do. All the time.

    The i286 introduced the BOUND opcode in 1982... Where is it used ???

    Oh yes, segmented and paginated memory is the solution...

    But wait, flat space is sexier and segment registers are PITA.

    Let's make fast SW and throw all the compiler warnings through the window...

    IF you use a language that implements boundary checks. Some are more stringent than others... but speed always ends up being the excuse for indulging in the safety-sin again and again, and processors are tired and tired of checking the same pointers for nothing. Because it can never occur right ?

    iAPX432 wanted to solve this with a micro-managed system with complex properties and it was slow. I understand why... but it would have been safer, right ?

    But wait !

    We're in 2020 and in hindsight (hahaha) 70% of all security bugs are memory safety issues and half of them "are use-after-free vulnerabilities, a type of security issue that arises from incorrect management of memory pointers"

    So you use a block, free it and ... it can still be read ? Why would that be possible ?

    Oh wait : we are bound to use the standard libs, even if our programming language has its own system.

    "There will always be bugs" is an old refrain, and it is not false.

    It is however a lousy pretext to accept preventable bugs and incapable platforms.

    I see a kind of bugs that is not completely dependent on the tools and the programming languages : check the goddamned access rights. Check which process requests what service and in which context.

    This is not the perfect solution but any system call should be easily traced by the callee to the root of the process tree. The whole history and tree of who calls what and with which access rights must be visible by the whole computer. So it's another hardware-assisted feature that must be etched in silicon (or whatever).

    I am not blaming anyone here, and I know extensive efforts are being made, for example with Rust. However this is not a complete solution, as long as the whole computer system co-evolves around old premises dating from the 60s, when things were so simple and laid back.

View all 4 project logs

Enjoy this project?



Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates