ROMfs and Remus, and the Founding of Rome

A project log for NYETduinoPlusLua

Wherein a Netduino Plus 2 is repurposed with alternative firmware based on Lua

ziggurat29ziggurat29 01/05/2018 at 23:290 Comments


Made a separate tool to generate 'romfs'.


I set down to reverse-engineer the Lua-based build system to see where the 'romfs' is generated.  Happily, this wound up being a relatively easy exercise, performed by a deviously named routing 'make_romfs' and an auxilliary module named 'utils.mkfs'.  This step is executed later in the build process, and was being blocked by the toolchain detection stuff.  I eventually just decided to cruft together a bespoke tool that performs that single step, rather than fooling around with trying to hook in the System Workbench toolchain at this juncture.

I copied build_elua.lua to a new file build_romfs.lua and then commented-out everything except for that top-level routine: make_romfs().  Then I ran it with Lua5.1 and observed how it croaked.  I then incrementally added things back until it stopped croaking (and took out a couple things that I know I didn't need that wired into the build system) and wound up with a minimal tool that generates the file.

The build_romfs ostensibly can operate in three modes:
  1. 'verbatim', which copies all files as-is into the image
  2. 'compressed', which runs the files through a minizer first, before coping into the image
  3. 'compiled', which compiles them to bytecode first, before coping into the image.

The first two are straightforward, and are supported.  The last one is useful, but not supported at this time because I have to build a special version of the compiler 'luac.exe' to make this work.  This feature is expected to be beneficial because the parser for the Lua interpreter apparently can be RAM-intensive, and so it would be beneficial to avoid the compilation step on-device.  However, this will be some work on my part to get running, and if I am to ever get the patches in eLua ported forward into 5.3.4 (as I want to), then I'd rather tackle mess with a special luac after that happens.  This device should have enough RAM to allow me to at least continue the evaluation without prematurely performing that optimization.

OK, so the build_romfs tool emits a single file:  a header which defines a C array that is the filesystem image.  It is included in one module only (as it must since the definition is in the header), so I simply copy it over and start the build again.

To my delight (and no small surprise), the build finished compiling.  There are a few warnings, but they are minor.  It does not link, however.  I need to pull some more source files.  These appear to be platform support source files, so I guess platform_int.c is not the sole interface.  (In retrospect, I now know that this is just for interrupts.  There are many others for the other peripherals.)

This will probably be an incremental process, because I imagine that the board configuration will define what modules are actually needed.  I start pulling the currently required ones in, and stubbing the implementation out as I did for platform_int.c.

I needed to add the uIP library, and several other modules, but most of the work was the usual 'stub out implementation connecting to the platform' that was done in platform.c.  There is going to be a lot of work to do here, and I'll have to figure out what is the actual public interface.  I also had to twiddle the linker script to emit a few symbols that the eLua relied upon to assess flash usage and the extent of the read-only area for something related to strings.

In the end, I was able to once again finish linking, and I got these size results:

So we still have plenty of room for more code.  I don't know how much more will be required, but I don't think it will be much.  I suspect the networking and filesystem library code was evicted at link time, so that could add a chunk.  Also, if I include SSL support, that will incur a big chunk - maybe 300k or so (based on some previous experience with mbedTLS).

I have no idea about RAM usage, but I did not include the memory allocators from eLua because I have some from FreeRTOS.  This chip has two memory regions, so I need to think about how I would want to use those.  One is 128 KiB, so I'll pretend that's all I've got for the moment.

I'm sure none of this build works, as was the case before, but it does mean that I've got the eLua source building, along with all the peripherals modules that it adds.  This is a good thing.  Now I need to understand the platform.c and platform_int.c, which is where I need to glue the eLua to the System Workbench code.


Attempt to implement the platform-specific glue logic for the system and all the peripherals.