It has been many hours of compiling, booting, and debugging and I have not made much progress. In the last log, I had the very start of the Linux kernel code running on the 68008, but it was failing early and without much information. I needed more output.
After digging through the kernel code, it became clear that printk() was the tool for the job. This function is the kernel's version of the familiar printf(), but it handles logs a little differently. Instead of directly outputting messages, printk() acts a log buffer until the kernel boots far enough to create a serial console. Once the console is available, any logs stored up from before that will be printed. I spent some time writing a simple console driver and attempting to get the kernel to use it, but my issues started before the console is even loaded.
To get around this, I cheated. I replaced the contents of the printk() function with dumb printf()-style code, immediately writing the log messages to the serial port and skipping all the buffering and console business. Now the kernel can at least talk. This is what it says:
5Linux version 4.4.0-uc0 (mackerel@c14f97401618) (gcc version 4.9.2 (GCC) ) #1 Fri Jul 22 20:27:01 UTC 2022 6 uClinux/MC68000 6Flat model support (C) 1998,1999 Kenneth Albanowski, D. Jeff Dionne 7On node 0 totalpages: 512 7free_area_init_node: node 0, pgdat 000b50cc, node_mem_map 000cd000 7 DMA zone: 4 pages used for memmap 7 DMA zone: 0 pages reserved 7 DMA zone: 512 pages, LIFO batch:0 7pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768 7pcpu-alloc:  0 6Built 0 zonelists in order, mobility grouping off. Total pages: 0 5Kernel command line: console=debug,earlyprintk=debug 6PID hash table entries: 1024 (order: 0, 4096 bytes) 6Dentry cache hash table entries: 1024 (order: 0, 4096 bytes) 6Inode-cache hash table entries: 1024 (order: 0, 4096 bytes) 6Memory: 1144K/0K available (613K kernel code, 31K rwdata, 80K rodata, 36K init, 45K bss, 4294966152K reserved, 0K cma-reserved) 5Virtual kernel memory layout: vector : 0x00000000 - 0x00000400 ( 1 KiB) kmap : 0x00000000 - 0xffffffff (4095 MiB) vmalloc : 0x00000000 - 0xffffffff (4095 MiB) lowmem : 0x00000000 - 0x00200000 ( 2 MiB) .init : 0x000b6000 - 0x000bf000 ( 36 KiB) .text : 0x00000400 - 0x00099bc0 ( 614 KiB) .data : 0x00099bc0 - 0x000b5a60 ( 112 KiB) .bss : 0x000bf000 - 0x000ca674 ( 46 KiB)
This is as far as it gets. Occasionally it will also crash and show a stack trace related to the mm_init() code. It seems pretty clear that there is an issue related to the memory. My first thought is that there is something wrong with the image file I'm generating or with the kernel configuration settings related to ROM/RAM sizes and locations. Seeing the "0K available" message is concerning. I have 2 full megabytes of RAM available from 0x000000 to 0x200000. My bootloader copies the Linux image into RAM starting at 0x400 (leaving the vector addresses alone) and then jumps there.
There is definitely more debugging work I can do in this area, but the other problem I'm having is harder to solve: hardware instability. Unsurprisingly, a bunch of vintage components on perfboards soldered together and connected on a backplane is not the pinnacle of reliability I would like. My RAM board in particular seems finicky. Rearranging the individual chips in the sockets can change the stability. The crashes I see (outside of Linux kernel problems) show up as illegal instruction exceptions or address errors, more evidence that something is off with the RAM hardware. One other possibility is that MFP or decoder CPLD is showing up on the CPU buses when it shouldn't be. This would also look a lot like a RAM problem.
In a last ditch effort to avoid blaming the hardware, I built and booted an older version of Linux (3.10.0) with and older version of gcc, but I got almost identical results. Turns out the Linux kernel startup changes very slowly between versions. The strategy at this point is two-fold: I need to deep dive into the Linux image structure and ROM/RAM configurations and I need to spend some time stress testing the hardware and improving reliability. A good first step might be to design a PCB for this RAM board.