I got the ray-tracer to fit - into the PIC, anyway. At 8473 bytes, it's way over the 1kB limit, unfortunately. The good news is that this is without any serious attempts at size optimization - it's using the free XC8 compiler's native 24-bit floating point in pure C, and it was written more for simplicity than compactness. I figured once I got it working, I could start cutting corners to make a version fit.
The bad news is that since this thing is going to take so long to run, I need to start running the code I have if I want a chance at it finishing before the contest deadline - my rough estimates put it at 8 hours, but it could be twice as long or more. So, I kicked it off a few minutes ago.
So, it won't be in 1kB, but I might get a ray-tracer on an 8-bit PIC in true VGA resolution. In some universe, that's worth doing anyway :-)
I'll post an image when it appears. I uploaded the code already - you can see that it could use some serious optimization.
The one issue I had was that the XC8 compiler doesn't support recursive functions, which make ray tracing so much easier. I had to convert to an iterative approach. I've written four (or maybe five) ray tracers of varying complexity for different applications (graphics / optics / solar concentrator design) since the late 1980s, but I think this was the first time I couldn't use recursion. It was a neat little twist.