I've been drooling over the chance to explore this idea for almost a year now: image processing in hardware.
Open source image processing has been developed and tested primarily on CPUs and GPUs. I'd like to come at the same problem with an FPGA-based approach. While CPUs and GPUs serve as excellent, general-purpose processors for developing algorithms, they lack one interesting characteristic: these approaches run on fixed logic designed for general-purpose programs. With an FPGA, this logic is flexible; it can be redefined for the particular problem at hand. Over the next several months, I'd like to try some well-known image processing algorithms that lend themselves to being easily implemented on FPGAs. (I'll start with point operations: thresholding, differencing, etc.)
Along the way, I've discovered that I'll be writing a lot of drivers; hence, night-vision goggles are a great way to make sure that I can talk to cameras, parse the data, and display it all in real time.
I got pretty far into this project before the end of the year, but I had to restart somewhat. With the classic SPI ILI9341 display board, I was able to display an image on the screen at 15FPS. (The image was blue, but--heck, that was fixable with a few more hours of debugging.)
Sadly, driving an image over to the display at 15FPS was pushing the limits of the SPI interface. Pushing an image onto the display over an SPI interface at 15 FPS required an SPI transfer speed of 75 MHz. While I can synthesize a PLL on the FPGA to give me a clock capable of that speed, the SPI interface in the ILI9341 just can't handle that rate of transfer.
Luckily, the ILI9341 chip supports multiple interfaces: SPI, and 8080 Parallel variants. By transferring 8 bits at once, I can reduce the overall clock speed and still display images at 30 FPS. WIth an SPI driver already written and working, I abandoned that approach and wrote an 8080 parallel interface driver instead.
To access the parallel port, I needed a different breakout board. Luckily, Adafruit sells one. I had a couple of bugs along the way, but this time the worst ones were actually in my wiring (forgot to tie the RD line HIGH) and not in my SystemVerilog code. The waveforms look very pretty on the logic analyzer. In case you'd like to try it out, it's on the Gits under ILI9341_MCU_Parallel_Ctrl in the master branch.
How complicated do you think it would be to capture HDMI and pipe it into the display?How much FPGA RAM would one need to do that?