Also check out the #OpenMV project!

I found this module which looks promising! https://www.olimex.com/Products/SOM/A13/A13-SOM-256/

Something like this isn't easy, but it's not impossible because it's been done before (check this out: http://www.extremetech.com/extreme/174449-novena-a-leather-bound-open-source-hacker-laptop-that-you-can-build-yourself). I can help you get started, but I've never actually done this myself. At least, not yet anyway.

First off, you have to find a CPU to use. I suggest the IMX6, as far as I know it has a CSI interface. And they come in lots of different models to fit your needs. If you don't need something extreme, a  cheap IMX6SL is perfect and goes for about $16 from digikey. Or you could get a nice 1GHz quad core for about $50. As far as I know, all IMX6 models have a 3D GPU except the SL.

Next you have to know how to route multilayer boards. 4 layers is the minimum, but 8 is recommended.

You also have to know how to route all sorts of high speed signals. Differential pairs, DDR memory lines,  and all that. Transmission lines have to be the same length/impedance and be terminated through resistors. This is the case most of the time. If your lines are really short, you can ignore that safely. But differential pairs HAVE to be routed parallel to each other. There are no exceptions.

Then you have to know what buck converters are. Linear regulators are waaaay too inefficient to power ARM processors, unless we're talking about the low powered ones found in microcontrollers. But an IMX6 will burn up that regulator in no time.

I'm not going to be that guy you usually see on the internet saying this isn't worth it and you should forget about it. I'm going to say you should totally go for it. All you need is dedication, and you will make it. I wish you the best of luck!

Thank you for the effort you put into this comment. :)

I have already looked at the IMX processors. My target project price point is pretty low (under 20$) so I'm looking for something really cheap.

Somethign like the Allwinner R8 on the CHIP (http://www.cnx-software.com/2015/05/08/chip-is-a-9-linux-development-board-powered-by-allwinner-r8-crowdfunding/) would be great, but I haven't found any modules like that, which would have a low price point. 

Ooh, $20 is VERY low. That's about how much custom PCBs of 2 square inches cost from OSHPark (a really good fab house). Trust me, I've tried to design an ARM board for less than $20, and it is not easy. The reason why the Chip is so cheap is because they make several thousand of them at a time, which is much cheaper than making one at a time.

The cheapest ARM CPU I can find is the IMX233, and that's already half your budget. It doesn't have a camera interface.

One thing I've learned over the years is to be flexible with my budget. That doesn't mean set a budget of $50 and end up spending $2000.

Oh, I forgot to mention I was talking about the cost per unit if one were to make 1000 units.