Close

Does this project spark your interest?

Become a member to follow this project and don't miss any updates

OpenMV

Python-powered machine vision modules

Similar projects worth following

This project was created on 06/03/2014 and last updated 4 months ago.

Description
The OpenMV project is a low-cost, extensible, Python-powered machine vision modules, that aims at becoming the Arduino of machine vision...

When I started this project, about a year ago, I was very disappointed with proprietary, overpriced, limited serial cameras/machine vision modules available to makers and hobbyists, so I decided to build better ones...

A few months later, the first OpenMV was born, It was able to do basic image processing algorithms and later more advanced stuff (face detection, key points extraction, template matching etc..).

The camera was programmable in C, and had a simple serial protocol and SPI/I2C/PWM..but I wanted to make it really easy and fun, so I used MicroPython to script the camera and I wrote an IDE that can view the frame buffer, run scripts or upload them to the camera...

Currently, I'm working on the second OpenMV, it's based on the same stuff, but will address a few issues with the first one, specifically, more Mhz and lots of RAM :)
Details

Features:

  • Scriptable in Python3.
  • $15 BOM/1000's including 4-layer PCB (OpenMV1).
  • On board uSD or internal flash storage for storing scripts/images/video.
  • 2MP RGB/YUV/JPEG sensor (OV2640).
  • Recording/Streaming MJPEG: to SD or via external WiFi shield.
  • Extension Header: breaks UART/PWM/SPI/I2C
  • Friendly IDE: upload/execute scripts, upload templates, view the framebuffer.
  • 16MB SDRAM:  on-board enables uClinux to run on OpenMV2
  • Image processing:
    • Viola-jones object detection (comptatible with OpenCV's cascades)
    • Template matching with NCC (normalized cross correlation) 
    • FAST/FREAK: keypoint detector/descriptor and matching.
    • Face Recognition: With LBP (Local Binary Patterns) work in progress.
    • Misc: RGB->LAB CLUT, kmeans clustering, histogram, median filter, scaling, sub-image and blitting, alpha blending.

The Hardware:

  • MCU: The MCU I choose is the STM32F4xx, an ARM Cortex-M4 micro running at 168-180MHz. It has a single precision FPU, DSP capabilities and a DCMI (Digital Camera Interface). Having a hardware camera interface along with the FPU/DSP made this particular MCU a perfect match for the project.
  • PCB: A 4-layer PCB is used, which costs more but much better signal integrity/EMI wise, especially when both sides have components and not much reference planes left over, plus, using 4-layers made it possible to fit everything on a 1.0x1.3 inches board. Prototypes are all done by OSHPark.
  • Image Sensor: OpenMV1 supports many single package lens/sensors, such as the OV965x(1.3MP), OV2640(2MP/JPEG). Using a single package sensor allows experimenting with different sensors easily. OpenMV2 supports only the OV2640 with an external lens.
  • Debugging/Firmware update: The serial wire debugging (SWD) is broken out on a 2.00mm header, and the DFU is easily accessible to upload new firmware images via USB. 
  • I/O Expansion: The main 2.54mm header breaks SPI, I2C, USART and PWM, for example, an SPI LCD viewing the framebuffer (the LCD driver is written completely in Python):
  • Wireless expansion:  I designed WiFi shields based on the CC3K module from TI, for OpenMV1/2. The camera can stream MJPEG to compatible browsers, and here's a custom Android demo viewing the camera stream from my phone, the CC3K module is fast enough (54Mbps) to stream live video from the camera:

The Software:

OpenMV uses a lot of cool SW, for example, ChaN's FatFS, CC3K SDK, ARM's DSP/Math libraries etc.. And of course it's completely programmable in Python 3! Yes, you can write Python scripts that have access to peripherals (SPI/I2C/UART), uSD, wireless, and of course the image processing code.

The IDE:

OpenMV has a friendly IDE that I made with Python/Glade/PyGTK. The IDE has syntax highlighting, it can upload or run scripts on the camera, view the framebuffer, update the firmware and can even help with some image processing tasks, such as saving templates or keypoint descriptors to the camera:

I plan on extending the IDE with more features, to communicate with the camera using sockets when WiFi shields are connected, this way the IDE will have remote access to the camera to stream images or upload scripts over the air.

OpenMV2:

The second OpenMV uses the newer STM32F429 runing at 180MHz, with more built-in SRAM, external 16MB SDRAM, external lens for the sensor, 2xIR LEDs, 2xServo headers and 20 I/Os.. see full specs below:

OpenMV1 Specs:

  • Processor: STM32F407 (168MHz)
  • Features: FPU/DSP/DCMI
  • RAM:196KB SRAM
  • Flash: 512KBs
  • Sensor: OV965x(1.3MP)/OV2640(2MP/JPEG)
  • I/O: USART/SPI/I2C/PWM
  • USB 2.0 FS
  • uSD interface: SPI
  • Power consumption: 120 mA
  • Dimensions: 1.0" x 1.3"

OpenMV2 Specs:

  • Processor: STM32F429 (180MHz)
  • Features: FPU/DSP/DCMI/2D Acceleration
  • RAM:256KB SRAM/16MB SDRAM(up to 64MB)
  • Flash: 2MB
  • Sensor: OV2640(2MP/JPEG)
  • 2x IR LEDS 
  • 2x Servo headers
  • 20 I/Os: USART/SPI/I2C/PWM
  • USB 2.0 FS
  • uSD interface: SDIO (4-bit mode)
  • Power consumption: TBD
  • Dimensions:...
Read more »

Components
  • 1 × STM32F4xx MCU ARM Cortex-M4 FPU DSP 168/180MHz
  • 1 × OV2640 CMOS 2MP/RGB/YUV/JPEG
  • 1 × 24LC128 I2C EEPROM 128KBIT 400KHZ 8MSOP
  • 1 × CRYSTAL 12MHz 12PF SMD 2.5x2.0
  • 1 × ADP222ACPZ PM/REG LDO 3.3v/2.5V 16LFCSP
  • 1 × IS42S81600E 16MB SDRAM (OpenMV2)
  • 1 × USB MICRO B USB Micro B Connector
  • 1 × uSD Connector
  • 1 × FPC Connector FPC 24POS 0.5mm
  • 1 × RGB LED

Project logs
  • OpenMV1 Prototypes

    4 months ago • 0 comments

    We're making a small batch of OpenMV1 prototypes for beta testing, if anyone's interested, they're available now for preorder on Tindie:

    https://www.tindie.com/products/bot_thoughts/openmv-cam/

    We're only making a few of them (10-30) so the costs are higher, hopefully there will be a bigger run in the future.

  • OpenMV on Quadcopter

    4 months ago • 4 comments

    Just got my hands on a small quad, I was think this would be the perfect application for a tiny camera like OpenMV, now all I need is an open place to record some videos from the quad :)

  • FAST/FREAK Keypoint Detection

    5 months ago • 3 comments

    Feature/Keypoints detection is a very interesting and useful tool to have around when doing image processing,  it has endless applications from tracking objects, matching and searching images to more advanced stuff...So far I've been using SURF for this, but other than being complicated and very resource consuming, the algorithm itself is patented in the US and cannot be used for commercial stuff without a license, in addition to that, the only implementation I could find (libopensurf) is GPL'd..

    So I removed all SURF code from the repo and I'm using this relatively new super fast/lightweight algorithm, called FREAK....Note that FREAK is a keypoint descriptor, so it needs something to find keypoints first, for this I'm using a corner detection algorithm called FAST.

    To give you an idea of what you can do with this FAST/FREAK detector, I made this short video:

View all 17 project logs

Discussions

Derek Simkowiak wrote 22 days ago null point

This is awesome! But I am confused by the blog page format... What is the latest news? Is there an actual product "home page" instead of a 4 month old blog post?

I have many questions:

Is version 2 available yet? How many units were made? How much do they cost? Where can I order one?

Is version 1 still available? The provided link to tindie.com says there is a "wait list"... What is the wait time? Will more be produced in the future? Or is this abandoned for the new version 2?

What is the commercial viability of this as a business... Is this just vaporware? Did people who ordered version 1 actually get them? Where can I read some customer reviews? Do you plan to build this into a business with support?

Any info is greatly appreciated. Thanks!

Are you sure? [yes] / [no]

i.abdalkader wrote 22 days ago null point

Hi, that's actually my personal blog, I wrote about the camera a couple of times, but it's not dedicated to the project, we'll have a working home page eventually, for now, I post major updates here and there's also a google group(http://bit.ly/1Am0ty3) for updates/questions...

We sold some (OMV1) cameras/prototypes for testing and feedback, we got about 100 more backorders, so there's a KS in the works, really soon, working on the content right now.

V2 is not available and will not be used, neither will V1 actually, we'll be using a new one which has more bang for buck (external lens, IR LEDs, more I/O, small form and the 180MHz MCU) will cost roughly the same as OMV1. Continued support depends on the outcome of the KS.

Are you sure? [yes] / [no]

gshmakov wrote 23 days ago null point

Hi, great project.

I wonder if you managed to find detailed description of OV2640 registers. All I could find is preliminary data sheet with very limited info and nothing on JPEG.

Are you sure? [yes] / [no]

i.abdalkader wrote 22 days ago null point

Search for ov2640 software application notes

Are you sure? [yes] / [no]

J Groff wrote a month ago null point
I would like to see something that just pumps DCT and IDCT on the parameter space of my choosing and hands me an array of correlated cosines

Are you sure? [yes] / [no]

Robert wrote 2 months ago null point
'">

Are you sure? [yes] / [no]

Radomir Dopieralski wrote 2 months ago null point
I can't wait to get my hands on one of those, and put them in my walking robots. This would be the perfect robot brains for me: not only small and already including a camera, but also programmable in Python! I'm trying to hack together a board with camera using VoCore and a laptop camera module, but it's not going very well, and it will probably never be able to actually process the image much.

Are you sure? [yes] / [no]

Rebelj12a wrote 2 months ago null point
Loving the progress on this. Seeing this as having a lot of potential. Have you looked into branching out to direct sensor driving with this separate from the camera board you have there?

Are you sure? [yes] / [no]

99guspuppet wrote 2 months ago null point
Wonderful Project the comments are full of valuable information Thanks for your efforts

Are you sure? [yes] / [no]

candidomolino wrote 2 months ago null point
Hi, really new here!

Great project!

Have you tried to used it as a microscope to identify and count cells? Would have a lot applications in biotech!

Good luck!

Are you sure? [yes] / [no]

i.abdalkader wrote 2 months ago -1 point
No, haven't tried that before, it could work.

Are you sure? [yes] / [no]

99guspuppet wrote 2 months ago null point
Great idea..... I have used webcams to make digital microscopes... the OpenMV would work great for that I think.

Are you sure? [yes] / [no]

hashg wrote 3 months ago null point
Where do I buy this board?

Are you sure? [yes] / [no]

i.abdalkader wrote 2 months ago null point
It will be available soon, hopefully :)

Are you sure? [yes] / [no]

Ali Vadoud wrote 3 months ago null point
Hi, I also wanted to know if you used the BSP for stm32f429 discovery from Emcraft because I was able to run uclinux on my stm32f429 board but I am not able to display anything on the tft, I have just access to busybox.

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
Probably need to enable the driver somewhere, I'm using FBTFT for the small LCD.

This fork has better support for discovery and binaries too https://github.com/robutest

Are you sure? [yes] / [no]

Ali Vadoud wrote 3 months ago null point
Hi, thanks again for your subgestions, I was finally able to see the linux pinguin on the tft. Just a last question, how did you import the files from https://github.com/robutest to a project in your IDE compiler ?. I am trying to export the files for uboot to a Thunderbench project in order to compile and edit them.
regards
p.s sorry for all those technical questions but it seems that it is the only way that I can learn something about uclinux.

Are you sure? [yes] / [no]

Ali Vadoud wrote 3 months ago null point
Hi, I really like your projec. I wanted to know how you have put linux into the microcontroller, did you used a linux based computer?

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
Thank you. No it's actually running on the STM32 there's an external SDRAM. uClinux supports cortex-m and work to support STM32 is done by emcraft.

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
I missed your other reply, just saw it in feed, the kernel is XIP (execute in
place) kernel, so code is running from flash (not relocated in RAM) kernel+fs
reside on flash, data goes into RAM. This works out great because SDRAM is slow
compared to the on-chip flash access time (which is cached and accelerated).

Are you sure? [yes] / [no]

Ali Vadoud wrote 3 months ago null point
Ok thanks

Are you sure? [yes] / [no]

seme1 wrote 4 months ago null point
Hi, This project sounds very interesting. In the rc vehicles domain, it would be very useful for First Person View (FPV), in addition to sense and avoid. However, it is important to for the module to have a very low latency. I see that another user (matt venn) has already asked about the latency. Did you have a chance to measure it yet ?

Are you sure? [yes] / [no]

i.abdalkader wrote 4 months ago null point
I've been focusing more on documentation, videos, adding new features, fixing old issues, improving the IDE etc. This week, more people will have OpenMVs so I expect there will be more testing going on, and more issues, but like I said to Matt, latency depends on the algorithm + frame size, it can take any where between 30ms-100ms to output the results. That's acceptable to me for now, if I can optimize it more, or do some upgrades in the future I will.

Are you sure? [yes] / [no]

DZ85 wrote 4 months ago null point
Hi, I have been trying a lot to get the OV2640 click (1600x1200)2MP image but I
haven't been able to do so.In your code you have not used the uxga_regs in
ov2640.c.Is this array supposed to be used instead of svga_regs in reset() ? I
would appreciate your help in this matter.Thanks in advance

edit - I am using a 16Mhz MCO out on a pin to clock the camera instead of the
12Mhz pwm signal that you are generating in source code.Should this make a
difference?

edit- Line 302 of ov2640 shouldnt it be UXGA_HSIZE instead of SVGA_HSIZE?

Are you sure? [yes] / [no]

i.abdalkader wrote 4 months ago null point
Yes, you should use svga_regs when (res <= SVGA ) and uxga_regs when (SVGA
<res<= UXGA) you could also use cif_regs and subsample from there down to QQCIF
(I subsample from SVGA so I didn't actually bother implementing the cif_regs).

Line 302 is a typo, I don't use high res much, so the code was outdated, sorry
for the confusion, but I fixed it now (checkout github, ENABLE UXGA commit) and
I captured an SXGA and UXGA images.

You can use higher clocks, but I don't see the point since there's an internal
PLL so you can double the 12MHz internally if you set CLKRC=0x80 you'll get
24MHz, which is what I do for low resolutions and I use 12MHz (0x81 or 0x00) for
high resolutions, in fact I think it's better this way to keep high freqs inside
the sensor. Also, you will need to change some registers.

One final note, you might want to lower the quality when using UXGA to avoid
overflowing the frame buffer, the code doesn't detect/handle that yet.

*If you find any other issues, please report them on github, and nice catch :)

Are you sure? [yes] / [no]

DZ85 wrote 3 months ago null point
Thank you for your response , why have you used external eeprom when you could
have stored settings on stm32f4 using IAP, wouldn't it save cost?
Can you also share sourcing for the Ov2640 BGA camera and lense?

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
The EEPROM is not currently used, I just left there just in case it's needed, the idea was to use it to manage the internal flash FS, which would save some RAM.
More details here:
https://github.com/micropython/micropython/issues/341

The Lens (20mm/22mm) and BGA chip (actually called CSP2) in eagle/omnivision.lbr

Are you sure? [yes] / [no]

DZ85 wrote 3 months ago null point
I'm sorry what I meant to ask was from which vendor have you sourced the BGA camera ?

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
That's available from many suppliers on aliexpress just search for ov2640 ($2-$3), like this one:
http://www.aliexpress.com/item/OV2640-BGA/1016825336.html

I had the lens, but you can find one there too (M12 I think)

Are you sure? [yes] / [no]

DZ85 wrote 3 months ago null point
Thanks again for your response , it takes 500 milliseconds to capture 2MP JPEG, is this due to the processing inside the camera DSP? Can I increase this speed by setting DVP prescaler to 0 in R_DVP_SP(0xd3) register?Are there any other settings that will help me reduce JPEG capture time?

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
It should be able to do 15FPS @2MPs according to the datasheet, so DSP is probably not the reason. I haven't tried much I'm not really concerned with high FPS at this resolution but you could try setting CLKRC=0x80 that's 24MHz or using a higher xclk.

Are you sure? [yes] / [no]

DZ85 wrote 3 months ago null point
Can you please check if you are able to click an image with CLKRC=0x80, in my current set-up with the discovery board and wires I am able to go until CLKRC=0x81.In CLKRC=0x80 I recieve some JPEG data but do not get DCMI_IT_FRAME interrupt.
Are you able to click a JPEG image immediately after setting the I2c registers?
I have to wait 3-5 seconds for the brightness of the image to settle.If I click an image before that I get much darker images.Thanks

With CLKRC=0x81(12Mhz) I am able to capture image in 200m secs thats about
(5-7 fps)

Is running CLKRC=0x80 with external 12Mhz same as connecting a 24Mhz external clock with CLKRC=0x00 due to internal multiplier/pll?

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
Maybe the I/O speeds need to be higher at this frequency, and yes the first image(s) are darker it takes time for the auto functions (AGC,AWB etc..) to work/settle down (I think you can change the speed of some functions check the datasheet)..
24MHz External clock and 0x00 *should* be the same as 12MHz and 0x80.

Are you sure? [yes] / [no]

DZ85 wrote 3 months ago null point
I tried over-clocking the STM32F to 240Mhz and also supplying 48Mhz clock to camera, however it only works for clocks below or equal to 24Mhz.I am able to capture image in CLKRC=0x80 however I get a lot of green pixels on the top left of the image. Could'nt find any functions to speed the auto functions please let me know if you have come across any.
I am able to capture within 300miliseconds using manual exposure mode and setting a high shutter speed however picture as whole is a bit dark.I need to check how exposure and shutter is calculated I guess.
Strangely in manual exposure mode the size of the image increases and I had to reduce the quality factor

Are you sure? [yes] / [no]

i.abdalkader wrote 3 months ago null point
You don't need to overclock for that, just try increasing I/O edges (I/O speeds) or check with scope make sure it's actually 24MHz (which is the typical clock btw, so you don't need to supply 48 either)
For the AEC/AGC speeds check "VV" register in the datasheet (I played with that once on the OV965x)

Are you sure? [yes] / [no]

i.abdalkader wrote 4 months ago null point
To everyone who commented on my project logs, I just saw your comments today! it seems that notifications for comments on project logs wasn't working until recently, so I'll try to reply to all comments starting with the oldest ones, I hope you see this and I'm really sorry about that.

Are you sure? [yes] / [no]

justinStitches wrote 4 months ago null point
Very cool work. FYI groupgets.com exists to group buy parts and BOMs in case that you can't get on a different platform. We just kicked off a FLIR Thermal Imager campaign here in case you want to a thermal version of openMV: https://www.groupgets.com/campaigns/27#details

Are you sure? [yes] / [no]

Viper-Gtr wrote 5 months ago null point
Great project! I saw that you used the "MLX90620". How many fps you get with it on the screen?

Are you sure? [yes] / [no]

i.abdalkader wrote 5 months ago 1 point
About 13-14FPS, this includes reading an RGB image, scaling it to 128x160 (lcd resolution), reading and normalizing temperature data, scaling the IR image and blending with RGB, the final step is writing to LCD.

Are you sure? [yes] / [no]

Pixel Pirate wrote 5 months ago null point
Really cool project! Perhaps you could squeeze an ARM9 on there and get buildroot running, once you've done that you could run python scripts from the SD card!

Are you sure? [yes] / [no]

i.abdalkader wrote 5 months ago null point
Thank you.. actually I managed to run uClinux on the big camera, but it's still missing some core drivers, might work on that when I have more time.

http://hackaday.io/project/1313/log/6346-running-uclinux-on-openmv2

Are you sure? [yes] / [no]

Pixel Pirate wrote 5 months ago null point
Yeah, uClinux is a really stripped down version of Linux, and therefore is limited in what it can do (Plus it's missing a MMU!), so I don't really think it would be suitable for what you're doing.
But if you're willing to mess around with BGA components, you could squeeze a 64MB RAM chip, 400MHz CPU, and 512MB flash chip all in roughly the same area as the 144-LQFP, for around the same price as the uC, too!

Are you sure? [yes] / [no]

i.abdalkader wrote 5 months ago null point
It has its limitations, but all I need is just a single process running MicroPython with OpenCV bindings and the MCU's built-in flash is enough for a romfs so no need for an external flash... I am willing to do something more powerful, I was actually considering a cortex with a serious SIMD/FPU, but I don't have all the resources I need right now :)

Are you sure? [yes] / [no]

Pixel Pirate wrote 5 months ago null point
True enough, I've just been 'that guy' recently because I've been experimenting with ARM9's.

Are you sure? [yes] / [no]

sophi--e wrote 5 months ago null point
Dude amazing ! i have to build one at some point. Are you open for questions if run into any problems?

Are you sure? [yes] / [no]

i.abdalkader wrote 5 months ago null point
sure :)

Are you sure? [yes] / [no]

visualkev wrote 5 months ago null point
I have to build one. It looks to be a very cool device. I have ordered some boards from Osh park. I may have missed some details, but the parts list from the schematic does not seem to describe tolerances for the values of the passives nor the value for Y1 nor the type of diodes for D1 and D2. I look forward to using openmv.

Are you sure? [yes] / [no]

i.abdalkader wrote 5 months ago 1 point
Some values/tolerances are usually not shown on schematics, because they can change later, but anyway, I will share the most recent BOM in the links section, it has DigiKey part numbers, except for the sensor, which you can get from ebay...

Are you sure? [yes] / [no]

mistertime wrote 5 months ago 2 points
Holy frack. I want one.

Are you sure? [yes] / [no]

Similar projects